Different tasks of episodic memory (EM) are only moderately correlated with each other. Furthermore, various EM tasks exhibit disproportional relationships with the hippocampus. This study examined the covariance structure of EM tasks and assessed whether this structure relates differently to hippocampal volume (HV) in a sample of 648 cognitively unimpaired older adults (mean age = 69.88). A confirmatory factor analysis (CFA) and linear regression models were used to test the associations between the observed factors of EM and HV. A model with three first-order subfactors (immediate verbal recall, delayed verbal recall, and visuospatial) derived from a second-order EM domain factor satisfied model fit (χ p value ≥ 0.05, CFI > 0.90, RMSEA < 0.08, SRMR < 0.08). Total, left, and right HV explained a similar amount of variance in all EM subfactors. CA1, CA3, subiculum, and entorhinal cortex volume were associated with all subfactors, while CA2 and dentate gyrus volume were not associated with EM. These results suggest that EM tasks are measuring the same construct, but different complex processes contribute to EM. Furthermore, HV accounted for a small portion of the variance in EM, suggesting that HV might not be a useful marker of EM in cognitively unimpaired older adults. Finally, this study provides evidence that various hippocampal subfield volumes may not be purely associated with any one aspect of EM processing.