Diffusion tensor imaging (DTI) is used to detect tissue pathology. In Alzheimer's disease (AD) research, DTI has been used to elucidate differences in disease stages and to track progression over time and clinical severity. Many of these studies have identified the fornix as particularly vulnerable in the early stages of pathology associated with memory decline in prodromal AD. Emerging research suggests principal tensor components, axial (DA) and radial (DR) diffusivity, are more sensitive to underlying tissue pathology than are mean diffusivity (MD) and fractional anisotropy (FA). Given the established regionally specific tissue decline in MCI, we examined components of the full diffusion tensor (MD, FA, DR, and DA) for sensitivity to regional pathology associated with specific memory deficits in 18 individuals with MCI. We investigated multiple regions of interest, including fornix, temporal stem, and control regions for association with severity of impairment on multiple memory measures, including a type of neuropsychological task shown to be particularly sensitive to early memory decline in MCI. Better paired associate learning was selectively associated with lower DA (β = -0.663, p = 0.003), but not with DR, MD, or FA of the temporal stems. Conversely, better paired associate learning was associated with lower DR (β = -0.523, p = 0.026), higher FA (β = 0.498, p = 0.036), and lower MD (β = -0.513, p = 0.030), but not DA in the fornix. No association was found for control regions, or for control cognitive measures. These findings suggest disparate pathology of temporal stems and fornix white matter in association with early memory impairment in MCI. Further, they highlight the methodological importance of evaluating the full tensor, rather than only summative metrics in research using DTI.