Aging-related neurocognitive disorders, including Alzheimer's disease (AD) and mild cognitive impairment (MCI), have been characterised by altered brain white matter (WM), relying widely on diffusion tensor imaging (DTI). DTI's limited accuracy in assessing crossing fibres prompted novel methods that distinguish fibres crossing through same voxel-spaces, such as fixel-based analysis (FBA), highlighting subtle macrostructural and microstructural alterations in AD and MCI. We examined the FBA and DTI's specificity in determining WM features relevant to memory in the neurocognitive aging spectrum. Diffusion-weighted images were analysed in 560 participants with various neurocognitive diagnoses from the Alzheimer's Disease Neuroimaging Initiative (F:297; mean age: 73.2 ± 8). Verbal memory was measured in 488 participants using the Rey Auditory Verbal Learning Test. FBA-derived measures of fibre density (FD), fibre-bundle cross-section (FC), and their combination (FDC), DTI fractional anisotropy (FA) and mean diffusivity (MD) were examined in relation to diagnoses and memory scores, controlling for age, sex, and intracranial volume. MCI and AD groups significantly differed from controls, with lower FD and FDC in the fornix and bilateral fibres extending to the medial temporal lobes (MTL). Memory was positively associated with FD and FDC in the fornix and MTL fibres, and FC in the anterior commissure (AC). Widespread FA reductions and MD increases were observed in AD and MCI and widely associated with memory. Fixel-wise measures highlight fibre tracts that are altered distinctly at the macroscopic and microscopic level in neurocognitive aging, and reveals structures associated with memory performance that are more specifically located than tensor-derived measures.