Normal ageing is associated with gradual brain atrophy. Determining spatial and temporal patterns of change can help shed light on underlying mechanisms. Neuroimaging provides various measures of brain structure that can be used to assess such age-related change but studies to date have typically considered single imaging measures. Although there is consensus on the notion that brain structure deteriorates with age, evidence on the precise time course and spatial distribution of changes is mixed. We assessed grey matter (GM) and white matter (WM) structure in a group of 66 adults aged between 23 and 81. Multimodal imaging measures included voxel-based morphometry (VBM)-style analysis of GM and WM volume and diffusion tensor imaging (DTI) metrics of WM microstructure. We found widespread reductions in GM volume from middle age onwards but earlier reductions in GM were detected in frontal cortex. Widespread age-related deterioration in WM microstructure was detected from young adulthood onwards. WM decline was detected earlier and more sensitively using DTI-based measures of microstructure than using markers of WM volume derived from conventional T1-weighted imaging.