The medial nucleus of the trapezoid body (MNTB) is an integral component of the auditory brainstem circuitry involved in sound localization. The giant presynaptic nerve terminal with multiple active zones, the calyx of Held (CH), is a hallmark of this nucleus, which mediates fast and synchronized glutamatergic synaptic transmission. To delineate how these synaptic structures adapt to reduced auditory afferents due to aging, we acquired and reconstructed circuitry-level volumes of mouse MNTB at different ages (3 weeks, 6, 18, and 24 months) using serial block-face electron microscopy. We used C57BL/6J, the most widely inbred mouse strain used for transgenic lines, which displays a type of age-related hearing loss. We found that MNTB neurons reduce in density with age. Surprisingly we observed an average of approximately 10% of poly-innervated MNTB neurons along the mouse lifespan, with prevalence in the low frequency region. Moreover, a tonotopy-dependent heterogeneity in CH morphology was observed in young but not in older mice. In conclusion, our data support the notion that age-related hearing impairments can be in part a direct consequence of several structural alterations and circuit remodeling in the brainstem.