Fragile X syndrome (FXS), the most common monogenic cause of autism, is often associated with hypersensitivity to sound. Several studies have shown abnormalities in the auditory brainstem in FXS; however, the emergence of these auditory phenotypes during development has not been described. Here, we investigated the development of phenotypes in FXS model [ knockout (KO)] mice in the ventral cochlear nucleus (VCN), medial nucleus of the trapezoid body (MNTB), and lateral superior olive (LSO). We studied features of the brainstem known to be altered in FXS or KO mice, including cell size and expression of markers for excitatory (VGLUT) and inhibitory (VGAT) synapses. We found that cell size was reduced in the nuclei with different time courses. VCN cell size is normal until after hearing onset, while MNTB and LSO show decreases earlier. VGAT expression was elevated relative to VGLUT in the KO mouse MNTB by P6, before hearing onset. Because glial cells influence development and are altered in FXS, we investigated their emergence in the developing KO brainstem. The number of microglia developed normally in all three nuclei in KO mice, but we found elevated numbers of astrocytes in KO in VCN and LSO at P14. The results indicate that some phenotypes are evident before spontaneous or auditory activity, while others emerge later, and suggest that Fmr1 acts at multiple sites and time points in auditory system development.