Abnormal brain development and cognitive dysfunction have been reported both in children and in adults with fragile X syndrome (FXS). However, few studies have examined neuroanatomical abnormalities in FXS during adolescence. In this study we focus on adolescent subjects with FXS (N = 54) as compared to age- and sex-matched subjects with idiopathic intellectual disability (Comparison Group) (N = 32), to examine neuroanatomical differences during this developmental period. Brain structure was assessed with voxel-based morphometry and independent groups t-test in SPM8 software. Results showed that the FXS group, relative to the comparison group, had significantly larger gray matter volume (GMV) in only one region: the bilateral caudate nucleus, but have smaller GMV in several regions including bilateral medial frontal, pregenual cingulate, gyrus rectus, insula, and superior temporal gyrus. Group differences also were noted in white matter regions. Within the FXS group, lower FMRP levels were associated with less GMV in several regions including cerebellum and gyrus rectus, and less white matter volume (WMV) in pregenual cingulate, middle frontal gyrus, and other regions. Lower full scale IQ within the FXS group was associated with larger right caudate nucleus GMV. In conclusion, adolescents and young adults with FXS demonstrate neuroanatomical abnormalities consistent with those previously reported in children and adults with FXS. These brain variations likely result from reduced FMRP during early neurodevelopment and mediate downstream deleterious effects on cognitive function.