Puberty represents the phase of sexual maturity, signaling the change from childhood into adulthood. During childhood and adolescence, prominent changes take place in the brain. Recently, variation in frontal, temporal, and parietal areas was found to be under varying genetic control between 5 and 19 years of age. However, at the onset of puberty, the extent to which variation in brain structures is influenced by genetic factors (heritability) is not known. Moreover, whether a direct link between human pubertal development and brain structure exists has not been studied. Here, we studied the heritability of brain structures at 9 years of age in 107 monozygotic and dizygotic twin pairs (N = 210 individuals) using volumetric MRI and voxel-based morphometry. Children showing the first signs of secondary sexual characteristics (N = 47 individuals) were compared with children without these signs, based on Tanner-stages. High heritabilities of intracranial, total brain, cerebellum, and gray and white matter volumes (up to 91%) were found. Regionally, the posterior fronto-occipital, corpus callosum, and superior longitudinal fascicles (up to 93%), and the amygdala, superior frontal and middle temporal cortices (up to 83%) were significantly heritable. The onset of secondary sexual characteristics of puberty was associated with decreased frontal and parietal gray matter densities. Thus, in 9-year-old children, global brain volumes, white matter density in fronto-occipital and superior longitudinal fascicles, and gray matter density of (pre-)frontal and temporal areas are highly heritable. Pubertal development may be directly involved in the decreases in gray matter areas that accompany the transition of our brains from childhood into adulthood.