Structural neuroimaging studies suggest the presence of subtle abnormalities in the brains of patients with bipolar disorder. The influence of genetic and/or environmental factors on these brain abnormalities is unknown. To investigate the contribution of genetic and environmental factors on grey and white matter brain densities in bipolar disorder, monozygotic and dizygotic twins concordant and discordant for bipolar disorder were scanned using 1.5 Tesla magnetic resonance imaging and compared with healthy twin pairs. A total of 232 subjects: 49 affected twin pairs (8 monozygotic concordant, 15 monozygotic discordant, 4 dizygotic concordant, 22 dizygotic discordant) and 67 healthy twin pairs (39 monozygotic and 28 dizygotic) were included. After correcting for the effect of lithium, the liability for bipolar disorder was associated with decreased grey matter density in widespread areas of the brain, but most prominent in frontal and limbic regions, and with decreased white matter density in (frontal parts of) the superior longitudinal fasciculi. The genetic risk to develop bipolar disorder was related to decreased grey matter density in the right medial frontal gyrus, precentral gyrus and insula and with decreased white matter density in the superior longitudinal fasciculi bilaterally. In conclusion, pathology in the frontal lobe, especially in parts of the superior longitudinal fasciculus, may be central to the genetic risk to develop bipolar disorder, while widespread grey matter abnormalities appear related to the illness itself.