Our aim was to assess in primary open angle glaucoma (POAG), a major cause of irreversible blindness worldwide, whether diffuse brain changes recently shown in advanced stage can be detected since the early stage. We used multimodal magnetic resonance imaging (MRI) in 57 patients with the three POAG stages and in 29 age-matched normal controls (NC). Voxelwise statistics was performed with nonparametric permutation testing. Compared with NC, disrupted anatomical connectivity (AC) was found in the whole POAG group along the visual pathway and in nonvisual white matter tracts (P < 0.001). Moreover, POAG patients showed decreased functional connectivity (FC) in the visual (P = 0.004) and working memory (P < 0.001) networks whereas an increase occurred in the default mode (P = 0.002) and subcortical (P < 0.001) networks. Altered AC and FC were already present in early POAG (n = 14) in both visual and nonvisual systems (P ≤ 0.01). Only severe POAG (n = 30) showed gray matter atrophy and this mapped on visual cortex (P < 0.001) and hippocampus (P < 0.001). Increasing POAG stage was associated with worsening AC in both visual and nonvisual pathway (P < 0.001), progressive atrophy in the hippocampus and frontal cortex (P < 0.003). Most of the structural and functional alterations within and outside the visual system showed correlation (P < 0.001 to 0.02) with computerized visual field and retinal nerve fiber layer thickness. In conclusion, the complex pathogenesis of POAG includes widespread damage of AC and altered FC within and beyond the visual system since the early disease stage. The association of brain MRI changes with measures of visual severity emphasizes the clinical relevance of our findings. Hum Brain Mapp 37:4581-4596, 2016. © 2016 Wiley Periodicals, Inc.