Neuroplasticity following deafness has been widely demonstrated in both humans and animals, but the anatomical substrate of these changes is not yet clear in human brain. However, it is of high importance since hearing loss is a growing problem due to aging population. Moreover, knowing these brain changes could help to understand some disappointing results with cochlear implant, and therefore could improve hearing rehabilitation. A systematic review and a coordinate-based meta-analysis were realized about the morphological brain changes highlighted by MRI in severe to profound hearing loss, congenital and acquired before or after language onset. 25 papers were included in our review, concerning more than 400 deaf subjects, most of them presenting prelingual deafness. The most consistent finding is a volumetric decrease in gray matter around bilateral auditory cortex. This change was confirmed by the coordinate-based meta-analysis which shows three converging clusters in this region. The visual areas of deaf children is also significantly impacted, with a decrease of the volume of both gray and white matters. Finally, deafness is responsible of a gray matter increase within the cerebellum, especially at the right side. These results are largely discussed and compared with those from deaf animal models and blind humans, which demonstrate for example a much more consistent gray matter decrease along their respective primary sensory pathway. In human deafness, a lot of other factors than deafness could interact on the brain plasticity. One of the most important is the use of sign language and its age of acquisition, which induce among others changes within the hand motor region and the visual cortex. But other confounding factors exist which have been too little considered in the current literature, such as the etiology of the hearing impairment, the speech-reading ability, the hearing aid use, the frequent associated vestibular dysfunction or neurocognitive impairment. Another important weakness highlighted by this review concern the lack of papers about postlingual deafness, whereas it represents most of the deaf population. Further studies are needed to better understand these issues, and finally try to improve deafness rehabilitation.