Recognizing other persons is a key skill in social interaction, whether it is with our family at home or with our colleagues at work. Due to brain lesions such as stroke, or neurodegenerative disease, or due to psychiatric conditions, abilities in recognizing even personally familiar persons can be impaired. The underlying causes in the human brain have not yet been well understood. Here, we provide a comprehensive overview of studies reporting locations of brain damage in patients impaired in person-identity recognition, and relate the results to a quantitative meta-analysis based on functional imaging studies investigating person-identity recognition in healthy individuals. We identify modality-specific brain areas involved in recognition from different person characteristics, and potential multimodal hubs for person processing in the anterior temporal, frontal, and parietal lobes and posterior cingulate. Our combined review is built on cognitive and neuroscientific models of face- and voice-identity recognition and revises them within the multimodal context of person-identity recognition. These results provide a novel framework for future research in person-identity recognition both in the clinical as well as basic neurosciences.