A longstanding controversy concerns the functional organization of high-level vision, and the extent to which the recognition of different classes of visual stimuli engages a single system or multiple independent systems. We examine this in the context of congenital prosopagnosia (CP), a neurodevelopmental disorder in which individuals, without a history of brain damage, are impaired at face recognition. This paper reviews all CP cases from 1976 to 2016, and explores the evidence for the association or dissociation of face and object recognition. Of the 238 CP cases with data permitting a satisfactory evaluation, 80.3% evinced an association between impaired face and object recognition whereas 19.7% evinced a dissociation. We evaluate the strength of the evidence and correlate the face and object recognition behaviour. We consider the implications for theories of functional organization of the visual system, and offer suggestions for further adjudication of the relationship between face and object recognition.