Many studies have reported sex differences in empathy and social skills. In this review, several lines of empirical evidences about sex differences in functions and anatomy of social brain are discussed. The most relevant differences involve face processing, facial expression recognition, response to baby schema, the ability to see faces in things, the processing of social interactions, the response to the others' pain, interest in social information, processing of gestures and actions, biological motion, erotic, and affective stimuli. Sex differences in oxytocin-based parental response are also reported. In conclusion, the female and male brains show several neuro-functional differences in various aspects of social cognition, and especially in emotional coding, face processing, and response to baby schema. An interpretation of this sexual dimorphism is provided in the view of evolutionary psychobiology.