Previous research on the neural underpinnings of empathy has been limited to affective situations experienced in a similar way by an observer and a target individual. In daily life we also interact with people whose responses to affective stimuli can be very different from our own. How do we understand the affective states of these individuals? We used functional magnetic resonance imaging to assess how participants empathize with the feelings of patients who reacted with no pain to surgical procedures but with pain to a soft touch. Empathy for pain of these patients activated the same areas (insula, medial/anterior cingulate cortex) as empathy for persons who responded to painful stimuli in the same way as the observer. Empathy in a situation that was aversive only for the observer but neutral for the patient recruited areas involved in self-other distinction (dorsomedial prefrontal cortex) and cognitive control (right inferior frontal cortex). In addition, effective connectivity between the latter and areas implicated in affective processing was enhanced. This suggests that inferring the affective state of someone who is not like us can rely upon the same neural structures as empathy for someone who is similar to us. When strong emotional response tendencies exist though, these tendencies have to be overcome by executive functions. Our results demonstrate that the fronto-cortical attention network is crucially involved in this process, corroborating that empathy is a flexible phenomenon which involves both automatic and controlled cognitive mechanisms. Our findings have important implications for the understanding and promotion of empathy, demonstrating that regulation of one's egocentric perspective is crucial for understanding others.