Self-referential and social processing are often engaged concurrently in naturalistic judgements and elicit activity in overlapping brain regions. We have termed this integrated processing 'self-other referential processing' and developed a task to measure its neural correlates. Ninety-eight healthy young people aged 16-25 (M = 21.5 years old, 67% female) completed our novel functional magnetic resonance imaging task. The task had two conditions, an active self-other referential processing condition in which participants rated how much they related to emotional faces and a control condition. Rating relatedness required thinking about oneself (self-referential processing) and drawing a comparison to an imagined other (social processing). Self-other referential processing elicited activity in the default mode network and social cognition system; most notably in the 'core self' regions of the medial prefrontal cortex and posterior cingulate cortex. Relatedness and emotional valence directly modulated activity in these core self areas, while emotional valence additionally modulated medial prefrontal cortex activity. This shows the key role of the medial prefrontal cortex in constructing the 'social-affective self'. This may help to unify disparate models of medial prefrontal cortex function, demonstrating its role in coordinating multiple processes-self-referential, social and affective processing-to allow the self to exist in a complex social world.