Adults with severe traumatic brain injury (TBI) often suffer poor social cognition. Social cognition is complex, requiring verbal, non-verbal, auditory, visual and affective input and integration. While damage to focal temporal and frontal areas has been implicated in disorders of social cognition after TBI, the role of white matter pathology has not been examined. In this study 17 adults with chronic, severe TBI and 17 control participants underwent structural MRI scans and Diffusion Tensor Imaging. The Awareness of Social Inference Test (TASIT) was used to assess their ability to understand emotional states, thoughts, intentions and conversational meaning in everyday exchanges. Track-based spatial statistics were used to perform voxelwise analysis of Fractional Anisotropy (FA) and Mean Diffusivity (MD) of white matter tracts associated with poor social cognitive performance. FA suggested a wide range of tracts were implicated in poor TASIT performance including tracts known to mediate, auditory localisation (planum temporale) communication between nonverbal and verbal processes in general (corpus callosum) and in memory in particular (fornix) as well as tracts and structures associated with semantics and verbal recall (left temporal lobe and hippocampus), multimodal processing and integration (thalamus, external capsule, cerebellum) and with social cognition (orbitofrontal cortex, frontopolar cortex, right temporal lobe). Even when controlling for non-social cognition, the corpus callosum, fornix, bilateral thalamus, right external capsule and right temporal lobe remained significant contributors to social cognitive performance. This study highlights the importance of loss of white matter connectivity in producing complex social information processing deficits after TBI.