The role of the human insula in facial emotion recognition is controversially discussed, especially in relation to lesion-location-dependent impairment following stroke. In addition, structural connectivity quantification of important white-matter tracts that link the insula to impairments in facial emotion recognition has not been investigated. In a case-control study, we investigated a group of 29 stroke patients in the chronic stage and 14 healthy age- and gender-matched controls. Lesion location of stroke patients was analysed with voxel-based lesion-symptom mapping. In addition, structural white-matter integrity for tracts between insula regions and their primarily known interconnected brain structures was quantified by tractography-based fractional anisotropy. Our behavioural analyses showed that stroke patients were impaired in the recognition of fearful, angry and happy but not disgusted expressions. Voxel-based lesion mapping revealed that especially lesions centred around the left anterior insula were associated with impaired recognition of emotional facial expressions. The structural integrity of insular white-matter connectivity was decreased for the left hemisphere and impaired recognition accuracy for angry and fearful expressions was associated with specific left-sided insular tracts. Taken together, these findings suggest that a multimodal investigation of structural alterations has the potential to deepen our understanding of emotion recognition impairments after stroke.