Deficits in facial emotion recognition have frequently been established in temporal lobe epilepsy (TLE). However, static, rather than dynamic emotion recognition paradigms have been applied. Affective prosody has been insufficiently studied in TLE, and there is a lack of studies investigating associations between auditory and visual emotion recognition. We wished to investigate potential deficits in a dynamic morph task of facial emotion recognition and in an affective prosody recognition task, as well as associations between both tasks. 25 patients with TLE and 24 healthy controls (CG) performed a morph task with faces continuously changing in their emotional intensity. They had to press a button, as soon as they were able to recognize the emotion expressed, and label it accordingly. In the auditory task, subjects listened to neutral sentences spoken in varying emotional tones, and labeled the emotions. Correlation analyses were conducted across both tasks. TLE patients showed significantly reduced prosody recognition compared to CG, and in the morph task, there was a statistical trend towards significantly reduced performance for TLE. Recognition rates in both tasks were significantly associated. TLE patients show deficits in affective prosody recognition, and they may also be impaired in a morph task with dynamically changing facial expressions. Impairments in basic social-cognitive tasks in TLE seem to be modality-independent.