Emotional difficulties in alexithymia and their social consequences have been linked to alterations in autonomic nervous system. However, most of previous studies did not take into account the distinction between the affective and the cognitive dimensions of the alexithymia, leading to inconsistent results. In this study, we compared the effects of both dimensions of alexithymia on the autonomic arousal to emotional and social visual stimulations. Skin conductance responses (SCRs) to items of the International Affective Pictures System characterized by emotional (unpleasant, neutral, and pleasant), social (with humans) or non-social (without humans) content were recorded in non-alexithymic (NA), affective (AA) and cognitive alexithymic (CA) participants, selected on the basis of the Toronto Alexithymia Scale and the Bermond-Vorst Alexithymia Questionnaire. All participants responded to questionnaires of empathy, social phobia, depression, and anxiety before the experiment and evaluated the arousal of the pictures after it. Cognitive alexithymic group showed lower amplitudes of SCRs to pictures with social than without social relevance whereas the opposite pattern was observed for the NA group. Arousal emotional effects of the pictures on SCRs did not differ among groups. In addition, CA participants showed lower scores than NA in the Personal Taking sub-scale of the empathy questionnaire, while AA showed lower scores than NA in the fantasy sub-scale. The CA group showed higher social phobia, depression and anxiety scores, than the other two groups. This work has two original outcomes: first, affective alexithymics expressed lower empathic affective scores than other groups; second, alexithymia modulated the impact of the social relevance of the stimuli on the autonomic reactivity, this impact vanishing in affective alexithymics and reversing in cognitive alexithymics. Thus, though the groups could not be distinguished on the basis of emotional effect on SCRs, they clearly differed when the empathic characteristics and the autonomic impact of social relevance were considered. Finally, the described autonomic signature to social relevant information could contribute to elucidate the difficulty of alexithymics to deal with emotions during social transactions.