This review assessed the specificity of facial emotion recognition impairment and the role of task characteristics in facial emotion recognition in autism spectrum disorder (ASD). Based on subsets of 148 studies identified in PubMed and PsycINFO, random-effects meta-analyses showed significant impairment in recognizing all basic facial emotions in ASD. Additionally, ASD involves poorer facial emotion recognition than other clinical conditions and has similar impairment in recognizing emotional and nonemotional facial attributes, as well as in recognizing emotion in faces and other modalities. Furthermore, there are significant moderating effects for emotion complexity and holistic processing, a statistical trend for task type, and no significant effect for motion, social relevance, or stimulus salience on facial emotion recognition in ASD. Altogether, this review suggests nonselective facial emotion recognition impairment in ASD. Such impairment is relatively specific to ASD but is not specific to the recognition of emotional facial attributes or emotion recognition in the face modality. Identifying the role of task characteristics improves our understanding of the mechanisms underlying facial emotion recognition in ASD.