Responses to smiling and nonsmiling expressions are influenced by sex of both viewer and expresser. This study investigated the stage of neural processing at which the sexes of viewer and expresser modulate the recognition of smiling and nonsmiling expressions by measuring event-related potentials. The results showed that late positive component was larger to neutral expression of own-sex faces than to that of opposite-sex faces. These results indicate that neural correlates of facial expression recognition are influenced by the sexes of both viewer and expresser of facial expression at the stage of cognitive evaluation.