It is known that the temporal cortex is involved in perception of emotional facial expressions, and the involvement is relatively independent of the emotional valence of those expressions. The present study revealed a valence-dependent aspect of the temporal cortex through individual differences analyses involving the neuroticism trait, one of the representative affective personality traits. Functional MRI was administered while subjects classified expressions of faces, and neuroticism scores were obtained from individual subjects. Significant brain activity was observed in the temporal pole (TP) during perception of both happy and sad expressions relative to neutral expressions. Correlational analyses revealed that TP activity during perception of sad expressions, but not happy expressions, correlated with the neuroticism scores. These results demonstrate differential roles for the temporal cortex in perception of happy and sad faces, and suggest that TP recruitment during understanding of negative emotions is dependent on the personality of the individuals.