Specific nutrients or dietary patterns influence an individual’s psychological stress. As a major aspect of a healthy diet, the influence of dietary diversity on psychological stress remains uncertain. Within these contexts, we aimed to examine the association between the dietary diversity score and psychological stress, using prospective data from the China Health and Nutrition Survey (CHNS). We included 7434 adult participants, with complete dietary information, in the 2011 wave, and followed-up with perceived stress scale (PSS-14) in the 2015 wave. The dietary intake of foods was coded into 29 food groups, using the DQQ for China, and the dietary diversity scores were obtained, using DQQ, by calculating the number of food groups consumed during one 24-h dietary recall. The univariate analysis, and logistic regression model were used to examine the relationship between psychological stress and diet diversity. Approximately half of the participants (4204, 56.55%) perceived a higher level of stress (PSS-14 total score > 25). Dietary diversity was lower in the higher-stress group (p for trend <0.0001). Unconditional multivariate logistic regression demonstrated that participants with higher daily dietary diversity were less likely to experience higher-level psychological stress, compared with participants with lower daily dietary diversity (ORs range: 0.480−0.809). Dietary diversity was found to be inversely associated with psychological stress, in this prospective analysis of a national population. Further studies are required to figure out the mechanism and effectiveness of dietary diversity on psychological stress.