Air pollution, especially the fine particulate matter (PM), may impair cognitive performance, but its short-term impact remains poorly understood. We investigated the short-term associations of PM with the cognitive performances of 954 white males measured as the global cognitive function (GCF) and Mini-Mental State Examination (MMSE) scores, and further explored whether taking nonsteroidal anti-inflammatory drugs (NSAIDs) could modify their relationships. Higher short-term exposure to PM demonstrated non-linear negative associations with cognitive function. Compared with the lowest quartile of the 28-day average PM concentration, the 2, 3, and 4 quartiles were associated with 0.378-, 0.376-, and 0.499-unit decreases in GCF score, 0.484-, 0.315-, and 0.414-unit decreases in MMSE score, and 69%, 45%, and 63% greater odds of low MMSE scores (≤25), respectively. Such adverse effects were attenuated among NSAIDs users compared to non-users. This study elucidates the short-term impacts of air pollution on cognition and warrants further investigations on the modifying effects of NSAIDs.