The issue of air pollution is gaining increasing attention worldwide, and mounting evidence has shown an association between air pollution and cognitive decline. The aim of this study was to investigate the relationships between air pollutants and cognitive impairment using the Mini-Mental State Exam (MMSE) and its sub-domains. In this study, we used data from the Taiwan Biobank combined with detailed daily data on air pollution. Cognitive function was assessed using the MMSE and its five subgroups of cognitive functioning. After multivariable linear regression analysis, a high level of particulate matter with an aerodynamic diameter of ≤2.5 μm (PM), low ozone (O), high carbon monoxide (CO), high sulfur dioxide (SO), high nitric oxide (NO), high nitrogen dioxide (NO), and high nitrogen oxide (NO) were significantly associated with low total MMSE scores. Further, high SO and low O were significantly associated with low MMSE G1 scores. Low O, high CO, high SO, high NO, and high NO were significantly associated with low MMSE G4 scores, and high PM, high particulate matter with an aerodynamic diameter of ≤10 μm (PM), high SO, high NO, and high NO were significantly associated with low MMSE G5 scores. Our results showed that exposure to different air pollutants may lead to general cognitive decline and impairment of specific domains of cognitive functioning, and O may be a protective factor. These findings may be helpful in the development of policies regarding the regulation of air pollution.