The gut microbiota is reportedly involved in neurodegenerative disorders, and exploration of differences in the gut microbiota in different cognitive status could provide clues for early detection and intervention in cognitive impairment. Here, we used data from the Taizhou Imaging Study (N = 516), a community-based cohort, to compare the overall structure of the gut microbiota at the species level through metagenomic sequencing, and to explore associations with cognition. Interestingly, bacteria capable of producing short-chain fatty acids (SCFAs), such as Bacteroides massiliensis, Bifidobacterium pseudocatenulatum, Fusicatenibacter saccharivorans and Eggerthella lenta, that can biotransform polyphenols, were positively associated with better cognitive performance (p < 0.05). Although Diallister invisus and Streptococcus gordonii were not obviously related to cognition, the former was dominant in individuals with mild cognitive impairment (MCI), while the later was more abundant in cognitively normal (CN) than MCI groups, and positively associated with cognitive performance (p < 0.05). Functional analysis further supported a potential role of SCFAs and lactic acid in the association between the gut microbiota and cognition. The significant associations persisted after accounting for dietary patterns. Collectively, our results demonstrate an association between the gut microbiota and cognition in the general population, indicating a potential role in cognitive impairment. The findings provide clues for microbiome biomarkers of dementia, and insight for the prevention and treatment of dementia.