Higher visit-to-visit cholesterol has been associated with cognitive decline. However, the association between long-term increase or decrease in cholesterol and cognitive decline remains unclear. A total of 4,915 participants aged ≥45 years with normal cognition in baseline were included. The participants were divided into four groups, namely low-low, low-high, high-low, and high-high, according to the diagnostic thresholds of total cholesterol (TC), non-high-density lipoprotein cholesterol (NHDL-C), low-density lipoprotein cholesterol, and high-density lipoprotein cholesterol (HDL-C) after 4 years of follow-up. Cognitive function was assessed by episodic memory and mental intactness. Binary logistic regression was used to analyse the association of cholesterol variation with cognitive decline. Among the participants, 979 (19.9%) experienced global cognitive decline. The odds ratio (OR) of global cognitive and memory function decline were remarkably lower in participants in the low-high NHDL-C group than those in the low-low group [OR and 95% confidence interval (CI): 0.50 [0.26-0.95] for global cognitive decline, 0.45 [0.25-0.82] for memory function decline]. The lower OR was also significant in females (OR [95% CI]: 0.38 [0.17-0.87] for global cognitive decline; 0.44 [0.19-0.97] for memory function decline) and participants without cardiovascular disease (OR [95% CI]: 0.31 [0.11-0.87] for global cognitive decline; 0.34 [0.14-0.83] for memory function decline). The increases in other cholesterol were also negatively associated with the risk of cognitive decline although not significantly. A longitudinal increase in NHDL-C may be protective for cognition in females or individuals without cardiovascular disease.