Cognitive aging is a complex process influenced by diverse life experiences and environmental factors. However, some traditional studies have oversimplified this process by assuming that cognitive aging trajectories follow a uniform process and that all individuals will experience similar declines. This framework minimizes the impact of external factors, neglecting the diversity observed in the aging population. In fact, research has shown significant inter- and intraindividual variability in cognitive trajectories, with some individuals maintaining stable or even improving cognitive function, while others experience rapid decline. To address this gap, emerging research proposes promising alternatives to the homogenous modelling approaches used, focusing on the identification of latent classes of cognitive trajectories. In this work, we build on this by examining the complex interaction of heterogeneous cognitive trajectories with external factors during the aging process, using episodic memory as a measure of cognitive function. We use longitudinal data from 1746 individuals aged 60 and older, assessed at three times over eight years. Our findings revealed three distinct cognitive trajectories - low cognitive performance with early decline , unmodulated cognitive change and high cognitive performance with late decline - each uniquely influenced by specific lifestyle factors. These findings challenge the current theoretical model of cognitive aging by identifying that factors such as concentration activities and social engagement significantly influence the trajectories of low cognitive performance with early decline and high cognitive performance with late decline, whereas the trajectory of unmodulated cognitive change is largely unaffected by environmental influences. Overall, our results highlight the critical role of individual environmental susceptibility in shaping cognitive trajectories. This research provides key insights into the heterogeneity of cognitive aging and underscores the need for a research paradigm shift in understanding cognitive trajectories' heterogeneity. While further research is required to determine how these findings translate into practice, tailoring interventions to these newly identified cognitive trajectories, we can significantly improve individual and public health outcomes through more precise and effective social prescribing interventions.