Memory T cells are a highly heterogeneous collection of antigen-experienced cells that undergo dynamic adaptations upon antigen re-encounter and environmental signals. This heterogeneity hinders studies on memory T cell durability and age-related dysfunction. Using chronic Epstein-Barr virus (EBV) infection and barcode-enabled antigen tracing, we assess the influence of age on memory states at the level of single antigen-specific CD8 T cells. In young adults (<40 years), EBV-specific CD8 T cells recognizing different antigenic peptides assume divergent preferred differentiation phenotypes. In older adults (>65-years), antigen-specific cells show largely distinct phenotypic and transcriptomic aging trajectories. Common to many albeit not all antigen-specific populations are maintained TCR diversity, gained natural killer cell-like, innate signatures and lost stem-like features while no evidence is seen for cellular senescence or exhaustion. TCR avidity contributes to these phenotypic differences and aging-related changes. Collectively, our data uncover divergent antigen-guided aging shifts in memory T cell phenotypes, which are informative for antigen selection in optimizing vaccine design and adoptive T cell therapy.