In response to infection, T cells adopt a range of differentiation states, creating numerous heterogeneous subsets that exhibit different phenotypes, functions, and migration patterns. This T cell heterogeneity is a universal feature of T cell immunity, needed to effectively control pathogens in a context-dependent manner and generate long-lived immunity to those pathogens. Here, we review new insights into differentiation state dynamics and population heterogeneity of CD8+ T cells in acute and chronic viral infections and cancer and highlight the parallels and distinctions between acute and chronic antigen stimulation settings. We focus on transcriptional and epigenetic networks that modulate the plasticity and terminal differentiation of antigen-specific CD8+ T cells and generate functionally diverse T cell subsets with different roles to combat infection and cancer.