Repetitive pathogen exposure leads to the dominant outgrowth of T cell clones with high T cell receptor (TCR) affinity to the relevant pathogen-associated antigens. However, low-affinity clones are also known to expand and form immunological memory. While these low-affinity clones contribute less immunity to the original pathogen, their role in protection against pathogens harboring immune escape mutations remains unclear. Based on identification of the TCR repertoire and functionality landscape of naive epitope-specific CD8 T cells, we reconstructed defined repertoires that could be followed as polyclonal populations during immune responses in vivo. We found that selective clonal expansion is governed by clear TCR avidity thresholds. Simultaneously, initial recruitment of broad TCR repertoires provided a polyclonal niche from which flexible secondary responses to mutant epitopes could be recalled. Elucidating how T cell responses develop "from scratch" is informative for the development of enhanced immunotherapies and vaccines.