Disruption of T cell memory during severe immune suppression results in reactivation of chronic viral infections, such as Epstein Barr virus (EBV) and Cytomegalovirus (CMV). How different subsets of memory T cells contribute to the protective immunity against these viruses remains poorly defined. In this study we examined the compartmentalization of virus-specific, tissue resident memory CD8+ T cells in human lymphoid organs. This revealed two distinct populations of memory CD8+ T cells, that were CD69+CD103+ and CD69+CD103-, and were retained within the spleen and tonsils in the absence of recent T cell stimulation. These two types of memory cells were distinct not only in their phenotype and transcriptional profile, but also in their anatomical localization within tonsils and spleen. The EBV-specific, but not CMV-specific, CD8+ memory T cells preferentially accumulated in the tonsils and acquired a phenotype that ensured their retention at the epithelial sites where EBV replicates. In vitro studies revealed that the cytokine IL-15 can potentiate the retention of circulating effector memory CD8+ T cells by down-regulating the expression of sphingosine-1-phosphate receptor, required for T cell exit from tissues, and its transcriptional activator, Kruppel-like factor 2 (KLF2). Within the tonsils the expression of IL-15 was detected in regions where CD8+ T cells localized, further supporting a role for this cytokine in T cell retention. Together this study provides evidence for the compartmentalization of distinct types of resident memory T cells that could contribute to the long-term protection against persisting viral infections.