The clinical success of cancer immune checkpoint blockade (ICB) has refocused attention on tumor-infiltrating lymphocytes (TILs) across cancer types. The outcome of immune checkpoint inhibitor therapy in cancer patients has been linked to the quality and magnitude of T cell, NK cell, and more recently, B cell responses within the tumor microenvironment. State-of-the-art single-cell analysis of TIL gene expression profiles and clonality has revealed a remarkable degree of cellular heterogeneity and distinct patterns of immune activation and exhaustion. Many of these states are conserved across tumor types, in line with the broad responses observed clinically. Despite this homology, not all cancer types with similar TIL landscapes respond similarly to immunotherapy, highlighting the complexity of the underlying tumor-immune interactions. This observation is further confounded by the strong prognostic benefit of TILs observed for tumor types that have so far respond poorly to immunotherapy. Thus, while a holistic view of lymphocyte infiltration and dysfunction on a single-cell level is emerging, the search for response and prognostic biomarkers is just beginning. Within this review, we discuss recent advances in the understanding of TIL biology, their prognostic benefit, and their predictive value for therapy.