In some instances, unsuppressed HIV has been associated with severe COVID-19 disease, but the mechanisms underpinning this susceptibility are still unclear. Here, we assessed the impact of HIV infection on the quality and epitope specificity of SARS-CoV-2 T cell responses in the first wave and second wave of the COVID-19 epidemic in South Africa. Flow cytometry was used to measure T cell responses following peripheral blood mononuclear cell stimulation with SARS-CoV-2 peptide pools. Culture expansion was used to determine T cell immunodominance hierarchies and to assess potential SARS-CoV-2 escape from T cell recognition. HIV-seronegative individuals had significantly greater CD4 T cell responses against the Spike protein compared to the viremic people living with HIV (PLWH). Absolute CD4 count correlated positively with SARS-CoV-2-specific CD4 and CD8 T cell responses (CD4 r=0.5, p=0.03; CD8 r=0.5, p=0.001), whereas T cell activation was negatively correlated with CD4 T cell responses (CD4 r=-0.7, p=0.04). There was diminished T cell cross-recognition between the two waves, which was more pronounced in individuals with unsuppressed HIV infection. Importantly, we identify four mutations in the Beta variant that resulted in abrogation of T cell recognition. Taken together, we show that unsuppressed HIV infection markedly impairs T cell responses to SARS-Cov-2 infection and diminishes T cell cross-recognition. These findings may partly explain the increased susceptibility of PLWH to severe COVID-19 and also highlights their vulnerability to emerging SARS-CoV-2 variants of concern.