COVID-19 vaccine design and vaccination rollout need to take into account a detailed understanding of antibody durability and cross-neutralizing potential against SARS-CoV-2 and emerging variants of concern (VOCs). Analyses of convalescent sera provide unique insights into antibody longevity and cross-neutralizing activity induced by variant spike proteins, which are putative vaccine candidates. Using sera from 38 individuals infected in wave 1, we show that cross-neutralizing activity can be detected up to 305 days pos onset of symptoms, although sera were less potent against B.1.1.7 (Alpha) and B1.351 (Beta). Over time, despite a reduction in overall neutralization activity, differences in sera neutralization potency against SARS-CoV-2 and the Alpha and Beta variants decreased, which suggests that continued antibody maturation improves tolerance to spike mutations. We also compared the cross-neutralizing activity of wave 1 sera with sera from individuals infected with the Alpha, the Beta or the B.1.617.2 (Delta) variants up to 79 days post onset of symptoms. While these sera neutralize the infecting VOC and parental virus to similar levels, cross-neutralization of different SARS-CoV-2 VOC lineages is reduced. These findings will inform the optimization of vaccines to protect against SARS-CoV-2 variants.