SARS-CoV-2 Delta and Omicron are globally relevant variants of concern. Although individuals infected with Delta are at risk of developing severe lung disease, infection with Omicron often causes milder symptoms, especially in vaccinated individuals. The question arises of whether widespread Omicron infections could lead to future cross-variant protection, accelerating the end of the pandemic. Here we show that without vaccination, infection with Omicron induces a limited humoral immune response in mice and humans. Sera from mice overexpressing the human ACE2 receptor and infected with Omicron neutralize only Omicron, but not other variants of concern, whereas broader cross-variant neutralization was observed after WA1 and Delta infections. Unlike WA1 and Delta, Omicron replicates to low levels in the lungs and brains of infected animals, leading to mild disease with reduced expression of pro-inflammatory cytokines and diminished activation of lung-resident T cells. Sera from individuals who were unvaccinated and infected with Omicron show the same limited neutralization of only Omicron itself. By contrast, Omicron breakthrough infections induce overall higher neutralization titres against all variants of concern. Our results demonstrate that Omicron infection enhances pre-existing immunity elicited by vaccines but, on its own, may not confer broad protection against non-Omicron variants in unvaccinated individuals.