Since the emergence of SARS-CoV-2 Omicron BA.1 and BA.2, several Omicron sublineages have emerged, supplanting their predecessors. Here we compared the neutralization of Omicron sublineages BA.1, BA.2, BA.4 and BA.5 by human sera collected from individuals who were infected with the ancestral B.1 (D614G) strain, who were vaccinated (3 doses) or with breakthrough infection with pre-Omicron strains (Gamma or Delta). All Omicron sublineages exhibited extensive escape from all sera when compared to the ancestral B.1 strain and to Delta, albeit to different levels depending on the origin of the sera. Convalescent sera were unable to neutralize BA.1, and partly neutralized BA.2, BA.4 and BA.5. Vaccinee sera partly neutralized BA.2, but BA.1, BA.4 and BA.5 evaded neutralizing antibodies (NAb). Some breakthrough infections (BTI) sera were non-neutralizing. Neutralizing BTI sera had similar neutralizing ability against all Omicron sublineages. Despite similar levels of anti-Spike and anti-Receptor Binding Domain (RBD) antibodies in all groups, BTI sera had the highest cross-neutralizing ability against all Omicron sublineages and convalescent sera were the least neutralizing. Antibody avidity inferred from the NT50:antibody titer ratio was highest in sera from BTI patients, underscoring qualitative differences in antibodies elicited by infection or vaccination. Together, these findings highlight the importance of vaccination to trigger highly cross-reactive antibodies that neutralize phylogenetically and antigenically distant strains, and suggest that immune imprinting by first generation vaccines may restrict, but not abolish, cross-neutralization.