The spike glycoprotein of severe acute respiratory syndrome coronavirus-2 (SARS-CoV-2) continues to accumulate substitutions, leading to breakthrough infections of vaccinated individuals. It remains unclear if exposures to antigenically distant SARS-CoV-2 variants can overcome memory B cell biases established by initial SARS-CoV-2 encounters. We determined the specificity and functionality of antibody and B cell responses following exposure to BA.5 and XBB variants in individuals who received ancestral SARS-CoV-2 mRNA vaccines. BA.5 exposures elicited antibody responses that targeted epitopes conserved between the BA.5 and ancestral spike. XBB exposures also elicited antibody responses that primarily targeted epitopes conserved between the XBB.1.5 and ancestral spike. However, unlike BA.5, a single XBB exposure elicited low frequencies of XBB.1.5-specific antibodies and B cells in some individuals. Pre-existing cross-reactive B cells and antibodies were correlated with stronger overall responses to XBB but weaker XBB-specific responses, suggesting that baseline immunity influences the activation of variant-specific SARS-CoV-2 responses.