The protective component of specific memory B cells (MBCs) response relative to serum antibody response in primary SARS-CoV-2 infection is not well understood. Using a relatively unbiased B-cell culture method with a limited number of MBCs in each well (100 cells/well), we characterized the fine specificity of MBC responses against SARS-CoV-2 infection. While serum spike antibody is predominantly against S2 domain, the memory B cells mainly recognize S1 domain. The 44.4-85.3% of S-binding MBCs are specific to S1 domain. High frequency of MBCs (30-62% of SARS-CoV-2 S-specific MBCs) cross-reacting with SARS-CoV S has also been demonstrated. 22-33% of S1-binding MBCs were cross-reactive with the SARS-CoV RBD. In addition, a panel of human monoclonal Ab was derived from S1-binding MBCs recognizing six group epitopes (groups 1-6). Among them, RBD-specific Ab (826) in group 4 and cross-reactive Ab (808) could resist the neutralizing escape of omicron. Herein, we demonstrated that a dominant S1-directed MBC response was generated during primary SARS-CoV-2 infection. More importantly, the cross-reactive RBD-directed MBCs against SARS-CoV may protect against emerging SARS-CoV-2 variants.