A rapid and accurate method for detection of virus (SARS-CoV-2)-specific antibodies is important to contain the 2019 coronavirus disease (COVID-19) outbreak, which is still urgently needed. Here, we develop a colorimetric-fluorescent dual-mode lateral flow immunoassay (LFIA) biosensor for rapid, sensitive, and simultaneous detection of SARS-CoV-2-specific IgM and IgG in human serum using spike (S) protein-conjugated SiO@Au@QD nanobeads (NBs) as labels. The assay only needs 1 μL of the serum sample, can be completed within 15 min, and is 100 times more sensitive than the colloidal gold-based LFIA. Two detection modes of our biosensor are available: the colorimetric mode for rapid screening of the patients with suspected SARS-CoV-2 infection without any special instrument and the fluorescent mode for sensitive and quantitative analyses to determine the concentrations of specific IgM/IgG in human serum and detect the infection early and precisely. We validated the proposed method using 16 positive serum samples from patients with COVID-19 and 41 negative samples from patients with other viral respiratory infections. The results demonstrated that combined detection of virus-specific IgM and IgG via SiO@Au@QD LFIA can identify 100% of patients with SARS-CoV-2 infection with 100% specificity.