Serological detection of antibodies for diagnosing infectious diseases has advantages in facile diagnostic procedures, thereby contributing to controlling the spread of the pathogen, such as in the recent SARS-CoV-2 pandemic. Lateral flow immunoassay (LFIA) is a representative serological antibody detection method suitable for on-site applications but suffers from low clinical accuracy. To achieve a simple and rapid serological screening as well as the sensitive quantification of antibodies against SARS-CoV-2, a colorimetric and fluorescent dual-mode serological LFIA sensor incorporating metal-enhanced fluorescence (MEF) was developed. For the strong fluorescence signal amplification, fluorophore Cy3 was immobilized onto gold nanoparticles (AuNPs) with size-controllable spacer polyethyleneglycol (PEG) to maintain an optimal distance to induce MEF. The sensor detects the target IgG with a concentration as low as 1 ng mL within 8 minutes. The employment of the MEF into the dual-mode serological LFIA sensor shows a 1000-fold sensitivity improvement compared with that of colorimetric LFIAs. The proposed serological LFIA sensor was tested with 73 clinical samples, showing sensitivity, specificity, and accuracy of 95%, 100%, and 97%, respectively. In conclusion, the dual-mode serological LFIA has great potential for application in diagnosis and an epidemiological survey of vaccine efficacy and immunity status of individuals.