Antibody detection methods for viral infections have received broad attention due to the COVID-19 pandemic. In addition, there remains an ever-increasing need to quantitatively evaluate the immune response to develop vaccines and treatments for COVID-19. Here, we report an analytical method for the rapid and quantitative detection of SARS-CoV-2 antibody in human serum by fluorescence polarization immunoassay (FPIA). A recombinant SARS-CoV-2 receptor binding domain (RBD) protein labeled with HiLyte Fluor 647 (F-RBD) was prepared and used for FPIA. When the anti-RBD antibody in human serum binds to F-RBD, the degree of polarization (P) increases by suppressing the rotational diffusion of F-RBD. The measurement procedure required only mixing a reagent containing F-RBD with serum sample and measuring the P value with a portable fluorescence polarization analyzer after 15 min incubation. We evaluated analytical performance of the developed FPIA system using 30 samples: 20 COVID-19 positive sera and 10 negative sera. The receiver operating characteristic curve drawn with the obtained results showed that this FPIA system had high accuracy for discriminating COVID-19 positive or negative serum (AUC = 0.965). The total measurement time was about 20 min, and the serum volume required for measurement was 0.25 μL. Therefore, we successfully developed the FPIA system that enables rapid and easy quantification of SARS-CoV-2 antibody. It is believed that our FPIA system will facilitate rapid on-site identification of infected persons and deepen understanding of the immune response to COVID-19.