The rapid diagnosis of SARS-CoV-2 is an essential aspect in the detection and control of the spread of COVID-19. We evaluated the accuracy of the rapid antigen test (RAT) using samples from the nasal cavity and nasopharynx based on sample collection timing and viral load. We enrolled 175 patients, of which 71 patients and 104 patients had tested positive and negative, respectively, based on real time-PCR. Nasal cavity and nasopharyngeal swab samples were tested using STANDARD Q COVID-19 Ag tests (Q Ag, SD Biosensor, Korea). The sensitivity of the Q Ag test was 77.5% (95% confidence interval [CI], 67.8−87.2%) for the nasal cavity and 81.7% (95% [CI, 72.7−90.7%) for the nasopharyngeal specimens. The RAT results showed a substantial agreement between the nasal cavity and nasopharyngeal specimens (Cohen’s kappa index = 0.78). The sensitivity of the RAT for nasal cavity specimens exceeded 89% for <5 days after symptom onset (DSO) and 86% for Ct of E and RdRp < 25. The Q Ag test performed fairly well, especially in the early DSO when a high viral load was present, and the nasal cavity swab can be considered an alternative site for the rapid diagnosis of COVID-19.