The coronavirus disease 2019 (COVID-19) pandemic has emphasized the need for development of a rapid diagnostic device for the effective treatment and its mitigation. Lateral flow immunoassay (LFIA) belongs to a class of diagnostic devices, which has the benefit of providing quick results, easy to handle, low cost, and on-site applicable. So far, several LFIA has been developed for the detection of infectious severe acute respiratory syndrome-coronavirus-2 (SARS-CoV-2), however, only a few of them are antigen (Ag)-based. Here, we describe an antibody (Ab)-labeled gold nanoparticles (AuNPs)-based LFIA (AuNPs-LFIA) for the detection of Receptor-Binding Domain (RBD) of SARS-CoV-2. For this, RBD Ab of SARS-CoV-2 was conjugated with the AuNPs, which served as a detecting probe. The fabricated LFIA strip was optimized for different parameters such as membrane pore size, blocking conditions, Ab coating concentration, and conjugate incubation. The optimized LFIA strips were validated in spiked buffer samples and the optimal limit of detection was found to be 1 ng/ml, which was confirmed by a smartphone-based application. Moreover, the developed AuNPs-LFIA strips effectively detected RBD Ag in 100 clinical samples with 94.3% sensitivity and 90.9% specificity in clinical samples when compared with the gold standard (RT-PCR). The fabricated LFIAs are reported to have storage stability of up to 21 days at 4°C and room temperature (RT). Hence, the developed LFIA can be used as a portable, cost-effective diagnostic device for rapid detection of SARS-CoV-2.