There is an urgent need for effective treatment and preventive vaccine to contain this devastating global pandemic, which requires a comprehensive understanding of humoral responses specific to SARS-CoV-2 during the disease progression and convalescent phase of COVID-19 patients. We continuously monitored the serum IgM and IgG responses specific to four SARS-CoV-2 related antigens, including the nucleoprotein (NP), receptor binding domain (RBD), S1 protein, and ectodomain (ECD) of the spike protein among non-severe and severe COVID-19 patients for seven weeks since disease onset. Most patients generated humoral responses against NP and spike protein-related antigens but with their distinct kinetics profiles. Combined detection of NP and ECD antigens as detecting antigen synergistically improved the sensitivity of the serological assay, compared to that of using NP or RBD as detection antigen. 80.7% of convalescent sera from COVID-19 patients revealed that the varying extents of neutralization activities against SARS-CoV-2. S1-specific and ECD-specific IgA responses were strongly correlated with the neutralization activities in non-severe patients, but not in severe patients. Moreover, the neutralizing activities of the convalescent sera were shown to significantly decline during the period between 21 days to 28 days after hospital discharge, accompanied by a substantial drop in RBD-specific IgA response. Our data provide evidence that are crucial for serological testing, antibody-based intervention, and vaccine design of COVID-19.