The SARS-CoV-2 virus poses a significant risk to immunocompromised patients, who display weakened immunity and reduced seroconversion following infection and vaccination. In this study, we recruited 19 hospitalized patients with immune disorders (ImCo) and 4 immunocompetent controls (ICC) with COVID-19. We evaluated their serological, humoral, and cellular immune responses at <30 days and >90 days post-symptom onset. ICC patients showed robust B and T cell responses against SARS-CoV-2, indicated by detectable antibody levels, memory antibody-secreting cells (mASCs) towards the spike protein and spike-specific CD4 and CD8 T cells. ImCo patients showed impaired immune responses, with lower levels of B cell responses. Further subdivision of the ImCo patients demonstrates that solid organ transplant (SOT) patients generated B cell responses similar to ICC patients, whereas the other ImCo patients, including patients with hematological malignancies and anti-CD20 therapy, did not. Absolute T cell numbers and spike-specific CD4 and CD8 T cell responses were low in the ImCo patients at <30 days but increased at later time points. Our findings suggest that even when B cell responses were reduced, patients could present a T cell response, suggesting a more successful line of passive immunization for immunocompromised individuals focusing on boosting T cell responses.