The relationship between SARS-CoV-2 viral load and risk of disease progression remains largely undefined in coronavirus disease 2019 (COVID-19). Here, we quantify SARS-CoV-2 viral load from participants with a diverse range of COVID-19 disease severity, including those requiring hospitalization, outpatients with mild disease, and individuals with resolved infection. We detected SARS-CoV-2 plasma RNA in 27% of hospitalized participants, and 13% of outpatients diagnosed with COVID-19. Amongst the participants hospitalized with COVID-19, we report that a higher prevalence of detectable SARS-CoV-2 plasma viral load is associated with worse respiratory disease severity, lower absolute lymphocyte counts, and increased markers of inflammation, including C-reactive protein and IL-6. SARS-CoV-2 viral loads, especially plasma viremia, are associated with increased risk of mortality. Our data show that SARS-CoV-2 viral loads may aid in the risk stratification of patients with COVID-19, and therefore its role in disease pathogenesis should be further explored.