Studying the early events that occur after viral infection in humans is difficult unless one intentionally infects volunteers in a human challenge study. Here, we use data about severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) in such a study in combination with mathematical modeling to gain insights into the relationship between the amount of virus in the upper respiratory tract and the immune response it generates. We propose a set of dynamic models of increasing complexity to dissect the roles of target cell limitation, innate immunity, and adaptive immunity in determining the observed viral kinetics. We introduce an approach for modeling the effect of humoral immunity that describes a decline in infectious virus after immune activation. We fit our models to viral load and infectious titer data from all the untreated infected participants in the study simultaneously. We found that a power-law with a power < 1 describes the relationship between infectious virus and viral load. Viral replication at the early stage of infection is rapid, with a doubling time of ~2 h for viral RNA and ~3 h for infectious virus. We estimate that adaptive immunity is initiated ~7 to 10 d postinfection and appears to contribute to a multiphasic viral decline experienced by some participants; the viral rebound experienced by other participants is consistent with a decline in the interferon response. Altogether, we quantified the kinetics of SARS-CoV-2 infection, shedding light on the early dynamics of the virus and the potential role of innate and adaptive immunity in promoting viral decline during infection.