HIV can persist in a latent form as integrated DNA (provirus) in resting CD4 T cells of infected individuals and as such is unaffected by antiretroviral therapy (ART). Despite being a major obstacle for eradication efforts, the genetic variation and timing of formation of this latent reservoir remains poorly understood. Previous studies on when virus is deposited in the latent reservoir have come to contradictory conclusions. To reexamine the genetic variation of HIV in CD4 T cells during ART, we determined the divergence in envelope sequences collected from 10 SIV infected rhesus macaques. We found that the macaques displayed a biphasic decline of the viral divergence over time, where the first phase lasted for an average of 11.6 weeks (range 4-28 weeks). Motivated by recent observations that the HIV-infected CD4 T cell population is composed of short- and long-lived subsets, we developed a model to study the divergence dynamics. We found that SIV in short-lived cells was on average more diverged, while long-lived cells harbored less diverged virus. This suggests that the long-lived cells harbor virus deposited starting earlier in infection and continuing throughout infection, while short-lived cells predominantly harbor more recent virus. As these cell populations decayed, the overall proviral divergence decline matched that observed in the empirical data. This model explains previous seemingly contradictory results on the timing of virus deposition into the latent reservoir, and should provide guidance for future eradication efforts.