HIV-positive people who use illicit drugs typically achieve lower levels of adherence to antiretroviral therapy and experience higher rates of sub-optimal HIV/AIDS treatment outcomes. Given the dearth of longitudinal research into ART adherence dynamics, we sought to identify factors associated with transitioning into and out of optimal adherence to ART in a longitudinal study of HIV-infected people who use illicit drugs (PWUD) in a setting of universal no-cost HIV/AIDS treatment. Using data from a prospective cohort of community-recruited HIV-positive illicit drug users confidentially linked to comprehensive HIV/AIDS treatment records, we estimated longitudinal factors associated with losing or gaining ≥95% adherence in the previous six months using two generalized linear mixed-effects models. Among 703 HIV-infected ART-exposed PWUD, becoming non-adherent was associated with periods of homelessness (adjusted odds ratio [AOR] = 2.52, 95% confidence interval [95% CI]: 1.56-4.07), active injection drug use (AOR = 1.25, 95% CI: 1.01-1.56) and incarceration (AOR = 1.54, 95% CI: 1.10-2.17). Periods of sex work (AOR = 0.51, 95% CI: 0.34-0.75) and injection drug use (AOR = 0.62, 95% CI: 0.50-0.77) were barriers to becoming optimally adherent. Methadone maintenance therapy was associated with becoming optimally adherent (AOR = 1.87, 95% CI: 1.50-2.33) and was protective against becoming non-adherent (AOR = 0.52, 95% CI: 0.41-0.65). In conclusion, we identified several behavioural, social and structural factors that shape adherence patterns among PWUD. Our findings highlight the need to consider these contextual factors in interventions that support the effective delivery of ART to this population.