We conducted four rounds of cognitive testing of self-report items that included 66 sociodemographically diverse participants, then field tested the three best items from the cognitive testing in a clinic waiting room (N = 351) and in an online social networking site for men who have sex with men (N = 6,485). As part of the online survey we conducted a randomized assessment of two versions of the adherence questionnaire-one which asked about adherence to a specific antiretroviral medication, and a second which asked about adherence to their "HIV medicines" as a group. Participants were better able to respond using adjectival and adverbial scales than visual analogue or percent items. The internal consistency reliability of the three item adherence scale was 0.89. Mean scores for the two different versions of the online survey were similar (91.0 vs. 90.2, p < 0.05), suggesting that it is not necessary, in general, to ask about individual medications in an antiretroviral therapy regimen when attempting to describe overall adherence.