This study investigated whether HIV-positive participants, stable on combined antiretroviral therapy (cART), showed cognitive impairments relative to HIV-negative controls; and whether clinical and neuroimaging factors correlated with cognitive function in the HIV-positive participants. One hundred and twenty-six white men who have sex with men, of whom 78 were HIV-positive and stable on cART and 48 were HIV negative, were recruited to this cross-sectional study. The median age of HIV-positive participants in this study was 47. They underwent clinical and neuropsychological evaluation and magnetic resonance imaging of the brain, including diffusion tensor imaging (DTI). Cognitive scores for both groups were compared, and regression models were run to explore the influence of clinical, psychiatric, lifestyle, and neuroimaging variables on cognition. The prevalence of neurocognitive impairment, using the multivariate normative comparison criteria, was 28% in HIV-positive participants and 5% in HIV-negative participants. After covarying for age, years of education, and non-English speaking background, there were significant differences between the HIV group and the controls across four cognitive domains. The HIV group showed significantly higher mean diffusivity (MD) and lower fractional anisotropy (FA) than the control group on DTI. Although anxiety levels were clinically low, anxiety and DTI measures were the only variables to show significant correlations with cognitive function. In the HIV group, poorer cognitive performance was associated with higher MD and lower FA on DTI and higher (albeit clinically mild) levels of anxiety. Our findings suggest that white matter changes and subtle anxiety levels contribute independently to cognitive impairment in HIV.