This cross-sectional study assessed determinants of HIV clinic appointment attendance and antiretroviral treatment (ART) adherence among 300 male fisherfolk on ART in Wakiso District, Uganda. Multi-level factors associated with missed HIV clinic visits included those at the individual (age, AOR = 0.98, 95% CI 0.97-0.99), interpersonal (being single/separated from partner, AOR: 1.25, 95% CI 1.01-1.54), normative (anticipated HIV stigma, AOR: 1.55, 95% CI 1.05-2.29) and physical/built environment-level (travel time to the HIV clinic, AOR: 1.11, 95% CI 1.02-1.20; structural-barriers to ART adherence, AOR: 1.27, 95% CI 1.04-1.56; accessing care on a landing site vs. an island, AOR: 1.35, 95% CI 1.08-1.67). Factors associated with ART non-adherence included those at the individual (age, β: - 0.01, η = 0.03; monthly income, β: - 0.01, η = 0.02) and normative levels (anticipated HIV stigma, β: 0.10, η = 0.02; enacted HIV stigma, β: 0.11, η = 0.02). Differentiated models of HIV care that integrate stigma reduction and social support, and reduce the number of clinic visits needed, should be explored in this setting to reduce multi-level barriers to accessing HIV care and ART adherence.