The aim of the study was to examine and describe the individual and structural-environmental factors that shape the vulnerability of brothel-based female sex workers (FSWs) in Ibadan, southwest Nigeria to HIV infection. A descriptive qualitative research design was utilised to elicit data, using in-depth interviews and focus group discussions, from 60 randomly selected participants in four brothels. A thematic analysis of data was undertaken following transcription and validation of interviews. Five themes emerged from the data: (i) flawed knowledge and fatalistic attitudes; (ii) the psychosocial and economic context of sex work; (iii) religious beliefs, stigma and risk taking; (iv) barriers to HIV testing; and (v) legal and policy constraints to sex work. We describe the complex interaction between these themes and how they combine to increase the risk of HIV infection among FSWs. The impact of previous interventions to reduce the risk of HIV infection among FSWs has been limited by personal and structural factors; hence we recommend that new strategies that recognise the practical constraints to HIV prevention among FSWs are urgently needed to make the environment of commercial work safer for FSWs, their clients, and by extension the general population.