Residential segregation is a common aspect of the urban experiences of African-Americans in the United States (US), yet few studies have considered how segregation might influence perinatal health. Here, we develop a conceptual model of relationships between segregation and birth outcomes and test the implications of the model in a sample of 434,376 singleton births to African-American women living in 225 US Metropolitan Statistical Areas (MSAs). Data from the National Center for Health Statistics 2002 birth files were linked to data from the 2000 US Census and two distinct measures of segregation: an index of isolation (the probability that an African-American resident will encounter another African-American resident in any random neighborhood encounter) and an index of clustering (the extent to which African-Americans live in contiguous neighborhoods). Using multilevel regression models, controlling for individual- and MSA-level socioeconomic status and other covariates, we found higher isolation was associated with lower birthweight, higher rates of prematurity and higher rates of fetal growth restriction. In contrast, higher clustering was associated with more optimal outcomes. We propose that isolation reflects factors associated with segregation that are deleterious to health including poor neighborhood quality, persistent discrimination and the intra-group diffusion of harmful health behaviors. Associations with clustering may reflect factors associated with segregation that are health-promoting such as African-American political power empowerment, social support and cohesion. Declines in isolation could represent positive steps toward improving birth outcomes among African-American infants while aspects of racial contiguity appear to be mitigating or indeed beneficial. Segregation is a complex multidimensional construct with both deleterious and protective influences on birth outcomes, depending on the dimensions under consideration. Further research to understand racial/ethnic and economic health disparities could benefit from a focus on the contributory role of neighborhood attributes associated with the dimensions segregation and other social geographies.