Physical distancing reduces transmission risks and slows the spread of COVID-19. Yet compliance with shelter-in-place policies issued by local and regional governments in the United States was uneven and may have been influenced by science skepticism and attitudes towards topics of scientific consensus. Using county-day measures of physical distancing derived from cell phone location data, we demonstrate that the proportion of people who stayed at home after shelter-in-place policies went into effect in March and April 2020 in the United States was significantly lower in counties with a high concentration of science skeptics. These results are robust to controlling for other potential drivers of differential physical distancing, such as political partisanship, income, education and COVID severity. Our findings suggest that public health interventions that take local attitudes towards science into account in their messaging may be more effective.