Physical activity has been suggested to be beneficial in preventing disease and improving body function in older people. Older people's leisure-time physical activity (LTPA) is affected by various factors, especially environmental factors. However, the differences in the association between older people's LTPA and the built environment in different sex groups remain unclear. Perceived built environment scores and older people's LTPA were collected for 240 older people in Jinhua using the Neighborhood Environment Walkability Scale and International Physical Activity Questionnaire, respectively. A linear regression method was used to analyze the associations between older people's LTPA and the built environment in men, women, and all participants. The results showed that land use mix diversity was associated with LTPA in older people for both sexes. In men, LTPA was also associated with access to services. However, in women, LTPA was associated with residential density, street connectivity, and crime safety. The relationship varied when demographic variables were incorporated into the regression analysis. Those results indicated that a shorter perceived distance from home to destination would motivate older people to engage more in LTPA. Older people's LTPA was affected by various built environment factors according to different sex groups. Women's LTPA was generally more sensitive to the built environment. More studies are needed to confirm the association between LTPA in older people and the built environment in men and women in mid- or small-sized Chinese cities in the future.