Physical activity (PA) is one of the most fundamental traits in the animal kingdom, has pervasive health benefits, and is genetically influenced. Using data from the Million Veteran Program, we conducted genetic analyses of leisure, work and home-time PA. For leisure, we included 189,812 individuals of European ancestry (SNP-based heritability (h) = 0.083 ± 0.005), 27,044 of African ancestry (h = 0.034 ± 0.017) and 10,263 of Latin American ancestry (h = 0.083 ± 0.036) in a cross-ancestry meta-analysis with UK Biobank data, identifying 70 lead variants. Leisure-time PA was genetically distinct from PA at home or work, with the latter two showing less health benefit with respect to health outcomes and lifespan. Mendelian randomization analyses showed a protective role of leisure-time PA against COVID-19 hospitalization (β = -0.067 ± 0.016; P = 2.8 × 10), and with other traits including cardiovascular and respiratory system diseases, metabolic traits and aging. These findings provide new insights into the biology of PA, showing specific causal health benefits of leisure-time PA.