Isolation and loneliness are related to various aspects of health. Physical performance is a central component of health. However, its relationship with isolation and loneliness is not well understood. We therefore assessed the relationship between loneliness, different aspects of social isolation, and physical performance over time. 8,780 participants from the English Longitudinal Study of Ageing, assessed three times over 8 years of follow-up, were included. Measures included physical performance (Short Physical Performance Battery), loneliness (modified UCLA Loneliness Scale), and isolation considered in three ways (domestic isolation, social disengagement, low social contact). Fixed effects regression models were used to estimate the relationship between changes in these parameters. Missing data were imputed to account for variable response and ensure a representative sample. Loneliness, domestic isolation and social disengagement were longitudinally associated with poorer physical performance when accounting for both time-invariant and time-variant confounders (loneliness: coef = - 0.06, 95% CI - 0.09 to - 0.02; domestic isolation: coef = - 0.32, 95% CI - 0.46 to - 0.19; social disengagement: coef = - 0.10, 95% CI - 0.12 to - 0.07). Low social contact was not associated with physical performance. These findings suggest social participation and subjectively meaningful interpersonal interactions are related to physical performance, and highlight additional considerations regarding social distancing related to COVID-19 control measures.