Little is known about the prognostic significance of specific characteristics of magnetic resonance imaging (MRI) measured plaque in the superficial femoral artery (SFA). Associations of MRI-measured plaque quantity, lumen area, and plaque composition in the SFA with subsequent mobility loss were studied in people with lower extremity peripheral artery disease (PAD). Participants with an ankle-brachial index (ABI) < 1.00 were identified from Chicago medical centers and underwent direct visualization of atherosclerotic plaque in the SFA using MRI. Participants were followed annually for up to 4 years. Mobility loss was defined as becoming unable to walk up and down a flight of stairs or walk one-quarter of a mile without assistance among participants without mobility impairment at baseline. Analyses adjusted for age, sex, race, comorbidities, ABI, physical activity, and other confounders. Of 308 PAD participants without baseline mobility impairment, 100 (32.5%) developed mobility loss during follow-up. Compared to the lowest mean plaque area tertile at baseline, participants in the highest (worst) plaque area tertile had a higher rate of mobility loss (hazard ratio (HR) = 2.08, 95% confidence interval (CI) = 1.14-3.79, p = 0.018). Compared to the highest mean lumen area tertile, the smallest (worst) mean lumen area tertile was associated with greater mobility loss (HR = 2.18, 95% CI = 1.20-3.96, p = 0.011). Neither lipid rich necrotic core nor calcium in the SFA were associated with mobility loss. In conclusion, greater plaque quantity and smaller lumen area in the proximal SFA, but not lipid rich necrotic core or calcium, were associated with higher mobility loss in people with PAD.