The objective of this study was to characterize bone, muscle, and fat measurements in early adulthood by youth sport participation. The study sample included 328 Iowa Bone Development Study participants (184 females). Organized sport participation was longitudinally assessed (14 times on average) using a physical activity questionnaire from age 6 to 17 years. At age 23 years, bone, lean mass (a proxy measure of muscle mass), and fat mass were assessed using dual-energy X-ray absorptiometry (DXA). Tibial bone stiffness, a bone strength indicator, was determined using high-resolution multi-row detector computed tomography (CT) and Finite Element Analysis. Longitudinal youth sport participation patterns were categorized into consistent participation, drop-out, and no participation. Sex-specific multivariable linear regression analyses were conducted to examine the associations between youth sport participation patterns and bone mineral content (BMC), lean mass index (LMI), fat mass index (FMI), BMC-to-lean ratio, lean-to-fat ratio, and tibial bone stiffness at age 23 years. After adjusting for covariates, males in the consistent youth sport participation and drop-out groups had 377 g and 192 g higher BMC, 1.6 kg/m and 1.5 kg/m higher LMI, and 112 kN/mm and 76 kN/mm higher bone stiffness at age 23, compared to males in the no-participation group ( < 0.01). Females in the consistent youth sport participation group had 1.4 kg/m lower FMI at age 23, compared to females in the no-participation group ( = 0.04). The BMC-to-lean ratio was higher among males ( = 0.02) and females ( < 0.01) in the consistent participation group, compared to their counterparts in the no-participation group; the lean-to-fat ratio also tended to be higher in males ( = 0.06) and females ( = 0.11). This study suggests sex differences in the benefits of youth sport participation on adulthood body compartments: healthier bone and muscle for males and healthier body fat for females. This study provides evidence to support the promotion of youth sport participation for healthy body composition later in life.