Testosterone and insulin-like growth factor-1 (IGF-1) are essential factors for the maintenance of bone health in men. However, the results for the association of testosterone and IGF-1 with bone parameters were not consistent in prior studies. We evaluated the relationship of testosterone, sex hormone-binding globulin (SHBG), and IGF-1 with bone mineral density (BMD) and bone turnover markers (BTMs) in Korean men. We enrolled 1227 men aged ≥50 years in this cross-sectional study. Serum levels of total testosterone (TT), SHBG, IGF-1, osteocalcin, and C-terminal cross-linking telopeptide of type I collagen (CTX) were measured. Free testosterone (FT) was calculated using Vermeulen's method. BMD was measured by dual-energy X-ray absorptiometry. TT level was not related to BMD or BTMs in the unadjusted model; however, after adjusting for SHBG and IGF-1, the association between TT and BTMs was significant (β = -0.139 for osteocalcin and β = -0.204 for CTX). SHBG levels were negatively associated with lumbar BMD, and positively associated with BTMs in all models. As SHBG level increased, the prevalence of osteopenia or osteoporosis defined by BMD significantly increased (OR of 1SD change, 1.24). IGF-1 levels were significantly related with BMD, but not with BTMs. Meanwhile, FT levels were positively associated with BMD and negatively associated with BTMs. In conclusion, SHBG levels were independently related with bone parameters and osteopenia in men aged ≥50 years. IGF-1 levels were positively associated with BMD, but not with BTMs. SHBG may play a role in regulating age-related bone loss in men after middle-age.