This study reports the incidence of hearing loss and associated factors, and differences across sex and race groups. Participants were from a longitudinal community-based cohort study. Hearing loss was defined as a pure-tone average of thresholds at frequencies 0.5, 1.0, 2.0, and 4.0 kHz > 25 dB HL in the worse ear. We used Cox proportional hazards regression models to determine demographic and health-related factors associated with incident hearing loss, in the entire sample and across sex (male/female) and race (White/Black) groups, separately. This study included 754 participants without hearing loss at baseline (mean age 56.7 [SD 16.2] years; 67.6% female; 24.2% racial Minority [22.9% Black]), contributing 3,771 person years (PY). The incidence rate of hearing loss was 44.8 (95% CI 38.5, 52.1) per 1000 PY, and incidence increased with age. In age-adjusted models, older age (unadjusted), male sex, noise exposure, and > 5 to 15 smoking pack years were associated with incident hearing loss; associations with noise exposure and smoking pack years did not remain in age-sex adjusted models. Some factors associated with incident hearing loss varied across sex and race groups. Interventions to prevent or manage hearing loss, which is a common public health concern, may vary across demographic groups.