We evaluated the heterogeneity of the effect of known risk factors on breast cancer development based on breast density by using the Breast Imaging-Reporting and Data System (BI-RADS). In total, 4,898,880 women, aged 40-74 years, who participated in the national breast cancer screening program in 2009-2010 were followed up to December 2018. Increased age showed a heterogeneous association with breast cancer (1-year hazard ratio (HR) = 0.92, 1.00 (reference), 1.03, and 1.03 in women with BI-RADS density category 1, 2, 3, and 4, respectively; P-heterogeneity < 0.001). More advanced age at menopause increased breast cancer risk in all BI-RADS categories. This was more prominent in women with BI-RADS density category 1 but less prominent in women in other BI-RADS categories (P-heterogeneity = 0.009). In postmenopausal women, a family history of breast cancer, body mass index ≥ 25 kg/m, and smoking showed a heterogeneous association with breast cancer across all BI-RADS categories. Other risk factors including age at menarche, menopause, hormone replacement therapy after menopause, oral contraceptive use, and alcohol consumption did not show a heterogeneous association with breast cancer across the BI-RADS categories. Several known risk factors of breast cancer had a heterogeneous effect on breast cancer development across breast density categories, especially in postmenopausal women.