We previously demonstrated that intake of low-fat dairy, but not high-fat dairy, was associated with a decreased colorectal cancer (CRC) recurrence risk. These risks, however, may differ by sex, primary tumour location, and disease stage. Combining data from two similar prospective cohort studies of people with stage I-III CRC enabled these subgroup analyses. Participants completed a food frequency questionnaire at diagnosis (n = 2283). We examined associations between low- and high-fat dairy intake and recurrence risk using multivariable Cox proportional hazard models, stratified by sex, and primary tumour location (colon and rectum), and disease stage (I/II and III). Upper quartiles were compared to lower quartiles of intake, and recurrence was defined as a locoregional recurrence and/or metastasis. During a median follow-up of 5.0 years, 331 recurrences were detected. A higher intake of low-fat dairy was associated with a reduced risk of recurrence (hazard ratio [HR]: 0.60, 95% confidence interval [CI]: 0.43-0.83), which seemed more pronounced in men (HR: 0.51, 95% CI: 0.34-0.77) than in women (HR: 0.84, 95% CI: 0.47-1.49). A higher intake of high-fat dairy was associated with an increased risk of recurrence in participants with colon cancer (HR: 1.60, 95% CI: 1.03-2.50), but not rectal cancer (HR: 0.88, 95% CI: 0.54-1.45). No differences in associations were observed between strata of disease stage. Concluding, our findings imply that dietary advice regarding low-fat dairy intake may be especially important for men with CRC, and that dietary advice regarding high-fat dairy intake may be specifically important in people with colon cancer.