Recent analyses have suggested decreases over time in colorectal cancer incidence at older ages (≥55 years) but increases at younger ages (20-54 years). Understanding the geographic heterogeneity of incidence facilitates resource allocation for potential interventions and advances our knowledge of differential etiologies for these cancers. We performed age-period-cohort analysis using 2000-2014 county-level incidence from the Surveillance, Epidemiology, and End Results (SEER) database, estimating relative risk (RR) and age-adjusted annual percent change (Net Drifts) simultaneously for 612 counties via a hierarchical model, separately for colon and rectum cancer, stratified by age group (20-54 vs. 55-84). We also studied correlates of RR and Net Drift with various county-level characteristics. In all SEER counties, colon and rectum cancer incidence rates increased at ages 20-54, whereas rates decreased at ages 55-84. There was marked heterogeneity in both RR and Net Drift among states and counties for both cancer types. Maps of county RR and Net Drift revealed localized clusters in several states. For both cancer types, counties with high RR and unfavorable Net Drift tended to have higher prevalence of obesity and diabetes and to be of a lower socioeconomic status. Counties with higher overall screening rates tended to have lower Net Drifts for both cancer types. Increasing colorectal cancer incidence in the younger age group is geographically widespread, although there is significant heterogeneity in temporal trends and risk both within and between states. These geographic patterns correlate with different county-level characteristics depending on cancer type and age group.