The incidence of rectal cancer is increasing in patients younger than 50 years. Locally advanced rectal cancer is still treated with neoadjuvant radiation, chemotherapy and surgery, but recent evidence suggests that patients with a complete response can avoid surgery permanently. To define correlates of response to neoadjuvant therapy, we analyzed genomic and transcriptomic profiles of 738 untreated rectal cancers. APC mutations were less frequent in the lower than in the middle and upper rectum, which could explain the more aggressive behavior of distal tumors. No somatic alterations had significant associations with response to neoadjuvant therapy in a treatment-agnostic manner, but KRAS mutations were associated with faster relapse in patients treated with neoadjuvant chemoradiation followed by consolidative chemotherapy. Overexpression of IGF2 and L1CAM was associated with decreased response to neoadjuvant therapy. RNA-sequencing estimates of immune infiltration identified a subset of microsatellite-stable immune hot tumors with increased response and prolonged disease-free survival.