Rectal cancer invading adjacent organs (T4) and locally recurrent rectal cancer (LRRC) pose a special challenge for surgical resection. We investigate the diagnostic performance of MRI and the results that can be achieved with MRI-guided surgery. All consecutive patients who underwent MRI-based multivisceral resection for T4 rectal adenocarcinoma or LRRC between 2005 and 2019 were included. Pelvic MRI findings were reviewed according to a seven-compartment staging system and correlated with histopathology. Outcomes were investigated by comparing T4 tumors and LRRC with respect to cause-specific survival in uni- and multivariate analysis. We identified 48 patients with T4 tumors and 28 patients with LRRC. Overall, 529 compartments were assessed with an accuracy of 81.7%, a sensitivity of 88.6%, and a specificity of 79.2%. Understaging was as low as 3.0%, whereas overstaging was 15.3%. The median number of resected compartments was 3 (interquartile range 3-4) for T4 tumors and 4 (interquartile range 3-5) for LRRC ( = 0.017). In 93.8% of patients with T4 tumors, a histopathologically complete (R0(local)-) resection could be achieved compared to 57.1% in LRRC ( < 0.001). Five-year overall survival for patients with T4 tumors was 53.3% vs. 32.1% for LRRC ( = 0.085). R0-resection and M0-category emerged as independent prognostic factors, whereas the number of resected compartments was not associated with prognosis in multivariate analysis. MRI predicts compartment involvement with high accuracy and especially avoids understaging. Surgery based on MRI yields excellent loco-regional results for T4 tumors and good results for LRRC. The number of resected compartments is not independently associated with prognosis, but R0-resection remains the crucial surgical factor.