After neoadjuvant (chemo)radiotherapy for rectal cancer, contact X-ray brachytherapy (CXB) can be applied aiming at organ preservation. This explorative study describes the early features on endoscopy and MRI after CXB. Patients treated with CXB following (chemo)radiotherapy and a follow-up of ≥12 months were selected. Endoscopy and MRI were performed every 3 months. Expert readers scored all the images according to structured reporting templates. Thirty-six patients were included, 15 of whom obtained a cCR. On endoscopy, the most frequently observed feature early in follow-up was an ulcer, regardless of whether patients developed a cCR. A flat, white scar and tumor mass were common at 6 months. Focal tumor signal on T2W-MRI and mass-like high signal on DWI were generally absent in patients with a cCR. An ulceration on T2W-MRI and "reactive" mucosal signal on DWI were observed early in follow-up regardless of the final tumor response. The distinction between a cCR and a residual tumor generally can be made at 6 months. Features associated with a residual tumor are tumor mass on endoscopy, focal tumor signal on T2W-MRI, and mass-like high signal on DWI. Early recognition of these features is necessary to identify patients who will not develop a cCR as early as possible.