Objective This study aims to investigate the risk factors for lymph node metastasis (LNM) and postoperative recurrence in patients undergoing surgery for pT1 colorectal cancer (pT1-CRC). Materials and methods We retrospectively analyzed 150 patients who underwent bowel resection with lymph node dissection for pT1-CRC at our department between September 2011 and December 2021. Univariate and multivariate analyses were performed to examine the effects of sex, depth of tumor invasion, venous invasion, lymphatic invasion, tumor budding (BD), and histological type on LNM and recurrence. We analyzed recurrence-free survival (RFS) curves. Results LNM was observed in 21 (14.0%) patients. Univariate analysis identified female sex, undifferentiated histological type, positive lymphatic invasion, and tumor budding grade 2/3 (BD2/3) as significant risk factors for LNM, whereas multivariate analysis identified female sex, undifferentiated histological type, and BD2/3 as independent risk factors. No cancer-related deaths were observed during the median observation period of 60.7 months. The five-year RFS rate differed significantly between LNM- and LNM+ patients, at 97.3% and 66.4%, respectively (p=0.0005). BD2/3 was also the significant risk factor for recurrence in the univariate analysis (p<0.0001). In LNM- patients, the five-year RFS was 98.7% for BD1 and 88.2% for BD2/3 (p=0.0014), while in LNM+ patients, it was 100% for BD1 and 37.0% for BD2/3 (p=0.036), with significant differences observed. Conclusion In pT1-CRC patients, female sex, undifferentiated histological type, and BD2/3 were the risk factors for LNM. The recurrence rate was higher in patients with LNM than in those without LNM. Regardless of LNM, BD2/3 was the risk factor for the postoperative recurrence of pT1-CRC.