Tumor-infiltrating lymphocytes (TILs) have potential value for stratifying the treatment of breast cancer (BC), though their precise use remains unclear. We aimed to investigate the utility of TILs using an alternative approach in different settings. We reviewed patients with triple-negative (TN) or human epithelial growth factor receptor 2 (HER2)-positive invasive ductal carcinomas from a single institutional cohort and classified archived hematoxylin-eosin-stained samples in terms of TIL score as low (<10 %), intermediate, and high (>50 %). The prognostic and predictive values of TILs were analyzed retrospectively in both adjuvant and neo-adjuvant settings. In the adjuvant setting, the presence of TILs at primary surgery was a significant favorable prognostic factor among 154 TNBCs [relapse-free survival (RFS): p = 0.015], but not among 183 HER2+ BCs (RFS: p = 0.097). The TNBC low-TIL group tended to relapse earlier. In the neo-adjuvant setting, detection of TILs on biopsy before primary systemic therapy was associated with the ratio of patients achieving pathological complete response among 48 TNBCs (p = 0.024), but not among 58 HER2+ BCs (p = 0.30). The presence of TILs in surgical specimens after systemic therapy had prognostic value in HER2+ BC (RFS: p = 0.007). The impact of TILs differs between patients with TN and HER2+ BC treated with standard therapies. Our three-grade scale for TILs may contribute to our understanding of the importance of the tumor microenvironment in routine practice. TILs after primary systemic therapy may be useful for the further stratification of treatment of HER2+ BC.