This study included patients with primary triple-negative breast cancer (TNBC) who underwent resection without neoadjuvant chemotherapy between January 2004 and December 2014. Among the 248 TNBCs studied, programmed cell death ligand-1 (PD-L1) expression was detected in 103 (41.5%) tumors, and high levels of tumor-infiltrating lymphocytes (TILs) were present in 118 (47.6%) tumors. PD-L1 expression correlated with high levels of TILs, but was not a prognostic factor. Patients with TILs-high tumors had better overall survival than those with TILs-low tumors (P = 0.016). There was a strong interaction between PD-L1 expression and TILs that was associated with both recurrence-free survival (P = 0.0018) and overall survival (P = 0.015). Multivariate Cox proportional hazards model analysis showed that PD-L1-positive/TILs-low was an independent negative prognostic factor for both recurrence-free survival and overall survival. Our findings suggest that PD-L1-positive/TILs-low tumors are associated with a poor prognosis in patients with TNBC, and that it is important to focus on the combination of PD-L1 expression on tumor cells and TILs present in the tumor microenvironment. These biomarkers may be useful for stratification of TNBCs and for predicting prognosis and developing novel cancer immunotherapies.