Immune cells have essential auxiliary functions and influence clinical outcomes in cancer, with high immune infiltration being associated with improved clinical outcomes and better response to treatment in breast cancer (BC). However, studies to date have not fully considered the tumor-infiltrating immune cell (TIIC) landscape in tumors. This study investigated potential biomarkers based on TIICs to improve prognosis and treatment effect in BC. We enrolled 5112 patients for analysis and used cell type identification by estimating relative subsets of RNA transcripts (CIBERSORT), a new computational algorithm, to quantify 22 TIICs in primary BC. From the results of univariate Cox regression, 12 immune cells were determined to be significantly related to the overall survival (OS) of BC patients. Furthermore, least absolute shrinkage and selection operator (LASSO) and multivariate Cox regression analyses were applied to construct an immune prognostic model based on six potential biomarkers. By dividing patients into low- and high-risk groups, a significant distinction in OS was found in the training cohort, with 20-year survival rates of 42.6% and 26.3%, respectively. Applying a similar protocol to validation and test cohorts, we found that OS was significantly shorter in the high-risk group than in the low-risk group, regardless of the molecular subtype of BC. Using the immune score model to predict the effect of BC patients to chemotherapy, the survival advantage for the low-risk group was evident among those who received chemotherapy, regardless of the chemotherapy regimen. In evaluating the predictive value of the nomogram, a decision curve showed better predictive accuracy than the standard tumor-node-metastasis (TNM) staging system. The immune cell infiltration-based immune score model can be effectively and efficiently used to predict the prognosis of BC patients as well as the effect of chemotherapy.