Breast cancer is poorly immunogenic due to immunosuppressive mechanisms produced in part by the tumor microenvironment (TME). The TME is a peritumoral area containing significant quantities of (1) cancer-associated fibroblasts (CAF), (2) tumor-infiltrating lymphocytes (TIL) and (3) tumor-associated macrophages (TAM). This combination protects the tumor from effective immune responses. How these protective cell types are generated and how the changes in the developing tumor relate to these subsets is only partially understood. Immunotherapies targeting solid tumors have proven ineffective largely due to this protective TME barrier. Therefore, a better understanding of the interplay between the tumor, the tumor microenvironment and immune cells would both advance immunotherapeutic research and lead to more effective immunotherapies. This review will summarize the current understanding of the microenvironment of breast cancer giving implications for future immunotherapeutic strategies.