Immune checkpoint blockade (ICB) has revolutionized the treatment of cancer patients. The main focus of ICB has been on reinvigorating the adaptive immune response, namely, activating cytotoxic T cells. ICB has demonstrated only modest benefit against advanced breast cancer, as breast tumors typically establish an immune suppressive tumor microenvironment (TME). Triple-negative breast cancer (TNBC) is associated with infiltration of tumor infiltrating lymphocytes (TILs) and patients with TNBC have shown clinical responses to ICB. In contrast, hormone receptor positive (HR+) breast cancer is characterized by low TIL infiltration and minimal response to ICB. Here we review how HR+ breast tumors establish a TME devoid of TILs, have low HLA class I expression, and recruit immune cells, other than T cells, which impact response to therapy. In addition, we review emerging technologies that have been employed to characterize components of the TME to reveal that tumor associated macrophages (TAMs) are abundant in HR+ cancer, are highly immune-suppressive, associated with tumor progression, chemotherapy and ICB-resistance, metastasis and poor survival. We reveal novel therapeutic targets and possible combinations with ICB to enhance anti-tumor immune responses, which may have great potential in HR+ breast cancer.