Cancer evolution is a complex process influenced by genetic factors and extracellular stimuli that trigger signaling pathways to coordinate the continuous and dynamic interaction between tumor cells and the elements of the immune system. For over 20 years now, the immune mechanisms controlling cancer progression have been the focus of intensive research. It is well established that the immune system conveys protective antitumor immunity by destroying immunogenic tumor variants, but also facilitates tumor progression by shaping tumor immunogenicity in a process called "immunoediting". It is also clear that immune-guided tumor editing is associated with tumor evasion from immune surveillance and therefore reinforcing the endogenous antitumor immunity is a desired goal in the context of cancer therapies. The tumor microenvironment (TME) is a complex network which consists of various cell types and factors having important roles regarding tumor development and progression. Tumor infiltrating lymphocytes (TILs) and other tumor infiltrating immune cells (TIICs) are key to our understanding of tumor immune surveillance based on tumor immunogenicity, whereby the densities and location of TILs and TIICs in the tumor regions, as well as their functional programs (comprising the "immunoscore") have a prominent role for prognosis and prediction for several cancers. The presence of tertiary lymphoid structures (TLS) in the TME or in peritumoral areas has an influence on the locally produced antitumor immune response, and therefore also has a significant prognostic impact. The cross-talk between elements of the immune system with tumor cells in the TME is greatly influenced by hypoxia, the gut and/or the local microbiota, and several metabolic elements, which, in a dynamic interplay, have a crucial role for tumor cell heterogeneity and reprogramming of immune cells along their activation and differentiation pathways. Taking into consideration the recent clinical success with the application immunotherapies for the treatment of several cancer types, increasing endeavors have been made to gain better insights into the mechanisms underlying phenotypic and metabolic profiles in the context of tumor progression and immunotherapy. In this review we will address (i) the role of TILs, TIICs and TLS in breast cancer (BCa); (ii) the different metabolic-based pathways used by immune and breast cancer cells; and (iii) implications for immunotherapy-based strategies in BCa.