Immune checkpoint therapy in breast cancer remains restricted to triple negative patients, and long-term clinical benefit is rare. The primary aim of immune checkpoint blockade is to prevent or reverse exhausted T cell states, but T cell exhaustion in breast tumors is not well understood. Here, we use single-cell transcriptomics combined with imaging mass cytometry to systematically study immune environments of human breast tumors that either do or do not contain exhausted T cells, with a focus on luminal subtypes. We find that the presence of a PD-1 exhaustion-like T cell phenotype is associated with an inflammatory immune environment with a characteristic cytotoxic profile, increased myeloid cell activation, evidence for elevated immunomodulatory, chemotactic, and cytokine signaling, and accumulation of natural killer T cells. Tumors harboring exhausted-like T cells show increased expression of MHC-I on tumor cells and of CXCL13 on T cells, as well as altered spatial organization with more immature rather than mature tertiary lymphoid structures. Our data reveal fundamental differences between immune environments with and without exhausted T cells within luminal breast cancer, and show that expression of PD-1 and CXCL13 on T cells, and MHC-I - but not PD-L1 - on tumor cells are strong distinguishing features between these environments.