The understanding of the tumor microenvironment (TME) has been expanding in recent years in the context of interactions among different cell types, through direct cell-cell communication as well as through soluble factors. It has become evident that the development of a successful antitumor response depends on several TME factors. In this context, the number, type, and subsets of immune cells, as well as the functionality, memory, and exhaustion state of leukocytes are key factors of the TME. Both the presence and functionality of immune cells, in particular T cells, are regulated by cellular and soluble factors of the TME. In this regard, one fundamental reason for failure of antitumor responses is hijacked immune cells, which contribute to the immunosuppressive TME in multiple ways. Specifically, reactive oxygen species (ROS), metabolites, and anti-inflammatory cytokines have central roles in generating an immunosuppressive TME. In this review, we focused on recent developments in the immune cell constituents of the TME, and the micromilieu control of antitumor responses. Furthermore, we highlighted the current challenges of T cell-based immunotherapies and potential future strategies to consider for strengthening their effectiveness.