Recent exciting progress in cancer immunotherapy has ushered in a new era of cancer treatment. Immunotherapy can elicit unprecedented durable responses in advanced cancer patients that are much greater than conventional chemotherapy. However, such responses only occur in a relatively small fraction of patients. A positive response to immunotherapy usually relies on dynamic interactions between tumor cells and immunomodulators inside the tumor microenvironment (TME). Depending on the context of these interactions, the TME may play important roles to either dampen or enhance immune responses. Understanding the interactions between immunotherapy and the TME is not only critical to dissect the mechanisms of action but also important to provide new approaches in improving the efficiency of current immunotherapies. In this review, we will highlight recent work on how the TME can influence the efficacy of immunotherapy as well as how manipulating the TME can improve current immunotherapy regimens in some cases.