The tumor microenvironment (TME), which is composed of cancer cells, stromal cells, immune cells, and extracellular matrices, plays an important role in tumor growth and progression. Thus, targeting the TME using a well-designed nano-drug delivery system is emerging as a promising strategy for the treatment of solid tumors. Compared to normal tissues, the TME presents several distinguishable physiological features such as mildly acidic pH, hypoxia, high level of reactive oxygen species, and overexpression of specific enzymes, that are exploited as stimuli to induce specific changes in the nanocarrier structures, and thereby facilitates target-specific delivery of imaging or chemotherapeutic agents for the early diagnosis or effective treatment, respectively. Recently, smart nanocarriers that respond to more than one stimulus in the TME have also been designed to elicit a more desirable spatiotemporally controlled drug release. This review highlights the recent progress in TME-sensitive nanocarriers designed for more efficient tumor therapy and imaging. In particular, the design strategies, challenges, and critical considerations involved in the fabrication of TME-sensitive nanocarriers, along with their in vitro and in vivo evaluations are discussed.