The tumor microenvironment (TME) in the liver plays an important role in primary and metastatic liver tumor formation and tumor growth promotion. Cellular and non-cellular components of the TME significantly influence tumor development, growth, metastatic spread, anti-tumor immunity and response to tumor therapy. The cellular components of the TME in the liver not only consist of infiltrating immune cells, but also of liver-resident cells such as liver sinusoidal endothelial cells (LSEC) and hepatic stellate cells (HSC), which promote tumor growth by negatively regulating tumor-associated immune responses. In this review, we characterize cells of the TME with pro- and anti-tumor function in primary and metastatic liver tumors. Furthermore, we summarize mechanisms that permit growth of hepatic tumors despite the occurrence of spontaneous anti-tumor immune responses and how novel therapeutic approaches targeting the TME could unleash tumor-specific immune responses to improve survival of liver cancer patients.