Phototherapy and immunotherapy in combination is regarded as the ideal therapeutic modality to treat both primary and metastatic tumors. Immunotherapy uses different immunological approaches to stimulate the immune system to identify tumor cells for targeted elimination. Phototherapy destroys the primary tumors by light irradiation, which induces a series of immune responses through triggering immunogenic cancer cell death. Therefore, when integrating immunotherapy with phototherapy, a novel anti-cancer strategy called photoimmunotherapy (PIT) is emerging. This synergistic treatment modality can not only enhance the effectiveness of both therapies but also overcome their inherent limitations, opening a new era for the current anti-cancer therapy. Recently, the advancement of nanomaterials affords a platform for PIT. From all these nanomaterials, inorganic nanomaterials stand out as ideal mediators in PIT due to their unique physiochemical properties. Inorganic nanomaterials can not only serve as carriers to transport immunomodulatory agents in immunotherapy owing to their excellent drug-loading capacity but also function as photothermal agents or photosensitizers in phototherapy because of their great optical characteristics. In this review, the recent advances of multifunctional inorganic nanomaterial-mediated drug delivery and their contributions to cancer PIT will be highlighted.