The human gut is inhabited by hundreds of billions of commensal microbiota that collectively produce thousands of small molecules and metabolites with local and systemic effects on the physiology of the host. Much evidence from preclinical to clinical studies has gradually confirmed that the gut microbiota can regulate anti-tumor immunity and affect the efficacy of cancer immune checkpoint inhibitors (ICIs) therapy. In particular, one of the main modes of gut microbiota regulating anti-tumor immunity is through metabolites, which are small molecules that can be transported in the body and act on local and systemic anti-tumor immune responses to promote ICIs immunotherapy efficacy. We discuss the functions of microbial metabolites in humans, focusing on the effects and mechanisms of microbial metabolites on immunotherapy, and analyze their potential applications as immune adjuvants and therapeutic targets to regulate immunity and enhance ICIs. In summary, this review provides the basis for the rational design of microbiota and microbial metabolite-based strategies of enhancing ICIs.