An important impediment to the incorporation of artificial intelligence-based tools into healthcare is their association with so-called black box medicine, a concept arising due to their complexity and the difficulties in understanding how they reach a decision. This situation may compromise the clinician's trust in these tools, should any errors occur, and the inability to explain how decisions are reached may affect their relationship with patients. Explainable AI (XAI) aims to overcome this limitation by facilitating a better understanding of how AI models reach their conclusions for users, thereby enhancing trust in the decisions reached. This review first defined the concepts underlying XAI, establishing the tools available and how they can benefit digestive healthcare. Examples of the application of XAI in digestive healthcare were provided, and potential future uses were proposed. In addition, aspects of the regulatory frameworks that must be established and the ethical concerns that must be borne in mind during the development of these tools were discussed. Finally, we considered the challenges that this technology faces to ensure that optimal benefits are reaped, highlighting the need for more research into the use of XAI in this field.