Huo Bright, Collins Gary, Chartash David, Thirunavukarasu Arun, Flanagin Annette, Iorio Alfonso, Cacciamani Giovanni, Chen Xi, Liu Nan, Mathur Piyush, Chan An-Wen, Laine Christine, Pacella Daniela, Berkwits Michael, Antoniou Stavros A, Camaradou Jennifer C, Canfield Carolyn, Mittelman Michael, Feeney Timothy, Loder Elizabeth, Agha Riaz, Saha Ashirbani, Mayol Julio, Sunjaya Anthony, Harvey Hugh, Ng Jeremy Y, McKechnie Tyler, Lee Yung, Verma Nipun, Stiglic Gregor, McCradden Melissa, Ramji Karim, Boudreau Vanessa, Ortenzi Monica, Meerpohl Joerg, Vandvik Per Olav, Agoritsas Thomas, Samuel Diana, Frankish Helen, Anderson Michael, Yao Xiaomei, Loeb Stacy, Lokker Cynthia, Liu Xiaoxuan, Guallar Eliseo, Guyatt Gordon
Division of General Surgery, Department of Surgery, McMaster University, Hamilton, Canada.
UK EQUATOR Centre, University of Oxford, Oxford, UK; Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology & Musculoskeletal Sciences, Botnar Research Centre, University of Oxford, Oxford, UK.
Artif Intell Med. 2025 Oct;168:103222. doi: 10.1016/j.artmed.2025.103222. Epub 2025 Aug 1.
The Chatbot Assessment Reporting Tool (CHART) is a reporting guideline developed to provide reporting recommendations for studies evaluating the performance of generative artificial intelligence (AI)-driven chatbots when summarizing clinical evidence and providing health advice, referred to as Chatbot Health Advice (CHA) studies. CHART was developed in several phases after performing a comprehensive systematic review to identify variation in the conduct, reporting and methodology in CHA studies. Findings from the review were used to develop a draft checklist that was revised through an international, multidisciplinary modified asynchronous Delphi consensus process of 531 stakeholders, three synchronous panel consensus meetings of 48 stakeholders, and subsequent pilot testing of the checklist. CHART includes 12 items and 39 subitems to promote transparent and comprehensive reporting of CHA studies. These include Title (subitem 1a), Abstract/Summary (subitem 1b), Background (subitems 2ab), Model Identifiers (subitem 3ab), Model Details (subitems 4abc), Prompt Engineering (subitems 5ab), Query Strategy (subitems 6abcd), Performance Evaluation (subitems 7ab), Sample Size (subitem 8), Data Analysis (subitem 9a), Results (subitems 10abc), Discussion (subitems 11abc), Disclosures (subitem 12a), Funding (subitem 12b), Ethics (subitem 12c), Protocol (subitem 12d), and Data Availability (subitem 12e). The CHART checklist and corresponding methodological diagram were designed to support key stakeholders including clinicians, researchers, editors, peer reviewers, and readers in reporting, understanding, and interpreting the findings of CHA studies.
聊天机器人评估报告工具（CHART）是一项报告指南，旨在为评估生成式人工智能（AI）驱动的聊天机器人在总结临床证据和提供健康建议时的性能的研究提供报告建议，此类研究被称为聊天机器人健康建议（CHA）研究。在进行全面的系统评价以确定CHA研究在实施、报告和方法方面的差异后，CHART分几个阶段制定。该评价的结果被用于制定一份清单草案，该草案通过531名利益相关者参与的国际多学科改良异步德尔菲共识过程、48名利益相关者参与的三次同步小组共识会议以及随后对清单的预试验进行了修订。CHART包括12项和39个子项，以促进CHA研究的透明和全面报告。这些项目包括标题（子项1a）、摘要/总结（子项1b）、背景（子项2ab）、模型标识符（子项3ab）、模型细节（子项4abc）、提示工程（子项5ab）、查询策略（子项6abcd）、性能评估（子项7ab）、样本量（子项8）、数据分析（子项9a）、结果（子项10abc）、讨论（子项11abc）、披露（子项12a）、资金（子项12b）、伦理（子项12c）、方案（子项12d）和数据可用性（子项12e）。CHART清单和相应的方法图旨在支持包括临床医生、研究人员、编辑、同行评审人员和读者在内的关键利益相关者报告、理解和解释CHA研究的结果。
