Gravel Jocelyn, Dion Chloé, Fadaei Kermani Mandana, Mousseau Sarah, Osmanlliu Esli
Department of Pediatric Emergency Medicine, CHU Sainte-Justine, Université de Montréal, Montréal, Québec.
Faculté de médecine, Université de Montréal, Montréal, Québec.
Paediatr Child Health. 2024 Sep 12;30(3):116-121. doi: 10.1093/pch/pxae062. eCollection 2025 Jun.
ChatGPT received attention for medical writing. Our objective was to evaluate whether ChatGPT 4.0 could improve the quality of abstracts submitted to a medical conference by clinical researchers.
This was an experimental study involving 24 international researchers (the participants) who provided one original abstract intended for submission at the 2024 Pediatric Academic Society (PAS) conference. We asked ChatGPT-4 to improve the quality of the abstract while adhering to PAS submission guidelines. Participants received the revised version and were tasked with creating a final abstract. The quality of each version (original, ChatGPT and final) was evaluated by the participants themselves using a numeric scale (0-100). Additionally, three co-investigators assessed abstracts blinded to the version. The primary analysis focused on the mean difference in scores between the final and original abstracts.
Abstract quality varied between the three versions with mean scores of 82, 65 and 90 for the original, ChatGPT and final versions, respectively. Overall, the final version displayed significantly improved quality compared to the original (mean difference 8.0 points; 95% CI: 5.6-10.3). Independent ratings by the co-investigators confirmed statistically significant improvements (mean difference 1.10 points; 95% CI: 0.54-1.66). Participants identified minor (n = 10) and major (n = 3) factual errors in ChatGPT's abstracts.
ChatGPT 4.0 does not produce abstracts of better quality than the one crafted by researchers but it offers suggestions to help them improve their abstracts. It may be more useful for researchers encountering challenges in abstract generation due to limited experience or language barriers.
ChatGPT在医学写作方面受到关注。我们的目的是评估ChatGPT 4.0能否提高临床研究人员提交给医学会议的摘要质量。
这是一项实验性研究，涉及24名国际研究人员（参与者），他们提供了一篇拟提交给2024年儿科学术协会（PAS）会议的原始摘要。我们要求ChatGPT-4在遵循PAS提交指南的同时提高摘要质量。参与者收到修订版，并负责撰写最终摘要。每个版本（原始版、ChatGPT版和最终版）的质量由参与者自己使用数字评分量表（0-100）进行评估。此外，三名共同研究者在对版本不知情的情况下评估摘要。主要分析集中在最终摘要和原始摘要之间的得分平均差异。
三个版本的摘要质量各不相同，原始版、ChatGPT版和最终版的平均得分分别为82分、65分和90分。总体而言，最终版的质量与原始版相比有显著提高（平均差异8.0分；95%置信区间：5.6-10.3）。共同研究者的独立评分证实了统计学上的显著提高（平均差异1.10分；95%置信区间：0.54-1.66）。参与者在ChatGPT生成的摘要中发现了少量（n = 10）和大量（n = 3）事实性错误。
ChatGPT 4.0生成的摘要质量并不比研究人员撰写的摘要质量高，但它能提供建议帮助他们改进摘要。对于因经验有限或语言障碍而在摘要撰写中遇到困难的研究人员可能更有用。
