Jesus-Ribeiro Joana, Roza Eugenia, Oliveiros Bárbara, Melo Joana Barbosa, Carreño Mar
Coimbra Institute for Clinical and Biomedical Research (iCBR) - Center of Investigation on Environment Genetics and Oncobiology (CIMAGO), Faculty of Medicine, University of Coimbra, Coimbra, Portugal.
Neurology Department, Unidade Local de Saúde da Região de Leiria, Leiria, Portugal.
Epilepsia Open. 2025 Apr 1. doi: 10.1002/epi4.70022.
Artificial intelligence chatbots have been a game changer in healthcare, providing immediate, round-the-clock assistance. However, their accuracy across specific medical domains remains under-evaluated. Dravet syndrome remains one of the most challenging epileptic encephalopathies, with new data continuously emerging in the literature. This study aims to evaluate and compare the performance of ChatGPT 3.5 and Perplexity in responding to questions about Dravet Syndrome.
We curated 96 questions about Dravet syndrome, 43 from healthcare professionals and 53 from caregivers. Two epileptologists independently graded the chatbots' responses, with a third senior epileptologist resolving any disagreements to reach a final consensus. Accuracy and completeness of correct answers were rated on predefined 3-point scales. Incorrect responses were prompted for self-correction and re-evaluated. Readability was assessed using Flesch reading ease and Flesch-Kincaid grade level.
Both chatbots had the majority of their responses rated as "correct" (ChatGPT 3.5: 66.7%, Perplexity: 81.3%), with no significant difference in performance between the two (χ = 5.30, p = 0.071). ChatGPT 3.5 performed significantly better for caregivers than for healthcare professionals (χ = 7.27, p = 0.026). The topic with the poorest performance was Dravet syndrome's treatment, particularly for healthcare professional questions. Both models exhibited exemplary completeness, with most responses rated as "complete" to "comprehensive" (ChatGPT 3.5: 73.4%, Perplexity: 75.7%). Substantial self-correction capabilities were observed: ChatGPT 3.5 improved 55.6% of responses and Perplexity 80%. The texts were generally very difficult to read, requiring an advanced reading level. However, Perplexity's responses were significantly more readable than ChatGPT 3.5's [Flesch reading ease: 29.0 (SD 13.9) vs. 24.1 (SD 15.0), p = 0.018].
Our findings underscore the potential of AI chatbots in delivering accurate and complete responses to Dravet syndrome queries. However, they have limitations, particularly in complex areas like treatment. Continuous efforts to update information and improve readability are essential.
Artificial intelligence chatbots have the potential to improve access to medical information, including on conditions like Dravet syndrome, but the quality of this information is still unclear. In this study, ChatGPT 3.5 and Perplexity correctly answered most questions from healthcare professionals and caregivers, with ChatGPT 3.5 performing better for caregivers. Treatment-related questions had the most incorrect answers, particularly those from healthcare professionals. Both chatbots demonstrated the ability to correct previous incorrect responses, particularly Perplexity. Both chatbots produced text requiring advanced reading skills. Further improvements are needed to make the text easier to understand and address difficult medical topics.
人工智能聊天机器人已成为医疗保健领域的变革者，可提供即时、全天候的帮助。然而，它们在特定医学领域的准确性仍未得到充分评估。德雷维特综合征仍然是最具挑战性的癫痫性脑病之一，文献中不断有新数据出现。本研究旨在评估和比较ChatGPT 3.5和Perplexity在回答有关德雷维特综合征问题时的表现。
我们精心整理了96个有关德雷维特综合征的问题，其中43个来自医疗保健专业人员，53个来自护理人员。两名癫痫专家独立对聊天机器人的回答进行评分，第三名资深癫痫专家解决任何分歧以达成最终共识。正确答案的准确性和完整性根据预定义的3分制进行评分。错误回答会被要求自行纠正并重新评估。使用弗莱什易读性和弗莱什-金凯德年级水平来评估可读性。
两个聊天机器人的大多数回答都被评为“正确”（ChatGPT 3.5：66.7%，Perplexity：81.3%），两者表现无显著差异（χ = 5.30，p = 0.071）。ChatGPT 3.5对护理人员的表现明显优于医疗保健专业人员（χ = 7.27，p = 0.026）。表现最差的主题是德雷维特综合征的治疗，尤其是针对医疗保健专业人员的问题。两个模型都表现出了出色的完整性，大多数回答被评为“完整”到“全面”（ChatGPT 3.5：73.4%，Perplexity：75.7%）。观察到显著的自我纠正能力：ChatGPT 3.5改进了55.6%的回答，Perplexity改进了80%。这些文本通常很难阅读，需要较高的阅读水平。然而，Perplexity的回答比ChatGPT 3.5的回答可读性明显更高[弗莱什易读性：29.0（标准差13.9）对24.1（标准差15.0），p = 0.018]。
我们的研究结果强调了人工智能聊天机器人在提供有关德雷维特综合征问题的准确和完整回答方面的潜力。然而，它们也有局限性，特别是在治疗等复杂领域。持续努力更新信息和提高可读性至关重要。
人工智能聊天机器人有潜力改善获取医疗信息的途径，包括有关德雷维特综合征等病症的信息，但这些信息的质量仍不明确。在本研究中，ChatGPT 3.5和Perplexity正确回答了大多数来自医疗保健专业人员和护理人员的问题，ChatGPT 3.5对护理人员的表现更好。与治疗相关的问题错误答案最多，尤其是来自医疗保健专业人员的问题。两个聊天机器人都展示了纠正先前错误回答的能力，特别是Perplexity。两个聊天机器人生成的文本都需要较高的阅读技巧。需要进一步改进以使文本更易于理解并解决困难的医学主题。
