Evaluating the Use of ChatGPT 3.5 and Bard as Self-Assessment Tools for Short Answer Questions in Undergraduate Ophthalmology.

OBJECTIVE

This study aimed to evaluate the efficacy of ChatGPT 3.5 and Google Bard as tools for self-assessment of short answer questions (SAQs) in ophthalmology for undergraduate medical students.

METHODOLOGY

A total of 261 SAQs were randomly selected from previous university examination papers and publicly available ophthalmology question banks. The questions were classified according to the competency-based medical education (CBME) curriculum of the National Medical Commission (NMC) of India into three categories: short note task-oriented questions (SNTO, n = 169), short note reasoning questions (SNRQ, n = 15), and applied aspect SAQs (SN Applied, n = 77). Image-based questions were excluded. Three ophthalmologists collaboratively developed model answers for each question. The same questions were then submitted to ChatGPT 3.5 and Google Bard. The AI-generated responses were independently evaluated by three ophthalmologists using a 3-point scale based on correct diagnosis, accuracy of content, and relevance. The scores were compiled, and the data were analyzed to compare the overall and category-wise performance of the two AI tools.

RESULTS

Out of a total possible score of 783 (261 questions × 3 points), ChatGPT 3.5 scored 696 (88.8%), while Bard scored 685 (87.5%). Although the overall performance difference was insignificant, ChatGPT 3.5 performed significantly better in the SNTO category. However, both AI tools produced poor-quality or inadequate answers for a subset of questions: 50 (19%) by ChatGPT 3.5 and 44 (16.8%) by Bard. Some responses lacked essential information, even for high-yield topics.

CONCLUSION

ChatGPT 3.5 and Bard can generate accurate and relevant responses to ophthalmology SAQs in most cases. ChatGPT 3.5 demonstrated slightly better performance, particularly for task-oriented questions, suggesting it may be a more effective tool for undergraduate students' self-assessment. However, due to a notable error rate (~20%), AI-generated responses should not be used in isolation and must be cross-referenced with standard textbooks. These tools best suit rapid information retrieval during the early study phases.