Current Strengths and Weaknesses of ChatGPT as a Resource for Radiation Oncology Patients and Providers.

PURPOSE

Chat Generative Pre-Trained Transformer (ChatGPT), an artificial intelligence program that uses natural language processing to generate conversational-style responses to questions or inputs, is increasingly being used by both patients and health care professionals. This study aims to evaluate the accuracy and comprehensiveness of ChatGPT in radiation oncology-related domains, including answering common patient questions, summarizing landmark clinical research studies, and providing literature reviews with specific references supporting current standard-of-care clinical practice in radiation oncology.

METHODS AND MATERIALS

We assessed the performance of ChatGPT version 3.5 (ChatGPT3.5) in 3 areas. We evaluated ChatGPT3.5's ability to answer 28 templated patient-centered questions applied across 9 cancer types. We then tested ChatGPT3.5's ability to summarize specific portions of 10 landmark studies in radiation oncology. Next, we used ChatGPT3.5 to identify scientific studies supporting current standard-of-care practice in clinical radiation oncology for 5 different cancer types. Each response was graded independently by 2 reviewers, with discordant grades resolved by a third reviewer.

RESULTS

ChatGPT3.5 frequently generated inaccurate or incomplete responses. Only 39.7% of responses to patient-centered questions were considered correct and comprehensive. When summarizing landmark studies in radiation oncology, 35.0% of ChatGPT3.5's responses were accurate and comprehensive, improving to 43.3% when provided the full text of the study. ChatGPT3.5's ability to present a list of studies related to standard-of-care clinical practices was also unsatisfactory, with 50.6% of the provided studies fabricated.

CONCLUSIONS

ChatGPT should not be considered a reliable radiation oncology resource for patients or providers at this time, as it frequently generates inaccurate or incomplete responses. However, natural language programming-based artificial intelligence programs are rapidly evolving, and future versions of ChatGPT or similar programs may demonstrate improved performance in this domain.