Can ChatGPT provide parent education for oral immunotherapy?

BACKGROUND

Oral Immunotherapy (OIT) has exhibited great potential in the treatment of food allergy. However, there is no global consensus on best practices of OIT. Parents of allergic children often struggle with concerns regarding OIT methodology, safety, and lack of accessible educational resources. ChatGPT is a generative artificial intelligence chatbot from OpenAI recognized for its ability to formulate human-like conversations. Although applications of artificial intelligence in medical settings continue to be explored, the effectiveness of ChatGPT as an educational resource remains unknown for OIT.

OBJECTIVE

To assess the accuracy of ChatGPT as a self-guided educational resource for parents with children undergoing OIT.

METHODS

A total of 14 common questions from parents regarding OIT were entered into ChatGPT version 3.5 and answers were copied verbatim. These responses were then categorized into basic, advanced, or medical, and evaluated by Allergy-Immunology health care practitioners from North America and the United Kingdom using a 10-point Likert scale. Response readability, understandability, and reproducibility were assessed using the Flesch Reading Ease and Flesch-Kincaid Grade level scores, the Patient Education Materials Assessment Tool, and natural language processing tools, respectively.

RESULTS

The average median rankings by the practitioners per category were 8.6, 8.4, and 7.8 for basic, advanced, and medical, respectively. ChatGPT responses exhibited low readability scores, corresponding with a high-grade reading level. Understandability was between 73% to 84%, with scores decreased owing to response complexity. When assessing reproducibility, ChatGPT responses achieved rates between 83% and 93%.

CONCLUSION

Our results revealed that ChatGPT provides intelligible and comprehensive responses to patient questions. Health care practitioners polled were generally positive but identified important limitations.