Development of a Flexible Chain of Thought Framework for Automated Routing of Patient Portal Messages.

The increase in utilization of patient portal messages has imposed a considerable burden on healthcare providers, contributing to an increased incidence of provider burnout. This study introduces a framework for leveraging Large Language Models (LLMs) and Chain-of-Thought (CoT) prompting in order to automatically categorize and route messages to their appropriate location. The modeling framework, which utilizes gold standard annotations from triage nurses, not only facilitates the dynamic adaptation of the model to evolving healthcare workflows and emerging edge-case scenarios, but also significantly improves the model's classification accuracy compared to traditional zero-shot methods. In addition, the framework allows for flexibility in its task and continuous improvement via annotation of exemplar messages. The model is able to accurately categorize messages in an automated fashion, which has potential to dramatically ease the burden on providers and provide faster and safer responses to patients. This framework can also be readily extended to work in a variety of clinical and documentation settings.