Prompt engineering with a large language model to assist providers in responding to patient inquiries: a real-time implementation in the electronic health record.

BACKGROUND

Large language models (LLMs) can assist providers in drafting responses to patient inquiries. We examined a prompt engineering strategy to draft responses for providers in the electronic health record. The aim was to evaluate the change in usability after prompt engineering.

MATERIALS AND METHODS

A pre-post study over 8 months was conducted across 27 providers. The primary outcome was the provider use of LLM-generated messages from Generative Pre-Trained Transformer 4 (GPT-4) in a mixed-effects model, and the secondary outcome was provider sentiment analysis.

RESULTS

Of the 7605 messages generated, 17.5% ( = 1327) were used. There was a reduction in negative sentiment with an odds ratio of 0.43 (95% CI, 0.36-0.52), but message use decreased ( < .01). The addition of nurses after the study period led to an increase in message use to 35.8% ( < .01).

DISCUSSION

The improvement in sentiment with prompt engineering suggests better content quality, but the initial decrease in usage highlights the need for integration with human factors design.

CONCLUSION

Future studies should explore strategies for optimizing the integration of LLMs into the provider workflow to maximize both usability and effectiveness.