Recommended antibiotic treatment agreement between infectious diseases specialists and ChatGPT.

BACKGROUND

Antimicrobial resistance is a global threat to public health. Chat Generative Pre-trained Transformer (ChatGPT) is a language model tool based on artificial intelligence. ChatGPT could analyze data from antimicrobial susceptibility tests in real time, especially in places where infectious diseases (ID) specialists are not available. We aimed to evaluate the agreement between ChatGPT and ID specialists regarding appropriate antibiotic prescription in simulated cases.

METHODS

Using data from microbiological isolates recovered in our center, we fabricated 100 cases of patients with different infections. Each case included age, infectious syndrome, isolated organism and complete antibiogram. Considering a precise set of instructions, the cases were introduced into ChatGPT and presented to five ID specialists. For each case, we asked, (1) "What is the most appropriate antibiotic that should be prescribed to the patient in the clinical case?" and (2) "According to the interpretation of the antibiogram, what is the most probable mechanism of resistance?". We then calculated the agreement between ID specialists and ChatGPT, as well as Cohen's kappa coefficient.

RESULTS

Regarding the recommended antibiotic prescription, agreement between ID specialists and ChatGPT was observed in 51/100 cases. The calculated kappa coefficient was 0.48. Agreement on antimicrobial resistance mechanisms was observed in 42/100 cases. The calculated kappa coefficient was 0.39. In a subanalysis according to infectious syndromes and microorganisms, Agreement (range 25 - 80%) and kappa coefficients (range 0.21-0.79) varied.

CONCLUSION

We found poor agreement between ID specialists and ChatGPT regarding the recommended antibiotic management in simulated clinical cases.