Does chatGPT-4.0's awareness of conversing with a urologist affect the accuracy of responses to questions about "sexually transmitted urethritis in men"?

INTRODUCTION

This study delved into the responses generated by ChatGPT-4 (artificial intelligence-language model) regarding queries on sexually transmitted urethritis in men and investigated the impact of "knowledge of conversing with a urologist" on the accuracy of its responses.

METHODS

A total of 272 questions from the "sexually transmitted infections treatment guidelines" (US Centers for Disease Control and Prevention) were prepared by a urology specialist and arranged to cover various levels of difficulty. The questions were presented in the formats of multiple-choice and true/false. Two groups were created: In Group 1, ChatGPT-4 was only provided with the questions, whereas in Group 2, it was explicitly stated that ChatGPT-4 was engaging in a conversation with a urology specialist. The accuracy of ChatGPT-4's responses was evaluated.

RESULTS

In Group 1, the accuracy rate was 81% (94/116), whereas in Group 2, it was 77.5% (90/116). Subgroup A, which consisted of multiple-choice questions, had accuracy rates of 77.5% (45/58) for Group 1 and 74.1% (43/58) for Group 2. Subgroup B, which included true/false questions, had accuracy rates of 84.4% (49/58) for Group 1 and 81% (47/58) for Group 2. The mean accuracy score was higher in Group 1, whereas the mean completeness score was higher in Group 2.

CONCLUSIONS

Providing ChatGPT-4 with the information that it was conversing with a urologist did not enhance the accuracy of its responses regarding sexually transmitted urethritis in men. The consistently high accuracy observed in ChatGPT-4's responses demonstrates that this system can be reliably used as a question-and-answer tool.