Burke Hugh, Kazinka Rebecca, Gandhi Raghu, Murray Aimee
Medical School, University of Minnesota (Twin Cities), Minneapolis, MN, USA.
Abbott Northwestern Hospital, Minneapolis, MN, USA.
Acad Psychiatry. 2025 Feb;49(1):13-17. doi: 10.1007/s40596-024-02080-9. Epub 2024 Nov 6.
This study was designed to investigate if artificial intelligence (AI) detection software can determine the use of AI in personal statements for residency applications.
Previously written personal statements were collected from physicians who had already matched to residency through the Electronic Residency Application System. Physicians were recruited for the study through collegial relationships and were given study information via email. The study team constructed five parallel statements from the shared personal statements to prompt AI to create a personal statement of similar content. An online AI detection tool, GPTZero, was used to assess all the personal statements. Statistical analyses were conducted using R. Descriptive statistics, t-tests, and Pearson correlations were used to assess the data.
Eight physicians' statements were compared to eight AI-generated statements. GPTZero was able to correctly identify AI-generated writing, assigning them significantly higher AI probability scores compared to human-authored essays. Human-generated statements were considered more readable, used shorter words with fewer syllables, and had more sentences compared to AI-generated essays. Longer average sentence length, low readability scores, and high SAT word percentages were strongly associated with AI-generated essays.
This study shows the capacity of GPTZero to distinguish human-created versus AI-generated writing. Use of AI can pose significant ethical challenges and carries a risk of inadvertent harm to certain applicants and erosion of trust in the application process. Authors suggest standardization of protocol regarding the use of AI prior to its integration in post-graduate medical education.
本研究旨在调查人工智能（AI）检测软件能否确定在住院医师申请个人陈述中是否使用了AI。
从已通过电子住院医师申请系统匹配到住院医师岗位的医生那里收集以前撰写的个人陈述。通过同事关系招募医生参与研究，并通过电子邮件向他们提供研究信息。研究团队从共享的个人陈述中构建了五份平行陈述，以促使AI创建内容相似的个人陈述。使用在线AI检测工具GPTZero对所有个人陈述进行评估。使用R进行统计分析。描述性统计、t检验和Pearson相关性用于评估数据。
将8位医生的陈述与8份AI生成的陈述进行了比较。GPTZero能够正确识别AI生成的文章，与人工撰写的文章相比，赋予它们显著更高的AI概率分数。与AI生成的文章相比，人工生成的陈述被认为更具可读性，使用的单词更短、音节更少，并且句子更多。平均句子长度更长、可读性分数低和SAT词汇百分比高与AI生成的文章密切相关。
本研究表明GPTZero有能力区分人工创作与AI生成的写作。使用AI可能带来重大的伦理挑战，并有可能对某些申请人造成无意的伤害，以及破坏对申请过程的信任。作者建议在将AI整合到研究生医学教育之前，对AI使用的协议进行标准化。
