Aykut Aslan, Sezenoz Almila Sarigul
Department of Ophthalmology and Visual Sciences, Kellogg Eye Center, University of Michigan, 1000 Wall St, Rm 641, Ann Arbor, MI, 48105, USA.
Department of Ophthalmology, School of Medicine, Marmara University, Istanbul, 34854, Turkey.
Ophthalmol Ther. 2024 Oct;13(10):2697-2713. doi: 10.1007/s40123-024-01014-w. Epub 2024 Aug 14.
OpenAI recently introduced the ability to create custom generative pre-trained transformers (cGPTs) using text-based instruction and/or external documents using retrieval-augmented generation (RAG) architecture without coding knowledge. This study aimed to analyze the features of ophthalmology-related cGPTs and explore their potential utilities.
Data collection took place on January 20 and 21, 2024, and custom GPTs were found by entering ophthalmology keywords into the "Explore GPTS" section of the website. General and specific features of cGPTs were recorded, such as knowledge other than GPT-4 training data. The instruction and description sections were analyzed for compatibility using the Likert scale. We analyzed two custom GPTs with the highest Likert score in detail. We attempted to create a convincingly presented yet potentially harmful cGPT to test safety features.
We analyzed 22 ophthalmic cGPTs, of which 55% were for general use and the most common subspecialty was glaucoma (18%). Over half (55%) contained knowledge other than GPT-4 training data. The representation of the instructions through the description was between "Moderately representative" and "Very representative" with a median Likert score of 3.5 (IQR 3.0-4.0). The instruction word count was significantly associated with Likert scores (P = 0.03). Tested cGPTs demonstrated potential for specific conversational tone, information, retrieval and combining knowledge from an uploaded source. With these safety settings, creating a malicious GPT was possible.
This is the first study to our knowledge to examine the GPT store for a medical field. Our findings suggest that these cGPTs can be immediately implemented in practice and may offer more targeted and effective solutions compared to the standard GPT-4. However, further research is necessary to evaluate their capabilities and limitations comprehensively. The safety features currently appear to be rather limited. It may be helpful for the user to review the instruction section before using a cGPT.
OpenAI最近推出了一项功能，无需编码知识，就能使用基于文本的指令和/或使用检索增强生成（RAG）架构的外部文档来创建定制生成式预训练变换器（cGPT）。本研究旨在分析与眼科相关的cGPT的特征，并探索其潜在用途。
于2024年1月20日和21日进行数据收集，通过在网站的“探索GPT”部分输入眼科关键词来查找定制GPT。记录cGPT的一般和特定特征，例如GPT-4训练数据以外的知识。使用李克特量表分析指令和描述部分的兼容性。我们详细分析了李克特得分最高的两个定制GPT。我们试图创建一个呈现令人信服但可能有害的cGPT来测试安全功能。
我们分析了22个眼科cGPT，其中55%供一般使用，最常见的亚专业是青光眼（18%）。超过一半（55%）包含GPT-4训练数据以外的知识。通过描述对指令的呈现介于“中等代表性”和“非常代表性”之间，李克特得分中位数为3.5（四分位距3.0 - 4.0）。指令词数与李克特得分显著相关（P = 0.03）。经过测试的cGPT在特定对话语气、信息、检索以及整合上传源知识方面显示出潜力。在这些安全设置下，有可能创建恶意GPT。
据我们所知，这是第一项针对医学领域检查GPT商店的研究。我们的研究结果表明，这些cGPT可以立即在实践中实施，并且与标准的GPT-4相比，可能提供更具针对性和有效的解决方案。然而，需要进一步研究以全面评估其能力和局限性。目前安全功能似乎相当有限。用户在使用cGPT之前查看指令部分可能会有所帮助。
