Zou Min, Popko Leszek, Gaudio Michelle
F.Hoffmann-La Roche AG, 4070, Basel, Switzerland.
Hoffmann-La Roche Limited, 7070 Mississauga Road, Mississauga, ON, L5N 5M8, Canada.
Ther Innov Regul Sci. 2025 May 13. doi: 10.1007/s43441-025-00785-z.
As described in ICH E3 Q&A R1 (International Council for Harmonisation. E3: Structure and content of clinical study reports-questions and answers (R1). 6 July 2012. Available from: https://database.ich.org/sites/default/files/E3_Q%26As_R1_Q%26As.pdf ): "A protocol deviation (PD) is any change, divergence, or departure from the study design or procedures defined in the protocol". A problematic area in human subject protection is the wide divergence among institutions, sponsors, investigators and IRBs regarding the definition of and the procedures for reviewing PDs. Despite industry initiatives like TransCelerate's holistic approach [Galuchie et al. in Ther Innov Regul Sci 55:733-742, 2021], systematic trending and identification of impactful PDs remains limited. Traditional Natural Language Processing (NLP) methods are often cumbersome to implement, requiring extensive feature engineering and model tuning. However, the rise of Large Language Models (LLMs) has revolutionised text classification, enabling more accurate, nuanced, and context-aware solutions [Nguyen P. Test classification in the age of LLMs. 2024. Available from: https://blog.redsift.com/author/phong/ ]. An automated classification solution that enables efficient, flexible, and targeted PD classification is currently lacking.
We developed a novel approach using a large language model (LLM), Meta Llama2 [Meta. Llama 2: Open source, free for research and commercial use. 2023. Available from: https://www.llama.com/llama2/ ] with a tailored prompt to classify free-text PDs from Roches' PD management system. The model outputs were analysed to identify trends and assess risks across clinical programs, supporting human decision-making. This method offers a generalisable framework for developing prompts and integrating data to address similar challenges in clinical development.
This approach flagged over 80% of PDs potentially affecting disease progression assessment, enabling expert review. Compared to months of manual analysis, this automated method produced actionable insights in minutes. The solution also highlighted gaps in first-line controls, supporting process improvement and better accuracy in disease progression handling during trials.
如国际人用药品注册技术协调会（ICH）E3问答R1（国际人用药品注册技术协调会。E3：临床研究报告的结构和内容——问答（R1）。2012年7月6日。可从以下网址获取：https://database.ich.org/sites/default/files/E3_Q%26As_R1_Q%26As.pdf）所述：“方案偏离（PD）是指与方案中定义的研究设计或程序的任何变化、偏离或背离”。在人类受试者保护方面，一个存在问题的领域是各机构、申办者、研究者和机构审查委员会（IRB）在方案偏离的定义和审查程序方面存在很大差异。尽管有像TransCelerate的整体方法这样的行业举措[Galuchie等人，《治疗创新与监管科学》55：733 - 742，2021]，但对有影响的方案偏离进行系统的趋势分析和识别仍然有限。传统的自然语言处理（NLP）方法实施起来通常很繁琐，需要大量的特征工程和模型调整。然而，大语言模型（LLM）的兴起彻底改变了文本分类，实现了更准确、细致入微和上下文感知的解决方案[Nguyen P.《大语言模型时代的测试分类》。2024年。可从以下网址获取：https://blog.redsift.com/author/phong/]。目前缺乏一种能够实现高效、灵活和有针对性的方案偏离分类的自动化分类解决方案。
我们开发了一种新颖的方法，使用大语言模型（LLM）Meta Llama2[Meta。Llama 2：开源，可免费用于研究和商业用途。2023年。可从以下网址获取：https://www.llama.com/llama2/]以及一个定制的提示，对罗氏方案偏离管理系统中的自由文本方案偏离进行分类。对模型输出进行分析，以识别临床项目中的趋势并评估风险，为人类决策提供支持。这种方法为开发提示和整合数据提供了一个可推广的框架，以应对临床开发中的类似挑战。
这种方法标记出了超过80%可能影响疾病进展评估的方案偏离，以便进行专家审查。与数月的人工分析相比，这种自动化方法在几分钟内就能产生可采取行动的见解。该解决方案还突出了一线控制中的差距，支持在试验期间改进流程并提高疾病进展处理的准确性。
