Genentech, A Member of the Roche Group, Roche, South San Francisco, CA, USA.
IBM Watson Health, Cambridge, MA, USA.
Drug Saf. 2018 Jun;41(6):579-590. doi: 10.1007/s40264-018-0641-7.
There is increasing interest in social digital media (SDM) as a data source for pharmacovigilance activities; however, SDM is considered a low information content data source for safety data. Given that pharmacovigilance itself operates in a high-noise, lower-validity environment without objective 'gold standards' beyond process definitions, the introduction of large volumes of SDM into the pharmacovigilance workflow has the potential to exacerbate issues with limited manual resources to perform adverse event identification and processing. Recent advances in medical informatics have resulted in methods for developing programs which can assist human experts in the detection of valid individual case safety reports (ICSRs) within SDM.
In this study, we developed rule-based and machine learning (ML) models for classifying ICSRs from SDM and compared their performance with that of human pharmacovigilance experts.
We used a random sampling from a collection of 311,189 SDM posts that mentioned Roche products and brands in combination with common medical and scientific terms sourced from Twitter, Tumblr, Facebook, and a spectrum of news media blogs to develop and evaluate three iterations of an automated ICSR classifier. The ICSR classifier models consisted of sub-components to annotate the relevant ICSR elements and a component to make the final decision on the validity of the ICSR. Agreement with human pharmacovigilance experts was chosen as the preferred performance metric and was evaluated by calculating the Gwet AC1 statistic (gKappa). The best performing model was tested against the Roche global pharmacovigilance expert using a blind dataset and put through a time test of the full 311,189-post dataset.
During this effort, the initial strict rule-based approach to ICSR classification resulted in a model with an accuracy of 65% and a gKappa of 46%. Adding an ML-based adverse event annotator improved the accuracy to 74% and gKappa to 60%. This was further improved by the addition of an additional ML ICSR detector. On a blind test set of 2500 posts, the final model demonstrated a gKappa of 78% and an accuracy of 83%. In the time test, it took the final model 48 h to complete a task that would have taken an estimated 44,000 h for human experts to perform.
The results of this study indicate that an effective and scalable solution to the challenge of ICSR detection in SDM includes a workflow using an automated ML classifier to identify likely ICSRs for further human SME review.
社交数字媒体（SDM）作为药物警戒活动的数据来源，其重要性日益凸显；然而，SDM 被认为是安全性数据的低信息含量数据源。鉴于药物警戒本身在缺乏客观“黄金标准”的情况下运作，其定义仅限于流程，因此将大量 SDM 引入药物警戒工作流程可能会加剧由于手动资源有限而导致的识别和处理不良事件的问题。最近医学信息学的发展导致了开发程序的方法，这些方法可以帮助人类专家在 SDM 中检测有效的个体病例安全报告（ICSR）。
在这项研究中，我们开发了基于规则和机器学习（ML）的模型，用于对 SDM 中的 ICSR 进行分类，并将其性能与人类药物警戒专家的性能进行了比较。
我们使用来自 Roche 产品和品牌的 311189 个 SDM 帖子的随机抽样，以及来自 Twitter、Tumblr、Facebook 和一系列新闻媒体博客的常见医学和科学术语，开发并评估了三个迭代的自动化 ICSR 分类器。ICSR 分类器模型由子组件组成，用于注释相关的 ICSR 元素，以及组件用于对 ICSR 的有效性做出最终决定。选择与人类药物警戒专家的一致性作为首选性能指标，并通过计算 Gwet AC1 统计量（gKappa）进行评估。对表现最好的模型进行了盲测，使用 Roche 全球药物警戒专家对其进行了测试，并对整个 311189 个帖子数据集进行了时间测试。
在这项研究中，最初严格的基于规则的 ICSR 分类方法导致模型的准确率为 65%，gKappa 为 46%。添加基于 ML 的不良事件注释器将准确率提高到 74%，gKappa 提高到 60%。通过添加另一个 ML ICSR 检测器进一步提高了性能。在一个 2500 个帖子的盲测集上，最终模型的 gKappa 为 78%，准确率为 83%。在时间测试中，最终模型完成任务耗时 48 小时，而人类专家完成这项任务的估计时间为 44000 小时。
这项研究的结果表明，一种有效的、可扩展的解决方案，可以应对 SDM 中 ICSR 检测的挑战，该方案包括使用自动化 ML 分类器来识别可能的 ICSR，以供进一步的人类 SME 审查。
