Department of Clinical Laboratory, Qilu Hospital of Shandong University, Jinan, Shandong, P.R. China.
Clin Chem Lab Med. 2020 Nov 26;59(5):883-891. doi: 10.1515/cclm-2020-0716. Print 2021 Apr 27.
Autoverification systems have greatly improved laboratory efficiency. However, the long-developed rule-based autoverfication models have limitations. The machine learning (ML) algorithm possesses unique advantages in the evaluation of large datasets. We investigated the utility of ML algorithms for developing an artificial intelligence (AI) autoverification system to support laboratory testing. The accuracy and efficiency of the algorithm model were also validated.
Testing data, including 52 testing items with demographic information, were extracted from the laboratory information system and Roche Cobas IT 3000 from June 1, 2018 to August 30, 2019. Two rounds of modeling were conducted to train different ML algorithms and test their abilities to distinguish invalid reports. Algorithms with the top three best performances were selected to form the finalized ensemble model. Double-blind testing between experienced laboratory personnel and the AI autoverification system was conducted, and the passing rate and false-negative rate (FNR) were documented. The working efficiency and workload reduction were also analyzed.
The final AI system showed a 89.60% passing rate and 0.95 per mille FNR, in double-blind testing. The AI system lowered the number of invalid reports by approximately 80% compared to those evaluated by a rule-based engine, and therefore enhanced the working efficiency and reduced the workload in the biochemistry laboratory.
We confirmed the feasibility of the ML algorithm for autoverification with high accuracy and efficiency.
自动验证系统极大地提高了实验室效率。然而，长期开发的基于规则的自动验证模型具有局限性。机器学习（ML）算法在评估大型数据集方面具有独特的优势。我们研究了 ML 算法在开发人工智能（AI）自动验证系统以支持实验室测试中的应用。还验证了算法模型的准确性和效率。
从 2018 年 6 月 1 日至 2019 年 8 月 30 日，从实验室信息系统和罗氏 Cobas IT 3000 中提取了包括 52 项具有人口统计学信息的检测项目的测试数据。进行了两轮建模，以训练不同的 ML 算法并测试其区分无效报告的能力。选择表现最好的前三种算法组成最终的集成模型。对经验丰富的实验室人员和 AI 自动验证系统进行了双盲测试，并记录了通过率和假阴性率（FNR）。还分析了工作效率和工作量减少。
最终的 AI 系统在双盲测试中的通过率为 89.60%，假阴性率为 0.95 每千分。与基于规则的引擎评估的无效报告数量相比，AI 系统降低了约 80%，从而提高了生化实验室的工作效率并减少了工作量。
我们证实了 ML 算法用于自动验证的可行性，具有很高的准确性和效率。
