Department of Software and Information Systems Engineering, Ben Gurion University, Beer Sheva, Israel.
Endocrinology, Diabetes, and Metabolism Institute, Rambam Health Care Campus, Haifa, Israel.
PLoS One. 2024 May 20;19(5):e0303542. doi: 10.1371/journal.pone.0303542. eCollection 2024.
We introduce a new approach for automated guideline-based-care quality assessment, the bidirectional knowledge-based assessment of compliance (BiKBAC) method, and the DiscovErr system, which implements it. Our methodology compares the guideline's Asbru-based formal representation, including its intentions, with the longitudinal medical record, using a top-down and bottom-up approach. Partial matches are resolved using fuzzy temporal logic. The system was evaluated in the type 2 Diabetes management domain, comparing it to three expert clinicians, including two diabetes experts. The system and the experts commented on the management of 10 patients, randomly selected from 2,000 diabetes patients. On average, each record spanned 5.23 years; the data included 1,584 medical transactions. The system provided 279 comments. The experts made 181 different unique comments. The completeness (recall) of the system was 91% when the gold standard was comments made by at least two of the three experts, and 98%, compared to comments made by all three experts. The experts also assessed all of the 114 medication-therapy-related comments, and a random 35% of the 165 tests-and-monitoring-related comments. The system's correctness (precision) was 81%, compared to comments judged as correct by both diabetes experts, and 91%, compared to comments judged as correct by one diabetes expert and at least as partially correct by the other. 89% of the comments were judged as important by both diabetes experts, 8% were judged as important by one expert, and 3% were judged as less important by both experts. Adding the validated system comments to the experts' comments, the completeness scores of the experts were 75%, 60%, and 55%; the expert correctness scores were respectively 99%, 91%, and 88%. Thus, the system could be ranked first in completeness and second in correctness. We conclude that systems such as DiscovErr can effectively assess the quality of continuous guideline-based care.
我们引入了一种新的自动化基于指南的护理质量评估方法，即双向基于知识的依从性评估（BiKBAC）方法和实现它的 DiscovErr 系统。我们的方法比较了基于 Asbru 的指南的正式表示，包括其意图，以及使用自上而下和自下而上方法的纵向医疗记录。使用模糊时间逻辑解决部分匹配。该系统在 2 型糖尿病管理领域进行了评估，将其与三名专家临床医生（包括两名糖尿病专家）进行了比较。系统和专家对随机选择的 10 名患者的管理情况发表了评论，从 2000 名糖尿病患者中选择。平均而言，每个记录跨越 5.23 年；数据包括 1584 次医疗交易。系统提供了 279 条评论。专家共发表了 181 条不同的独特评论。当黄金标准是至少两名专家中的两名发表评论时，系统的完整性（召回率）为 91%，与三名专家发表的所有评论相比为 98%。专家还评估了所有 114 条药物治疗相关评论和 165 条测试和监测相关评论的随机 35%。与两名糖尿病专家判断为正确的评论相比，系统的正确性（精度）为 81%，与一名糖尿病专家判断为正确的评论相比为 91%，与另一名专家判断为部分正确的评论相比为 91%。两名糖尿病专家都认为 89%的评论很重要，一名专家认为 8%的评论很重要，两名专家都认为 3%的评论不太重要。将经过验证的系统评论添加到专家的评论中，专家的完整性评分分别为 75%、60%和 55%；专家的正确性评分分别为 99%、91%和 88%。因此，系统可以在完整性方面排名第一，在正确性方面排名第二。我们得出结论，像 DiscovErr 这样的系统可以有效地评估基于指南的连续护理质量。
