Haimerl Martin, Reich Christoph
Furtwangen University of Applied Sciences, Furtwangen, Germany.
BMC Med Inform Decis Mak. 2025 Mar 11;25(1):126. doi: 10.1186/s12911-025-02909-9.
In the future, more medical devices will be based on machine learning (ML) methods. In general, the consideration of risks is a crucial aspect for evaluating medical devices. Accordingly, risks and their associated costs should be taken into account when assessing the performance of ML-based medical devices. This paper addresses the following three research questions towards a risk-based evaluation with a focus on ML-based classification models.
First, we analyzed how often risk-based metrics are currently utilized in the context of ML-based classification models. This was performed using a literature research based on a sample of recent scientific publications. Second, we introduce an approach for evaluating such models where expected risks and associated costs are integrated into the corresponding performance metrics. Additionally, we analyze the impact of different risk ratios on the resulting overall performance. Third, we elaborate how such risk-based approaches relate to regulatory requirements in the field of medical devices. A set of use case scenarios were utilized to demonstrate necessities and practical implications, in this regard.
First, it was shown that currently most scientific publications do not include risk-based approaches for measuring performance. Second, it was demonstrated that risk-based considerations have a substantial impact on the outcome. The relative increase of the resulting overall risks can go up to 196% when the ratio between different types of risks (false negatives vs. false positives) changes by a factor of 10.0. Third, we elaborated that risk-based considerations need to be included into the assessment of ML-based medical devices, according to the relevant EU regulations and standards. In particular, this applies when a substantial impact on the clinical outcome / in terms of the risk-benefit relationship occurs.
In summary, we demonstrated the necessity of a risk-based approach for the evaluation of medical devices which include ML-based classification methods. We showed that currently many scientific papers in this area do not include risk considerations. We developed basic steps towards a risk-based assessment of ML-based classifiers and elaborated consequences that could occur, when these steps are neglected. And, we demonstrated the consistency of our approach with current regulatory requirements in the EU.
未来，更多的医疗设备将基于机器学习（ML）方法。一般来说，风险考量是评估医疗设备的关键方面。因此，在评估基于ML的医疗设备性能时，应考虑风险及其相关成本。本文针对基于风险的评估提出了以下三个研究问题，重点关注基于ML的分类模型。
首先，我们分析了基于ML的分类模型背景下当前使用基于风险的指标的频率。这是通过对近期科学出版物样本进行文献研究来完成的。其次，我们引入了一种评估此类模型的方法，即将预期风险和相关成本纳入相应的性能指标中。此外，我们分析了不同风险比率对最终整体性能的影响。第三，我们阐述了这种基于风险的方法与医疗器械领域监管要求的关系。在这方面，利用一组用例场景来展示必要性和实际意义。
首先，结果表明目前大多数科学出版物都没有包括用于衡量性能的基于风险的方法。其次，结果表明基于风险的考量对结果有重大影响。当不同类型风险（假阴性与假阳性）之间的比率变化10.0倍时，最终整体风险的相对增加可高达196%。第三，我们阐述了根据欧盟相关法规和标准，基于风险的考量需要纳入对基于ML的医疗设备的评估中。特别是，当对临床结果/在风险效益关系方面产生重大影响时，情况更是如此。
总之，我们证明了对包括基于ML的分类方法的医疗设备进行基于风险的评估的必要性。我们表明目前该领域的许多科学论文都没有考虑风险因素。我们制定了对基于ML的分类器进行基于风险评估的基本步骤，并阐述了忽略这些步骤可能产生的后果。而且，我们证明了我们的方法与欧盟当前监管要求的一致性。
