Department of Psychiatry, University of Pittsburgh, 3811 O'Hara Street, Pittsburgh, PA, 15231, USA.
Department of Statistics, University of Pittsburgh, Pittsburgh, PA, USA.
BMC Med Res Methodol. 2023 Jun 19;23(1):144. doi: 10.1186/s12874-023-01965-x.
Machine learning tools such as random forests provide important opportunities for modeling large, complex modern data generated in medicine. Unfortunately, when it comes to understanding why machine learning models are predictive, applied research continues to rely on 'out of bag' (OOB) variable importance metrics (VIMPs) that are known to have considerable shortcomings within the statistics community. After explaining the limitations of OOB VIMPs - including bias towards correlated features and limited interpretability - we describe a modern approach called 'knockoff VIMPs' and explain its advantages.
We first evaluate current VIMP practices through an in-depth literature review of 50 recent random forest manuscripts. Next, we recommend organized and interpretable strategies for analysis with knockoff VIMPs, including computing them for groups of features and considering multiple model performance metrics. To demonstrate methods, we develop a random forest to predict 5-year incident stroke in the Sleep Heart Health Study and compare results based on OOB and knockoff VIMPs.
Nearly all papers in the literature review contained substantial limitations in their use of VIMPs. In our demonstration, using OOB VIMPs for individual variables suggested two highly correlated lung function variables (forced expiratory volume, forced vital capacity) as the best predictors of incident stroke, followed by age and height. Using an organized analytic approach that considered knockoff VIMPs of both groups of features and individual features, the largest contributions to model sensitivity were medications (especially cardiovascular) and measured medical risk factors, while the largest contributions to model specificity were age, diastolic blood pressure, self-reported medical risk factors, polysomnography features, and pack-years of smoking. Thus, we reach very different conclusions about stroke risk factors using OOB VIMPs versus knockoff VIMPs.
The near-ubiquitous reliance on OOB VIMPs may provide misleading results for researchers who use such methods to guide their research. Given the rapid pace of scientific inquiry using machine learning, it is essential to bring modern knockoff VIMPs that are interpretable and unbiased into widespread applied practice to steer researchers using random forest machine learning toward more meaningful results.
机器学习工具，如随机森林，为在医学中建模大型、复杂的现代数据提供了重要机会。不幸的是，在理解为什么机器学习模型具有预测性时，应用研究仍然依赖于“袋外”（OOB）变量重要性指标（VIMPs），这些指标在统计学界被认为存在相当大的缺陷。在解释了 OOB VIMPs 的局限性后——包括对相关特征的偏见和有限的可解释性——我们描述了一种称为“knockoff VIMPs”的现代方法，并解释了它的优势。
我们首先通过对 50 篇近期随机森林手稿的深入文献回顾，评估当前 VIMP 实践。接下来，我们建议使用 knockoff VIMPs 进行分析的有组织和可解释的策略，包括为特征组计算它们并考虑多个模型性能指标。为了演示方法，我们开发了一个随机森林来预测睡眠心脏健康研究中的 5 年中风事件，并比较基于 OOB 和 knockoff VIMPs 的结果。
文献综述中的几乎所有论文在使用 VIMPs 方面都存在重大局限性。在我们的演示中，使用 OOB VIMPs 对单个变量进行分析表明，两个高度相关的肺功能变量（用力呼气量、用力肺活量）是中风事件的最佳预测指标，其次是年龄和身高。使用一种有组织的分析方法，同时考虑了特征组和单个特征的 knockoff VIMPs，对模型敏感性贡献最大的是药物（特别是心血管药物）和测量的医疗风险因素，而对模型特异性贡献最大的是年龄、舒张压、自我报告的医疗风险因素、多导睡眠图特征和吸烟包年数。因此，我们使用 OOB VIMPs 和 knockoff VIMPs 得出了关于中风风险因素的非常不同的结论。
由于研究人员使用这些方法来指导他们的研究，因此近乎普遍依赖 OOB VIMPs 可能会提供误导性的结果。鉴于使用机器学习进行科学研究的快速步伐，必须将可解释和无偏的现代 knockoff VIMPs 广泛应用于应用实践中，以使使用随机森林机器学习的研究人员能够得出更有意义的结果。
