Vranken Jeroen F, van de Leur Rutger R, Gupta Deepak K, Juarez Orozco Luis E, Hassink Rutger J, van der Harst Pim, Doevendans Pieter A, Gulshad Sadaf, van Es René
Department of Cardiology, University Medical Center Utrecht, Heidelberglaan 100, 3584 CX Utrecht, The Netherlands.
Informatics Institute, University of Amsterdam, Amsterdam, The Netherlands.
Eur Heart J Digit Health. 2021 May 8;2(3):401-415. doi: 10.1093/ehjdh/ztab045. eCollection 2021 Sep.
Automated interpretation of electrocardiograms (ECGs) using deep neural networks (DNNs) has gained much attention recently. While the initial results have been encouraging, limited attention has been paid to whether such results can be trusted, which is paramount for their clinical implementation. This study aims to systematically investigate uncertainty estimation techniques for automated classification of ECGs using DNNs and to gain insight into its utility through a clinical simulation.
On a total of 526 656 ECGs from three different datasets, six different methods for estimation of aleatoric and epistemic uncertainty were systematically investigated. The methods were evaluated based on ranking, calibration, and robustness against out-of-distribution data. Furthermore, a clinical simulation was performed where increasing uncertainty thresholds were applied to achieve a clinically acceptable performance. Finally, the correspondence between the uncertainty of ECGs and the lack of interpretational agreement between cardiologists was estimated. Results demonstrated the largest benefit when modelling both epistemic and aleatoric uncertainty. Notably, the combination of variational inference with Bayesian decomposition and ensemble with auxiliary output outperformed the other methods. The clinical simulation showed that the accuracy of the algorithm increased as uncertain predictions were referred to the physician. Moreover, high uncertainty in DNN-based ECG classification strongly corresponded with a lower diagnostic agreement in cardiologist's interpretation ( < 0.001).
Uncertainty estimation is warranted in automated DNN-based ECG classification and its accurate estimation enables intermediate quality control in the clinical implementation of deep learning. This is an important step towards the clinical applicability of automated ECG diagnosis using DNNs.
近年来，使用深度神经网络（DNN）对心电图（ECG）进行自动解读备受关注。虽然初步结果令人鼓舞，但对于这些结果是否可信却关注有限，而这对于其临床应用至关重要。本研究旨在系统地研究用于基于DNN的ECG自动分类的不确定性估计技术，并通过临床模拟深入了解其效用。
在来自三个不同数据集的总共526656份ECG上，系统地研究了六种不同的用于估计偶然不确定性和认知不确定性的方法。基于排名、校准以及对分布外数据的稳健性对这些方法进行了评估。此外，进行了一项临床模拟，应用不断增加的不确定性阈值以实现临床上可接受的性能。最后，估计了ECG的不确定性与心脏病专家之间缺乏解读一致性之间的对应关系。结果表明，在对认知不确定性和偶然不确定性进行建模时收益最大。值得注意的是，变分推理与贝叶斯分解相结合以及与辅助输出的集成优于其他方法。临床模拟表明，随着将不确定的预测提交给医生，算法的准确性提高。此外，基于DNN的ECG分类中的高不确定性与心脏病专家解读中的较低诊断一致性密切相关（<0.001）。
在基于DNN的ECG自动分类中进行不确定性估计是必要的，其准确估计能够在深度学习的临床应用中实现中间质量控制。这是朝着使用DNN进行自动ECG诊断的临床适用性迈出的重要一步。
