Max Kelsen, Brisbane, QLD, Australia.
ARC Training Centre for Information Resilience (CIRES), Brisbane, Australia.
Sci Rep. 2023 May 6;13(1):7395. doi: 10.1038/s41598-023-31126-5.
Uncertainty estimation is crucial for understanding the reliability of deep learning (DL) predictions, and critical for deploying DL in the clinic. Differences between training and production datasets can lead to incorrect predictions with underestimated uncertainty. To investigate this pitfall, we benchmarked one pointwise and three approximate Bayesian DL models for predicting cancer of unknown primary, using three RNA-seq datasets with 10,968 samples across 57 cancer types. Our results highlight that simple and scalable Bayesian DL significantly improves the generalisation of uncertainty estimation. Moreover, we designed a prototypical metric-the area between development and production curve (ADP), which evaluates the accuracy loss when deploying models from development to production. Using ADP, we demonstrate that Bayesian DL improves accuracy under data distributional shifts when utilising 'uncertainty thresholding'. In summary, Bayesian DL is a promising approach for generalising uncertainty, improving performance, transparency, and safety of DL models for deployment in the real world.
不确定性估计对于理解深度学习（DL）预测的可靠性至关重要，对于将 DL 部署到临床中也至关重要。训练数据集和生产数据集之间的差异可能导致不确定性估计不足的错误预测。为了研究这一陷阱，我们使用三个 RNA-seq 数据集对 57 种癌症类型的 10968 个样本进行了单点和三种近似贝叶斯 DL 模型对未知原发性癌症的预测进行了基准测试。我们的结果突出表明，简单且可扩展的贝叶斯 DL 显著提高了不确定性估计的泛化能力。此外，我们设计了一种原型指标——开发和生产曲线之间的面积（ADP），该指标评估了将模型从开发部署到生产时的准确性损失。使用 ADP，我们证明了当利用“不确定性阈值”时，贝叶斯 DL 可以在数据分布转移下提高准确性。总之，贝叶斯 DL 是一种很有前途的方法，可以用于推广不确定性，提高 DL 模型在现实世界中的性能、透明度和安全性。
