Institut Pasteur, Université de Paris, Département de neuroscience, F-75015 Paris, France; Center for Research and Interdisciplinarity (CRI), Université Paris Descartes, Paris, France.
Institut Pasteur, Université de Paris, Département de neuroscience, F-75015 Paris, France; Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany; Center for Research and Interdisciplinarity (CRI), Université Paris Descartes, Paris, France.
Neuroimage. 2022 Jul 15;255:119171. doi: 10.1016/j.neuroimage.2022.119171. Epub 2022 Apr 10.
MRI has been extensively used to identify anatomical and functional differences in Autism Spectrum Disorder (ASD). Yet, many of these findings have proven difficult to replicate because studies rely on small cohorts and are built on many complex, undisclosed, analytic choices. We conducted an international challenge to predict ASD diagnosis from MRI data, where we provided preprocessed anatomical and functional MRI data from > 2,000 individuals. Evaluation of the predictions was rigorously blinded. 146 challengers submitted prediction algorithms, which were evaluated at the end of the challenge using unseen data and an additional acquisition site. On the best algorithms, we studied the importance of MRI modalities, brain regions, and sample size. We found evidence that MRI could predict ASD diagnosis: the 10 best algorithms reliably predicted diagnosis with AUC∼0.80 - far superior to what can be currently obtained using genotyping data in cohorts 20-times larger. We observed that functional MRI was more important for prediction than anatomical MRI, and that increasing sample size steadily increased prediction accuracy, providing an efficient strategy to improve biomarkers. We also observed that despite a strong incentive to generalise to unseen data, model development on a given dataset faces the risk of overfitting: performing well in cross-validation on the data at hand, but not generalising. Finally, we were able to predict ASD diagnosis on an external sample added after the end of the challenge (EU-AIMS), although with a lower prediction accuracy (AUC=0.72). This indicates that despite being based on a large multisite cohort, our challenge still produced biomarkers fragile in the face of dataset shifts.
磁共振成像（MRI）已被广泛用于识别自闭症谱系障碍（ASD）的解剖学和功能差异。然而，由于这些研究依赖于小样本量，并且建立在许多复杂的、未公开的、分析性的选择之上，许多发现都难以复制。我们开展了一项国际挑战，旨在从 MRI 数据中预测 ASD 诊断，我们提供了来自>2000 个人的预处理解剖学和功能 MRI 数据。预测评估严格进行盲法评估。146 名挑战者提交了预测算法，在挑战结束时，使用未见过的数据和额外的采集地点对这些算法进行了评估。在最好的算法上，我们研究了 MRI 模态、脑区和样本量的重要性。我们发现 MRI 可以预测 ASD 诊断的证据：10 种最佳算法可靠地预测诊断的 AUC 值约为 0.80——远优于目前使用 20 倍大的队列中的基因分型数据获得的结果。我们观察到，功能 MRI 比解剖学 MRI 对预测更重要，增加样本量可稳步提高预测准确性，为改善生物标志物提供了一种有效的策略。我们还观察到，尽管有强烈的动机将数据推广到未见数据，但在给定数据集上开发模型存在过度拟合的风险：在手边的数据上进行交叉验证表现良好，但不具有泛化能力。最后，我们能够在挑战结束后添加的外部样本（EU-AIMS）上预测 ASD 诊断，尽管预测准确性较低（AUC=0.72）。这表明，尽管我们的挑战基于一个大型多站点队列，但我们的生物标志物仍然容易受到数据集转移的影响。
