School of Biological Sciences, University of Reading, Whiteknights, Reading, RG6 6EX, United Kingdom.
Bioinformatics. 2024 Aug 2;40(8). doi: 10.1093/bioinformatics/btae491.
Despite an increase in protein modelling accuracy following the development of AlphaFold2, there remains an accuracy gap between predicted and observed model quality assessment (MQA) scores. In CASP15, variations in AlphaFold2 model accuracy prediction were noticed for quaternary models of very similar observed quality. In this study, we compare plDDT and pTM to their observed counterparts the local distance difference test (lDDT) and TM-score for both tertiary and quaternary models to examine whether reliability is retained across the scoring range under normal modelling conditions and in situations where AlphaFold2 functionality is customized. We also explore plDDT and pTM ranking accuracy in comparison with the published independent MQA programmes ModFOLD9 and ModFOLDdock.
plDDT was found to be an accurate descriptor of tertiary model quality compared to observed lDDT-Cα scores (Pearson r = 0.97), and achieved a ranking agreement true positive rate (TPR) of 0.34 with observed scores, which ModFOLD9 could not improve. However, quaternary structure accuracy was reduced (plDDT r = 0.67, pTM r = 0.70) and significant overprediction was seen with both scores for some lower quality models. Additionally, ModFOLDdock was able to improve upon AF2-Multimer model ranking compared to TM-score (TPR 0.34) and oligo-lDDT score (TPR 0.43). Finally, evidence is presented for increased variability in plDDT and pTM when using custom template recycling, which is more pronounced for quaternary structures.
The ModFOLD9 and ModFOLDdock quality assessment servers are available at https://www.reading.ac.uk/bioinf/ModFOLD/ and https://www.reading.ac.uk/bioinf/ModFOLDdock/, respectively. A docker image is available at https://hub.docker.com/r/mcguffin/multifold.
尽管 AlphaFold2 的开发提高了蛋白质建模的准确性，但预测和观察到的模型质量评估 (MQA) 得分之间仍然存在准确性差距。在 CASP15 中，注意到 AlphaFold2 模型准确性预测的变化，对于观察到的质量非常相似的四级模型。在这项研究中，我们将 plDDT 和 pTM 与其观察到的对应物局部距离差异测试 (lDDT) 和 TM 分数进行比较，无论是三级还是四级模型，以检查在正常建模条件下以及在定制 AlphaFold2 功能的情况下，可靠性是否在评分范围内保留。我们还比较了 plDDT 和 pTM 与已发布的独立 MQA 程序 ModFOLD9 和 ModFOLDdock 的排名准确性。
与观察到的 lDDT-Cα 分数相比，plDDT 被发现是三级模型质量的准确描述符（Pearson r=0.97），并且与观察到的分数相比，排名协议真正阳性率 (TPR) 达到 0.34，而 ModFOLD9 无法提高。然而，四级结构的准确性降低了（plDDT r=0.67，pTM r=0.70），对于一些质量较低的模型，两个分数都出现了显著的过预测。此外，ModFOLDdock 能够提高与 TM 分数（TPR 0.34）和寡聚 lDDT 分数（TPR 0.43）相比的 AF2-Multimer 模型排名。最后，当使用自定义模板回收时，plDDT 和 pTM 的变异性增加的证据被呈现出来，对于四级结构，这种变异性更加明显。
ModFOLD9 和 ModFOLDdock 质量评估服务器可在 https://www.reading.ac.uk/bioinf/ModFOLD/ 和 https://www.reading.ac.uk/bioinf/ModFOLDdock/ 获得。一个 Docker 镜像可在 https://hub.docker.com/r/mcguffin/multifold 获得。
