a Plasma Chemistry Research Group , Research Centre for Natural Sciences, Hungarian Academy of Sciences , Budapest, Hungary.
b Medicinal Chemistry Research Group , Research Centre for Natural Sciences, Hungarian Academy of Sciences , Budapest, Hungary.
SAR QSAR Environ Res. 2018 Sep;29(9):661-674. doi: 10.1080/1062936X.2018.1505778. Epub 2018 Aug 30.
Prediction performance often depends on the cross- and test validation protocols applied. Several combinations of different cross-validation variants and model-building techniques were used to reveal their complexity. Two case studies (acute toxicity data) were examined, applying five-fold cross-validation (with random, contiguous and Venetian blind forms) and leave-one-out cross-validation (CV). External test sets showed the effects and differences between the validation protocols. The models were generated with multiple linear regression (MLR), principal component regression (PCR), partial least squares (PLS) regression, artificial neural networks (ANN) and support vector machines (SVM). The comparisons were made by the sum of ranking differences (SRD) and factorial analysis of variance (ANOVA). The largest bias and variance could be assigned to the MLR method and contiguous block cross-validation. SRD can provide a unique and unambiguous ranking of methods and CV variants. Venetian blind cross-validation is a promising tool. The generated models were also compared based on their basic performance parameters (r and Q). MLR produced the largest gap, while PCR gave the smallest. Although PCR is the best validated and balanced technique, SVM always outperformed the other methods, when experimental values were the benchmark. Variable selection was advantageous, and the modelling had a larger influence than CV variants.
预测性能往往取决于所应用的交叉验证和测试验证协议。为了揭示其复杂性，使用了几种不同的交叉验证变体和建模技术的组合。检查了两个案例研究（急性毒性数据），应用了五重交叉验证（随机、连续和威尼斯盲人形式）和留一法交叉验证（CV）。外部测试集显示了验证协议之间的效果和差异。使用多元线性回归（MLR）、主成分回归（PCR）、偏最小二乘回归（PLS）、人工神经网络（ANN）和支持向量机（SVM）生成模型。通过排序差异总和（SRD）和方差因子分析（ANOVA）进行比较。最大的偏差和方差可以归因于 MLR 方法和连续块交叉验证。SRD 可以提供方法和 CV 变体的独特且明确的排序。威尼斯盲人交叉验证是一种很有前途的工具。还根据其基本性能参数（r 和 Q）对生成的模型进行了比较。MLR 产生的差距最大，而 PCR 产生的差距最小。尽管 PCR 是验证和平衡效果最好的技术，但当实验值作为基准时，SVM 始终优于其他方法。变量选择是有利的，建模比 CV 变体的影响更大。
