Research Centre for Cheminformatics, Jasenova 7, 11030, Beograd, Serbia.
Laboratory for Molecular Biomedicine, Institute of Molecular Genetics and Genetic Engineering, University of Belgrade, Vojvode Stepe 444a, 11010, Beograd, Serbia.
J Cheminform. 2014 Mar 29;6(1):10. doi: 10.1186/1758-2946-6-10.
We address the problem of selecting and assessing classification and regression models using cross-validation. Current state-of-the-art methods can yield models with high variance, rendering them unsuitable for a number of practical applications including QSAR. In this paper we describe and evaluate best practices which improve reliability and increase confidence in selected models. A key operational component of the proposed methods is cloud computing which enables routine use of previously infeasible approaches.
We describe in detail an algorithm for repeated grid-search V-fold cross-validation for parameter tuning in classification and regression, and we define a repeated nested cross-validation algorithm for model assessment. As regards variable selection and parameter tuning we define two algorithms (repeated grid-search cross-validation and double cross-validation), and provide arguments for using the repeated grid-search in the general case.
We show results of our algorithms on seven QSAR datasets. The variation of the prediction performance, which is the result of choosing different splits of the dataset in V-fold cross-validation, needs to be taken into account when selecting and assessing classification and regression models.
We demonstrate the importance of repeating cross-validation when selecting an optimal model, as well as the importance of repeating nested cross-validation when assessing a prediction error.
我们解决了使用交叉验证选择和评估分类和回归模型的问题。目前最先进的方法可能会产生方差较大的模型，使得它们不适合许多实际应用，包括 QSAR。在本文中，我们描述并评估了提高所选模型可靠性和信心的最佳实践。所提出方法的一个关键操作组件是云计算，它使以前不可行的方法得以常规使用。
我们详细描述了一种用于分类和回归的参数调整的重复网格搜索 V 折交叉验证算法，并且我们定义了一种用于模型评估的重复嵌套交叉验证算法。关于变量选择和参数调整，我们定义了两种算法（重复网格搜索交叉验证和双交叉验证），并为在一般情况下使用重复网格搜索提供了论据。
我们在七个 QSAR 数据集上展示了我们算法的结果。在选择和评估分类和回归模型时，需要考虑在 V 折交叉验证中选择不同数据集划分的预测性能的变化。
我们证明了当选择最优模型时重复交叉验证的重要性，以及当评估预测误差时重复嵌套交叉验证的重要性。
