Naeini Mahdi Pakdaman, Cooper Gregory F
Intelligent Systems Program, University of Pittsburgh, Pittsburgh, USA.
Department of Biomedical Informatics, University of Pittsburgh, Pittsburgh, USA.
Proc IEEE Int Conf Data Min. 2016 Dec;2016:360-369. doi: 10.1109/ICDM.2016.0047. Epub 2017 Feb 2.
Learning accurate probabilistic models from data is crucial in many practical tasks in data mining. In this paper we present a new non-parametric calibration method called (ENIR). The method can be considered as an extension of BBQ [20], a recently proposed calibration method, as well as the commonly used calibration method based on isotonic regression (IsoRegC) [27]. ENIR is designed to address the key limitation of IsoRegC which is the monotonicity assumption of the predictions. Similar to BBQ, the method post-processes the output of a binary classifier to obtain calibrated probabilities. Thus it can be used with many existing classification models to generate accurate probabilistic predictions. We demonstrate the performance of ENIR on synthetic and real datasets for commonly applied binary classification models. Experimental results show that the method outperforms several common binary classifier calibration methods. In particular on the real data, ENIR commonly performs statistically significantly better than the other methods, and never worse. It is able to improve the calibration power of classifiers, while retaining their discrimination power. The method is also computationally tractable for large scale datasets, as it is ( log ) time, where is the number of samples.
从数据中学习准确的概率模型在数据挖掘的许多实际任务中至关重要。在本文中，我们提出了一种名为（ENIR）的新的非参数校准方法。该方法可被视为最近提出的校准方法BBQ [20]以及基于等渗回归的常用校准方法（IsoRegC）[27]的扩展。ENIR旨在解决IsoRegC的关键局限性，即预测的单调性假设。与BBQ类似，该方法对二元分类器的输出进行后处理以获得校准概率。因此，它可以与许多现有的分类模型一起使用以生成准确的概率预测。我们在合成数据集和真实数据集上展示了ENIR对于常用二元分类模型的性能。实验结果表明，该方法优于几种常见的二元分类器校准方法。特别是在真实数据上，ENIR通常在统计上比其他方法表现得显著更好，且从不更差。它能够提高分类器的校准能力，同时保留其区分能力。该方法对于大规模数据集在计算上也是易于处理的，因为它的时间复杂度为（log），其中是样本数量。
