Repairing concavities in ROC curvesP.A. Flach, S. Wu, Repairing concavities in ROC curves. Proc. 2003 UK Workshop on Computational Intelligence, pp. 38–44. August 2003. PDF, 309 Kbytes.
In this paper we propose methods to detect and repair concavities in ROC curves by manipulating model predictions. We introduce two model assembly algorithms. Algorithm SwapOne aims to improve the Area Under the ROC Curve (AUC) of a probabilistic classifier by investigating three models from different thresholds of a probabilistic model, such that one is below the line connecting the other two, and assembles a hybrid model combining the two better models and an inversion of the poorer model. Algorithm SwapCurve aims to improve AUC by identifying part of a probabilistic ROC curve that is below its convex hull, and inverting the ranking of test instances in that part of the curve. Experimental results on 24 UCI datasets demonstrate that the second algorithm gives small but significant improvements on 10 of these datasets. The novelty of our approach lies in that, unlike blind ensemble methods, it investigates the performance of the model in order to decide where performance needs to be improved.