An analysis of rule evaluation metricsJ. Furnkranz, P.A. Flach, An analysis of rule evaluation metrics. Proc. 20th International Conference on Machine Learning (ICML'03). ISBN 1-57735-189-4, pp. 202–209. January 2003. PDF, 153 Kbytes. External information
In this paper we analyze the most popular evaluation metrics for separate-and-conquer rule learning algorithms. Our results show that all commonly used heuristics, including accuracy, weighted relative accuracy, entropy, Gini index and information gain, are equivalent to one of two fundamental prototypes: precision, which tries to optimize the area under the ROC curve for unknown costs, and a cost-weighted difference between covered positive and negative examples, which tries to find the optimal point under known or assumed costs. We also show that a straight-forward generalization of the m-estimate trades off these two prototypes.