Skip to main content

WekaMetal

WekaMetal is a Meta-Learning extension to the data-mining package Weka. It offers advice on algorithm selection, based on expected accuracy and time performance. WekaMetal uses algorithms based on those produced by the MetaL consortium. More specifically, the paper Zoomed Ranking: Selection of Classification Algorithms Based on Relevant Performance Information. by C. Soares and P. Brazdil is very relevant to WekaMetal.

Introduction

WekaMetal provides a ranking which predicts how well a set of algorithms will perform on a particular dataset. (Roughly speaking, this is what meta-learning is about.) The ranking is generated using knowledge of the performance of those algorithms on some "benchmark" datasets. The ranker first "characterizes" the previously unseen dataset. Based on this characterisation, and a similar characterisation of the benchmark datasets, datasets are selected which are similar to the unseen dataset. WekaMetal then uses its knowledge of the performance of the algorithms on the benchmark datasets to predict the performance of the algorithms on the unseen dataset.

Download

The current version of WekaMetal is available for download. You will also need a version of Weka. WekaMetal is known to work with Weka 3.2. It may also work with later version although your milage may vary. Weka and WekaMetal are written in Java, and have been tested with Java 1.3. WekaMetal may work with earlier versions of Java that include Swing but this is untested.

Installation

You need to extract the above:
jar -tvf WekaMetal-0.11.jar
		
This will create a WekaMetal directory. This directory will contain a jarfile which holds the WekaMetal class files, and some configuration and working files (described below).

You will probably need to edit the benchmarks.list file to make it point to some datasets. I recommend you use the same benchmarks as those listed, to avoid having to regenerate the benchmark data (this takes a long time). Update the paths to wherever the UCI datasets are stored on your machine. WekaMetal may still complain that it needs to generate the benchmarks, in which case try disabling the "Strict Benchmark Checking" (see below).

Launching WekaMetal

To run WekaMetal, change to the WekaMetal directory and execute the following command:
java -classpath WekaMetal.jar: weka.gui.explorer.Explorer
	
For example, I have Weka installed in /usr/local/weka/weka-3-2, so I run the command:
java -classpath WekaMetal.jar:/usr/local/weka/weka-3-2/weka.jar weka.gui.explorer.Explorer
	

Usage

If you are not familiar with Weka you will need to read the Weka documentation. WekaMetal provides an extra tab in Weka (MetaL) which allows rankers to be run. Once a dataset has been loaded, this tab will become usable:

[WekaMetal Screenshot]

The top-centre widget allows different rankers to be selected. Currently there are two available. The SRR (Simple Ratio of Ratios) ranker ranks algorithms based on expected accuracy. The ARR (Adjusted Ratio of Ratios) ranks algorithms based on expected accuracy and algorithm run time.

Characteristics can be (de)activated in the central panel. A characteristic is a meta-level attribute which describes a dataset. If unsure, leave them all selected. The right hand panel selects the algorithms which should be ranked.

Clicking "Start" causes the ranker to run. The Ranking will be generated in a few seconds, providing that WekaMetal has previously computed the performance of the selected algorithms on the benchmark datasets. If your told that the benchmark data is not available you have two options:

Known Bugs and Limitations