WCCI Performance Prediction Challenge

The challenge is over, but a new challenge is on-going using the same datasets, check it out!

Results File Formats

The results on each dataset should be formatted in ASCII files according to the following table. If you are a Matlab user, you may find some of the sample code routines useful for formatting the data. You can view an example of each format from the filename column. Optionally, you may submit your models in Matlab format.

Filename Development Challenge Description File Format
[dataname]_train.resu Optional Compulsory Classifier outputs for training examples +/-1 indicating class prediction.
[dataname]_valid.resu Compulsory Compulsory Classifier outputs for validation examples
[dataname]_test.resu Optional Compulsory Classifier outputs for test examples
[dataname]_train.conf Optional+ Optional+ Classifier confidence for training examples Non-negative real numbers indicating the confidence in the classification (large values indicating higher confidence). They do not need to be probabilities, and can be simply absolute values of discriminant values. Optionally they can be normalized between 0 and 1 to be interpreted as abs(P(y=1|x)-P(y=-1|x)).
[dataname]_valid.conf Optional+ Optional+ Classifier confidence for validation examples
[dataname]_test.conf Optional+ Optional+ Classifier confidence for test examples
[dataname].guess Optional* Compulsory* Your prediction of the BER (Balanced Error Rate) that you will achieve on test data A single number between 0 and 1.
[dataname]_model.mat Optional Optional The trained CLOP model used to compute the submitted results A Matlab learning object saved with the command save('[dataname]_model.mat', 'modelname').
+ If no confidence file is supplied, equal confidence will be assumed for each classification. If confidences are not between 0 and 1, they will be divided by their maximum value.
* If no guess file is supplied it will be assumed that the predicted BER is 1 (which is highly detrimental to your submission). Optionally, you may add a second number indicating the error bar on your guess.

Results Archive Format

Submitted files must be in either a .zip or .tar.gz archive format. You can download the example zip archive or the example tar.gz archive to help familiarise yourself with the archive structures and contents (the results were generated with the sample code). Submitted files must use exactly the same filenames as in the example archive. If you use tar.gz archives please do not include any leading directory names for the files. Use

zip results.zip *.resu *.conf *.guess *.mat
tar cvf results.tar *.resu *.conf *.guess *.mat; gzip results.tar
to create valid archives.

Challenge Submissions

If you wish that your method is ranked on the overall table you should include classification results on ALL the datasets for the five tasks, but this is mandatory only for final submissions.

The method of submission is via the form on the submissions page. Please limit yourself to 5 submissions per day maximum. If you encounter problems with submission, please contact the Challenge Webmaster.

Your last 5 valid submissions will count towards the final ranking. (There are no more "bonus entries"). The deadline for submissions is March 1, 2006.