You will probably want to use the batchtest tool for large experiments; xvalidate will help you to quickly test things, perhaps as a debugging aid.
You can use xvalidate with large datasets, but you will need enough disk space to hold 'folds' copies of the data. The learner you use with xvalidate must also be able to work with large datasets.
Xvalidate takes input in C4.5 format and uses folddata, which must be in your path. You use the -c option to tell xvalidate how to run the learner. Xvalidate will append the names of the folds of the datasets to the end of the -c string, the learner must accept the name and read input appropriately.
Xvalidate expects the learner to output results in the following format:
error-rate size
The learner's error rate on the test set, followed by some whitespace, followed by the size of the learned model (in whatever unit you want), followed by a newline.
Xvalidate will collect the output of the runs of the learner, average them, and report:
mean-error-rate (standard deviation of error rate) mean-size (standard deviation of size) average-utime (standard deviation of utime) average-stime (standard deviation of stime)
for example:
26.111 (5.500) 0.000 (0.000) 0.013 (0.005) 0.010 (0.008)
The times are very accurate on UNIX. Under CYGNUS (windows) utime will be slightly overestimated and stime will be zero.
xvalidate -source datasets/mushroom -f mushroom -folds 15 -seed 100 -c "mostcommonclass -u -f"
Does 15-fold cross-validation of the 'mostcommonclass' learner on the dataset called 'mushroom' in the 'datasets/mushroom' directory. The mostcommonclass learner will be invoked as: mostcommonclass -u -f <constructed-dataset-name> for each of the 15 constructed datasets. It will use a seeded random number generator so the exact experiment could be reproduced.
hosted by