Decision trees can be intelligible, but do they perform well enough that you should use them? Have SVMs replaced neural nets, or are neural nets still best for regression and SVMs best for classification? Boosting maximizes a margin similar to SVMs, but can boosting compete with SVMs? If it does, is it better to boost weak models or to boost stronger models? Bagging is easier than boosting - how well does it stack up against boosting? Breiman said Random Forests are better than bagging and as good as boosting. Was he right? And what about old friends like logistic regression, KNN, and naive Bayes? Should they be put out to pasture, or do they fill important niches?
In this talk we'll compare the performance of ten supervised learning methods on nine criteria: Accuracy, F-score, Lift, Precision/Recall Break-Even Point, Area under the ROC, Average Precision, Squared Error, Cross-Entropy, and Probability Calibration. The results show that no one learning method does it all, but some methods can be 'repaired' to do very well across all performance metrics. In particular, we show how to obtain good probabilities from max-margin methods such as SVMs and boosting via Platt's Method and Isotonic Regression. We also describe a meta-ensemble method that combines select models from these ten learning methods to yield even better performance than any of the individual learning methods. Although these ensembles perform extremely well, they are too complex for some real-world applications. We'll describe a model compression method that tries to fix that. Finally, if time permits, we'll discuss how the nine performance metrics relate to each other, and on which of the metrics you probably should (or shouldn't) depend.