A Combination of Methods for Building Ensemble of Classifiers

In this paper we make an extensive study of different methods for building ensembles of classifiers. We examine variants of ensemble methods that are based on perturbing features. We illustrate the power of using these variants by applying them to a number of different problems. We find that the best performing ensemble is obtained by combining an approach based on random subspace with a cluster based input decimated ensemble and the principal direction oracle. Compared with other state-of-the-art stand-alone classifiers and ensembles, this method consistently performed well across twelve diverse benchmark datasets. Another useful finding is that this approach does not need parameters to be carefully tuned for each dataset (in contrast to the fundamental importance of parameters tuning when using SVM and extreme learning machines), making our ensemble method well suited for practitioners since there is less risk of over-training. Another interesting finding is that random subspace can be coupled with several other ensemble methods to improve performance.