Classification is one of the most popular topics in remote sensing. Consider the problems that the remote sensing data are complicated and few labeled training samples limit the performance and efficiency in the classification of remote sensing image. For these problems, a huge number of methods were proposed in the last two decades. However, most of them do not yield good performance. In this paper, a remote sensing image classification algorithm based on the ensemble of extreme learning machine (ELM) neural network, namely, stacked autoencoder (SAE)-ELM, is proposed. First, due to improve the ensemble classification accuracy, we adopt feature segmentation and SAE in the sample data to create high diversity among the base classifiers. Furthermore, ELM neural network is chosen as a base classifier to improve the learning speed of the algorithm. Finally, to determine the final ensemble-based classifier, Q-statistics is adopted. The experiment compares the proposed algorithm with Bagging, Adaboost, Random Forest et al., which results show that the proposed algorithm not only gets high classification accuracy on low resolution, medium resolution, high resolution and hyperspectral remote sensing images, but also has strong stability and generalization on UCI data.