The performance of the typical scene categorization approach based on spatial pyramid model (SPM) and support vector machine (SVM) is limited in high-dimensional image representations and kernelized classifiers. In this article, a novel method combined with compact spatial pyramid model (CSPM) and ensemble of extreme learning machines (ELM) is proposed. The method consists of two major steps: First, Agglomerative Information Bottleneck (AIB) algorithm is applied to construct a compact spatial pyramid model (CSPM), which can compress the vocabulary size that maintains the performance of original SPM. Second, an effective ensemble method combined with Rotation Forest and weighted voting scheme for ELM (RFWV-ELM) is applied to enhance the classification performance. This ensemble method could solve the instability caused by the randomly assigned weights and biases of original ELM and improve the generalization ability of ELM neural network simultaneously. The experimental results on two benchmark datasets illustrated that the proposed framework combined with CSPM and RFWV-ELM can achieve better classification performance than several existing scene categorization algorithms.