Representing the scenes by learning the subtle variations in the spatial content of different classes is crucial for scene classification in remote sensing images. In this paper, we propose a scene attribute modeling to obtain a discriminative and compact representation for scene classification. First, we construct a scene attribute model (SAM) by training a Gaussian mixture model (GMM) using convolutional features to capture the scene attributes implicitly. Then, we perform a maximum a posteriori (MAP) adaptation to enhance the contribution of significant attributes in each scene resulting in a high-dimensional feature vector which contains redundant attributes from all the scenes. Hence, we use factor analysis to obtain a compact representation of the high-dimensional feature vector termed scene-vector, which retains only the significant attributes specific to a scene. The proposed approach is demonstrated on three benchmark datasets, namely, UC Merced Land Use, AID, and NWPU-RESISC45 datasets. We further show that, being a compact representation, our scene-vector outperforms state-of-the-art methods for scene classification in remote sensing images.