The key problem of remote sensing image classification is to understand the semantic content of the images effectively. Up to now, most methods are to improve the accuracy of the final classification through the convolutional neural networks model. However, the structure of the whole model is complex and contains a large number of parameters. To address this problem, in this study, we proposed a lightweight method of the intrinsic mean within Lie groups, which can achieve high accuracy. We apply the computational advantage of Lie group manifold space and design the kernel function that is suitable for both Lie group sample and vector sample. Experiments are performed on two publicly available and challenging data sets [aerial image dataset (AID) and Northwestern Polytechnical University-REmote Sensing Image Scene Classification, contains 45 scene classes (NWPU-RESISC45)], and the results show that our method is more accurate than the state-of-the-art methods.