In recent years, deep learning methods have made great progress in hyperspectral image classification. However, the current models obtain deep-seated global feature by deepening the number of network layers, ignore the local neighborhood environment. To solve this problem, we propose a double-branch network model, which extracts spatial and spectral local context feature respectively, fuses different levels of feature adaptively and has strong ability to aggregate context information. Specifically, this model uses self-calibrated convolution to extract spatial information and multi-scale dense convolution to extract spectral information. We propose that local context feature extraction (LCFM) module makes full use of the background information and mutual information between positions of each pixel to obtain local context feature information. In addition, focal-loss is used to solve the problem of different classification difficulty of each sample. Simulation results showed that the proposed module not only achieves high classification accuracy, but also greatly reduces the amount of calculation and parameters.