Deep learning is a popular and effective technique for the hyperspectral image (HSI) classification. Current deep learning-based methods have numerous free parameters to be trained. They may be unavailable once lacking training samples. In addition, these approaches only use the features from the deepest layer and exclude shallow features, which is a kind of information loss. To remedy such deficiencies, in this work, we construct a novel deep encoder with kernel-wise Taylor series (EKTS) for the HSI classification. More specifically, we introduce the Taylor series to approximate the role of deep networks for feature extraction. Because the original Taylor series is linear, the kernel theory is used to build the kernel-wise Taylor series to encode the HSI data and extract deep nonlinear features. Furthermore, an alternating iterative optimization strategy is developed to obtain the outputs of all layers of the built deep encoder. Subsequently, we stack the outputs of all layers to obtain the final features that integrate both shallow and deep features. At last, the support vector machine (SVM) is adopted to deal with the obtained final features so that label results can be predicted. Unlike current deep learning methods, our EKTS has no free parameters to be trained and combines the advantages of both shallow and deep features to predict labels. Sufficient experimental analysis has been performed to verify the greater classification performance of our EKTS method compared with many state-of-the-art approaches.