Early fault diagnosis of rolling bearings is crucial to operating and maintenance cost reduction of the equipment with bearings. This paper aims to propose a novel early fault feature extraction method based on the proposed hierarchical symbol dynamic entropy (HSDE) and the binary tree support vector machine (BT-SVM). Multiscale symbolic dynamic entropy (MSDE) has been recently proposed to characterize the dynamical behavior of time series. MSDE has several merits comparing with multiscale sample entropy (MSE) and multiscale permutation entropy (MPE), such as high computational efficiency and robustness to noise. However, MSDE only utilizes the fault information in the low frequency components and consequently the fault information hidden in the high frequency components is discarded. To address this shortcoming, a new method, namely HSDE, is proposed to extract the fault information in the high frequency components. Then, the BT-SVM is utilized to automatically complete the fault type identification. The effectiveness of the proposed method is validated using simulated and experimental vibration signals. Meanwhile, a comparison is conducted between MPE, hierarchical permutation entropy (HPE), MSE, hierarchical sample entropy (HSE), MSDE and HSDE. Results show that the proposed method performs best to recognize the early fault types of rolling bearings.