To ensure safe operation, the high-precision estimation of the state of charge (SOC) in the battery management system (BMS) is relied on. The classical recurrent neural network (RNN) has a gradient and poor accuracy problem, and the RNN with additional gates is complex and hard to apply in engineering. To address these issues, the RNN with small sequence trained by multi-level optimization is proposed in this paper to improve the accuracy and the gradient problem of the SOC estimation. First, the small sequence is introduced to improve the gradient problem and running speed of RNN and the SOC post filter is used to increase the continuity of SOC estimation. Then, the particle swarm optimization (PSO) is used to pre-train the RNN to obtain the optimal weight and threshold value, and the hybrid optimizer of Adam and stochastic gradient descent (SGD) is utilized to improve the accuracy of SOC estimation. The experimental data of the charging and discharging test, and the performances of the proposed method, the nonlinear autoregressive with exogeneous inputs neural network (NARX), RNN, long short-term memory (LSTM), and gated recurrent unit (GRU) for estimating the SOC of lithium batteries are compared. The results show that the estimation error of the RNN is within 3.66% and that of NARX is within 6%, and the proposed method, LSTM, and GRU reduce the error to 0.47%, 1.04%, and 0.48%, respectively. Moreover, the calculation time of the proposed method is 1/3 and 1/9 of that of LSTM and GRU. Besides, the proposed method is robust to different vehicle cycles, measurement noises, temperature variations, and battery aging.