This paper aims to improve the tracking control performance of the three-stage valve (TSV) controlled electro-hydraulic servo system (EHSS) with parameter uncertainties and other lumped unknown nonlinearities, including unknown dynamics and disturbances. A more accurate nonlinear model of the TSV-controlled EHSS is established and a neural network-based finite-time command-filtered adaptive backstepping control (NNFCABC) method is proposed for the EHSS. Adaptive control is used to deal with the system parameter uncertainties, and the radial basis function neural network (RBFNN) algorithm is introduced to approximate the lumped unknown nonlinearities. The prediction errors of serial-parallel estimation models (SPEMs) and the tracking errors are utilized together to design adaptive laws to estimate the system parameters and the weights of the RBFNNs. The entire control framework utilizes command-filtered control and backstepping techniques. By applying Levant differentiators as command filters and introducing fractional power terms into the virtual control laws and the SPEMs, the proposed NNFCABC theoretically guarantees the tracking performance of the closed-loop control system with finite-time convergence. Comparative simulations and experiments verify the feasibility and superiority of the proposed control scheme.