This paper addresses the control problem of the flexible joint manipulator (FJM) with unmeasurable system states and mismatched uncertainties. First, the control system is transformed as a matched uncertain system with unmeasurable states based on differentiation method. Then, the uncertainties and unmeasurable states are estimated by designing a fixed-time observer (FTO). Based on the integral sliding-mode control (SMC), the bi-limit homogeneity technique and the estimation of FTO, a finite-time SMC is proposed for FJM. Compared with the existing finite-time SMC method for FJM, the most attractive feature of the proposed method is that not only the finite-time convergence is guaranteed but also the two angular velocity sensors for rotation angles of link and motor are simplified. Moreover, the proposed SMC can suppress the mismatched and matched uncertainties by using chattering-free control input. The fixed-time stability of FTO and finite-time stability of proposed controller are proved. Finally, the efficiency of proposed scheme is shown by the numerical simulation.