Abstract Considering the features of the fractional Klein-Kramers equation (FKKE) in phase space, only the unilateral boundary condition in position direction is needed, which is different from the bilateral boundary conditions in [Cartling B., Kinetics of activated processes from nonstationary solutions of the Fokker-Planck equation for a bistable potential, J. Chem. Phys., 1987, 87(5), 2638–2648] and [Deng W., Li C., Finite difference methods and their physical constrains for the fractional Klein-Kramers equation, Numer. Methods Partial Differential Equations, 2011, 27(6), 1561–1583]. In the paper, a finite difference scheme is constructed, where temporal fractional derivatives are approximated using L1 discretization. The advantages of the scheme are: for every temporal level it can be dealt with from one side to the other one in position direction, and for any fixed position only a tri-diagonal system of linear algebraic equations needs to be solved. The computational amount reduces compared with the ADI scheme in [Cartling B., Kinetics of activated processes from nonstationary solutions of the Fokker-Planck equation for a bistable potential, J. Chem. Phys., 1987, 87(5), 2638–2648] and the five-point scheme in [Deng W., Li C., Finite difference methods and their physical constrains for the fractional Klein-Kramers equation, Numer. Methods Partial Differential Equations, 2011, 27(6), 1561–1583]. The stability and convergence are proved and two examples are included to show the accuracy and effectiveness of the method.