Numerical discretization of a regularized inverse source problem leads to a non-symmetric saddle point linear system. Interestingly, the Schur complement of the non-symmetric saddle point system is Hermitian positive definite (HPD). Then, we propose a preconditioner matching the Schur complement (MSC). Theoretically, we show that the preconditioned conjugate gradient (PCG) method for a linear system with the preconditioned Schur complement as coefficient has a linear convergence rate independent of the matrix size and value of the regularization parameter involved in the inverse problem. Fast implementations are proposed for the matrix-vector multiplication of the preconditioned Schur complement so that the PCG solver requires only quasi-linear operations. To the best of our knowledge, this is the first solver with guarantee of linear convergence for the inversion of Schur complement arising from the discrete inverse problem. Combining the PCG solver for inversion of the Schur complement and the fast solvers for the forward problem in the literature, the discrete inverse problem (the saddle point system) is solved within a quasi-linear complexity. Numerical results are reported to show the performance of the proposed matching Schur complement (MSC) preconditioning technique.