Optimization problems involving the sum of three convex functions have received much attention in recent years, where one is differentiable with Lipschitz continuous gradient, one is composed of a linear operator and the other is proximity friendly. The primal-dual fixed point algorithm is a simple and effective algorithm for such problems. To exploit the second-order derivatives information of the objective function, we propose a primal-dual fixed point algorithm with an adapted metric method. The proposed algorithm is derived from the idea of establishing a generally fixed point formulation for the solution of the considered problem. Under mild conditions on the iterative parameters, we prove the convergence of the proposed algorithm. Further, we establish the ergodic convergence rate in the sense of primal-dual gap and also derive the linear convergence rate with additional conditions. Numerical experiments on image deblurring problems show that the proposed algorithm outperforms other state-of-the-art primal-dual algorithms in terms of the number of iterations.