The classical Pauli particle (CPP) serves as a slow manifold, substituting the conventional guiding center dynamics. Based on the CPP, we utilize the averaged vector field (AVF) method in the computations of drift orbits. Demonstrating significantly higher efficiency, this advanced method is capable of accomplishing the simulation in less than one-third of the time of directly computing the guiding center motion. In contrast to the CPP-based Boris algorithm, this approach inherits the advantages of the AVF method, yielding stable trajectories even achieved with a tenfold time step and reducing the energy error by two orders of magnitude. By comparing these two CPP algorithms with the traditional RK4 method, the numerical results indicate a remarkable performance in terms of both the computational efficiency and error elimination. Moreover, we verify the properties of slow manifold integrators and successfully observe the bounce on both sides of the limiting slow manifold with deliberately chosen perturbed initial conditions. To evaluate the practical value of the methods, we conduct simulations in non-axisymmetric perturbation magnetic fields as part of the experiments, demonstrating that our CPP-based AVF method can handle simulations under complex magnetic field configurations with high accuracy, which the CPP-based Boris algorithm lacks. Through numerical experiments, we demonstrate that the CPP can replace guiding center dynamics in using energy-preserving algorithms for computations, providing a new, efficient, as well as stable approach for applying structure-preserving algorithms in plasma simulations.