This study proposes an enhanced integration algorithm that combines the magnetic field-based positioning system (MPS—Magnetic Pose Estimation System) with an inertial system with the advantage of an invariant filter structure. Specifically, to mitigate the nonlinearity of the propagation model and perturbing effect from the estimated uncertainty, the formulation of the invariant Kalman filter was derived in detail. Then, experiments were conducted to validate the algorithm with an unmanned vehicle equipped with an IMU and MPS receiver. As a result, the navigation performance of the IEKF-based inertial and magnetic field integration system was presented and compared with the conventional Kalman filter results. Furthermore, the convergence and navigation performance were evaluated in the presence of state variable initialization errors. The findings indicate that the inertial and magnetic field coupled with the IEKF outperforms the typical KF approach, particularly when dealing with initial estimate uncertainties.