Vector magnetometers play an essential role in aerial and marine navigation due to their advantages of low cost, portability, and high sensitivity. However, the unstable magnetic field anomaly restricts the performance improvement of the magnetic compass. The existing methods have done a lot of optimization and improvement work, but the test results are less than expected in a dynamic environment. In order to solve this problem, a bionic magnetic compass algorithm based on the theory of “radical pairs” in birds was proposed. This article finds the stability convergence of the magnetic compass deviation (MCD) at the critical angle. The theoretical analysis and experiment results show that the heading accuracy at the critical angle is trustworthy. Then, the MCD distribution and the magnetic heading correction model are established. Finally, the current magnetic heading is corrected according to the angle difference between the critical angle and the current heading. The sea trial shows that compared with the traditional compensation algorithm, the reservoir environment is improved by 83.1%, and the effect in real ocean environment is improved by 30.12% and 58.49%.