This paper proposed a new compensation method of underwater magnetic vector measurement system which innovatively using the geomagnetic total field as benchmark. Moreover, this method does not require the use of heading angle data in the calculation of the compensation parameters, which avoids errors caused by insufficient accuracy of underwater heading angle. The implementation of these features relies on the new objective function constructed according to the characteristics of the spatial projection of the magnetic vector field on the horizontal plane of the geographical coordinate. The core matrix of the geomagnetic vector ellipsoid model is used as a constraint to determine the range of function solutions. The Levenberg-Marquardt algorithm is used to find the optimal solution to the new objective function. Three experiments were carried out to verify the proposed method. In simulation experiment, the deviation between the calculated value and the preset value is less than 0.01%. In field experiment, the standard deviation value of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">x, y, z</i> magnetic components and the root mean square error of magnetic total field after compensation are reduced from 1527.38nT, 1397.35nT, 1529.16nT, 574.94nT to 9.86nT (0.65%), 9.99nT (0.72%), 5.63nT (0.38%), 2.14nT (0.37%). The proposed method is finally verified by marine experiment. The standard deviation value of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">x, y, z</i> magnetic components and the root mean square error of magnetic total field after compensation are reduced from 575.02 nT, 830.60 nT, 620.95 nT, 1702.35 nT to 17.43 nT (1.11%), 14.14 nT (1.70%), 25.17nT (4.10%), 7.18 nT (0.42%).