Airborne magnetic and gradient measurements are commonly used geophysical remote sensing tools to obtain the distribution features of ore mineral bodies. It is known that ore mineral bodies generally contain remanent magnetization, and magnetization vector inversion (MVI) can produce the magnetization intensity and direction of the source, which is more suitably used to interpret measured airborne magnetic and gradient data. To accurately reveal the underground magnetization vector distribution, we proposed a high-precision method with double constraints on the data and physical structure, and we used the cross-gradient inversion of airborne magnetic anomalies and the combination matrix of airborne magnetic and gradient (CMG) data to recover the physical parameters of the sources with different depths. We used the combination matrix to produce the different component data constraints and the cross-gradient function to finish the inversion to provide structural constraints. For anomaly sources at similar depths, joint inversion based on the cross-gradient of magnetic gradient data and CMG data is more suitably used. The superiority of the double constraints method is proven by theoretical model tests. We apply the proposed method to interpret airborne magnetic and gradient data in Shandong Province to detect iron mineral resources, and we select the cross-gradient inversion of airborne magnetic anomalies and CMG data depending on the nonlinear features of the power spectrum. The main ore bodies have a northeast distribution with a depth range of 1048–1800 m, successfully giving the distribution range of the high-magnetic bodies; a better mineral potential is in the northern part of the survey area.