In this study, we developed a technique to utilize phase nonlinearity as a quantifiable confidence measure of the reliability of component displacement vectors extracted from multi-scale and multi-direction phases. The proposed approach includes three novel concepts. First, relative confidence values are established based on phase nonlinearity to quantify the reliability of the displacement components across various environmental conditions. Second, by analyzing the distribution of the relative confidence values, adaptive confidence measures are extracted and are used to reject unreliable displacement components; these confidence measures can dynamically adapt to various environmental conditions, such as varying image noise and large amplitudes of motion. Third, phase nonlinearity–weighted motion integration is conducted to ensure accurate estimation of the full displacement vector from the component vectors. By integrating these three approaches, a remarkable enhancement was achieved in full-field displacement measurement in terms of accuracy, robustness, and signal-to-noise ratio, especially in environments with high noise levels. The capability of the proposed technique was evaluated through numerical experiments using artificial patterns that simulate identical motions under varying noise levels. Experimental validation was also performed on an air compressor to substantiate the accuracy and robustness of the proposed technique. The findings verify its improvements over conventional techniques for full-field displacement measurement.