Non-contact measurement methods based on computer vision have shown great potential for measuring structural surface displacement and strain due to their low cost and high efficiency. However, the traditional computer vision technology based on digital image correlation (DIC) is limited by its dependence on natural texture features of the surface and its susceptibility to environmental conditions, making it challenging to obtain effective and robust measurements of surface deformation. Additionally, these methods typically only focus on the structural response in the image plane, which makes it difficult to perform a comprehensive health assessment of complex structural components with asymmetry. Therefore, a time-domain visual measurement technique based on image phase information is proposed for measuring three-dimensional (3D) deformation of structures using stereophotogrammetry. The method involves extracting image phase information using 2D Gabor filters and applying it to dense optical flow estimation based on polynomial approximation and semi-global block matching (SGBM) for obtaining robust 3D displacement measurements. The support vector regression (SVR) is then used to perform hyperplane fitting of the original displacements for improving the smoothness of the 3D displacements. Finally, an intuitive strain conversion method is introduced to convert the 3D displacements into 3D strains. The performance of the proposed technique has been validated through numerical simulation experiments based on a physics-based graphics model (PBGM) in a synthetic environment. The robustness of the proposed method has been further proved through free attenuation vibration experiments on a steel ruler. The initial evaluation of the expected performance of the measurement plan has been enabled by performing a 1:1 reduction of the field measurement process in a synthetic environment. The results indicate that the proposed method is a promising method for accurately and reliably measuring structural deformation in both synthetic and real-world environments, especially for complex structural components with asymmetry.