To enhance the accuracy and efficiency of the deflection response measurement of concrete bridges with a non-contact scheme and address the ill-conditioned nature of the inverse problem in influence line (IL) identification, this study introduces a computer-vision-aided deflection IL identification method that integrates edge detection and time-domain forward inference (TDFI). The methodology proposed in this research leverages computer vision technology with edge detection to surpass traditional contact-based measurement methods, greatly enhancing the operational efficiency and applicability of IL identification and, in particular, addressing the challenge of accurately measuring small deflections in concrete bridges. To mitigate the limitations of the Lucas–Kanade (LK) optical flow method, such as unclear feature points within the camera’s field of view and occasional point loss in certain video frames, an edge detection technique is employed to identify maximum values in the first-order derivatives of the image, creating virtual tracking points at the bridge edges through image processing. By precisely defining the bridge boundaries, only the essential structural attributes are preserved to enhance the reliability of minimal deflection deformations under vehicular loads. To tackle the ill-posed nature of the inverse problem, a TDFI model is introduced to identify IL, recursively capturing the static bridge response generated by the bridge under the influence of successive axles of a multi-axle vehicle. The IL is then computed by dividing the response by the weight of the preceding axle. Furthermore, an axle weight ratio reduction coefficient is proposed to mitigate noise amplification issues, ensuring that the weight of the preceding axle surpasses that of any other axle. To validate the accuracy and robustness of the proposed method, it is applied to numerical examples of a simply supported concrete beam, indoor experiments on a similar beam, and field tests on a three-span continuous concrete beam bridge.