AbstractThis paper presents the implementation of a novel monitoring system in which video images and conventional sensor network data are simultaneously analyzed to identify the structural flexibility from the ambient vibrations. The magnitude ratio between the flexibility estimated from known/unknown input force are theoretically derived and decomposed into two parts: and . The first scale factor related to basic modal parameters can be acquired using the general modal identification methods. Aiming to tackle the difficulty in identifying the second scale factor related to the force intensity, a video stream of traffic is processed to detect and classify vehicles to determine the vehicle's location while displacement measurements are simultaneously collected. By integrating the toll station data, the vehicle loads are assigned to the vehicle on the bridge deck through the uniqueness of the license plate number. Thus, a structural input–output relationship is established to solve the second scale factor . Finally, the flexibility estimated from the ambient vibration are scaled by and , respectively to obtain the exact flexibility , which are same as the analytical ones . Both numerical example and a laboratory test are performed to demonstrate the accuracy of the proposed methodology. The algorithms, approaches, and results given in the paper demonstrate its effectiveness and shows great potential for its application on a real‐life bridge's condition assessment.