Vibration-based Structural Health Monitoring (SHM) is widely recognized as a prominent approach for evaluating the safety and integrity of civil infrastructure. This research presents a novel algorithm for vibration measurement using a movable camera, effectively overcoming the issue of signal drift caused by camera motion. The proposed algorithm comprises three modules: optical flow computation, camera pose estimation, and motion compensation. Validation experiments by virtual images demonstrate accurate camera pose estimation and effective mitigation of drifting signals. The dynamic characteristics of the frame structure are extracted using a movable camera and the proposed algorithm. Several challenges are addressed, including accurately estimating the scale factor to compensate for the drifting signals. Experimental validation is conducted to verify the proposed approach. The results demonstrate the efficacy and promising potential of the proposed approach. Finally, this study demonstrates the practical application through damage localization experiments.