Cameras capture the dynamic behaviors of vibrating structures or systems in videos, and the object in videos can respond to new forces to explore more dynamics by visual simulation methods. However, existing visual simulation models usually suffer from strong noise interference and lack rigorous verification of the simulation accuracy in the field of structural dynamics analysis. In this article, we present an interactive visual simulation modeling (IVSM) method, aiming to build an effective and accurate visual simulation model for structure dynamic response analysis. The IVSM endows the object in videos with the ability to directly and visually respond to new forces, in a way of human–computer interaction. We demonstrate the excellent performance of the IVSM in two application scenarios, i.e., the response prediction and damage detection. In response prediction, experiments, and IVSM simulations are compared to verify the effectiveness and accuracy of the proposed IVSM. The synthesized animations of the IVSM also present high fidelity to the actual experiment. In damage detection, we propose a novel damage magnification technique to build the IVSM. The IVSM simulation results demonstrate the superior performance in detecting and visualizing abnormal behaviors caused by damage.