Deflection, as an intuitive index, plays a pivotal role in assessing the load-bearing capability and structural integrity of complex and sizable steel structures. Notwithstanding, the conventional contact measurement is limited to static deflection and necessitates work stands and manual readings. Therefore, it hinders the attainment of dynamic deflection and fails to cater to the engineering demands of real-time and prolonged monitoring. By leveraging the advancements in computer vision technology, we propose an innovative system for real-time deflection monitoring of complex and sizable steel structures, particularly suitable for monotonic deformation induced by prolonged loading. Specifically, the off-axis-based displacement measurement method was adopted to surmount the constraint of requiring the optical axis of the camera to be perpendicular to the target. Additionally, the inverse compositional Gauss-Newton (IC-GN) algorithm and parallel computation based on seed point diffusion were exploited to boost the matching and computing efficiency for attaining real-time monitoring of multi-points. To validate the efficacy of the proposed system, we conducted static load monitoring tests on a Bailey beam, with a height of 29.5 m and length of 16.5 m per span, as part of the Alibaba Jiangsu headquarters project in Nanjing. The collected test data were compared with the results from a laser displacement sensor and the ABAQUS model. The outcomes substantiate that the system is capable of non-contact, expeditious, and straightforward installation, besides achieving high accuracy and real-time deflection monitoring. This system serves as a sophisticated solution for health monitoring in constructing complex and sizeable steel structures and further contributes to realizing construction intelligence.