Monitoring the actual vibration coverage is critical for preventing over- or under-vibration and ensuring concrete’s strength. However, the current manual methods and sensor techniques fail to meet the requirements of on-site construction. Consequently, this study proposes a novel approach for estimating the pose of concrete vibrator racks. This method integrates the Linear Spatial Kernel Aggregation (LSKA) module into the You Only Look Once (YOLO) framework to accurately detect the keypoints of the rack and then employs the vanishing point theorem to estimate the rotation angle of the rack without any 3D datasets. The method enables the monitoring of the vibration impact range for each vibrator’s activity and is applicable to various camera positions. Given that measuring the rotation angle of a rack in reality poses is challenging, this study proposes employing a simulation environment to validate both the feasibility and accuracy of the proposed method. The results demonstrate that the improved YOLOv8-Pose achieved a 1.4% increase in accuracy compared with YOLOv8-Pose, and the proposed method monitored the rotation angle with an average error of 6.97° while maintaining a working efficiency of over 35 frames per second. This methodology was successfully implemented at a construction site for a high-arch dam project in China.