Compared with traditional vibration measurement sensors, visual vibration displacement measurement technology has many advantages, such as long-distance, non-contact and non-interference. However, when it comes to complex working conditions or inconvenient pre-processing scene, visual measuring techniques usually can’t produce results with the high precision needed for vibration analysis. In this paper, we proposed a visual measurement algorithm for vibration displacement of rotating body using semantic segmentation network by taking a high-speed industrial camera as the image acquisition medium and rotor as the experimental object. To make the network better applicable to segmentation of rotating body targets in different illumination, blur and similarity contexts, the cross stage partial (CSP) connections, concat channel fusion module and shuffle attention (SA) are purposefully incorporated into proposed CMCS convolution module. During calculating the loss in the training network stage, we add the binary cross-entropy loss and Dice-loss to solve the problem of serious imbalance between positive and negative samples caused by remote photography. The vibration signal synchronously collected by eddy current sensor is exploited as standard signal to compare the different segmentation algorithms quantitatively and qualitatively. The experimental results show that the RMSE between vibration displacement signal regressed by our segmentation algorithm and standard signal is the smallest, and the time domain diagram, frequency domain diagram and shaft orbit diagram are also more fitted to the standard signal curve, which provides valuable guidance for the visual vibration displacement measurement of rotating body in complex industrial setting.