The vibration process applied to fresh concrete is an important link in the construction process, but the lack of effective monitoring and evaluation methods results in the quality of the vibration process being difficult to control and, therefore, the structural quality of the resulting concrete structures difficult to guarantee. In this paper, according to the sensitivity of internal vibrators to vibration acceleration changes under different vibration media, the vibration signals of vibrators in air, concrete mixtures, and reinforced concrete mixtures were collected experimentally. Based on a deep learning algorithm for load recognition of rotating machinery, a multi-scale convolution neural network combined with a self-attention feature fusion mechanism (SE-MCNN) was proposed for medium attribute recognition of concrete vibrators. The model can accurately classify and identify vibrator vibration signals under different working conditions with a recognition accuracy of up to 97%. According to the classification results of the model, the continuous working times of vibrators in different media can be further statistically divided, which provides a new method for accurate quantitative evaluation of the quality of the concrete vibration process.