Sports injuries of high-level athletes restrict the improvement of sports performance. Under this premise, an efficient and accurate sports injury assessment method is needed to detect potential sports injuries and conduct injury prevention training. Therefore, this paper proposes a novel sports injury prediction algorithm based on visual analysis technology. The proposed algorithm first takes the time-frequency of sensed data as the convolutional neural network (CNN) input. The one-dimensional time series collected by the sensor is converted into two-dimensional images using the Gram angle domain algorithm. The one-dimensional sensed data provides a new perspective and provides a basis for better use of convolutional neural networks and computer vision technology. Second, combining the residual network’s structure and advantages and hole convolution and multihole convolution kernel residual module is proposed. It improves the model’s ability to extract features at different scales while effectively controlling the parameter scale. Based on these modules, a single-sensor-based athlete action recognition algorithm is proposed. Several comparative experiments have been conducted on a public data set containing only acceleration sensors to verify the proposed algorithm’s effectiveness.