A lightweight bearing fault detection approach based on an improved residual network is presented to solve the shortcomings of previous fault diagnostic methods, such as inadequate feature extraction and an excessive computational cost due to high model complexity. First, the raw data are turned into a time–frequency map using the continuous wavelet transform, which captures all of the signal’s time- and frequency-domain properties. Second, an improved residual network model was built, which incorporates the criss-cross attention mechanism and depth-separable convolution into the residual network structure to realize the important distinction of the extracted features and reduce computational resources while ensuring diagnostic accuracy; simultaneously, the Meta-Acon activation function was introduced to improve the network’s self-adaptive characterization ability. The study findings indicate that the suggested approach had a 99.95% accuracy rate and a floating point computational complexity of 0.53 GF. Compared with other networks, it had greater fault detection accuracy and stronger generalization ability, and it could perform high-precision fault diagnostic jobs due to its lower complexity.