The proposal of deep neural network achieves intelligent detection of abnormal fish behaviors. However, with the increase of network depth, the defects of large training memory and poor real-time performance restrict the deployment of the algorithm in aquaculture end devices. Therefore, this paper proposes a high-precision and lightweight end-to-end target detection model based on deformable convolution and improved YOLOv4. First of all, replacing the YOLOv4 backbone network with the lightweight network MobileNetV3 and replacing the standard convolution with a deep separable convolution have achieved a significant reduction in network parameters and calculations; Secondly, deformable convolution is used to improve the target feature extraction ability and increase the detection accuracy of the model in underwater images; Finally, an ablation experiment is conducted to compare the detection effect under different deformable convolution layers and network positions. Experimental results show that the combination of three-layer deformable convolution and standard convolution has the best performance. Compared with the YOLO series, the proposed model has an accuracy of 95.47% while the parameter amount is reduced by 10 times and the FPS is doubled. Rapid detection of dead fish is achieved in real circulating aquaculture systems.