Bovine respiratory disease (BRD) is the leading cause of death in calves. Abnormal respiratory behavior (ARB) is one of the prevalent clinical signs of BRD in calves, manifesting as tachypnea and visible chest and abdominal respiration. This paper proposed a calf ARB detection method based on frame difference (FD) and improved YOLOv5 to increase detection efficiency and achieve automatic detection. A total of 1400 video segments, each lasting about 30 s, were collected from 10 calves with ARB and 40 with normal respiratory behavior (NRB). First, a squeeze-and-excitation (SE) model was added to the YOLOv5 network to realize key feature extraction of the calf head and trunk. Next, the model was trained and tested using 5520 images obtained after You Only Cut Once (YOCO) image enhancement. Next, FD was used to obtain the shaking pixels and determine the proportion of trunk shake (PTS) for each frame of the calf image. Then, 1250 video segments were divided into 3 s video units. Finally, 2500 ARB and 2500 NRB 3 s video units were manually selected for the calf ARB detection set. The maximum, minimum, mean, variance and standard deviation of PTS within each 3 s video unit were extracted as feature values and input to support vector machine (SVM). The results showed that the improved YOLOv5 network achieved high mAP50 of 98.2% and mAP50:95of 92.6%, which was 0.9% and 2.3% higher than the baseline YOLOv5. Meanwhile, ARB could be detected with an accuracy of 96.80%, a sensitivity of 95.20%, a specificity of 98.40%, and a precision of 98.35% after 5-fold cross-validation. The results indicated that the proposed method could effectively detect calf trunk and head and be used for calf ARB detection, providing a technical reference for calf ARB detection.