Evaluation of inter-rater agreement of the clinical signs used to diagnose bovine respiratory disease in individually housed veal calves

Abstract

In dairy calves raised for veal, typical clinical signs of bovine respiratory disease (BRD) are ocular discharge, nasal discharge, ear droop or head tilt, abnormal respiration, cough, and increased rectal temperature. Despite the existence of several clinical scoring systems, there are few studies on the variability of human recognition of individual BRD clinical signs. The objective of this study was therefore to assess the inter-rater agreement of BRD clinical signs in veal calves. We hypothesized that BRD clinical signs were not detected equally between veterinarians, technicians, and producers of the veal industry and that some clinical signs have higher inter-rater agreement than others. During 2017-2018, we prospectively recorded 524 videos of physical examinations of random veal calves from 48 different batches in Québec, Canada. A researcher, not involved in the inter-rater assessment, classified each video as presence/absence of each BRD clinical sign except rectal temperature. For each of the 5 clinical signs, 15 videos with and 15 videos without the clinical signs were randomly selected to avoid kappa paradoxes. Those 30 videos were then presented in a random order to experienced raters of BRD in veal calves: 6 veterinarians, 6 technicians, and 6 producers. The raters assessed the clinical signs using scores based on the Wisconsin and California scoring system with modifications (0 = absent, 1 = mild, 2 = moderate, 3 = severe for nasal discharge, ocular discharge, and ear droop or head tilt; and 0 = absent, 1 = moderate, 2 = severe for abnormal respiration and induced cough). We used median percentage agreement (Pa), median Cohen's kappa (κ), and Gwet's agreement coefficient 1 (AC1) to assess inter-rater agreement. The effect of scale combination was also tested to determine the optimal combination (4-scale 0/1/2/3 vs. 3-scale 0/1/2 vs. 2-scale 0/1,2,3; 0,1/2,3; or 0/1,2). The differences of inter-rater agreement between veterinarians, technicians, and producers were estimated by a Wilcoxon rank-sum test. The 2-scale combination (0,1/2,3 or 0/1,2) had the highest inter-rater agreement for all clinical signs. With this combination, induced cough was the clinical sign with the highest inter-rater agreement (Pa = 0.93; κ = 0.79; AC1 = 0.87) and abnormal respiration was the sign with the lowest inter-rater agreement (Pa = 0.77; κ = 0.20; AC1 = 0.74). According to Pa and AC1 values, the 2-scale inter-rater agreement of the 5 clinical signs was good (value > 0.6). According to κ, only ear droop or head tilt and induced cough had a substantial 2-scale inter-rater agreement (κ > 0.6). In general, the 2-scale inter-rater agreement was better among veterinarians than among technicians and producers, except for the ear droop/head tilt, where agreement was better among producers. We concluded that with severity scores assessed on a scale of 2 (0,1/2,3 or 0/1,2), the inter-rater agreement of BRD clinical signs was variable according to the sign in veal calves. BRD clinical signs were not detected equally between veterinarians, technicians, and producers of the veal industry. Future research could determine if this discrepancy could be improved by standardization training.