The inter-rater reliability and individual reviewer performance of the 2012 world heart federation guidelines for the echocardiographic diagnosis of latent rheumatic heart disease

Abstract

BackgroundIn 2012, the World Heart Federation (WHF) published guidelines for the echocardiographic diagnosis of rheumatic heart disease (RHD). This study assesses individual reviewer performance and inter-rater agreement and reliability on the presence of any RHD, as well classification of RHD based on the 2012 WHF criteria. MethodsFour cardiologists individually reviewed echocardiograms in the context of a randomized clinical trial (ClinicalTrials.gov:NCT03346525) and participated in a blinded adjudication panel. Panel decision was the reference standard for diagnosis. Performance of individual reviewers to panel adjudication was compared through sensitivity and specificity analyses and inter-rater reliability was assessed between individual panelists using Fleiss free marginal multirater kappa. ResultsEchocardiograms from 784 children had two independent reports and panel adjudication. The accuracy of independent reviewers for any RHD had high sensitivity (94%, 95% CI 93–95%) and moderate specificity (62%, 95% CI 53–70%). Sensitivity and specificity for definite RHD was 61.3 (95% CI, 55.3–67.1) and 93.1 (95% CI, 91.6–94.4), with 86.8 (84.7–88.7) and 65.8 (61.0–70.4) for borderline RHD. There was moderate inter-rater agreement (κ = 0.66) on the presence of any RHD while agreement for specific 2012 WHF classification was only fair (κ = 0.51). ConclusionsThe 2012 WHF guidelines are moderately reproducible when used by expert cardiologists. More cases of RHD were diagnosed by an consensus panel than by individual reviewers. A revision to the criteria is now warranted to further increase the reliability of the WHF criteria.