Abstract
This study aims at introducing a method for individual agreement evaluation to identify the discordant raters from the experts' group. We exclude those experts and decide the best experts selection method, so as to improve the reliability of the constructed tongue image database based on experts' opinions. Fifty experienced experts from the TCM diagnostic field all over China were invited to give ratings for 300 randomly selected tongue images. Gwet's AC1 (first-order agreement coefficient) was used to calculate the interrater and intrarater agreement. The optimization of the interrater agreement and the disagreement score were put forward to evaluate the external consistency for individual expert. The proposed method could successfully optimize the interrater agreement. By comparing three experts' selection methods, the interrater agreement was, respectively, increased from 0.53 [0.32-0.75] for original one to 0.64 [0.39-0.80] using method A (inclusion of experts whose intrarater agreement>0.6), 0.69 [0.63-0.81] using method B (inclusion of experts whose disagreement score=“0”), and 0.76 [0.67-0.83] using method C (inclusion of experts whose intrarater agreement>0.6& disagreement score=“0”). In this study, we provide an estimate of external consistency for individual expert, and the comprehensive consideration of both the internal consistency and the external consistency for each expert would be superior to either one in the tongue image construction based on expert opinions.
Highlights
Traditional Chinese medicine (TCM), as a kind of complementary and alternative medicine with a history of five thousand years, has been gradually accepted and embraced by the western medicine system, while, in traditional Chinese medicine, tongue diagnosis plays an important role in the clinical syndrome differentiation and therapeutic evaluation
Through the large-scale agreement study of expert opinions, our study focuses on identifying the discordant raters from the experts’ group and if there exist related method to assess the external consistency of individual expert and, in addition, what the best method for experts selection would be in order to construct a more reliable database
In the 50 experts’ rating results from the 230 nontesting tongue images, the discordant experts for all the 25 tongue features were recognized, and the interrater agreements before and after the removal of those discordant experts are manifested in Table 2, from which we can see that the interrater agreement for all the 25 tongue features were increased after the first exclusion of those identified experts
Summary
Traditional Chinese medicine (TCM), as a kind of complementary and alternative medicine with a history of five thousand years, has been gradually accepted and embraced by the western medicine system, while, in traditional Chinese medicine, tongue diagnosis plays an important role in the clinical syndrome differentiation and therapeutic evaluation. Evidence-Based Complementary and Alternative Medicine from experts, a reliability and agreement evaluation of the obtained data is essential. Kappa statistics has been widely used in the clinical agreement and reliability studies for nominal and ordinal measurement, such as in the neurology [1], pathology [2], epidemiology [3], clinical diagnostics especially for medical images [4, 5], and clinical therapeutic evaluation [6]. The benchmarks for the range of varied agreement coefficients’ values are provided by Landis and Koch with 0–0.20 as poor, 0.21–0.40 as fair, 0.41–0.60 as moderate, 0.61–0.80 as substantial, and 0.81–1.0 as almost perfect [8]
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Free) 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
