Reliability Kappa Coefficient Research Articles

The use of multiple informants to assess children's maltreatment experiences

Most researchers rely exclusively on the reports of protective service workers to determine children's abuse history. In this report, information about children's maltreatment experiences is obtained from protective service workers and three supplementary sources of data: parents, medical records, and clinical observations. Fifty-six children from 34 families receiving protective services for verified reports of physical abuse, neglect, sexual abuse, and/or emotional maltreatment participated in the study, with most children known to have experienced more than one type of abuse. The supplementary data provided important information about the range and severity of children's maltreatment experiences. Review of the parent and medical record data led to identification of 28 children who had additional types of maltreatment experiences that were not reported by their protective services workers—nine cases of physical abuse, two cases of neglect, five cases of sexual abuse, and 12 cases of emotional maltreatment. Supplementary data also revealed information about incidents of specific types of abuse that were more severe than those reported by the children's protective service workers in an additional 24 cases. A method was devised to synthesize information provided from the different sources of data examined in this report to derive 0–4 point ratings of four categories of maltreatment experiences: (1) physical abuse, (3) neglect, (3) sexual abuse, and (4) emotional maltreatment. The Kappa reliability coefficients for each of these scales were .88, .73, .83, and .90, respectively. Data demonstrating the relationship between these maltreatment ratings and various indices of the children's socioemotional and cognitive functioning were also presented to provide preliminary support for the validity of these scales. The findings from this study suggest that multiple sources of data should be examined in order to obtain accurate assessments of children's maltreatment experiences, and that independent raters can synthesize discrepant data to obtain reliable and valid estimates of children's abuse history. Clinical and methodological issues relevant to the improved assessment of children's maltreatment experiences are also discussed.

Kappa and Attenuation of the Odds Ratio

This paper explores the relation between reliability and validity for binary variables. An equation is first derived for expressing the kappa coefficient of reliability as a function of true prevalence, sensitivity, specificity, and the degree of correlation of errors within true positives and within true negatives. It is then shown that under certain circumstances kappa is closely related to an index of the validity of the odds ratio as a means for quantifying an association between an imperfectly classified binary factor and membership in two subpopulations. This index is defined as the ratio of the observed odds ratio minus one to the true odds ratio minus one. Numerical results indicate that, when error rates are high, kappa can be of smaller magnitude than this index of validity. When the value of kappa reflects substantial correlation of errors, however, the value of kappa typically exceeds that of the index of validity. Correlated errors among the true negatives are found to affect the correspondence between kappa and the index of validity rather differently than do correlated errors among the true positives. Although kappa often exceeds the index of validity, use of kappa to correct the odds ratio for attenuation owing to nondifferential misclassification generally leads to undercorrection rather than overcorrection. Consequently, the relation between kappa and the index of the validity of the odds ratio may have practical value for reducing bias in the estimation of exposure-disease associations in epidemiologic research.

Schizophrenia, Schizotypal and Delusional Disorders (Section F2): Results of the ICD-10 Field Trial

To determine whether the use of ICD-10 provides greater diagnostic reliability than ICD-9, a field trial of the ICD-10 draft (WHO, 1987) was carried out. A total of 1,778 diagnoses made on the basis of ICD-10, ICD-9 and, in part, DSM-III were established by 134 clinicians in 10 centres using case reports and interviews. The corrected reliability coefficients of schizophrenic disorders obtained with ICD-10, with kappa-coefficients of reliability of 0.69 for the 2-character category and 0.67 for the 3-character category are higher than those reported for the diagnosis of schizophrenia made without specified criteria, but lower than those reported for DSM-III. Despite the operationally defined diagnostic guidelines, diagnostic discrepancies still persist when organic disturbances, substance abuse or psychogenic features are additionally present. The diagnoses of each rater remain stable on change over from ICD-9 to ICD-10. Our data suggest that ICD-10 is suitable for worldwide use in diagnosing schizophrenia.