Inter-rater Reliability For Individual Items Research Articles

Intrarater and interrater reliability of the dynamic gait index in post stroke patients with eye movement disorders

BackgroundThe Dynamic Gait Index (DGI) is a useful tool that has been evaluated for its reliability in patients with vestibular disorders, elderly people and, in chronic stroke population. Present study was aimed to evaluate the intrarater and interrater reliability of the DGI to measure dynamic balance and gait performance in stroke patients with eye movement disorders. MethodsA sample of 30 stroke patients suffering from eye movement disorders were recruited. Two Physical therapists assessed the intrarater and interrater reliability of the DGI in two testing sessions three days apart. In the later session, two raters assessed the patients’ performance simultaneously on the DGI. The reliability was calculated using the intra-class correlation coefficient (ICC2, 1). Standard error of measurement (SEM) and minimal detectable change (MDC95) at 95% confidence interval were also calculated. A significance level was set at p-value <0.05. ResultsThe (ICC2, 1) for intrarater and interrater reliability of total DGI scores was 0.86 and 0.91 respectively. While (ICC2, 1) for intrarater and interrater reliability of individual items ranged from 0.73 to 0.91 to 0.73–0.93, respectively. The (SEM) and (MDC95) for intrarater reliability of total DGI scores were 0.76 and 2.10, respectively. Corresponding values for interrater reliability were 0.62 and 1.71, respectively. ConclusionsThe DGI is a reliable tool for evaluating the dynamic balance and gait performance in stroke patients with eye movement disorders. This tool showed good to excellent intrarater and interrater reliability of total DGI scores and moderate to good intrarater and interrater reliability of individual items of the DGI.

Translation, cultural adaptation, and validation of the Integrated Palliative Outcome Scale-renal (IPOS-r) to Czech

BackgroundPatients with advanced kidney disease suffer from burdensome symptoms, which should be assessed by valid and reliable patient-reported outcome measures.This study aimed to provide a translation, cultural adaptation, and validation of the Czech version of the IPOS-r.MethodsThe IPOS-r was translated to Czech and culturally adapted using cognitive interviews. During the validation phase, patients and staff in dialysis centres and outpatient renal clinics completed the IPOS-r. Internal consistency was tested with Cronbach’s alpha, its reliability via intraclass correlation coefficient for total IPOS-r score, and weighted Kappa (for test-retest and interrater reliability of individual items). Convergent validity was tested with Spearman correlation to Kidney Disease Quality of Life Survey-Short Form 1.2 (KDQOL-SF 1.2). We assessed sensitivity to change using a distribution-based approach.ResultsTwo sets of translators independently performed forward and backward translations of the IPOS-r. Ten patients and ten health care professionals participated in cognitive pre-testing. The sample size for validation included 88 patients (mean age 66 ± SD13.8; 58% men) who were treated with haemodialysis (70.5%), home haemodialysis (5.5%), peritoneal dialysis (3%), and conservative management (21%). Cronbach’s alpha was 0.72, and the intraclass correlation was 0.84 for test-retest reliability and 0.73 for interrater reliability. The IPOS-r correlated with KDQOL-SF 1.2 had a rho between 0.4–0.8 for most of the IPOS-r items, showing good convergent validity. The IPOS-r measure is feasible and takes 9 minutes to complete. Patients who reported a change in health status after 1 month demonstrated a total IPOS-r score change of eight points in both positive and negative directions.ConclusionsThe process of translation and cultural adaptation of the IPOS-r was successful, and the Czech IPOS-r measure is a valid and reliable tool. The Czech IPOS-r can be used to assess symptoms in patients with advanced chronic kidney disease.Trial registrationGAUK [82121].

Development and Testing of a Sedation Scale for Use in Rabbits (Oryctolagus cuniculus).

In biomedical research, rabbits are commonly sedated to facilitate a variety of procedures. Developing a sedation assessment scale enables standardization of levels of sedation and comparisons of sedation protocols, and may help in predicting sedation level requirements for different procedures. The goal of this study was to develop a rabbit sedation assessment scale using a psychometric approach. We hypothesized that the sedation scale would have construct validity, good internal consistency, and reliability. In a prospective, randomized, blinded study design, 15 (8 females, 7 males) healthy 1-y-old New Zealand white rabbits received 3 intramuscular treatments: midazolam (0.5 mg/kg; n = 6); midazolam (1.5 mg/kg)-ketamine (5 mg/kg; n = 7); and alfaxalone (4 mg/kg)-dexmedetomidine (0.1 mg/kg)-midazolam (0.2 mg/kg; n = 3). One rabbit received 2 treatments. A sedation scale was developed by using psychometric methods, with assessment performed by 6 independent raters who were blind to treatment. Final sedation scale items included posture, palpebral reflex, orbital tightening, lateral recumbency, loss of righting reflex, supraglottic airway device placement, toe pinch, and general appearance. The scale showed construct validity, good to very good interrater reliability for individual items (6 raters; intraclass correlation coefficient, 0.671 to 0.940), very good intrarater reliability (5 raters; intraclass correlation coefficient, 0.951 to 0.987), and excellent internal consistency (Cronbach α, 0.947). The sedation scale performed well under the conditions tested, suggesting that it can be applied in a wider range of settings (different populations, raters, sedation protocols).

Evaluation of the McMahon Competence Assessment Instrument for Use with Midwifery Students During a Simulated Shoulder Dystocia.

Intrapartum emergencies occur infrequently but require a prompt and competent response from the midwife to prevent morbidity and mortality of the woman, fetus, and newborn. Simulation provides the opportunity for student midwives to develop competence in a safe environment. The purpose of this study was to determine the inter-rater reliability of the McMahon Competence Assessment Instrument (MCAI) for use with student midwives during a simulated shoulder dystocia scenario. A pilot study using a nonprobability convenience sample was used to evaluate the MCAI. Content validity indices were calculated for the individual items and the overall instrument using data from a panel of expert reviewers. Fourteen student midwives consented to be video recorded while participating in a simulated shoulder dystocia scenario. Three faculty raters used the MCAI to evaluate the student performance. These quantitative data were used to determine the inter-rater reliability of the MCAI. The intraclass correlation coefficient (ICC) was used to assess the inter-rater reliability of MCAI scores between 2 or more raters. The ICC was 0.86 (95% confidence interval, 0.60-0.96). Fleiss's kappa was calculated to determine the inter-rater reliability for individual items. Twenty-three of the 42 items corresponded to excellent strength of agreement. This study demonstrates a method to determine the inter-rater reliability of a competence assessment instrument to be used with student midwives. Data produced by this study were used to revise and improve the instrument. Additional research will further document the inter-rater reliability and can be used to determine changes in student competence. Valid and reliable methods of assessment will encourage the use of simulation to efficiently develop the competence of student midwives.

Comparing the ability of OPTION12 and OPTION5 to assess shared decision-making in genetic counselling.

ObjectivesOPTION12 is the most widely used tool to measure shared decision-making (SDM) in health care. A newer scale, OPTION5, has been proposed as a more parsimonious measure that better addresses core concepts of SDM. This study compares OPTION5 to OPTION12 in prenatal genetic counselling. MethodsTwo raters independently used OPTION12 and OPTION5 to score 27 clinical encounters between genetic counsellors (GC) and women with pregnancies at increased risk for genetic conditions. Global and item scores on the two instruments were compared to test concurrent validity and to identify usability in this context. Inter-rater reliability was also assessed for both instruments. ResultsMean scores for OPTION12 were 43.8 (SD=9.7), and for OPTION5 were=60.6 (SD=12.5). The correlation between OPTION12 and OPTION5 scores was r=0.70. Inter-rater reliability was 0.70 and 0.85 for OPTION12 and OPTION5 respectively, however mean inter-rater reliability for individual items was 0.31 and 0.63 for OPTION12 and OPTION5 respectively. ConclusionsGCs exhibit SDM as measured by both OPTION instruments. OPTION5 exhibits improved psychometric performance relative to OPTION12, and more specifically targets the core constructs of SDM. However, refinement of OPTION instruments or manuals is needed to improve reliability and validity in GC assessment.

A Visual Rating Scale for the Hummingbird Sign with Adjustable Diagnostic Validity.

Although the hummingbird sign (HBS) is a distinctive feature of progressive supranuclear palsy (PSP) VS. idiopathic Parkinson's disease (IPD) and other parkinsonian disorders, there are no consensus criteria for its identification. To develop consensus criteria for the HBS to enable accurate communication between researchers and physicians, we developed a new radiologic rating scale for the HBS (the HBS-RS), and the inter-rater reliability (IRR) and diagnostic validity of the HBS-RS were evaluated. Two raters blinded to the clinical diagnoses reviewed T1 midsagittal magnetic resonance images of 133 patients with IPD (n = 93) or PSP (n = 40). The existence of the HBS was assessed in two steps that were separated by two weeks; the first was based on their own experience and the second was according to the HBS-RS. The HBS-RS comprises 4 items (contour of the third ventricle floor, shape of the beak, shape of the hummingbird head, and midbrain atrophy), with weighted scores from 0 to 2. The IRR of individual items in the HBS-RS and of the composite scores showed moderate-to-good agreement (Cohen's kappa [κ], 0.479-0.766) and were observed to be highest for the contour of the third ventricle floor. The sensitivities and specificities varied depending on the cut-off for each item or for the composite scores. The sensitivities for each item were high (85.0-92.5) at a low cut-off (0 VS. 1 or 2). The specificities reached more than 80% when the composite scores of the HBS-RS were used. Receiver operating characteristic curves for the total HBS-RS scores showed fair diagnostic accuracy for PSP (AUC, 0.76 and 0.73). The HBS-RS is a simple and measurable visual assessment tool to identify the HBS, with adjustable diagnostic validity for PSP. The results suggest that the HBS-RS may be used for objective measurements of the HBS in research and in the clinic.

Psychometric Evaluation of the Ostomy Complication Severity Index

The purpose of this study was to evaluate the psychometric properties of a new instrument to measure incidence and severity of ostomy complications early in the postoperative period. 71 participants were enrolled, most were men (52%), white (96%), and married or partnered (55%). The mean age of participants was 57 ± 15.09 years (mean ± SD). Fifty-two participants (84%) experienced at least 1 ostomy complication in the 60-day postoperative period. The research setting was 3 acute care settings within a large healthcare system in the Midwestern United States. We developed an evidence-based conceptual model to guide development and evaluation of a new instrument, the Pittman Ostomy Complication Severity Index (OCSI). The OCSI format includes Likert-like scale with 9 individual items scored 0 to 3 and a total score computed by summing the individual items. Higher scores indicate more severe ostomy complications. This study consisted of 2 phases: (1) an expert review, conducted to establish content validity; and (2) a prospective, longitudinal study design, to examine psychometric properties of the instrument. A convenience sample of 71 adult patients who underwent surgery to create a new fecal ostomy was recruited from 3 hospitals. Descriptive analyses, content validity indices, interrater reliability testing, and construct validity testing were employed. Common complications included leakage (60%), peristomal moisture-associated dermatitis (50%), stomal pain (42%), retraction (39%), and bleeding (32%). The OCSI demonstrated acceptable evidence of content validity index (CVI = 0.9) and interrater reliability for individual items (k = 0.71-1.0), as well as almost perfect agreement for total scores among raters (ICC = 0.991, P ≤ .001). Construct validity of the OCSI was supported by significant correlations among variables in the conceptual model (complications, risk factors, stoma care self-efficacy, and ostomy adjustment). OCSI demonstrated acceptable validity and reliability and can be used to assess incidence and severity of ostomy complications in the early postoperative period. We found the OCSI to be brief, easy-to-use, and clinically practical. It can be used to (a) identify priority areas for nursing intervention related to the ostomy, (b) determine appropriate interventions to prevent or treat complications, and (c) evaluate the effects of nursing interventions designed to improve outcomes for patients with ostomies.

Standardizing Assessment of Elderly People in Acute Care: The interRAI Acute Care Instrument

To examine the frequency distributions and interrater reliability of individual items of the interRAI Acute Care instrument. Observational study of a representative sample of older inpatients; duplicate assessments conducted on a subsample by independent assessors to examine interrater reliability. Acute medical, acute geriatric and orthopedic units in 13 hospitals in nine countries. Five hundred thirty-three patients aged 70 and older (mean age 82.4, range 70-102) with an anticipated stay of 48 hours or longer of whom 161 received duplicate assessments. Sixty-two clinical items across 11 domains. Premorbid (3-day observation period before onset of the acute illness) and admission (the first 24 hours of hospital stay) assessments were conducted. The frequency of deficits exceeded 30% for most items, ranging from 1% for physically abusive behavior to 86% for the need for support in activities of daily living after discharge. Common deficits were in cognitive skills for daily decision-making (38% premorbid, 54% at admission), personal hygiene (37%, 65%), and walking (39%, 71%). Interrater reliability was substantial in the premorbid period (average kappa=0.61) and admission period (average kappa=0.66). Of the 69 items tested, less than moderate agreement (kappa<0.4) was recorded for six (9%), moderate agreement (kappa=0.41-0.6) for 14 (20%), substantial agreement (kappa=0.61-0.8) for 40 (58%), and almost perfect agreement (kappa>0.8) for nine (13%). Initial assessment of the psychometric properties of the interRAI Acute Care instrument provided evidence that item selection and interrater reliability are appropriate for clinical application. Further studies are required to examine the validity of embedded scales, diagnostic algorithms, and clinical protocols.

Case management for the mentally ill: a comparative evaluation of client satisfaction

This paper reports on client satisfaction with community mental health services. A Client Satisfaction Interview was used as one outcome measure in a randomized controlled study of clinical case management vs. standard community care. The Interview had high internal consistency reliability (Cronbach's alpha = 0.89) with very high inter-rater reliability for overall scores (r= 0.99) and good inter-rater reliability for individual items (kappa over 0.9 for 18 of 22 items). Case management clients had significantly higher total satisfaction scores than control group clients who received standard community care at 9 months (t= 2.02 P= 0.05) but not at baseline (t= 0.59, P= 0.55) or 18 months (t= 1.76, P= 0.09). In addition to a quantitative measure of satisfaction, clients were asked to provide more qualitative accounts of their experience with services. The findings are discussed in relation to the team's model of case management and the differences between case management and control services. The limitations of this study are considered in the context of methodological issues surrounding measurement of client satisfaction and the study's own methodological shortcomings.

Behavioral assessment of social performance: a rating system for social phobia

The Social Performance Rating Scale (SPRS) is a modification of the rating system for behavioral assessment of social skills, originally developed by Trower et al. (Trower, P., Bryant, B., & Argyle, M. (1978). Social skills and mental health. Pittsburgh: University of Pittsburgh Press) and subsequently adapted by Turner and colleagues (e.g., Turner, S. M., Beidel, D. C., Dancu, C. V., & Keys, D. J. (1986). Psychopathology of social phobia and comparison to avoidant personality disorder. Journal of Abnormal Psychology, 95, 389–394). Designed to yield ratings of social performance appropriate for use in a socially phobic population and based on videotaped role plays, the five SPRS ratings are gaze, vocal quality, speech length, discomfort, and conversation flow. The sum of these ratings provides an internally consistent total score. In an initial study of the psychometric properties of the SPRS, three groups were assessed: individuals with social phobia, another anxiety disorder, or no psychological disorder. Inter-rater reliability for individual items and the total score proved excellent, and positive evidence for convergent, divergent, and criterion-related validity was obtained.

Inter-rater reliability of two depression rating scales, MADRS and DRRS, based on videotape records of structured interviews

The inter-rater reliability of the French versions of the MADRS and the DRRS was studied on the basis of 58 videotape records of structured standardised interviews of depressed inpatients under antidepressant treatment. Each patient was assessed by two trained raters, from the same videotape recording. The inter-rater reliability of total scores was high with both scales (intra-class correlation coefficients: 0.86 for MADRS and 0.77 for DRRS). However, the inter-rater reliability for individual items was higher and more homogeneous for the MADRS than for the DRRS. Finally, the structured interview in French appears to be relevant for the MADRS, but it should be improved for the DRRS.

Instruments for Assessing the Quality of Drug Studies Published in the Medical Literature

To develop valid and reliable instruments to assess the methodologic quality and clinical relevance of drug studies. We developed an instrument to assess the methodologic quality of articles reporting clinical research and an instrument to measure nonmethodologic measures of quality, such as clinical relevance, generalizability, and adherence to ethical standards. Each instrument was pretested by seven independent, masked reviewers and modified based on interrater agreement and content validity of individual items. We determined correlational validity of the final methodologic quality instrument by comparing quality scores assigned to 10 articles by means of our instrument and a previously published one. Clinical drug studies published in symposium proceedings and peer reviewed biomedical literature. Interrater reliability of overall quality scores, measured by intraclass correlation (r) and Kendall's coefficient of concordance (W), and interrater reliability of individual items, by percentage agreement. The interrater reliability of the pretest methodologic quality instrument was high (r = .89 [95% confidence interval, .73 to .96]; W = 0.64). Correlational validity of the final instrument was suggested by the high degree of concordance with another previously published one (W = 0.74). The interrater reliability of the pretest clinical relevance instrument was moderate (r = .41 [95% confidence interval, .18 to .64]; W = 0.47). Reviewers confirmed the content validity of both instruments. The two instruments we developed, one measuring methodologic quality and one measuring clinical relevance of articles reporting clinical research, are reliable, valid, and applicable to a variety of research designs.