Inter-rater Reliability Values Research Articles

Reliability of Safe Maximum Lifting Determinations of a Functional Capacity Evaluation

Functional capacity evaluations (FCEs) are measurement tools used in predicting readiness to return to work following injury. The interrater and test-retest reliability of determinations of maximal safe lifting during kinesiophysical FCEs were examined in a sample of people who were off work and receiving workers' compensation. Twenty-eight subjects with low back pain who had plateaued with treatment were enrolled. Five occupational therapists, trained and experienced in kinesiophysical methods, conducted testing. A repeated-measures design was used, with raters testing subjects simultaneously, yet independently. Subjects were rated on 2 occasions, separated by 2 to 4 days. Analyses included intraclass correlation coefficients (ICCs) and 95% confidence intervals. The ICC values for interrater reliability ranged from.95 to.98. Test-retest values ranged from.78 to.94. Inconsistencies in subjects' performance across sessions were the greatest source of FCE measurement variability. Overall, however, test-retest reliability was good and interrater reliability was excellent.

Reliability of isokinetic trunk muscle strength measurement.

To determine the intrarater and interrater reliability of reciprocal concentric trunk flexion and extension peak torque values at different angular velocities using the Cybex NORM isokinetic dynamometer. Trunk flexor and extensor muscles of 15 healthy subjects were assessed at 60 degrees/sec and 90 degrees/sec angular velocities. Each subject was tested by two physicians three times, with at least 48 hr between test sessions. Intraclass correlation coefficients were used to determine the intrarater and interrater reliability of the reciprocal concentric trunk flexion and extension peak torques at 60 degrees/sec and 90 degrees/sec angular velocities. For intrarater reliability intraclass correlation coefficients for trunk flexion, peak torque values ranged from 0.89 to 0.95; for the extensors, values ranged from 0.80 to 0.92. The intraclass correlation coefficient values for interrater reliability were also found to be highly reliable, and peak torque values demonstrated intraclass correlation coefficients values that ranged from 0.95 to 0.98. Reciprocal concentric trunk flexion and extension peak torque measurements at 60 degrees/sec and 90 degrees/sec angular velocities had a high intrarater and interrater reliability with the Cybex NORM isokinetic dynamometer in healthy subjects.

Assessor training strategies and their effects on accuracy, interrater reliability, and discriminant validity.

This study compares the effects of data-driven assessor training with schema-driven assessor training and control training. The sample consisted of 229 industrial and organizational psychology students and 161 managers who were randomly assigned to 1 of these training strategies. Participants observed and rated candidates in an assessment center exercise. The data-driven and schema-driven assessor training approaches outperformed the control training on all 3 dependent variables. The schema-driven assessor training resulted in the largest values of interrater reliability, dimension differentiation, and accuracy. Managers provided significantly more accurate ratings than students but distinguished less between the dimensions. Practical implications regarding the design of assessor trainings and the composition of assessor teams are proposed.

The developmental profile: preliminary results on interrater reliability and construct validity.

This study presents the preliminary results of research into the interrater reliability and construct validity of the Developmental Profile (DP). In the DP a number of developmental lines, such as Object-Relations, Self-Images, and Problem-Solving Capacities, are assessed and classified according to the level of functioning. A total of 108 profiles were assessed, drawn from three different categories of patients. The weighted kappa values for interrater reliability were sufficient. On the adaptive level, but also on the maladaptive levels Symbiosis and Resistance, significant differences were found between psychiatric patients, "normal controls" (dental patients) and somatic patients. No differences were recorded between the latter two groups. The conclusion is that the DP is a promising instrument, of which the reliability and validity has to be further investigated in order to contribute to scientific support for psychodynamic theory formation.

The development of a valid and reliable scale for rating anxiety in dementia (RAID)

A rating scale to measure anxiety in dementia sufferers was developed and evaluated in a sample of 51 inpatients and 32 day-hospital patients. Anxiety scores were not related to sex, age, accommodation or DSM-IV diagnosis of the type of dementia. However, both subjects with physical illnesses and subjects with insight into their memory problems had significantly higher anxiety scores.The kappa values for inter-rater reliability ranged from 0.51 to 1 and for test-retest reliability from 0.53 to 1, which indicates moderate to good reliability.The overall agreement on individual items ranged from 82-100% (inter-rater) and 84-100% (test-retest).The professionals working in the care of the elderly and carer groups felt that the scale was comprehensive and all the items in the scale were important, thereby confirming that it has good content validity. The scale significantly correlated with other anxiety scales and also with independent ratings both by a consultant psychiatrist and also nursing staff, indicating good concurrent validity. Anxiety scores were significantly higher in dementia patients who fulfilled modified DSM-IV criteria for anxiety and clinical diagnosis of anxiety disorder.This showed evidence of good criterion validity. Factor analysis showed five factors, including all items of the scale. Scores of 11 and above on the scale indicated significant clinical anxiety. Overall, the scale had good reliability and validity. It should be a useful clinical and research instrument for assessing anxiety in dementia sufferers.

Precision and reliability for measurement of change in MRI lesion volume in multiple sclerosis: a comparison of two computer assisted techniques

OBJECTIVEThe serial quantification of MRI lesion load in multiple sclerosis provides an effective tool for monitoring disease progression and this has led to its increasing use as an outcome measure...

The arm motor ability test: Reliability, validity, and sensitivity to change of an instrument for assessing disabilities in activities of daily living

Objective: To continue and expand determination of the reliability, validity, and sensitivity to change of the Arm Motor Ability Test (AMAT), an instrument for assessing deficits in activities of daily living (ADL). Design: The AMAT was administered twice to patients, with an interest interval of either 1 or 2 weeks, by one of two examiners assigned to patients in counterbalanced order. Patients' interest intervals and scores on the arm portion of the Motricity Index was unknown to the raters. Setting: A referral inpatient neurological rehabilitation center. Patients: Thirty-three subacute stroke inpatients with moderate to mild upper extremity motor deficit: median Motricity-index-Arm score = 89, median chronicity = 43d, median age = 66yr; 12 were women. Main Outcome Measure and Results: The AMAT was developed in 1987, and interrater reliabilities at that time were found to range from .95 to .99. The present values for interrater reliability (2 scales) from videotaped test performance were: kappas = .68 to .77, Spearman correlations = .97 to .99. For performance time, interscorer reliability from videotaped test performance was .99. Homogeneities for the three AMAT measures for the total sample (Cronbach's alpha and split-half reliability) were .93 to .99. The test-retest reliabilities for the total sample were .93 to .99. The correlations to the Motricity-lndexArm score were .45 to .61. The AMAT detected the difference in change occurring as a result of the passage of 1 versus 2 weeks in these subacute inpatients, presumably as a result of intensive therapy and/or spontaneous recovery, confirming the results of an earlier intervention study. Conclusion: The AMAT is an instrument with high interrater reliability, internal consistency, and sensitivity to change, as well as having satisfactory concurrent validity.

The Mobility Scale for Acute Stroke Patients: intra-rater and inter-rater reliability

Objective: With increasing pressure for accountability in the Australian health service, clinicians require reliable and valid measurement tools for assessing the physical status of stroke patients in the acute setting prior to entering rehabilitation, so that predictions can be made about the potential length of stay in rehabilitation and level of outcome. In response to the lack of suitable measures of physical disability for acute stroke patients, the Mobility Scale for Acute Stroke Patients was developed. The aim of this paper was to investigate the intra-rater and inter-rater reliability of the scale. Method: Five tasks, which incorporate functional movements from lying to walking, were assessed using a six-point scale based on the level of assis tance. Stroke patients were filmed while being assessed by an experienced examiner using the Mobility Scale. Twenty-one clinicians viewed 10 video- taped performances of each activity. Results: The intra-rater reliability from the viewing sessions one month apart was excellent for 92% of the judgements (weighted kappa >0.75). The weighted kappa values for inter-rater reliability were all in the excellent agreement range above 0.75. Conclusion: The results of this study showed that the Mobility Scale has a high degree of reliability using this testing procedure. The reliability of the Mobility Scale in the clinical setting and its validity in predicting length of stay and outcome are being investigated.

Interrater reliability of the Peabody Developmental Motor Scales: fine motor scale.

The interrater reliability of the Peabody Developmental Motor Scales (PDMS) fine motor scale was examined in 23 children with developmental disabilities who were between 2 years and 5 years of age. Three occupational therapists viewed videotapes of the children and scored each child's performance on the fine motor section of the test. Data were analyzed with the intraclass correlation (ICC) approach; ICC values ranged from 0.90 to 0.97 for the subskills of Grasping, Hand Use, Eye-Hand Coordination, and Manual Dexterity. The ICC interrater reliability value for the total fine motor score was 0.99. Reliability values were additionally computed with the PDMS fine motor scale age-equivalent scores and Z scores. The ICC values for these methods of scoring the PDMS were 0.99 and 1.00, respectively. The PDMS fine motor scale can be used consistently to evaluate fine motor delays in this population of young children.

Interrater Reliability of the Structural Interview

This study investigated the structural interview for its reliability. The interview was developed by Kernberg et al. and is intended to make a differential diagnosis between a neurotic, borderline, and psychotic personality organization. We studied two categories of psychiatric patients (outpatients and inpatients) by recruiting 37 subjects: 20 men and 17 women. Two assessment methods were used: observation and videotape evaluation. The three authors alternated their roles between interviewer, observer, and videotape evaluation. The results showed that it was possible to classify the patients into the categories neurosis, borderline, and psychosis. The ? values for interrater reliability ranged from moderate to substantial. Further study is indicated, and well-planned training in the structural interview would make the technique a useful tool within the psychodiagnostic process.

Reliability of Interpretation of The Paris Classification of Normal End Feel for Elbow Flexion and Extension

Identification of the anatomic and pathoanatomic structure which limits the range of motion of a joint help to determine the need for and type of treatment approach. The purpose of this study was to determine the interrater and intrarater reliability of two experienced physical therapists in the identification of the type of end feel for elbow flexion and extension when using the Paris classification system of normal end feel. Four trials of flexion end feel and four trials of extension end feel were conducted for each of the twenty subjects by each blindfolded examiner. A total of 160 movements were performed by each examiner. The intertester Kappa value for interrater reliability for flexion was .40 and for extension was .73 with a significance of p < .0001 for both flexion and extension. Intrarater agreement was measured by percent comparison. Examiner A demonstrated 80% agreement for flexion and 79% for extension. Examiner B showed 75% agreement for flexion and 78% for extension. This study demonstrated a fair to good level of reliability of interpretation of elbow end feel assessment in a non clinical setting.

Visual Assessment of Foot Type and Relationship of Foot Type to Lower Extremity Injury

The purpose of this study was 1) to establish the interrater reliability of classifying foot type by visual appraisal and 2) to determine any relationship between foot type and subsequent knee pain or ankle sprains. Seventy-seven athletes were evaluated by three trained physical therapists to determine interrater reliability of a visual appraisal to identify foot type. Feet were classified according to operational definitions, and specific criteria had to be met for the foot to be classified as supinated, pronated, or neutral. Questionnaires concerning knee pain were completed at the beginning of the season, and incidence of ankle sprain was followed throughout the football and cross country seasons for 55 athletes. The Kappa value for interrater reliability for visually assessing foot type was .72. There was a significant relationship between foot type and knee pain (X2 = 4.45, N = 55, df = 2, p < .05). There was no relationship between foot type and incidence of ankle sprain. These results indicate that 1) physical therapists trained in the procedure can reliably use visual appraisal to classify foot type, and 2) athletes with excessively pronated or supinated foot types may be more susceptible to knee pain than athletes with neutral foot types. J Orthop Sports Phys Ther 1991;14(2):70-74.

Abnormalities of Adult Personality and Behaviour (Section F 6) - Results of the ICD-10 Field Trial

Personality disorders, the most important category of section F6, were studied in the ICD-10 field trial. The overall Kappa values for interrater reliability of section F6 is 0.64, for personality disorders 0.61 and for dyssocial personality 0.73. These results are very close to those of the DSM-III field trials. Subjective assessments of feasibility, suitability and goodness of fit demonstrate good acceptance of the diagnostic guidelines by the clinicians. A comparison of ICD-9, DSM-III and ICD-10 diagnoses yielded broad agreement for diagnoses of personality disorders with overlapping Kappa values of 0.91 and 0.94. The acceptance and reliability of the new classification could be increased by a number of improvements and the planned introduction of a multiaxial system.