Inter-rater Reliability For Total Score Research Articles

Validity, inter-rater reliability, and feasibility of the Chelsea Physical Assessment Tool for assessing physical function in post-acute COVID-19 patients: A cross-sectional study.

Various tools have been created to measure physical function during intensive care unit (ICU) stay and after ICU discharge, but those have not been validated in coronavirus 2019 (COVID-19) patients. There is a need for a reliable, valid and feasible tool to define the rehabilitation needs of post-ICU COVID-19 patients entering the acute wards and then rehabilitation clinics. This study aims to investigate the validity, inter-rater reliability and feasibility of Chelsea Physical Assessment Tool (CPAx) in assessing the functional status of COVID-19 patients after discharge from the ICU. Demographic and clinical characteristics of the patients were recorded. Patients were evaluated using the modified Medical Research Council (MRC) dyspnea scale, Functional Oral Intake Scale, Glasgow Coma Scale, CPAx, Barthel Index, Katz Index and MRC sum score, measurements of grip strength obtained by dynamometer, the 5 time sit-to-stand test and 30 seconds and sit-to-stand test. CPAx and the other functional assessment tools were administered to 16 patients within 48hours following ICU discharge. For inter-rater reliability, another physiatrist independently re-assessed the patients. MRC sum score, Barthel and Katz indexes were used to assess construct validity of CPAx. The discriminative validity of CPAx was determined by its ability to differentiate between patients with and without ICU acquired muscle weakness based on MRC sum score. The intra-class correlation coefficients (ICC) were calculated to determine inter-rater reliability for total scores of the functional assessment tools. Cohen's Kappa (κ) coefficient and weighed Kappa (κw) were calculated to determine inter-rater reliability of individual CPAx items. Ceiling and flooring effects were calculated by percentage frequency of lowest or highest possible score achieved. The number and percentages of the patients who were able to complete each tool were calculated to assess feasibility. The CPAx score was strongly correlated with MRC sum score (rho: 0.83), Barthel Index (rho: 0.87) and Katz Index (rho: 0.89) (p< 0.001) showing construct validity. Area under the ROC curve demonstrated that cut off score for CPAx was ⩽ 12 to discriminate patients with MRC sum score < 48, with a sensitivity and a specificity of 100% and 63%, respectively (AUC = 0.859, p< 0.001). ICC was high for CPAx, MRC sum score, Barthel and Katz indexes, Glasgow Coma Scale, and hand grip strength measurement, with the highest value observed for CPAx (ICC, 0.96; 95% confidence interval (CI), 0.71-0.98). κ and κw analysis showed good to excellent inter-rater reliability for individual CPAx items. No floor or ceiling effect was observed at CPAx while floor effect was observed at Barthel Index scores (25%) and Katz Index scores (37.5%). All patients could be evaluated using CPAx while less were physically able to complete the 5 time sit-to-stand, 30 seconds sit-to-stand tests (n= 4) and MRC sum score (n= 14). CPAx is a valid, reliable, and feasible tool to assess the physical functional state in COVID-19 patients following discharge from the ICU.

95 Rating endometriosis apps using the applications scoring system

To identify the top-rated endometriosis applications (apps) using the previously validated APPLICATIONS scoring system. A list of endometriosis apps were identified from the apple app store, google play, and an internet search. Apps that were deemed inaccurate, non-English language, and duplicate were excluded. The APPLICATIONS scoring system, a standardized, validated tool, was used to assess the remaining apps for comprehensive medical information, accessibility, and ease of access. The apps were scored by three evaluators. Through the search conducted, 19 applications were identified. Cross-validation of these applications was performed using the Google search engine. Seven apps were excluded (36.8%). The remaining 12 (63%) applications were scored by three reviewers. Intra-class correlation coefficient (ICC) for assessing inter-rater reliability of total scores was 0.94 (95% CI = 0.85, 0.98 indicating good reliability). Two of the apps scored above 10 using the APPLICATIONS scoring system, Flutter, an iOS-only patient-directed app by Flutter Health Inc, scored 14.67 points and Endo App, an android only physician-directed app by ESHRE scored 13 points. Of the limited amount of endometriosis-related applications identified only 12 were found to be relevant. This highlights the lack of useful patient and physician-directed apps for the management and treatment of endometriosis. There is abundant potential for the development of mobile resources to aid in the diagnosis and treatment of endometriosis.

Conversations between Latina mothers and their child's mental health provider: An observational study of shared decision-making regarding pediatric patient mental health needs

ObjectiveTo evaluate shared decision-making (SDM) and delineate SDM processes in audio-recorded conversations between language congruent Spanish-/English-speaking clinicians and parents of pediatric mental health patients. MethodsTranscripts from audio-recorded consultations were rated using the 5-Item Observing Patient Involvement in Decision Making (Observer OPTION5) instrument. One hundred encounters between seventeen clinicians and 100 parents were rated. Interrater reliability for total score was 0.98 between two trained coders (ICC range: 0.799-0.879). ResultsScores ranged between 0 and 70 on a 100-point scale, with an average total Observer OPTION5 score of 33.2 (SD = 17.36). This corresponded to modest success at mutual shared decision-making. Clinicians and parents both showed effort at identifying a problem with treatment options and engaging in team talk. However, preference elicitation and integration were largely lacking. ConclusionThe present sample performed on par with other populations studied to date. It expands the evaluation of observed SDM to include Latino patients and new clinician populations. Practice implicationsUse of the Observer OPTION5 Item instrument highlights that eliciting and integrating parent/patient preferences is a skill that requires attention when delivering culturally competent interventions.

Self-assessment of team performance using T-NOTECHS in simulated pediatric trauma resuscitation is not consistent with expert assessment

BackgroundThe Trauma NOn-TECHnical Skills (T-NOTECHS) tool has been used to assess teamwork in trauma resuscitation, but its reliability and validity for self-assessment is unknown. Our purpose was to determine the reliability and validity of self-administered T-NOTECHS in pediatric trauma resuscitation. MethodsSimulated in situ resuscitations were evaluated using T-NOTECHS in real time by experts and immediately afterwards by team members. Reliability was analyzed with linear-weighted kappa and intra-class correlation. T-NOTECHS scores were compared between expert (gold-standard) and self-assessment. ResultsFifteen simulations were examined. T-NOTECHS scores were similar between self- and expert assessment for leadership. Self-assessment scores were higher than expert for the other domains and total composite score. Inter-rater reliability for total score was similar between the two groups, but differences were observed in the domains. ConclusionsSelf-assessment is not interchangeable with expert rating when using T-NOTECHS. Future studies need to determine how self-assessment can be best utilized. Level of evidenceStudies of diagnostic accuracy – Level 2.

Applicability of the Chinese version of the Personal and Social Performance scale in patients with severe mental disorders.

This study was designed to determine the reliability and validity of the Chinese version of the Personal and Social Performance scale (PSP-CHN) and assess the applicability of using the PSP-CHN in patients with severe mental disorders. A total of 285 outpatients with severe mental disorders, 220 with schizophrenia and 65 with major depressive disorder, were enrolled into the study. Both diagnoses were made using the DSM-IV. All the patients were assessed with the PSP-CHN, the GAF, and the CGI-S. In addition, the PANSS and the MADRS were used to assess the patients with schizophrenia and major depressive disorder, respectively. The PSP-CHN showed good internal consistency (Cronbach α=0.839, n=285), high interrater reliability for total scores (intraclass correlation coefficient, ICC=0.865, n=48), and good test-retest reliability (ICC=0.892, n=130). The PSP-CHN total score showed a statistically significant positive correlation with the GAF score (r=0.927, P<.01, n=285), as well as a significant negative correlation with the CGI-S total score (r=-0.793, P<.01, n=285), the PANSS total score (r=-0.694, P<.01, n=220), and the MADRS total score (r=-0.721, P<.01, n=65). We were able to demonstrate that the PSP-CHN is a reliable and valid measurement tool to assess the personal and social functioning in patients with severe mental disorders.

Exploration of the (Interrater) Reliability and Latent Factor Structure of the Alcohol Use Disorders Identification Test (AUDIT) and the Drug Use Disorders Identification Test (DUDIT) in a Sample of Dutch Probationers

Background: The use of brief, reliable, valid, and practical measures of substance use is critical for conducting individual (risk and need) assessments in probation practice. In this exploratory study, the basic psychometric properties of the Alcohol Use Disorders Identification Test (AUDIT) and the Drug Use Disorders Identification Test (DUDIT) are evaluated. Methods: The instruments were administered as an oral interview instead of a self-report questionnaire. The sample comprised 383 offenders (339 men, 44 women). A subset of 56 offenders (49 men, 7 women) participated in the interrater reliability study. Data collection took place between September 2011 and November 2012. Results: Overall, both instruments have acceptable levels of interrater reliability for total scores and acceptable to good interrater reliabilities for most of the individual items. Confirmatory factor analyses (CFA) indicated that the a priori one-, two- and three-factor solutions for the AUDIT did not fit the observed data very well. Principal axis factoring (PAF) supported a two-factor solution for the AUDIT that included a level of alcohol consumption/consequences factor (Factor 1) and a dependence factor (Factor 2), with both factors explaining substantial variance in AUDIT scores. For the DUDIT, CFA and PAF suggest that a one-factor solution is the preferred model (accounting for 62.61% of total variance). Conclusions: The Dutch language versions of the AUDIT and the DUDIT are reliable screening instruments for use with probationers and both instruments can be reliably administered by probation officers in probation practice. However, future research on concurrent and predictive validity is warranted.

Brazilian version of the Vestibular Disorders Activities of Daily Living Scale (VADL)

AbstractThe Vestibular Disorders Activities of Daily Living Scale (VADL) assesses the impact of dizziness and body imbalance on the everyday activities of patients with vestibulopathy. The scale encompasses 28 activities divided into three sub-scales (functional, ambulation and instrumental). ObjectiveTo translate and cross-culturally adapt the VADL to the Brazilian Portuguese language and verify its reliability. MethodQuestionnaire translation methodological research. Eighty elderly subjects (age ≥ 65 years) with chronic dizziness arising from vestibular disorders were enrolled, of which 40 participated in the pre-testing stage and 40 in reliability analysis. Concordance Correlation Coefficient (CCC) analysis was used to assess reliability. Internal consistency was estimated using Cronbach's alpha (α). ResultsPre-test analysis revealed 15% of incomprehension on two activities; these items had to be adapted. The VADL-Brazil had similar levels of test-retest and inter-rater reliability for total score and presented substantial agreement (CCC = 0.79). Internal consistency was excellent for total score (α= 0.92), good for the functional (α= 0.89) and locomotion (α= 0.86) sub-scales, and poor for the instrumental subscale (α= 0.56). ConclusionThe Brazilian version of the VADL was proven adequate, with good levels of reliability and internal consistency. It might be thus considered as an alternative to assess the functional capacity of vestibulopathy patients.

Interrater Reliability of the Melbourne Assessment of Unilateral Upper Limb Function for Children With Hemiplegic Cerebral Palsy

We examined the interrater reliability of the Melbourne Assessment of Unilateral Upper Limb Function. Three occupational therapists independently scored 34 videotaped assessments of children with hemiplegic cerebral palsy aged 6 yr, 1 mo, to 14 yr, 5 mo. Intraclass correlation coefficients (ICCs) at a 95% confidence interval were calculated for total scores, category scores, and item scores. The correlation between raters' total scores was high (ICC = .961). The highest correlation for test components between raters was found for fluency (ICC = .902), followed by range of movement (ICC = .866), and the lowest correlation was found for quality of movement (ICC = .683). The ICCs for individual test item scores varied and ranged from .368 to .899. This study demonstrated high interrater reliability for total scores, with scoring of some individual components and items requiring further consideration from both a clinical and a research perspective.

Reliability and Validity of the Dynamic Gait Index in Persons With Chronic Stroke

Jonsdottir J, Cattaneo D. Reliability and validity of the Dynamic Gait Index in persons with chronic stroke. Objective To establish the test-retest and interrater reliability as well as the concurrent construct validity of the Dynamic Gait Index (DGI) as a measure for dynamic balance in people with chronic stroke. Design Cohort study. Setting Day hospital and ambulatory care at a rehabilitation center. Participants A consecutive sample of 25 participants, at least 3 months poststroke and able to walk at least 10m with or without a walking aid, participated in the study. Two independent raters rated performances on the DGI. Interventions Not applicable. Main Outcome Measures The DGI was administered in 2 testing sessions 3 days apart. In the second session, the participants were rated by 2 raters. Intraclass correlation coefficients (ICCs), model 2,1, and the Bland and Altman method were used to analyze total scores and item scores. Concurrent construct validity was tested by correlating results to the Berg Balance Scale, the timed walking test, the Timed Up & Go test, and the Activities-specific Balance Confidence Scale. Results ICCs for test-retest and interrater reliability of total scores were good (.96, .96, respectively) whereas reliability for single item scores was moderate to good (range, .55−.93). The hypotheses for concurrent construct validity were confirmed with all measures (range, .68−.83). Conclusions The DGI showed high reliability and showed evidence of concurrent validity with other balance and mobility scales. It is a useful clinical tool for evaluating dynamic balance in ambulatory people with chronic stroke.

Development of a Multidimensional Balance Scale for Use With Functionally Independent Older Adults

Rose DJ, Lucchese N, Wiersma LD. Development of a multidimensional balance scale for use with functionally independent older adults. Objective To develop and evaluate the validity and reliability of a multidimensional balance scale—the Fullerton Advanced Balance (FAB) scale—suitable for use with functionally independent older adults. Design Psychometric evaluation of the scale’s content and convergent validity, test-retest and intra- and interrater reliability, and internal rater consistency. Setting Urban community. Participants Forty-six community-residing older adults (mean ± standard deviation, 75±6.2y), with (n=31) and without identified balance problems (n=15), participated in the study. Four physical therapists with expertise in the assessment and treatment of balance disorders in older adults also participated in the content validity and/or reliability phases of the study. Interventions Not applicable. Main Outcome Measures Spearman rank correlation coefficients for convergent validity, test-retest, intra- and interrater reliability, and homogeneity coefficient values for rater consistency. Results Test-retest reliability for the total balance scale score was high (ρ=.96). Interrater reliability for total score ranged from .94 to .97 whereas intrarater reliability coefficients ranged from .97 to 1.00. Homogeneity ( H) coefficients were greater than .90 for 6 of the 10 individual test items and all 10 test items had H coefficients of greater than .75 for both rating sessions. Conclusions Preliminary results suggest that the FAB scale is a valid and reliable assessment tool that is suitable for use with functionally independent older adults residing in the community.

RELIABILITY AND VALIDITY OF AN OBJECTIVE MEASURE OF SITTING BALANCE.

Purpose/Hypothesis: Available balance measures are primarily for people who can stand and are not valid for those who are primarily non-ambulators. The purpose of this study was to assess the reliability and validity of an objective measure of sitting balance. Number of Subjects: Participants were 14 primarily non-ambulatory older adults (mean age 81.5 years), 11 content expert clinicians, 4 raters who were PT students and 2 geriatric experienced PTs. Materials/Methods: Older participants performed 11 tasks ranging from quiet sitting to a sit to stand transfer, with 5 repeated on foam for a total of 16 items. The test was scored by student and experienced raters. Participants rated their steadiness on each item using a 5-point scale. Raters viewed video of the original testing and rescored participants performance to determine reliability. Content experts rated the importance of each item using a 5-point scale and made suggestions for improvement. Data analysis included ICCs, one way ANOVA, paired sample t-tests, Chronbachs alpha, and Pearson correlations. Results: Intrarater reliability of the raters ranged from ICC=.86 to 1.0. Interrater reliability for total scores was ICC=.97. There were no differences in the mean rating for each item and for the total score between the student and experienced raters. The entire tool demonstrated internal consistency (Chronbachs alpha=.76), but two items had an alpha below the desired level of .4. Most (61%) participants considered themselves fairly steady when performing the test as a whole. There was a strong relationship (r=.84, p=.001) between total score of self-assessed steadiness and total score on the balance tool. The mean rating of importance for each item ranged from 2.30 to 3.82. In general, the experts felt that items on foam were not useful. Conclusions: The 16-item tool appears to be reliable for assessing sitting balance in primarily non-ambulatory subjects. It demonstrates excellent intra and interrater reliability which is not dependent on rater experience. The tool appears to have good face and content validity for assessing balance in primarily non-ambulatory subjects. Based on the findings the tool will be modified to eliminate redundant items and items with poor internal consistency. Further study is necessary to determine reliability and validity of the revised 10-item tool. Clinical Relevance: This study demonstrates that this measure of sitting balance may be an effective tool for individuals who are primarily non-ambulatory. The future 10-item tool may provide clinicians with an objective measure of sitting balance and should be used in clinical practice to measure and monitor changes in sitting balance over time.

A clinical rating scale for Batten disease

Batten disease (juvenile neuronal ceroid lipofuscinosis [JNCL]) is an autosomal recessive neurodegenerative disorder characterized by blindness, seizures, and relentless decline in cognitive, motor, and behavioral function. Onset is in the early school years, with progression to death typically by late adolescence. Development of a clinical instrument to quantify severity of illness is a prerequisite to eventual assessment of experimental therapeutic interventions. To develop a clinical rating instrument to assess motor, behavioral, and functional capability in JNCL. A clinical rating instrument, the Unified Batten Disease Rating Scale (UBDRS), was developed by the authors to assess motor, behavioral, and functional capability in JNCL. Children with verified JNCL were evaluated independently by three neurologists. Intraclass correlation coefficients (ICCs) were used to estimate the interrater reliability for total scores in each domain. Interrater reliability for scale items was assessed with weighted kappa statistics. Thirty-one children with confirmed JNCL (10 boys, 21 girls) were evaluated. The mean age at symptom onset was 6.1 +/- 1.6 years, and the mean duration of illness was 9.0 +/- 4.4 years. The ICCs for the domains were as follows: motor = 0.83, behavioral = 0.68, and functional capability = 0.85. The Unified Batten Disease Rating Scale (UBDRS) is a reliable instrument that effectively tests for neurologic function in blind and demented patients. In its current form, the UBDRS is useful for monitoring the diverse clinical findings seen in Batten disease.

Inter-rater reliability of two depression rating scales, MADRS and DRRS, based on videotape records of structured interviews

The inter-rater reliability of the French versions of the MADRS and the DRRS was studied on the basis of 58 videotape records of structured standardised interviews of depressed inpatients under antidepressant treatment. Each patient was assessed by two trained raters, from the same videotape recording. The inter-rater reliability of total scores was high with both scales (intra-class correlation coefficients: 0.86 for MADRS and 0.77 for DRRS). However, the inter-rater reliability for individual items was higher and more homogeneous for the MADRS than for the DRRS. Finally, the structured interview in French appears to be relevant for the MADRS, but it should be improved for the DRRS.

Cross‐validation of the Delirium Rating Scale in Older Patients

To cross-validate the Delirium Rating Scale (DRS). Cross-sectional. Geriatric medicine and geriatric psychiatry assessment units and consultation services. A total of 104 older patients on the above services. Mini-Mental State Examination (MMSE) score, Barthel Index score, DRS score, Blessed Dementia Scale score, clinical diagnoses using DSM-III-R criteria. The mean DRS score was highest in the delirium group. Cronbach's alpha was .90, and inter-rater reliability of total scores was .91 (intra-class correlation). Receiver Operating Characteristic curve analysis showed that the area under the curve for the DRS was significantly higher than the MMSE as a test for delirium. At its published cutpoint of 10, the sensitivity of the DRS is .82 and the specificity is 94. The value at which the sensitivity of the DRS is .90 is 8, at which specificity is .82. The DRS appears to a feasible instrument. In a sample with a high proportion of delirious patients, it has acceptable measurement properties when used by expert observers.

Catatonia. I. Rating scale and standardized examination.

To facilitate the systematic description of catatonic signs, we developed a catatonia rating examination, rating scale and screening instrument. We constructed a 23-item rating scale and a truncated 14-item screening instrument using operationalized definitions of signs ascribed to catatonia in published sources. Inter-rater reliability was tested in 44 simultaneous ratings of 28 cases defined by the presence of > or = 2 signs on the 14-item screen. Inter-rater reliability for total score on the rating scale was 0.93, and mean agreement of items was 88.2% (SD 9.9). Inter-rater reliability for total score on the screening instrument was 0.95, and mean agreement of items was 92.7% (SD 4.9). Diagnostic agreement was high based on criteria for catatonia put forth by other authors. Seven per cent (15/215) of consecutively admitted patients to an academic psychiatric in-patient facility met criteria for catatonia. It is concluded that catatonia is a distinct, moderately prevalent neuropsychiatric syndrome. The rating scale and screening instrument are reliable and valid. Their use facilitates diagnosis, treatment protocols, and cross-study comparisons.

The neurological evaluation scale (NES): A structured instrument for the assessment of neurological signs in schizophrenia

The significance of neurological signs in schizophrenia is poorly understood. In part, this may reflect the marked variability in the methods of ascertainment in previous studies. The Neurological Evaluation Scale (NES) is designed to standardize the assessment of neurological impairment in schizophrenia. The battery consists of 26 items. Data on the interrater reliability for total score, functional areas of interest, and individual items are presented. Preliminary validity data demonstrate the ability of the battery to discriminate patients with schizophrenia from nonpsychiatric controls.