Use Of Rating Scales Research Articles

Introducing the Staff-Administered Meal Independence Rating Scale for Use Among Patients With Serious Mental Illnesses.

For individuals living with serious mental illnesses (SMIs), inadequate meal preparation skills can hinder the ability to live independently; yet rating scales tailored for this population are lacking. We describe development, item analysis, and initial reliability and validity of the Staff-Administered Meal Independence Rating Scale (SAMIRS). After item development involving expert consultation, two rounds of pilot testing ( n = 188, n = 293) were conducted in inpatient and residential settings (transitional living residences [TLRs] and community residences [CRs]). For initial testing of convergent validity, Pearson correlations with Specific Levels of Functioning (SLOF) scale items were computed. Exploratory factor analysis revealed a single factor; Cronbach's alpha was high (0.98). The mean SAMIRS score varied by setting: CR residents scored higher than those in TLRs or inpatient units. Scores were highly correlated with SLOF items measuring community living skills. Although further study is warranted, the SAMIRS could be a useful tool in rating functional needs pertaining to meal independence among individuals with SMI.

What's not to like about Likert? Developing a nonverbal animal preference scale (NAPS).

Rating scales, such as Likert scales, are incredibly flexible and intuitive tools for measuring individuals' rating of agreement with or relative preference for many types of stimuli. For humans, this typically involves ratings of agreement between end points representing distinct attitudes or beliefs; For example, strongly disagree to strongly agree. Nonverbal versions of Likert scales have also been presented to children, allowing them to indicate their degree of preference, pain, or happiness. However, before the current study, no known efforts had been made to develop a nonverbal rating scale for use with nonhuman animals. Such a scale would be a useful welfare tool, allowing nonverbal individuals to indicate not just relative preferences between pairs of items but their degree of liking for individual items. I present an outline of the steps taken to create such a scale for use with three zoo-housed gorillas. Two gorillas succeeded in associating preferred and less preferred foods with different response buttons but none of the gorillas were able to effectively use the neutral response button. It is possible that limits in gorillas' capacity for conditional discriminations and/or conceptualization of constructs as abstract as "liking" impeded training. These data are relevant for understanding gorilla cognition and can inform continued efforts to create a tool for nonhumans to communicate their preferences to human caregivers in a more nuanced way than is currently possible.

A secondary analysis to develop a scale for measuring unemployed workers' experiences of Australian employment services.

Research consistently shows that Australian employment services are failing those they are intended to serve. Based on findings in other human service areas, a valid and reliable instrument to measure unemployed workers' experiences may provide an opportunity for improvement in this sector. To establish a basis for developing a suitable rating scale. An exploratory factor analysis combined with qualitative cross check for face validity of an existing large survey of Australian unemployed workers. Six factors appear to be important elements of service delivery: (F1) useful and competent, (F2) client-centred, (F3) receptive to feedback, (F4) trustworthy, (F5) fair, and (F6) friendly. While each of these factors have been either described explicitly or referred to implicitly in previous studies, this study is the first to attempt to combine these factors and is a precursor to establishing a valid and reliable rating scale for use by unemployed workers in evaluating their employment service providers. At a time when Australia is exploring new approaches to employment services, such a scale using a robust set of factors may allow for the improvement of employment services and thus be held accountable to a significant stakeholder group whom they aim to serve -unemployed workers.

Validating a Series of Photonumeric Rating Scales for Use in Facial Aesthetics Using Statistical Analysis of Intra- and Inter-rater Reliability.

BackgroundGrowing demand for minimally invasive aesthetic procedures to correct age-related facial changes and optimize facial proportions has been met with innovation but has created an unmet need for objective assessment tools to evaluate results empirically.ObjectivesThe purpose of this study is to establish the intra- and inter-rater reliability of ordinal, photonumeric, 4-, or 5-point rating scales for clinical use to assess facial aesthetics.MethodsBoard-certified plastic surgeons and dermatologists (3 raters) performed live validation of jawline contour, temple volume, chin retrusion, nasolabial folds, vertical perioral lip lines, midface volume loss, lip fullness, and crow’s feet dynamic- and at rest-rating scales over 2 rounds, 2 weeks apart. Subjects selected for live validation represented the range of scores and included 54 to 83 subjects for each scale. Test-retest reliability was quantitated through intra- and inter-rater reliability, determined from the mean weighted kappa and round 2 intraclass correlation coefficients, respectively. The clinical significance of a 1-grade difference was assessed through rater comparison of 31 pairs of side-by-side photographs of subjects with the same grade or a different grade on the developed scales.ResultsThe study demonstrated substantial to near-perfect intra- and inter-rater reliability of all scales when utilized by trained raters to assess a diverse group of live subjects. Furthermore, the clinical significance of a 1-point difference on all the developed scales was established.ConclusionsThe high test-retest reliability and intuitive layout of these scales provide an objective approach with standardized ratings for clinical assessment of various facial features.

Development and Psychometric Evaluation of the PROMIS Pediatric Pain Intensity Measure in Children and Adolescents with Chronic Pain

The goal of the current study was to enhance the measurement of the pediatric chronic pain experience through a methodologically rigorous approach. This paper outlines the development and initial validation of a pain intensity measure for pediatric patients with chronic pain using Patient-Reported Outcomes Measurement Information System methodology. Measure development incorporated feedback from children with painful conditions. Based on input from pediatric participants and content experts, 4 candidate items assessing pain intensity were included for large scale testing. Children completed self-report items pertaining to their pain experience that were developed as part of a larger pool of new candidate Patient-Reported Outcomes Measurement Information System pediatric pain domain items as well as measures of pain interference, depressive symptoms, fatigue, pain behavior, pain intensity, and pain catastrophizing. The final sample for the large scale testing included N = 442 pediatric patients between the ages 8 to 18 years (Mean age = 13.54, Standard Deviation = 2.78; 71.27% female) experiencing chronic pain. Psychometric analysis resulted in a final measure that included 3 items with evidence of reliability (Cronbach alpha = .82) and convergent validity. The Likert format of the response options may be preferable to the traditional numeric rating scale for use in pediatric populations who experience chronic pain based on patients’ feedback, which was directly utilized in designing the scale. Further, the inclusion of fewer and clinically meaningful response options should reduce ambiguity for young respondents. PerspectiveWe have developed and evaluated a clinically sensitive and psychometrically precise 3-item pain intensity measure with Likert-type responses for self-report use among children and adolescents ages 8 to 18 years with chronic pain. Development of the item content and response options included input from children and adolescents with chronic pain. The development of pain intensity items with pediatric appropriate language, and labeled, fewer response options to yield maximal clinically meaningful information improves the precision of pain intensity measurement in children.

Adaptation and validation of a structured version of the Hamilton Depression Rating Scale for use by non-clinicians in South Africa (AFFIRM-HDRS).

The Hamilton depression rating scale (HDRS) is seen as an international gold standard for the measurement of depression. High rates of untreated depression, including perinatal depression, support the rationale to adapt and validate a structured version of the HDRS to be used by non-clinicians in low-income settings. Using previous structured versions of the HDRS as a foundation, the tool was adapted, translated, and then validated with an isiXhosa speaking perinatal population in South Africa (n = 187), using cognitive testing, test-retest reliability, and inter-rater assessments. The AFFIRM-HDRS was compared with the Edinburgh postnatal depression scale (EPDS) using the non-parametric Spearman Rho test to assess concurrent validity. Internal consistency was examined using Cronbach's Alpha, and inter-rater and test-retest reliability were assessed with the intra-class coefficient (ICC). Cohen's Kappa was used to assess the overall percentage agreement for each individual item of the AFFIRM-HDRS. The AFFIRM-HDRS showed good construct and content validity had significant associations with the EPDS (Rho = 0.60 and 0.43, P < .001), and acceptable internal consistency (Cronbach's alpha = 0.74.). Inter-rater reliability and test-retest scores were excellent, with intraclass correlations ranging from 0.97 (0.94-0.99) to 0.98 (0.97-0.99) between raters, and test-retest reliability being 0.90 (95% CI: 0.86-0.93). The tool performed similarly to previous structured versions. Individual item-rest correlations suggest that the items "Weight loss or gain," "Insight," and "Libido" did not fit well with the overall instrument, but that the rest of the items performed well. The AFFIRM-HDRS is adequately structured to be used by non-clinicians in an isiXhosa speaking perinatal population. It is hoped that this structured tool can be used to assist with identification and referral of these at-risk populations by non-clinicians in resource-constrained environments, thereby playing a role in addressing the treatment gap for perinatal depression in LMICs.

583 Psychometric validation and meaningful change threshold of the worst itching intensity numerical rating scale for use in hemodialysis patients with pruritus

583 Psychometric validation and meaningful change threshold of the worst itching intensity numerical rating scale for use in hemodialysis patients with pruritus

Adaptation of the Basso–Beattie–Bresnahan locomotor rating scale for use in a clinical model of spinal cord injury in dogs

BackgroundNaturally occurring acute spinal cord injury (SCI) in pet dogs provides an important clinical animal model through which to confirm and extend findings from rodent studies; however, validated quantitative outcome measures for dogs are limited. New methodWe adapted the Basso Beattie Bresnahan (BBB) scale for use in a clinical dog model of acute thoracolumbar SCI. Based on observation of normal dogs, modifications were made to account for species differences in locomotion. Assessments of paw and tail position, and trunk stability were modified to produce a 19 point scale suitable for use in dogs, termed the canine BBB scale (cBBB). Pet dogs with naturally occurring acute SCI were assigned cBBB scores at 3, 10 and 30days after laminectomy. ResultsScores assigned via the cBBB were stable across testing sessions in normal dogs but increased significantly between days 3 and 30 in SCI-affected dogs (p=0.0003). The scale was highly responsive to changes in locomotor recovery over a 30day period, with a standardized response mean of 1.34. Comparison with existing methodsConcurrent validity was good, with strong correlations observed between the cBBB and two other locomotor scales, the OSCIS (r=0.94; p<0.001) and the MFS (r=0.85; p<0.0001). cBBB scores inversely correlated with other assessments of recovery including mechanical sensory threshold (r=−0.68; p<0.0001) and coefficient of variation of stride length (r=−0.49; p<0.0001). ConclusionsThese results support the use of the cBBB to assess locomotor recovery in canine clinical translational models of SCI.

Maximizing measurement efficiency of behavior rating scales using Item Response Theory: An example with the Social Skills Improvement System — Teacher Rating Scale

Measurement efficiency is an important consideration when developing behavior rating scales for use in research and practice. Although most published scales have been developed within a Classical Test Theory (CTT) framework, Item Response Theory (IRT) offers several advantages for developing scales that maximize measurement efficiency. The current study provides an example of using IRT to maximize rating scale efficiency with the Social Skills Improvement System — Teacher Rating Scale (SSIS — TRS), a measure of student social skills frequently used in practice and research. Based on IRT analyses, 27 items from the Social Skills subscales and 14 items from the Problem Behavior subscales of the SSIS — TRS were identified as maximally efficient. In addition to maintaining similar content coverage to the published version, these sets of maximally efficient items demonstrated similar psychometric properties to the published SSIS — TRS.

Development of the psychometric property of a Minimum Data-Set-Based Depression Rating Scale for use in long-term care facilities in Taiwan

Objectives: Our aim is to develop the psychometric property of the Minimum Data-Set-Based Depression Rating Scale (MDS-DRS) to ensure its use to assess service needs and guide care plans for institutionalized residents.Methods: 378 residents were recruited from the Haoran Senior Citizen Home in northern Taiwan. The MDS-DRS and GDS-SF were used to identify observable features of depression symptoms in the elderly residents.Results: A total of 378 residents participated in this study. The receiver operating characteristic (ROC) curve indicated that the MDS-DRS has a 43.3% sensitivity and a 90.6% specificity when screening for depression symptoms. The total variance, explained by the two factors ‘sadness’ and ‘distress,’ was 58.1% based on the factor analysis.Conclusions: Reliable assessment tools for nurses are important because they allow the early detection of depression symptoms. The MDS-DRS items perform as well as the GDS-SF items in detecting depression symptoms. Furthermore, the MDS-DRS has the advantage of providing information to staff about care process implementation, which can facilitate the identification of areas that need improvement. Further research is needed to validate the use of the MDS-DRS in long-term care facilities.

Exploring Automated Scoring of Korean High School Students’ English Composition

The primary purpose of this study was to explore the efficacy of an automated essay scoring (AES) program, namely IntelliMetric, in scoring Korean high school students` English compositions in a large-scale standardized writing assessment setting. In addition, this study also examined the relative utility of 5-point analytic rating scales and 4-point analytic rating scales for use in scoring essays from this particular group of secondary EFL students in Korea. The results of this study showed that the computer scoring models built based on a large number of student essays with known scores (i.e., training set) could successfully predict the scores of essays set aside for the purpose of internal validation (i.e., validation set), providing evidence for the efficacy of the computer scoring models developed. This was backed up by three different measures comparing the scores derived from the automated essay scoring program and trained human raters: mean scores, agreement rates, and pearson correlation. Considering the computer program`s ability to apply scoring criteria or standards consistently over time, although far from being perfect, it seems to show the potential to score essays reliably large scale assessment contexts such as a nation-wide writing test.

Development and Validation of a Parent Report Measure for Assessing Social-Emotional Competencies of Children and Adolescents

Based on the premises that strength-based assessment of children and adolescents is an important emerging area, and that additional tools for this purpose are needed, this study details development and validation efforts on a new strength-based assessment: the Social-Emotional Assets and Resilience Scale, parent form (SEARS-P). Following careful development of a comprehensive research prototype assessment, a large and diverse nationwide sample of more than 2,000 ratings of school-age children and adolescents were obtained from their parents and other caregivers. Factor analytic procedures revealed a robust and replicable underlying factor structure, including Self-Regulation/Responsibility, Social Competence, and Empathy factors. The factor scores and total score of the SEARS-P were shown to have strong internal consistency reliability, as well as strong interrater reliability between mother-father pairs who rated the same child. Convergent construct validity of the SEARS-P was established through findings of significant correlations with two established strength-based rating scales for use by parents. Construct validity of the SEARS-P was further bolstered through findings of significant gender differences in scores (with females rated as having higher levels of competency than males), as well as significant differences in scores based on educational disability status. Limitations and future research needs are discussed, as are implications of this study for research and practice with children and families.

The interrater reliability of the modified Gait Abnormality Rating Scale for use with people with intellectual disability

Background Researching falls in persons with ID is limited by difficulties in applying standardised balance outcome measures. The modified Gait Abnormality Rating Scale (GARS-M), developed to identify falls risk in older adults, requires only that the participant walks and thus may be a feasible falls research tool to use with people with ID. In the present study, we consider the interrater reliability of the GARS-M.Method Nine people with ID were videotaped walking. These recordings were used by 10 physiotherapists to rate the person's walking on the 7 categories of the GARS-M. The intraclass correlation coefficient (ICC) was calculated for the physiotherapists' scores in each category and the total score; ICC ≥ 0.7 indicating good interrater reliability.Results The ICC was greater than 0.7 in only one category of the GARS-M, but ICC = 0.85 for the total score. Average time to rate one person was 8 minutes.Conclusions The GARS-M is potentially useful for falls prevention research with people with ID.

Apathy and Functioning in First-Episode Psychosis

Apathy and Functioning in First-Episode Psychosis

The psychometric properties of the Attention–Distraction, Inhibition–Excitation Classroom Assessment Scale (ADIECAS) in a sample of children with moderate and severe intellectual disabilities

The Attention–Distraction, Inhibition–Excitation Classroom Assessment Scale (ADIECAS) [Evans, P. L. C. (1975). Inhibition and stimulus generalization in the discrimination learning of ESN(S) and ESN(M) children. Unpublished Ph.D. thesis. Manchester, UK: University of Manchester] assesses attention-related difficulties in children with intellectual disabilities (ID). The present study explored the psychometric properties of the ADIECAS with a sample of 84 children with moderate and severe ID whose teachers completed the ADIECAS and the Conners’ Teachers Rating Scale [Goyette, C. H, Conners, C. K., & Ulrich, R. F. (1978). Normative data on revised Conners’ parent and teachers rating scales. Journal of Abnormal Child Psychology, 6, 221–236]. Following factor analysis the Evans and Hogg [Evans, P. L. C., & Hogg, J. (1984). A classroom rating scale for use with mentally retarded children. British Journal of Clinical Psychology, 23, 187–194] structure was the most appropriate for use with this sample. Moderate to strong positive relationships were found between the ADIECAS and the Conners’, and the ADIECAS factors were significant predictors of the Conners’ subscales. This study provides further support for the ADIECAS as a reliable and valid measure of ADHD-related symptoms among children with moderate and severe ID.

Childhood autism: translation and validation of the Childhood Autism Rating Scale for use in Brazil

To translate the Childhood Autism Rating Scale into Brazilian Portuguese and to determine the initial psychometric properties of the resulting version (CARS-BR). The methodology used to produce an adequate version included translation, backtranslation and evaluation of semantic equivalence. In order to determine its psychometric properties (internal consistency, validity and reliability), the CARS-BR was administered to 60 consecutive patients with autism, aged between 3 and 17 years and seen at a university hospital. Internal consistency was high, with a Cronbach's alpha of 0.82. Convergent validity, in comparison with the Autistic Traits Assessment Scale, exhibited a Pearson's correlation coefficient of r = 0.89. When correlated with the Global Assessment of Functioning Scale in order to evaluate discriminant validity, the CARS-BR exhibited a Pearson's coefficient of r = -0.75. Test-retest reliability exhibited a kappa coefficient of 0.90. These results suggest that the CARS-BR is a valid and reliable instrument for evaluating autism severity in Brazil.

Training Satisfaction Rating Scale

We describe the process of testing a measurement model of a satisfaction rating scale for use with training programs. The scale was developed by the Evaluation Unit of the University of Seville's Training Center. Following an initial content validity study in which we reviewed how training evaluation is modeled, a 12-item rating scale was developed and administered to a sample of 2746 subjects. The measurement model was examined through an exploratory factor analysis using polychoric correlations, the results of which were consistent with the previous dimensions obtained in the content validity study. In order to refine the model under study, a confirmatory factor analysis (also based on polychoric correlations) was then performed. The results present empirical evidence of the measurement model's validity. The scale offers simple, direct and reliable training satisfaction scores. Future development of the scale is discussed.

Development of an Agitation Rating Scale for Use with Acute Presentation Behavioral Management Patients

Background: Agitation is a condition frequently observed in behavior management patients presenting to the emergency department (ED). Despite the Joint Commission on the Accreditation of Health Care Organizations’ standards for the assessment and monitoring of behavioral health patients, the authors are unaware of a reliable, valid agitation rating scale appropriate for use in the emergency setting.

Validity and reliability of the Minimum Data Set Depression Rating Scale (MDSDRS) for older adults in nursing homes

this research examined the psychometric properties of the Minimum Data Set Depression Rating Scale for use among older adults living in nursing homes. interviews with 145 older adults in three nursing homes were conducted to complete the Hamilton Depression Rating Scale and the Geriatric Depression Scale. Information relevant to completing the Minimum Data Set Depression Rating Scale was gathered from the Minimum Data Set. the Minimum Data Set Depression Rating Scale did not perform well when validated against the Hamilton Depression Rating Scale and the Geriatric Depression Scale. Minimum Data Set Depression Rating Scale cut-off levels of > or =2 and > or =3 were associated with relatively low total score correlations and sensitivity rates, but acceptable specificity. findings suggest that the Minimum Data Set Depression Rating Scale may be of limited clinical value to identify depression among older adults living in nursing homes.

A DSM-IV—Referenced Teacher Rating Scale for Use in Clinical Management

Two studies were conducted to examine the psychometric properties of a DSM-IV-referenced teacher-completed rating scale in children receiving special education. To assess reliability, teachers rated 74 students on two separate occasions (test-retest) using the teacher version of the Child Symptom Inventory-4 (CSI-4T), and teacher aides also rated the children on the first occasion (interrater). In a second study, teacher CSI-4T and Teacher's Report Form (TRF) ratings and consultant diagnoses were obtained for 101 students. Internal consistency reliabilities (0.72-0.94), 2-week test-retest reliabilities (r = 0.61-0.88), and interrater agreement (r = 0.19-0.56) for the CSI-4T major symptom categories were comparable with dimensional rating scales. CSI-4T ratings showed a consistent pattern of convergent and divergent validity with TRF scores and with consultant diagnoses. Findings provide preliminary support for the reliability and validity of the CSI-4T as a measure of DSM-IV symptoms in children receiving special education.