Multi-rater Assessment Research Articles

Evaluating a novel method of scoring multirater assessments of leader competencies.

Evaluating a novel method of scoring multirater assessments of leader competencies.

Supplemental Material for Evaluating a Novel Method of Scoring Multirater Assessments of Leader Competencies

Supplemental Material for Evaluating a Novel Method of Scoring Multirater Assessments of Leader Competencies

Image quality comparison of 1.5T and 3T prostate MRIs of the same post-hip arthroplasty patients: multi-rater assessments including PI-QUAL version 2.

To compare the image quality of 1.5T and 3T prostate MRIs of the same post-hip arthroplasty patients, with a specific focus on the degree of susceptibility artifacts. This single-center retrospective study included post-hip arthroplasty patients who underwent 1.5T prostate MRIs between 2021 and 2023, as well as comparative 3T prostate MRIs. Three blinded abdominal radiologists retrospectively reviewed their diffusion-weighted imaging (DWI, 50s/mm2), T2-weighted imaging (T2WI), and dynamic contrast-enhanced imaging (DCE) to evaluate the image quality. The degree of susceptibility artifacts was categorized using a three-point scale, with 3 indicating the least artifact and 1 indicating the most. Image quality was also evaluated using Prostate Imaging Quality (PI-QUAL) version 2. The median of the three raters' scores was compared between 1.5T and 3T prostate MRIs using the Wilcoxon signed-rank test. The inter-rater agreement was evaluated using the multi-rater generalized kappa. Twenty pairs of 1.5T and 3T prostate MRI examinations from 20 unique patients were included. The DWI susceptibility artifact score at 1.5T was significantly higher than at 3T (mean score ± standard deviation, 2.80 ± 0.41 vs. 2.35 ± 0.93, p = 0.014). In contrast, no significant differences were observed in the susceptibility artifact scores in T2WI and DCE, or in the PI-QUAL score. The inter-reader agreement in the susceptibility artifact score was moderate (multi-rater generalized kappa: 0.60) in DWI, perfect in T2WI (not applicable), andsubstantial (0.65) in DCE. The inter-reader agreement wasfair (0.27) in the PI-QUAL score. Using 1.5T scanners may be preferable to reduce susceptibility artifacts from hip prostheses in DWI.

Comparative analysis of bibliometric indicators of Russian journals in agroengineering

The results of the bibliometric analysis of scientometric indicators of scientific journals in the sphere of agricultural mechanization and electrification are presented. The publications were selected based on SRSTI (State Rubricator of Sci-tech Information, GRNTI). All national journals in the area were analyzed. The selected bibliometric indicators provide for the multirater assessment of the journals. These are: citation five-year impact factor, loadings, and self-citation coefficient. Russian Science Citation Index (RSCI) Database served as the source for the data on every journal. Finally, five most significant publications were chosen, with three of them already included in the RSCI Database, and the remaining two – being recommended for inclusion. The completed comparative analysis permits to identify the most demanded journals for each indicator. The Scientific Journal of KubSAU is the most cited one, also leading in the number of loadings. The Agricultural Machinery and Technologies Journal demonstrates the highest impact factor. The Tractors and Agricultural Machinery Journal shows the lowest self-citation coefficient. The factors to increase the use of scientific serials are discussed. The obtained results may be used to increase authority and citation of publications in agroengineering, and to support their acquisition to the sci-tech library collections.

The convergent validity of the Romanian version of the Behavioral Assessment of Children: A Multitrait-Multimethod Analysis

This study assessed the validity of the Behavior Assessment System for Children 2nd Edition, by providing evidence based on a multitraitmultimethod (MTMM) analysis performed on data obtained with a culturally adapted form of the test. The study included two Romanian samples (161 adolescents aged 12-18 years and 91 children aged 8-12 years) that were each assessed with all the three age corresponding forms (self, parent, and teacher) of the test. We used a confirmatory factor analytic framework in order to test the convergent and discriminant validity of BASC-2, by generating a CFAbased multi-trait, multi-method latent factor correlation matrix. Results showed that, for both the Adolescent and Child forms, the coefficients falling under the validity diagonal (mono-trait, hetero-method) have the highest median values, compared to the coefficients encountered in the monomethod block and those in the heteromethod-heterotrait triangles. The study provides evidence for the construct validity of the multi-trait, multi-rater assessment system that is the basis of the BASC-2.

A multirater perspective on personality and performance: An empirical examination of the trait-reputation-identity model.

Organizational scientists have historically assessed personality via self-reports, but there is a growing recognition that personality ratings from observers offer superior prediction of job performance compared to targets' self-reports. Yet, the origin of these differences remains unclear: do observers show predictive validity advantages (a) because they have a clearer lens into how targets' thoughts, feelings, and desires translate to their behaviors (traits), (b) because they infer personality from how targets characteristically adapt their behaviors to situations (reputation), or (c) because they omit targets' unexpressed, internal aspects of personality (identity)? With a sample of 422 cadets at a highly selective military educational institute in South Korea, we applied (McAbee & Connelly, 2016) Trait-Reputation-Identity (TRI) Model to decompose consensus and discrepancy in multirater personality data. The variance associated with reputations (the unique personality insights held by observers) dominated the prediction for conscientiousness and agreeableness in predicting all criteria. Trait factors (reflecting the consensus between self- and observer-reports) were moderately related to ratings of job performance, citizenship, and grades for most theoretically aligned personality dimensions. Identity factors (targets' unique personality self-views) were generally unrelated to performance criteria, save for some modest positive relationships for conscientiousness (predicting work and academic performance) and agreeableness (predicting citizenship). These findings suggest that personality is an important determinant of success less by depicting "who employees are" (or "who they think they are") but more by describing "what employees do." We discuss the implications of these results for how multirater assessments can be built into organizational psychology research and practice. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Variations in Demonstrated Emotional Intelligence: Trainee to Experienced Faculty Member.

To identify differences in emotional intelligence (EI)-related competencies between fellows and faculty in a cohort of pediatric critical care physicians. Single-center, cross-sectional observation study. Seventy-two-bed multidisciplinary pediatric critical care unit at a quaternary children's hospital (Children's Hospital of Philadelphia, Philadelphia, PA). Forty-seven critical care physicians, including 19 fellows and 28 faculty members, were assessed. A multidisciplinary team of 83 physicians, nurses, and nurse practitioners contributed to the assessments. None. A multirater EI assessment tool (Emotional and Social Competency Inventory 360) was used to measure EI competencies of participating physicians across 12 core competencies. Utilizing a priori scoring definitions, physician EI competencies were classified as strengths or areas for growth. Results were stratified based on provider experience, generating comparisons between fellow and faculty cohorts. Ninety-four percent (177/188) of distributed assessments were completed. Fellow strengths were identified as organization awareness, achievement orientation, and teamwork; areas for growth were influence and emotional self-awareness. Compared with fellows, faculty members demonstrated additional strengths in the domains of adaptability, emotional self-control, coach and mentor, positive outlook, inspirational leadership, and influence. This study provides the first characterization of EI competencies among trainees and faculty members using a validated multirater assessment tool. The descriptions of physician EI, based on years of experience, are an important piece of the foundation for future explorations into the advancement of physician EI and effective leadership.

Gaming Leadership Development with Multi‐rater Assessment

Game development companies are founded based on a passion and love for gaming. Founders have spent a lifetime playing games, which has motivated education in a given development area like design or programming. The disruptive technology experienced in the gaming industry requires constant updating of these game development skills, which consumes the focus of founders, leaving skill gaps in running the business. Remaining sustainable in gaming leadership requires assessment of skills to identify a targeted plan for continuing development. Leadership development is a highly researched area in academia as well as a multi-billion-dollar industry annually. One of the most effective assessments used for leadership development is multi-rater assessments. Research and successful case studies indicate that when multi-rater-based leadership development is implemented properly with aligned competencies, it leads to individual behavioral change and collectively organizational change. Multi-rater assessments in the gaming industry appear to be a new concept due to the emerging nature of this industry. Our research first focuses on identification of leadership competencies in the gaming industry. We shift focus to multi-rater evaluation and exemplify key feedback that fosters permanent behavioral changes in gaming leaders, as well as process steps to support organizational change.

Experiences That Matter: Unraveling the Link Between Extracurricular Activities and Emotional and Social Competencies

Emotional and social competencies have been shown to be extremely desirable in young people for their successful entry into the labor market. Their development has been studied primarily as a result of formal training in the educational and work domains, whereas relatively little is known about the role played by extracurricular activities in promoting these types of competencies. Non-working personal experiences are often used as proxies to assess the emotional and social competencies of candidates in recruitment and selection phases. However, this inference is not based on clear scientific evidence. Drawing on experiential learning theory, this study investigated empirically the relationship between a range of extracurricular activities (volunteering, cultural activities, experience abroad, sport) and the competency portfolio of graduates. Data were collected from a sample of 324 graduates through a structured survey and a multi-rater assessment of their emotional and social competencies. The results of the Partial Least Square-Path Modeling in general provide support for the positive association between experiential extracurricular activities and emotional and social competencies, although not all relationships are supported. The present study contributes to advance in the understanding of the determinants of emotional and social competencies by examining their relationship with a broad range of extracurricular activities. Moreover, it discusses implications for higher education and human resource management.

Developing a Comprehensive Leadership Program for Extension County Directors

This article provides an overview of the development and components of LEAD95, University of Tennessee (UT) Extension’s comprehensive leadership program for county-level leaders. The program was designed in accordance with results from a Delphi study, which identified critical skills and resources new county directors need to be successful. The program consists of face-to-face workshops, an online resource tool kit, online learning opportunities, a multirater assessment, and mentoring. Expected outcomes include increased competency, increased job satisfaction, and more consistent leadership and direction among county-level leaders in the UT Extension system.

Cross-cultural adaptation of the recovery self-assessment instrument (RSA–R) person in recovery version to Brazilian Portuguese (Pt/Br)

Purpose The Recovery Self-Assessment (RSA–R) in Revised Version is an instrument designed to assess the degree to which mental health programs and services implement recovery-oriented practices. The purpose of this paper is to conduct a cross-cultural adaptation of the RSA–R instrument for use in local mental health services in the city of Campinas, State of São Paulo, Brazil. Design/methodology/approach This method for cross-cultural adaptation of the instrument included a series of iterative steps including preparation, translation, back translation, harmonization, expert evaluation, focus groups, in-depth interviews, expert opinion and pilot study. Findings A multi-rater assessment of the equivalence of content between the original RSA–R and the translated version revealed that each of the 32 items achieved at least 88% agreement in terms of equivalency. A multi-step harmonization process revealed additional suggestions for improvements in readability, comprehension and applicability to Brazilian context. An expert in youth and adult education provided additional stylistic recommendations. Combined, this iterative approach to cross-cultural translation resulted in an adapted version of the instrument that was well understood, culturally appropriate and adequate for further verification of psychometric properties. Originality/value The recovery process in Brazil and in the USA has culturally determined differences in terms of the way mental disorders are understood, diagnosed and treated. Moreover, there are different notions of what constitutes desirable results of recovery, health care and welfare. At the present time, there are few, if any, available cross-cultural instruments to assess the recovery-orientation of services between Brazil and the USA.

Multirater assessment of young children's social and emotional learning via the SSIS SEL Brief Scales – Preschool Forms

Interest in social and emotional learning (SEL) skills is growing rapidly with all 50 states adopting SEL standards for preschool children. However, data on these types of skills for young children are limited due to a paucity of psychometrically-sound assessments. Further, most available assessments are lengthy and minimally aligned with widely used SEL frameworks such as the model proposed by the Collaborative for Academic, Social, and Emotional Learning (CASEL). Thus, the current study focused on the development of valid and time-efficient rating scales of young children's SEL skills using teachers and parents as informants. We used item response theory to select items from the SSIS Social Emotional Learning Edition (SSIS SEL; Gresham & Elliott, 2017) using the national standardization sample of the measure. We then examined initial evidence of score reliability, validity, and fairness for the SSIS SEL Brief Scales – Preschool Forms resulting from this process. Results provide initial evidence for score reliability, validity, and fairness for both the Teacher and Parent versions of this measure.

MEASURING TEACHERS’ PEDAGOGICAL CONTENT KNOWLEDGE USING MANY-FACET RASCH MODEL

In the context of teacher professional development and teacher certification, an objective measurement of teachers’ Pedagogical Content Knowledge (PCK) is requried. This research is a case study to measure teachers’ PCK within prospective middle school science teachers participating in an Indonesian teacher professional development program. The study used quantitative approach which involved multirater assessment analysed by Many-Facet Rasch Model (MFRM) to assess teachers’ PCK in teaching a specific topic, i.e. “The Particulate Nature of Matter” (PNM). Researchers developed a rubric for items to measure teachers’ enactment PCK for teaching that specific topic. The MFRM analysis of the data revealed item reliability, item validity, separation and unidimensionality fall within good category. The findings show prospective teachers need to improve their PCK specifically in terms of “knowledge of instructional strategies and representations for teaching science” dimension. However, for knowledge of science curriculum and knowledge of students’ understanding in science dimension, the findings show that the participants were competent.

FinnTwin12 Cohort: An Updated Review.

This review offers an update on research conducted with FinnTwin12 (FT12), the youngest of the three Finnish Twin Cohorts. FT12 was designed as a two-stage study. In the first stage, we conducted multiwave questionnaire research enrolling all eligible twins born in Finland during 1983–1987 along with their biological parents. In stage 2, we intensively studied a subset of these twins with in-school assessments at age 12 and semistructured poly-diagnostic interviews at age 14. At baseline, parents of intensively studied twins were administered the adult version of the interview. Laboratory studies with repeat interviews, neuropsychological tests, and collection of DNA were made of intensively studied twins during follow-up in early adulthood. The basic aim of the FT12 study design was to obtain information on individual, familial and school/neighborhood risks for substance use/abuse prior to the onset of regular tobacco and alcohol use and then track trajectories of use and abuse and their consequences into adulthood. But the longitudinal assessments were not narrowly limited to this basic aim, and with multiwave, multirater assessments from ages 11 to 12, the study has created a richly informative data set for analyses of gene–environment interactions of both candidate genes and genomewide measures with measured risk-relevant environments. Because 25 years have elapsed since the start of the study, we are planning a fifth-wave follow-up assessment.

Multirater Assessment to Teacher Professionalism based on Pedagogical Content Knowledge

The Indonesian Government has performed teacher certification program since the establishment of the Teacher Law No. 14 year 2005 and PCK has been acknowledged as the indicator of science teachers’ professionalism in some countries. Therefore, this research aims to explore teachers’ PCK as a case study to 20 middle school science teachers with different teaching experiences in the special Region of Yogyakarta, Indonesia. This research was conducted in PCK planning and enacted PCK by collecting lesson plans, recorded the learning activities in the classroom, and structured interviews. The assessment to all research material was employed by 6 raters using PCK assessment rubric consisting of 12 items that was later analyzed with Many Facet Rasch Model. The analysis result informed that all items had good validity and reliability. This study found that only 15% of participants were classified as good teachers’PCK. In-depth analysis on the difficulty level item concluded that the teacher participant’s PCK ability on the representation and conceptual teaching strategies aspects still needed to be improved.

Psychometric Fundamentals of the Social Skills Improvement System: Social–Emotional Learning Edition Rating Forms

This study described the development of the Social Skills Improvement System Social Emotional Learning Edition Rating Forms (SSIS SEL RF) for teachers, parents, and students. This new multirater assessment is a reconfiguration of the SSIS Rating Scales items inspired by the CASEL Social Emotional Competency framework. The internal structure and score reliability estimates were examined across three raters for a common sample of more than 200 individual children ages 3 to 18 years. Confirmatory factor analyses tested against the CASEL five-dimensional SEL theoretical model demonstrated adequate fit for the SSIS SEL Parent and Student RFs and mediocre fit of the Teacher RF. Internal consistency, test-retest, and interrater reliability estimates for scores on each of the SSIS SEL RFs all met or exceeded acceptable criteria. Thus, researchers and practitioners interested in measuring the social–emotional behavior of children ages 3 to 18 can expect reliable scores and structurally meaningful behavior content within the Collaborative on Academic Social Emotional Learning (CASEL) SEL competency framework. Limitations to the present findings and suggestions for future research conclude the report.

Shared leadership in entrepreneurial teams: the impact of personality

PurposeThe purpose of this paper is to develop a model to better understand which personality traits and personal values impact transformational leadership qualities in self-directed entrepreneurial teams as perceived by team members.Design/methodology/approachA cohort consisting of six self-directed entrepreneurial teams was selected. A multi-rater system was applied to assess the perceived transformational leadership qualities. A model was developed, using three dimensions of transformational leadership as dependent variables: inspiring others, stimulating interaction among group members and communicating a strong vision.FindingsThe ability to inspire others was predicted by friendliness, measuring a positive labeling of social phenomena. In addition, two positive moderation effects emerged as being related to inspirational capacities: assertiveness and despondency, and assertiveness and emotional empathy interacted. The second moderation effect also impacted the capacity to stimulate group interaction. The personality traits “modesty” and the personal value “human relations” were negatively related to the perceived capacity to communicate a strong vision. Furthermore, a significant but moderate effect of team membership on the capacity to inspire others respectively stimulate interaction seems to exist.Practical implicationsThe research results offer valuable opportunities to enhance or to develop those informal, transformational leadership qualities positively influencing entrepreneurial effectiveness.Social implicationsAs entrepreneurship in small, self-directed teams is a popular phenomenon, research results add to the understanding of group interaction related to informal leadership.Originality/valueShared or informal transformational leadership in the context of self-directed, entrepreneurial teams is a relatively new phenomenon. Integrating a multi-rater assessment of leadership with personality combines interesting perspectives.

REFORMULATION AND GENERALISATION OF THE COHEN AND FLEISS KAPPAS

The assessment of consistency in the categorical or ordinal decisions made by observers or raters is an important problem especially in the medical field.  The Fleiss Kappa, Cohen Kappa and Intra-class Correlation (ICC), as commonly used for this purpose, are compared and a generalised approach to these measurements is presented.  Differences between the Fleiss Kappa and multi-rater versions of the Cohen Kappa are explained and it is shown how both may be applied to ordinal scoring with linear, quadratic or other weighting.  The relationship between quadratically weighted Fleiss and Cohen Kappa and pair-wise ICC is clarified and generalised to multi-rater assessments. The AC coefficient is considered as an alternative measure of consistency and the relevance of the Kappas and AC to measuring content validity is explored.

Three sides to a story: Child, parent, and nurse perspectives on the child's experience during hematopoietic stem cell transplantation.

The experience of children undergoing hematopoietic stem cell transplantation (HSCT), including the ways in which different participants (ie, children, parents, and nurses) contribute to the overall picture of a child's experience, is poorly characterized. This study evaluated parent, child, and nurse perspectives on the experience of children during HSCT and factors contributing to interrater differences. Participants were enrolled in a multicenter, prospective study evaluating child and parent health-related quality of life over the year after HSCT. Children (n = 165) and their parents and nurses completed the Behavioral, Affective, and Somatic Experiences Scale (BASES) at baseline (before/during conditioning), 7 days after the stem cell infusion (day+7), and 21 days after the stem cell infusion (day+21). The BASES domains included Somatic Distress, Mood Disturbance, Cooperation, and Getting Along. Higher scores indicated more distress/impairment. Repeated measures models by domain assessed differences by raters and changes over time and identified other factors associated with raters' scores. Completion rates were high (≥73% across times and raters). Multivariate models revealed significant time-rater interactions, which varied by domain. For example, parent-rated Somatic Distress scores increased from baseline to day+7 and remained elevated at day+21 (P < .001); children's scores were lower than parents' scores across time points. Nurses' baseline scores were lower than parents' baseline scores, although by day+21 they were similar. Older child age was associated with higher Somatic Distress and Mood Disturbance scores. Worse parent emotional functioning was associated with lower scores across raters and domains except for Cooperation. Multirater assessments are highly feasible during HSCT. Ratings differ by several factors; considering ratings in light of such factors may deepen our understanding of the child's experience. Cancer 2017;123:3159-66. © 2017 American Cancer Society.

Student Self-Assessment and Multisource Feedback Assessment

It has become common practice for management students to participate in some sort of self-assessment or multisource feedback assessment (MSF; also called 360-degree assessment or multirater assessment) during their management degree program. These assessments provide students invaluable feedback about themselves and assist students in their personal and professional development. This article draws on a conceptualization of the self as multifaceted to examine the benefits and limitations of self-assessment and MSF. Specifically, although an aid to students’ learning, self-assessment tools predominantly treat the self as an individual self, thereby ignoring the relational and collective aspects of the self. Moreover, self-assessments that people make tend to be inflated, unreliable, and biased. MSF improves on self-assessment by including others’ assessment of the self; however, it also possesses three limitations in its current treatment: MSF (a) conceptualizes the self as an individual self, (b) ignores the importance of context, and (c) relies primarily on a self-other rating agreement. These limitations are explored, and remedies are offered to better account for the multifaceted nature of the self.