Role-play Assessment Research Articles

The chemistry between us: Illuminating complementarity patterns in interpersonal role-play assessment via moment-to-moment analyses.

In assessment and selection, organizations often include interpersonal interactions because they provide insights into candidates' interpersonal skills. These skills are then typically assessed via one-shot, retrospective assessor ratings. Unfortunately, the assessment of interpersonal skills at such a trait-like level fails to capture the richness of how the interaction unfolds at the behavioral exchange level within a role-play assessment. This study uses the lens of interpersonal complementarity theory to advance our understanding of interpersonal dynamics in role-play assessment and their effects on assessor ratings. Ninety-six MBA students participated in four different flash role-plays as part of diagnosing their strengths and weaknesses. Apart from gathering assessor ratings and criterion measures, coders also conducted a fine-grained examination of how the behavior of the two interaction partners (i.e., MBA students and role-players) unfolded at the moment-to-moment level via the Continuous Assessment of Interpersonal Dynamics (CAID) measurement tool. In all role-plays, candidates consistently showed mutual adaptations in line with complementarity principles: Affiliative behavior led to affiliative behavior, whereas dominant behavior resulted in docile, following behavior and vice versa. For affiliation, mutual influence also occurred in that both interaction partners' temporal trends in affiliation were entrained over time. Complementarity patterns were significantly related to ratings of in situ (role-playing) assessors but not to ratings of ex situ (remote) assessors. The effect of complementarity on validity was mixed. Overall, this study highlights the importance of going beyond overall ratings to capture behavioral contingencies such as complementarity patterns in interpersonal role-play assessment. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Psychometric properties of the mock interview rating scale for schizophrenia and other serious mental illnesses.

Over the past 10 years, job interview training has emerged as an area of study among adults with schizophrenia and other serious mental illnesses who face significant challenges when navigating job interviews. The field of mental health services research has limited access to assessments of job interview skills with rigorously evaluated psychometric properties. We sought to evaluate the initial psychometric properties of a measure assessing job interview skills via role-play performance. As part of a randomized controlled trial, 90 adults with schizophrenia or other serious mental illnesses completed a job interview role-play assessment with eight items (and scored using anchors) called the mock interview rating scale (MIRS). A classical test theory analysis was conducted including confirmatory factor analyses, Rasch model analysis and calibration, and differential item functioning; along with inter-rater, internal consistency, and test-retest reliabilities. Pearson correlations were used to evaluate construct, convergent, divergent, criterion, and predictive validity by correlating the MIRS with demographic, clinical, cognitive, work history measures, and employment outcomes. Our analyses resulted in the removal of a single item (sounding honest) and yielded a unidimensional total score measurement with support for its inter-rater reliability, internal consistency, and test-retest reliability. There was initial support for the construct, convergent, criterion, and predictive validities of the MIRS, as it correlated with measures of social competence, neurocognition, valuing job interview training, and employment outcomes. Meanwhile, the lack of correlations with race, physical health, and substance abuse lent support for divergent validity. This study presents initial evidence that the seven-item version of the MIRS has acceptable psychometric properties supporting its use to assess job interview skills reliably and validly among adults with schizophrenia and other serious mental illnesses. NCT03049813.

Abstract P6-05-15: Standard of Care for HR+/HER2- Breast Cancer: Integrating The “Yale Model Shared Decision-Making Multidisciplinary Team Solution” into the Practice Setting

Abstract Background: Breast cancer patients are faced with treatment choices that can involve complex preference-sensitive decisions. The National Quality Forum initiated a “Call to Action” to integrate shared decision-making (SDM) processes into practice where clinicians and patients work together to make healthcare decisions that align with what matters most to patients. Projects In Knowledge, @Point of Care, Dartmouth and Yale collaborated to develop a pilot educational initiative to address and improve patient-centered care and SDM processes in the institutional cancer care setting. Methods: Training materials co-developed for the Yale Breast Cancer multidisciplinary team (N=11: oncologists, nurses/NPs, pharmacist) address SDM, CDK4/6 Inhibitor treatment of metastatic HR+ HER2- breast cancer, and clinician-patient role play methods implementing SDM in treatment discussions/decisions with patients. Reinforcement training, based on interim interview and case role play assessments, was customized to meet specific needs of the team. Qualitative semi-structured interviews and simulation case role play observational methods, using a two-rater system, were used to assess improved SDM performance. Baseline pre-intervention interviews and case role play assessments were compared to interim post-intervention and end of pilot (EOP) post-reinforcement training intervention interviews and case role play assessments (using a Likert scale 0-4 rating score: 0=0%; 1=25%, 2=50%; 3=75%; 4=100%). Following the training and assessments, a focus group of team members provided insights into the performance of the group, assessed the acceptability, feasibility, and repeatability of the program, and informed future education. Results: Semi-structured interview findings revealed that clinicians learned about nuances of CDK 4/6 inhibitors, crystallized their understanding of SDM through reinforcement training (customized in real time), and felt they were better able to implement SDM as a result of their case role play assessments. Training empowered the Yale Breast Cancer team to show pre- to post-education improvement in SDM case role play scenarios, ranging from 16% to 39%. Areas of greatest improvement: 1) determining decision style preference (+36%); 2) determining patients’ risk/burden tolerance (+32%); 3) determining patients’ activation, engagement, and self-efficacy (+34%), 4) determining trade-off decisions with patients (+39%), and 5) determining patients’ readiness to make a decision (+32%). Future research should explore how best to integrate SDM into the real world time restricted clinical practice. Conclusions: Educational training improved SDM skills for the multidisciplinary Yale Breast Cancer team, which can lead to improved clinician-patient decision-making and patient-centric care. The training process also facilitated team building and encouraged ongoing participation in SDM. Overall Yale Breast Cancer SDM Pilot Case Study Role-Play Assessments (Data reflect findings for 11 participants who completed their case role play) Citation Format: Elaine Rudell, Tarjani Agrawal, Patty Peterson, Michele Fallon Ingram, Brant Oliver, Kerin Adelson. Standard of Care for HR+/HER2- Breast Cancer: Integrating The “Yale Model Shared Decision-Making Multidisciplinary Team Solution” into the Practice Setting [abstract]. In: Proceedings of the 2022 San Antonio Breast Cancer Symposium; 2022 Dec 6-10; San Antonio, TX. Philadelphia (PA): AACR; Cancer Res 2023;83(5 Suppl):Abstract nr P6-05-15.

Grammar as validity evidence for assessing L2 interactional competence

Abstract This study examines how L2 learners of English at different pragmatic performance levels implement requests during role-play assessment interaction. In doing so, the role of grammar-for-interaction as validity evidence of assessing interactional competence is explicated. Using qualitative and quantitative data, the following research questions are investigated: (a) How do learners use grammar differently depending on the role-play request situations they are engaged in?; (b) To what extent are learners’ role-play performance levels associated with the ability to use context-sensitive grammar when co-constructing request sequences? The data come from a database of 102 L2 English learners’ role-play interactions with 45 hours of recorded interaction in total. The role-play assessment tasks include various real-life communicative situations, such as requests designed for specific interlocutors in a university context. Using conversation analysis, the role-play interactions at different performance levels were analyzed. Sequential analyses showed that higher-level learners utilized diverse grammatical formats that are sensitive to their sequential positions and contingencies associated with requests. As quantitative evidence, the relationship between selected grammatical constructions and the learners’ role-play performance levels determined by trained raters was examined. With this, I discuss the role of grammar-for-interaction in defining and assessing interactional competence.

Community implementation of contingency management to address stimulant use

IntroductionContingency management (CM) is efficacious for reinforcing stimulant abstinence, and technical assistance (TA) is increasingly sought to aid its community-based implementation. In an interagency partnership involving a sponsoring single-state authority and statewide treatment agency in Oregon, an intermediary purveyor organization provided a robust TA package to support design, implementation, and evaluation of CM programming for an opioid treatment program (OTP) over the course of a 12-month implementation service project. MethodsIn addition to an online training offering OTP leaders and staff conceptual foundation for CM, the TA package included purveyor-led activities to: 1) engage leaders in collaborative design to customize CM programming; 2) assemble a local implementation team to logistically prepare OTP systems for CM delivery; 3) provide virtual coaching-to-criterion to assure readiness of counseling staff to deliver CM programming; 4) compile a tailored CM resource library of implementation support materials; and 5) avail ongoing consultation during implementation. Stimulant abstinence was targeted via a voucher-based protocol with escalating reinforcement, for which gift cards from local vendors served as reinforcers. Virtual coaching eventuated in individual role-play assessments, wherein staff delivery of CM programming with a standardized patient was scored via Likert scale (1 = Very Poor, 7 = Excellent) on six CM fidelity dimensions. This observational cohort design subsequently assessed clinical effectiveness during active implementation via OTP records review for CM-exposed and comparison client groups. ResultsIn role-play assessments, all counseling staff exceeded an a priori fidelity benchmark signifying implementation readiness (M = 31.33, SD = 3.72). Among 73 clients enrolled in the CM programming, rate of stimulant-free urine drug screens was 11 % greater than among 120 clients serving as historical controls (p < .01; Cohen's D = 0.40). The study also identified secondary therapeutic benefit in six-month treatment retention, with clients enrolled in CM retained at a 14 % greater rate than 162 CM-ineligible clients concurrently enrolled in OTP services (p < .05). ConclusionsFindings from this interagency partnership offer reason for optimism regarding community-based implementation. Beyond the demonstrated empirical support for this TA package and resulting clinical effectiveness of the CM programming, an eventual sustainment decision by OTP leadership strengthens the rationale for customizing CM to clinical settings' local needs and resources.

Involving People with Lived Experiences in Role-Play Assessment in Social Work Education in England: Implications for Practice

The involvement of people with lived-experience in social work education in England is mandated by the social work regulator. Our colleagues from the ‘SUCI Group—Supporting Social Work Education with Lived Experience’, increasingly contribute to assessment processes in our social work programmes, as well as to other aspects of the curriculum. For some years our SUCI colleagues, as they are known, have supported the assessment process in the preparation-to-practice module delivered at our university through the medium of live role-plays; now, due to changes in delivery brought about by Covid-19, these assessments are conducted virtually as part of the summative procedures. In this article we highlight the need to implement a robust assessment to confirm students possess sufficient practice skills to progress to their initial placement. We explore how we manage this activity through a simulated role-play to ensure a safe experience for both students and our SUCI colleagues. We conclude by affirming that receiving feedback from people with lived-experience should be normalised for social workers throughout their professional development. Social work education; expertise-by-experience; role-play assessment; service participant involvement; inclusion

Role-Play Assessment of Therapist Adherence and Skill in Implementation of Trauma-Focused Cognitive-Behavioral Therapy.

Numerous efforts are underway to train clinicians in evidence-based practices. Unfortunately, the field has few practical measures of therapist adherence and skill with which to judge the success of these training and implementation efforts. One possible assessment method is using behavioral rehearsal, or role-play, as an analogue for therapist in-session behavior. The current study describes aspects of reliability, validity and utility of a behavioral role-play assessment developed to evaluate therapist adherence and skill in implementing Trauma-Focused Cognitive-Behavioral Therapy (TF-CBT). TF-CBT role-play assessments were conducted with a sample of 43 therapists as part of a larger training study. The TF-CBT role-play assessments were independently coded for TF-CBT adherence and skill by a certified TF-CBT trainer and three clinical psychology doctoral students. Findings indicated good interrater reliability for the individual items (ICC: M = .71, SD = .15). Regarding utility, 67.19% (n = 43/64) of contacted therapists completed the role-play assessment, which took an average of 30min (M = 31.42, SD = 5.65) to complete and 60min (M = 62.84, SD = 11.31) to code. Therapists with a master's degree were more likely to complete the role-play assessment than those with other degrees but no other differences in demographic variables, practice characteristics, or TF-CBT knowledge or training were found between participants and nonparticipants. Role-play assessments may offer an alternative to observational coding for assessing therapist adherence and skill, particularly in contexts where session recordings are not feasible.

Simulated Patient Role-Plays with Consumers with Lived Experience of Mental Illness Post-Mental Health First Aid Training: Interrater and Test Re-Test Reliability of an Observed Behavioral Assessment Rubric.

Mental Health First Aid (MHFA) training teaches participants how to assist people experiencing mental health problems and crises. Observed behavioral assessments, post-training, are lacking, and the literature largely focuses on self-reported measurement of behaviors and confidence. This study explores the reliability of an observed behavioral assessment rubric used to assess pharmacy students during simulated patient (SP) role-play assessments with mental health consumers. Post-MHFA training, pharmacy students (n = 528) participated in SP role-play assessments (n = 96) of six mental health cases enacted by consumers with lived experience of mental illness. Each assessment was marked by the tutor, participating student, and consumer (three raters). Non-parametric tests were used to compare raters’ means scores and pass/fail categories. Interrater reliability analyses were conducted for overall scores, as well as pass/fail categories using intra-class correlation coefficient (ICC) and Fleiss’ Kappa, respectively. Test re-test reliability analyses were conducted using Pearson’s correlation. For interrater reliability analyses, the intra-class correlation coefficient varied from poor-to-good to moderate-to-excellent for individual cases but was moderate-to-excellent for combined cases (0.70; CI 0.58–0.80). Fleiss’ Kappa varied across cases but was fair-to-good for combined cases (0.57, p < 0.001). For test re-test reliability analyses, Pearson’s correlation was strong for individual and combined cases (0.87; p < 0.001). Recommended modifications to the rubric, including the addition of barrier items, scoring guides, and specific examples, as well as the creation of new case-specific rubric versions, may improve reliability. The rubric can be used to facilitate the measurement of actual, observed behaviors post-MHFA training in pharmacy and other health care curricula.

Effects of Online Training on Educators' Knowledge and Use of System of Least Prompts to Support Augmentative and Alternative Communication.

The present study evaluated the effects of online training on educators' knowledge and use of system of least prompts (SLP) to support the communication of individuals with autism (ASD) who use augmentative and alternative communication (AAC). A pre/posttest group design with switching replications was utilized. Three dependent variables: (a) knowledge of SLP, (b) planning for SLP, and (c) performance of SLP implementation during teleconference role-plays, were analyzed using a two-way mixed analysis of variance (ANOVA). Results indicate that training was effective in increasing educators' knowledge as well as the ability to plan for SLP. Implementation of SLP also increased over time, however, data analysis did not support the conclusion that this result was directly related to the online training. Additional results suggest that the participants viewed online instruction, with teleconference role-play assessments and case study vignettes, as a socially valid activity. Participants reported learning a valuable skill and as a result of the training were more confident working with individuals who use AAC. Implications, limitations, and directions for future research are also discussed.

Survival analysis and quality of life of pediatric low grade glioma treated by postoperative radiotherapy

Objective To analyze the clinical characteristics of children with low grade glioma (LGG) and evaluate their survival status and quality of life in LGG pediatric patients after postoperative radiotherapy. Methods Clinical data of 27 LGG children aged≤14 years admitted to Department of Radiation Oncology of our hospital from January 2011 to December 2017 were retrospectively analyzed. The 5-year overall survival (OS) and progression-free survival (PFS) were estimated with the Kaplan-Meier method.The quality of life was evaluated by body mass index (BMI), WeeFIM and enrollment situation. Results The 5-year OS and PFS of 27 pediatric patients were 92% and 87%. BMI was measured in 25 children who were still alive and the results showed that 1 child was slightly emaciated, 3 were overweight and 21 were normal. The total score of WeeFIM was> 108 in 21 cases and< 90 in 4 cases. In terms of social role-playing assessment, 5 cases delayed enrollment for≥ 2 years, 2 cases had good adaptation except the difficulties due to lameness, 7 of them had poor study performance, but had no communication barrier. Conclusions Children with LGG can obtain relatively excellent OS and PFS after postoperative radiotherapy. Part of the children with long-term survival may experience a declined quality of life, which is mainly manifested with a decline in motor and cognitive functions. Key words: Pediatric low grade glioma/radiotherapy; Survival; Quality of life

Role Play: Let’s Act It Out!

Generally, most university students are reluctant to speak English in public due to lack of Englishproficiency. This situation has led to lost interest in role-play activity among UiTM Cawangan Kelantanstudents; particularly Integrated Language Skills 1 (ELC121) students. Hence, this project intends toincrease students’ interest in role-play activity as well as enhance students’ proficiency in Englishlanguage especially in role-play assessment. This project focuses on using Interactive DVD as a tool tohelp in motivating the student’s interest and enhancing student’s proficiency level. This interactive DVD iscomprised of notes, samples of dialogues and video presentations meant for teaching and learning roleplay.Based on the outcomes, the students show positive enthusiasms in joining the class and performoutstandingly in role-play test. The Interactive DVD motivates them to actively participate in role playactivity with confidence.

Evaluation of a Level I trauma center provider training in patient-centered alcohol brief interventions using the Behavior Change Counseling Index rated by standardized patients

BackgroundTraumatic injury requiring hospitalization is common in the USA and frequently related to alcohol consumption. The American College of Surgeons requires that Level I and II verified trauma centers implement...

Structured External Memory Aid Treatment for Mild Cognitive Impairment.

Purpose The aims of this study were (a) to examine the effects of a structured external memory aid (EMA) treatment for individuals with possible mild neurocognitive disorder on their use of EMAs to compensate for memory impairments, (b) to examine the extent that improved functional EMA use maintains following intervention, and (c) to measure the effects of treatment on participants' self-reported memory skills. Method The researchers employed a multiple-baseline design across subjects within dyads. Data were collected using the Roleplay Assessment. The graphed data were analyzed visually at baseline, treatment, and maintenance sessions. In addition, the researchers examined the participants' self-report of memory skills through an experimental pre-post design. Results Following intervention, a functional strategy use treatment effect was observed. All 6 participants' graphs revealed a large positive change in level, with no overlapping data. Statistical analyses also supported the positive effects of treatment. The participants maintained the use of strategies for 1-3 months. The participants self-reported a statistically significant increase in strategy use following intervention. Conclusions With treatment, individuals with mild memory impairments can learn to use EMAs to facilitate enhanced performance on functional activities. This study provides evidence for the importance of documenting individual treatment effects and using a functional measure to examine compensatory strategy use in everyday tasks.

Managing proposal sequences in role-play assessment: Validity evidence of interactional competence across levels

This qualitative study reports an investigation of the nature of interactional competence at various levels of achievement in the context of role-play speaking assessment. The focal point of this study is on how examinees jointly accomplish the interactional work involved in proposal sequences in role-play interaction. Based on a conversation analysis of a corpus of role-play interaction, I argue that distinct sequential organizations and interactional features found across examinees’ levels serve as critical validity evidence for assessing interactional competence. Various shift markers and stepwise transitions were present in higher-level examinees when they initiated and shifted actions in role-play interaction. However, lower-level examinees’ opening turns were typically forwarded without establishing a shared understanding relevant to an upcoming action. When the examinees responded to various proposal sequences, coherent and sufficient topic organizations were recurrent in higher-level performances. The examinees, regardless of levels, managed to close the role-play interaction well. I discuss the implications of the demonstrated link between the recurrent interactional features and examinees’ interactional competence for future research into speaking assessment and teaching.

発達障害児の保護者向け機能的アサーション・トレーニング : 相互作用を強調したロールプレイ・アセスメントによる追試的検討

発達障害児の保護者向け機能的アサーション・トレーニング : 相互作用を強調したロールプレイ・アセスメントによる追試的検討

A role-play assessment tool and drama-based social skills intervention for adults with autism or related social communication difficulties

This study adds to a small literature on social skills measures and interventions for adults with autism spectrum disorder (ASD) or related social communication difficulties (SCD) without intellectual disability (ID). In study 1, a new multimodal assessment tool, the role-play assessment of social skills (R-PASS), was used to measure real-time application of social skills. The scores of adults with ASD/SCD were marginally lower than those of neurotypical adults, with a large effect size, suggesting that the measure can identify differences between the two groups. Therefore, the R-PASS shows potential as an objective tool to assess dynamic and naturalistic social skills. In Study 2, a pre–post single-group design study, we measured the effectiveness of a drama-based social skills intervention for seven participants who self-identified as having ASD/SCD. The R-PASS was used by external raters blind to diagnosis and intervention status to compare the performance of intervention participants to that of neurotypical adults. R-PASS scores suggested substantial improvement of social skills in the majority of participants post-intervention. Furthermore, relatives’ and participants’ perception of their social communication and self-regulation skills improved from pre- to post-intervention. These results suggest that the intervention may have helped the participants improve their social skills.

Repetitions as communication strategies in L2 oral interactions

The implementation of communicative language teaching (CLT) approach in early 1970s has greatly impacted second language (L2) learning worldwide, including Malaysia. Since this approach mainly focuses on the conveyance of a message, CLT students are expected to use various communication strategies (CSs) in getting their message across. In this regard, past literature indicates that repetitions play various roles in oral communication. Hence, they became the focus of the current study. The data were elicited from role-play assessments in English classes involving six selected pairs of students. The analysis of the transcribed oral data revealed that, aside from being employed as time-gaining strategies, repetitions also occurred as a means to provide affirmation, to ask for confirmation and to emphasise a point in oral communication which indicates their role in enhancing effectiveness in communication as posited by Clennell. Consequently, Dörnyei and Scotts' taxanomy of communication is proposed to be revised.

Repetitions as communication strategies in L2 oral interactions

The implementation of communicative language teaching (CLT) approach in early 1970s has greatly impacted second language (L2) learning worldwide, including Malaysia. Since this approach mainly focuses on the conveyance of a message, CLT students are expected to use various communication strategies (CSs) in getting their message across. In this regard, past literature indicates that repetitions play various roles in oral communication. Hence, they became the focus of the current study. The data were elicited from role-play assessments in English classes involving six selected pairs of students. The analysis of the transcribed oral data revealed that, aside from being employed as time-gaining strategies, repetitions also occurred as a means to provide affirmation, to ask for confirmation and to emphasise a point in oral communication which indicates their role in enhancing effectiveness in communication as posited by Clennell. Consequently, Dörnyei and Scotts' taxanomy of communication is proposed to be revised.

“Are You Into Beer Pong?” Exploring Question-Answer Sequences in an L2 Oral Performance Assessment

ABSTRACTDespite the frequent occurrence of yes/no questions in naturalistic talk-in-interaction, there is widespread concern that yes/no questions only allow predetermined answers and limit L2 learners’ performance in language assessment contexts. Drawing upon audio-recordings of role-play conversations between an L1 American English speaker and 104 L2 English learners, we use conversation analysis to examine how L2 learners respond to yes/no questions in an oral performance assessment. The analysis was guided by the following research questions: (a) How do L2 English learners respond to a yes/no interrogative in a role-play assessment context? (b) How do L2 English learners’ different responses lead to distinct sequences of conversational turns? In our analysis we show that yes/no interrogatives elicit a range of responses (e.g., repair initiation, type-conforming responses, non-type-conforming responses) that result in negotiation of meaning between participants. While turn design and sequence organization features we observed largely align with the existing literature on naturally occurring conversations, we also discuss deviant cases that provide insights into L2 learner discourse. Moreover, we discuss the findings in terms of implications for L2 assessment.

Role-play assessments as a measure of parenting skills.

Various forms of assessments of parenting skill are used across studies including parent self-report, observations of parent and child interactions, and role-play assessments of parenting skill. Role-play assessments, though used in several studies, have not been sufficiently evaluated so as to unde