Video-based Situational Judgment Test Research Articles

Performance on Video-Based Situational Judgment Test Items: Simulated Interracial Interactions

Individuals are known to categorize others into social groups based on cues like race and gender and to experience relative discomfort when interacting with “outgroup” members. Two experimental studies were used to examine whether actor demographic cues in situational judgment assessment items completed by test takers in a simulated employee selection context may lead to differences in their performance and reactions to the hiring organization. In both studies, test takers assumed the perspective of actors shown in video-based scenarios and indicated how they would respond to interaction partners (IPs) to whom they were racially similar or dissimilar. In Study 1, a given test taker responded to IPs of a constant gender; in Study 2, IPs’ gender varied across scenarios within each condition. In Study 1, Black test takers spent more time and scored better on two of the four scenarios when responding to racially similar IPs. These effects were not found in Study 2, but demographic cues showed new interactive effects on performance and reactions. We discuss the implications of different findings across the two studies.

The Predictive Validity of a Text-Based Situational Judgment Test in Undergraduate Medical and Dental School Admissions.

Situational judgment tests (SJTs) can be used to assess the nonacademic attributes necessary for medical and dental trainees to become successful practitioners. Evidence for SJTs' predictive validity, however, relates predominantly to selection in postgraduate settings or using video-based SJTs at the undergraduate level; it may not be directly transferable to text-based SJTs in undergraduate medical and dental school selection. This preliminary study aimed to address these gaps by assessing the validity of the UK Clinical Aptitude Test (UKCAT) text-based SJT. Study participants were 218 first-year medical and dental students from four UK undergraduate schools who completed the first UKCAT text-based SJT in 2013. Outcome measures were educational supervisor ratings of in-role performance in problem-based learning tutorial sessions-mean rating across the three domains measured by the SJT (integrity, perspective taking, and team involvement) and an overall judgment of performance-collected in 2015. There were significant correlations between SJT scores and both mean supervisor ratings (uncorrected r = 0.24, P < .001; corrected r = 0.34) and overall judgments (uncorrected rs = 0.16, P < .05; corrected rs = 0.20). SJT scores predicted 6% of variance in mean supervisor ratings across the three nonacademic domains. The results provide evidence that a well-designed text-based SJT can be appropriately integrated, and add value to, the selection process for undergraduate medical and dental school. More evidence is needed regarding the longitudinal predictive validity of SJTs throughout medical and dental training pathways, with appropriate outcome criteria.

Occupation-specific social competences in vocational education and training (VET): the example of a technology-based assessment

The difficulties regarding the definition and measurement of social competences, crucial to the research in vocational education and training, are well known. In this paper, we show, based on the example of medical assistants in Germany, that these skills have to be assessed in an occupation-specific way, and develop an appropriate simulation-based Situational Judgment Test. Competence modeling, the translation into the test format and results of a data collection are discussed in detail for the dimension of communicative strategies. Based on empirical and theoretical analysis, we derived a normative competence model underlying the development of a video-based situational judgment test. The final assessment contained 2 × 6 prototypical interactional types of situations, presented in short videos. At the end of every scene, questions of multiple choice format or open questions were asked for each of the social competence dimensions (emotion regulation, perspective coordination, communication strategies). In regard to the evaluation, for the dimension of communication strategies, for example, open answers were rated by using a weighted checklist giving a final score for each situation or all over the test. After two pre-evaluations on N = 236 and N = 260 pupils at the end of their apprenticeship towards becoming medical assistants, validity aspects due to e.g. format were analyzed and edited. A final inquiry on N = 405 pupils confirmed the expected reliability and dimensionality aspects. Validity measures could additionally be confirmed by a structural equation model (N = 405, Χ2 = 594.319, df = 518, CFI = .96, RMSEA = .02 and SRMR = .07) showing unexpected but meaningful underlying factorial structures for the dimensions of emotion regulation and communication strategies. Results of the test show that emotional dissonance is present in every situation. Regarding test-scores, high levels are rarely achieved for all dimensions. By using an occupation-specific approach, we show that social competences can be defined and performance-relatedly assessed. However, results show that the competence levels are marginally satisfactory for all dimensions, considering the stressful environment of medical assistants. Hence, contents of this nature should be integrated more concretely in their educational pathway.

Assessing Perceptions of Interpersonal Behavior with a Video-Based Situational Judgment Test

Accurate appraisal of others’ behavior is critical for the production of skilled interpersonal behavior. We used an ecologically valid methodology, a video-based situational judgment test with true-false items, to assess the accuracy with which students (N = 947) perceive the interpersonal behavior of actors involved in workplace situations. Students also completed personality and cognitive measures, allowing for an examination of the individual difference correlates of behavioral perception skill. Results indicated that behavioral perception skill correlates strongly with mental ability, correlates weakly with openness to experience, and is unrelated to conscientiousness, extraversion, neuroticism, and agreeableness.

The validity of interpersonal skills assessment via situational judgment tests for predicting academic success and job performance.

This study provides conceptual and empirical arguments why an assessment of applicants' procedural knowledge about interpersonal behavior via a video-based situational judgment test might be valid for academic and postacademic success criteria. Four cohorts of medical students (N = 723) were followed from admission to employment. Procedural knowledge about interpersonal behavior at the time of admission was valid for both internship performance (7 years later) and job performance (9 years later) and showed incremental validity over cognitive factors. Mediation analyses supported the conceptual link between procedural knowledge about interpersonal behavior, translating that knowledge into actual interpersonal behavior in internships, and showing that behavior on the job. Implications for theory and practice are discussed.

De webcamtest als voorspeller van professioneel gedrag

The webcam test as a predictor of professional behavior The webcam test as a predictor of professional behavior Gedrag &amp; Organisatie, volume 24, September 2011, nr 3, pp. 257-285.The webcam test is a new video assessment designed to measure social competencies. The main purpose of this study was to investigate the criterion-related validity of the webcam test and its incremental validity over and above a cognitive ability test, a personality questionnaire and a video-based situational judgment test. The sample consisted of 106 psychology students. In line with our expectations, the webcam test showed a significant correlation with the criterion professional behavior, a questionnaire about students’ motivation, chairmanship and preparation. Furthermore, the webcam test showed incremental validity over and above the other selection tests. Participants perceived the webcamtest as more face valid than the cognitive ability test and the personality questionnaire. However, the video-based situational judgment test was perceived as more face valid than the webcamtest. Limitations of this study and suggestions for further study are discussed.

SITUATIONAL JUDGMENT TESTS: CONSTRUCTS ASSESSED AND A META-ANALYSIS OF THEIR CRITERION-RELATED VALIDITIES

Situational judgment tests (SJTs) are a measurement method that may be designed to assess a variety of constructs. Nevertheless, many studies fail to report the constructs measured by the situational judgment tests in the extant literature. Consequently, a construct-level focus in the situational judgment test literature is lacking, and researchers and practitioners know little about the specific constructs typically measured. Our objective was to extend the efforts of previous researchers (e.g., McDaniel, Hartman, Whetzel, & Grubb, 2007; McDaniel & Ngyuen, 2001; Schmitt & Chan, 2006) by highlighting the need for a construct focus in situational judgment test research. We identified and classified the construct domains assessed by situational judgment tests in the literature into a content-based typology. We then conducted a meta-analysis to determine the criterion-related validity of each construct domain and to test for moderators. We found that situational judgment tests most often assess leadership and interpersonal skills and those situational judgment tests measuring teamwork skills and leadership have relatively high validities for overall job performance. Although based on a small number of studies, we found evidence that (a) matching the predictor constructs with criterion facets improved criterion-related validity; and (b) video-based situational judgment tests tended to have stronger criterion-related validity than pencil-and-paper situational judgment tests, holding constructs constant. Implications for practice and research are discussed.

Construct-Driven Development of a Video-Based Situational Judgment Test for Integrity

In a field study conducted in a multi-ethnic selection setting at the Dutch police, we examined the construct validity of a video-based situational judgment test (SJT) aimed to measure the construct of integrity. Integrity is of central importance to productive work performance of police officers. We used a sample of police applicants, which consisted of a Dutch ethnic majority group and an ethnic minority group. The ethnic minority applicants came from one of the four largest ethnic minority groups in The Netherlands, namely groups with a Dutch Antillean, a Moroccan, a Surinamese, or a Turkish background. A critical issue is the often-found construct-heterogeneity of SJTs. However, we found that a construct-driven approach may be fruitful in the development of SJTs aiming to measure one single construct. Confirming our expectations, we found support for the construct validity of the SJT intended to measure the construct of integrity. These results held across ethnic majority and ethnic minority applicants. Therefore, the SJT is a promising test for personnel selection in a multi-ethnic setting.

Video-based versus written situational judgment tests: A comparison in terms of predictive validity.

In this study, the authors examined whether video-based situational judgment tests (SJTs) have higher predictive validity than written SJTs (keeping verbal content constant). The samples consisted of 1,159 students who completed a video-based version of an SJT and 1,750 students who completed the same SJT in a written format. The study was conducted in a high stakes testing context. The video-based version of an interpersonally oriented SJT had a lower correlation with cognitive ability than did the written version. It also had higher predictive and incremental validity for predicting interpersonally oriented criteria than did the written version. In this high stakes context, applicants also reacted relatively favorably to the SJTs, although there was no significant difference in face validity between the formats. These findings suggest that SJT format changes be made with caution and that validation evidence is needed when changes are proposed.

The Operational Validity of a Video-Based Situational Judgment Test for Medical College Admissions: Illustrating the Importance of Matching Predictor and Criterion Construct Domains.

This study is part of a trend of examining noncognitive predictors, for example, a situational judgment test (SJT), as supplements to cognitive predictors for making college admission decisions. The authors examined criterion data over multiple academic years and universities. The criterion domain was broadly conceptualized, including both cognitive and interpersonal domains. The sample consisted of 7,197 candidates of the Medical and Dental Studies Admission Exam in Belgium. Results confirmed the importance of cognitive predictors. A video-based SJT was differentially valid for predicting overall grade point average for different curricula. The SJT showed incremental validity over cognitively oriented measures for curricula that included interpersonal courses, but not for other curricula. The SJT became more valid through the years. This demonstrates the importance of carefully specifying predictor- criterion linkages and of differentiating both predictor and criterion constructs.

Predicting insurance agent turnover using a video-based situational judgment test

High turnover of newly contracted insurance agents is a wide-spread problem faced by many insurance companies. The objective of the present study was to examine the usefulness of a video-based selection test in the prediction of turnover for insurance agents. Newly contracted insurance agents (N=677) responded to items presented on a video-based situational judgment test. Each item presented a videotaped situation showing an agent interacting with client. Following the situation, the agents responded to a multiple-choice question indicating how they would handle the situation. Turnover information was gathered on the agents one year after contract. Using an empirical scoring method, data analyses indicated a significant relationship between test scores and the turnover criterion.