Situational judgement tests represent a measurement method and can be designed to minimise coaching effects

Abstract

Editor – Coaching is controversial in all selection contexts and we thank Rostom et al.1 for drawing attention to this because research in this area is lacking. Our review2 emphasises that situational judgement tests (SJTs) are a measurement method – in which there is no single approach to design – and that each SJT should be evaluated individually regarding issues relating to coaching, validity, fairness, and so on. We highlight three key areas to inform discussion. Situational judgement tests can be designed to be less susceptible to coaching by tailoring their content and the response formats used and instructions given. The study by Lievens et al.3 of an SJT for Belgian medical school admissions is welcome as a propensity scoring analysis resolves problems associated with self-selection in coaching studies. Here, results showed coaching effects in the form of a 0.5 standard deviation improvement; however, the SJT design specification differed significantly from others. For example, the UK Foundation Programme4 SJT uses a more complex response format that employs two types of response (‘Rank all five possible responses in order’ and ‘Choose the three best responses from a list of eight’) that are significantly more cognitively loaded than those used in the response format in the Belgian test (‘Pick the best response from a choice of four’). Using complex, cognitively loaded formats is likely to ‘make SJTs less prone to coaching effects’.3 Situational judgement tests of more heterogeneous content (i.e. capturing a variety of domains) are less susceptible to coaching as they increase ‘the range and specificity of strategies that must be learned and memorised by trainees’.3 The Belgian SJT targeted two domains (building relationships and communicating information), using one response format, across 30 items. By contrast, the UKFP SJT targets five domains, using two response formats, across 70 items.4 Other steps to reduce coaching effects include ensuring complexity is built into scenarios so that the candidate must engage with the question rather than employing a simple response strategy, and maintaining a large item bank in which items are continually renewed. Research also shows that SJTs are less susceptible to coaching effects when they are constructed from experts’ judgements or empirical keying rather than rules.5 The UKFP SJT also adopts a cognitively oriented response instruction (i.e. ‘What should you do?’) rather than a behaviourally oriented format (i.e. ‘What would you do?’). Cognitively oriented instructions are less susceptible to self-deception and impression management6,7 and therefore to coaching. Research must examine whether SJT coaching produces genuine or artificial effects. Formal education and training in important domains (such as communication) could and, indeed, should be beneficial to the development of learners over the longer term. By contrast, coaching is usually short-lived and is geared towards test-taking strategies. External providers (usually commercial) provide tips that are specifically intended to help a candidate to ‘pass’ the SJT, rather than to facilitate detailed understanding about what constitutes effective behaviour in job-relevant situations. However, research is required to explore these differential effects in SJTs (skills development versus test-taking strategies) and to examine whether coaching is actually linked to training and job outcomes. Situational judgement tests measure understanding of effective behaviour in a given situation, in relation to non-academic attributes, such as empathy, integrity and teamwork, depending on the test specification. These tests do not measure personality traits per se; they measure implicit trait policies and general experience (and, depending on level, specific job knowledge).8 Implicit trait policies are beliefs about the costs and benefits of expressing certain traits, such as knowing that being agreeable is likely to be better in many situations. Higher-order interactions between individual differences in cognition, intellect, personality and affect, and links to training outcomes and job performance, represent an exciting area for SJT research.