Test-taker Reactions Research Articles

Simulated job applicant test-taker reactions to rejection: comparing absolute and social-comparative feedback

Simulated job applicant test-taker reactions to rejection: comparing absolute and social-comparative feedback

Game on! Exploring reactions to game-based selection assessments

PurposeThe use of game-based assessments (GBAs) is growing in selection contexts, yet test-takers have varying reactions to such assessments, which have important implications for applicant behavior. This paper reviews the literature on applicant reactions and explores classic assessment models in the context of GBAs, identifying best practice recommendations and pitfalls for enhancing the candidate experience.Design/methodology/approachA sample of 374 participants from MTurk completed cognitive GBAs and questionnaires regarding test-taker reactions (job-relatedness, perceived opportunity to perform, provision of selection information, face validity, task engagement, task motivation and willingness to refer others to the company), technology self-efficacy, and game/technology experience.FindingsFairness mediated the relationship between procedural justice rules and willingness to recommend the company to others. Technology self-efficacy was significantly related to fairness perceptions and procedural justice perceptions. Males had significantly higher procedural justice perceptions of GBAs than females.Research limitations/implicationsThe study underscores the importance of considering fairness perceptions and individual differences in reactions to GBAs. Future research should study participants within high-stakes hiring situations and examine other individual difference factors such as ethnicity.Practical implicationsGBAs are a viable assessment method for personnel selection, yet organizations must recognize that individuals are more likely to respond positively to GBAs if they perceive such assessments as fair and job-related, and perceive themselves as capable of performing well on the assessment.Originality/valueThis study tests a classic model of procedural fairness in a novel and timely assessment context.

Test-taker reactions to ideal point measures of personality

Ideal point measures of personality, which include moderately worded items, have gained attention from researchers and assessment practitioners due to their potential for more accurately assessing personality relative to traditional dominance measures, which contain only positive and negative items. However, no research of which we are aware has investigated applicant reactions to this new measure type. Given the well-known influence of reactions on applicant and organizational outcomes, knowledge of reactions to ideal point measures will be critical to decisions regarding their use in organizations. The present study compares test-taker reactions to ideal point and dominance personality measures when taken as part of a research study and in a simulated selection context. We hypothesize that test-takers completing the ideal point measure will perceive lower accuracy and higher difficulty relative to a dominance measure, particularly in a simulated applicant condition. We then propose a conditional indirect effect of measure type on overall affective reactions via perceived difficulty and accuracy such that the indirect effect is stronger in the simulated applicant condition. The current study uses a mixed-methods approach. First, in an experimental, factorial design, we manipulate the factors of (a) measure type (dominance vs. ideal point); and (b) instruction set (honest vs. simulated applicant). Results support the hypotheses. Then, we investigate how test-takers’ qualitative explanations of reactions clarify findings. Results reveal that lower perceived accuracy and greater perceived difficulty may be driven by test-takers’ perceptions that responses to ideal point items have ambiguous interpretations. Implications for assessment and personnel selection are discussed.

New frontiers in cognitive ability testing: working memory

PurposeCognitive ability tests demonstrate strong relationships with job performance, but have several limitations; notably, subgroup differences based on race/ethnicity. As an alternative, the purpose of this paper is to develop a working memory assessment for personnel selection contexts.Design/methodology/approachThe authors describe the development of Global Adaptive Memory Evaluation (G.A.M.E.) – a working memory assessment – along with three studies focused on refining and validating G.A.M.E., including examining test-taker reactions, reliability, subgroup differences, construct and criterion-related validity, and measurement equivalence across computer and mobile devices.FindingsEvidence suggests that G.A.M.E. is a reliable and valid tool for employee selection. G.A.M.E. exhibited convergent validity with other cognitive assessments, predicted job performance, yielded smaller subgroup differences than traditional cognitive ability tests, was engaging for test-takers, and upheld equivalent measurement across computers and mobile devices.Research limitations/implicationsAdditional research is needed on the use of working memory assessments as an alternative to traditional cognitive ability testing, including its advantages and disadvantages, relative to other constructs and methods.Practical implicationsThe findings illustrate working memory’s potential as an alternative to traditional cognitive ability assessments and highlight the need for cognitive ability tests that rely on modern theories of intelligence and leverage burgeoning mobile technology.Originality/valueThis paper highlights an alternative to traditional cognitive ability tests, namely, working memory assessments, and demonstrates how to design reliable, valid, engaging and mobile-compatible versions.

Test-taker perception of and test performance on computer-delivered speaking tests: the mediational role of test-taking motivation

BackgroundResearch on the test-taker perception of assessments has been conducted under the assumption that negative test-taker perception may influence test performance by decreasing test-taking motivation. This assumption, however, has not been verified in the field of language testing. Building on expectancy-value theory, this study explored the relationships between test-taker perception, test-taking motivation, and test performance in the context of a computer-delivered speaking test.MethodsSixty-four Japanese university students took the TOEIC Speaking test and completed a questionnaire that included statements about their test perception, test-taking motivation, and self-perceived test performance. Five students participated in follow-up interviews.ResultsQuestionnaire results showed that students regarded the TOEIC Speaking test positively in terms of test validity but showed reservations about computer delivery, and that they felt sufficiently motivated during the test. Interview results revealed various reasons for their reservations about computer delivery and factors that distracted them during the test. According to correlation analysis, the effects of test-taker perception and test-taking motivation seemed to be minimal on test performance, and participants’ perception of computer delivery was directly related to test-taking effort, but their perception of test validity seemed to be related to test-taking effort only indirectly through the mediation of perceived test importance.ConclusionOur findings not only provide empirical evidence for the relationship between test-taker perception and test performance but also highlight the importance of considering test-taker reactions in developing tests.

Using pre-test explanations to improve test-taker reactions: Testing a set of “wise” interventions

The present research tested a set of “wise” interventions (Walton, 2014) designed to improve employee reactions to assessment tests. Drawing upon theories of test-taking reactions, fairness, and social exchange, we generated and pilot-tested pre-test explanations to facilitate positive reactions to the assessments. Across two experimental studies of working adults, we tested a control condition and four experimental groups: (1) an informational fairness condition, (2) a social fairness condition, (3) an uncertainty reduction condition, and (4) a combined condition. In the first study, 256 retail employees were randomly assigned to one of the pre-test explanation conditions before completing a work sample test. Findings indicated higher perceptions of fairness for test-takers in the combined explanation group. In addition, the effects of the test explanations depended upon two contextual variables: test-takers’ level of perceived organizational support and the quality of leader-member exchange relationships with their supervisors. In the second study, the mechanisms underlying pre-test explanations were examined using an online sample of 269 working adults. Consistent with our conceptual framework, findings demonstrated that pre-test explanations had direct effects on transparency, respect, and reassurance. Taken together, these findings have implications for understanding the effects of pre-test explanations in organizational settings as well as the boundary conditions for their use.

Impact of electronic warnings on online personality scores and test-taker reactions in an applicant simulation

This study investigates the impact of different types of electronic warnings during a simulated job application assessment. Results indicated that negatively worded warnings and accusations worked better than positively worded warnings at blunting faking. Although there was some evidence that test-takers engaging in higher levels of faking heeded warnings more, warnings tended to decrease scores for all test-takers. While positive warnings motivated test-takers to perform well, negative warnings and accusations increased test-taking anxiety. Whereas past research has failed to find a relationship between warnings and perceived fairness, current results suggest that this relationship depends on the level of test-taker distortion.

Cheating, Reactions, and Performance in Remotely Proctored Testing: An Exploratory Experimental Study

Purpose We sought to provide empirical insight and develop theory for a new organizational phenomenon: remote proctoring for Internet-based tests. We examined whether this technology is effective at decreasing cheating and whether it has unintended effects on test-taker reactions, performance, or selection procedures.

Test length and cognitive fatigue: An empirical examination of effects on performance and test-taker reactions.

Person and situational determinants of cognitive ability test performance and subjective reactions were examined in the context of tests with different time-on-task requirements. Two hundred thirty-nine first-year university students participated in a within-participant experiment, with completely counterbalanced treatment conditions and test forms. Participants completed three test sessions of different length: (a) a standard-length SAT test battery (total time 4(1/2) hr), (b) a shorter SAT test battery (total time 3(1/2) hr), and (c) a longer SAT test battery (total time 5(1/2) hr). Consistent with expectations, subjective fatigue increased with increasing time-on-task. However, mean performance increased in the longer test length conditions, compared with the shorter test length condition. Individual differences in personality/interest/motivation trait complexes were found to have greater power than the test-length situations for predicting subjective cognitive fatigue before, during, and at the end of each test session. The relative contributions of traits and time-on-task for cognitive fatigue are discussed, along with implications for research and practice. (PsycINFO Database Record (c) 2009 APA, all rights reserved).

Warning Against Faking on a Personality Test: Effects on Applicant Reactions and Personality Test Scores

To reduce faking on personality tests, applicants may be warned that a social desirability scale is embedded in the test. Although this procedure has been shown to substantially reduce faking, there is no data that addresses how such a warning may influence applicant reactions toward the selection procedure or the relationships among personality constructs. Using an organizational justice framework, this study examines the effect of warning on procedural justice perceptions. Additionally, the extent to which warning changes the relationships among personality variables, socially desirable responding, and organizational justice variables, was explored. The results suggest that warning did not negatively affect test‐taker reactions. However, the relationships among the justice measures and the personality variables and socially desirable responding differed across the warned and unwarned groups. The organizational justice model fit best and there was less multicollinearity among the personality variables in the warned condition, compared to the unwarned condition. Thus, providing a warning appears to have positive consequences when using personality measures.

Procedural justice and organizational staffing: a tale of two paradigms

This paper proposes that greater attention to procedural fairness can enhance the overall effectiveness of human resource assessment practices. Recent work on the attributes of fair assessment tests is examined. Following this review, we discuss test-taker reactions to a variety of widely used instruments and procedures. Some selection techniques that show acceptable levels of statistical validity are often seen as violating norms of social justice. To alleviate this problem, suggestions for increasing justice are offered. These include (a) substituting a test that is perceived to be unfair with one that is viewed as fair, (b) modifying an existing test to improve fairness perceptions, (c) explaining to or educating the public as to the validity of existing selection devices, and (d) developing composite assessment technologies through various combination strategies.

The role of test-taker feedback in the test development process: test-takers' reactions to a tape-mediated test of proficiency in spoken Japanese

This study explores how test-taker reactions in a specific purpose testing context may vary according to characteristics of the test-taker. Such reactions are of interest both theoretically to the researcher and practically to the test developer, who is concerned to ensure that the test is fair and appropriate for all candidates and acceptable to the range of test-takers. The reactions may also be of use in the improvement of test items (as a supplement to item analysis) and test rubrics. The article reports on the use of test-taker feedback in the development of the occupational foreign language test (Japanese), a tape-mediated test of spoken Japanese for the tourism and hospitality industry. Some 53 trial subjects completed a post-test questionnaire, providing reactions to the test as a whole, to task types, and to individual test items. Relationships are investigated between responses and a number of test-taker characteristics, including gender, type of course undertaken (general or specific purpose), amount of study of the language, time spent in Japan and relevant occupa tional experience. Responses are also considered for items and person shown by Rasch IRT analysis as indicating significant misfit. Aspects of the content, construct and face validity of the test are considered in the light of the analysis. The role of feedback in the revision of test items, in the writing of test rubrics and in the development of the test-user handbook is also discussed.