Groups Of Test Takers Research Articles

Unsupervised Anomaly Detection in Sequential Process Data

Abstract: In this study, we present three types of unsupervised anomaly detection to identify anomalous test-takers based on their action sequences in problem-solving tasks. The first method relies on the use of the Isolation Forest algorithm to detect anomalous test-takers based on raw action sequences extracted from process data. The second method transforms raw action sequences into contextual embeddings using the Bidirectional Encoder Representations from Transformers (BERT) model and then applies the Isolation Forest algorithm to detect anomalous test-takers. The third method follows the same procedure as the second method, but it includes an intermediary step of dimensionality reduction for the contextual embeddings before applying the Isolation Forest algorithm for detecting anomalous cases. To compare the outcomes of the three methods, we analyze the log files from test-takers in the US sample ( n = 2,021) who completed the problem-solving in technology-rich environments (PSTRE) section of the Programme for the International Assessment of Adult Competencies (PIAAC) 2012 assessment. The results indicated that different groups of test-takers were flagged as anomalous depending on the representation (raw action sequences vs. contextual embeddings) and dimensionality of action sequences. Also, when the contextual embeddings were used, a larger number of test-takers were flagged by the Isolation Forest algorithm, indicating the sensitivity of this algorithm to the dimensionality of input data.

Does financial risk tolerance change over time?

Financial planners work in an environment that requires the documentation of a client’s financial attitudes and preferences. Financial risk tolerance is one such attitudinal construct that is generally required by regulators to be evaluated. While there are numerous commercial and academic products used to assess client risk attitudes, questions have been raised over the past several decades regarding the stability of scores from risk-tolerance tools. Specifically, financial planners, as well as regulators, require evidence documenting to what extent risk tolerance changes over time, and if changes do occur, the variables associated with variability. The purpose of this study was to address these needs. Based on a model that included macroeconomic indicators, biopsychosocial and environmental factors, and measures of social support, it was determined that risk-tolerance attitudes remain generally stable over time. However, there are groups of test takers that exhibit significant shifts in risk tolerance. This article describes some of the variables associated with these score changes, as well as providing financial planning professionals with guidance on how to identify clients who may be prone to shifting their tolerance for financial risk.

The impact of input format on written performance in a listening-into-writing assessment

Over the last five decades, research in teaching and testing (academic) listening has investigated different foci. Initially, teaching listening involved bottom-up approaches (Dirven and Oakeshott-Taylor, 1984) then both higher- and lower-level processes were integrated (Voss, 1984). In the early 2000s, different input formats (Read, 2002) and discourse features of lectures (Thompson, 2003) were the subjects of academic listening research. More recently, EAP tests have increasingly taken an integrated approach to reflect real-world tasks, yet few studies have looked at integrated listening-into-writing tasks (Cubilo and Winke, 2013).This counter-balanced measures design study investigates how test taker performance differs on an integrated EAP listening-into-writing task when lecture input is presented as audio only in one half and video in the other half of the input. Two groups of test takers took part in the current study.A Hotelling's T2 test revealed a statistically significant effect on scores when test takers were presented with the audio only input first but there was no significant effect on scores when the video input was presented first. Data on test taker preferences revealed that more people preferred the video input to audio only.

A Retrospective Review of NCLEX Candidates’ Testing Behavior: Examining the Relationship Between Repeat Testing, Time-to-Test, and Discipline

<h3>Background</h3> Concerns about the competence of repeat NCLEX test takers have persisted, resulting in a patchwork of regulatory policies that limit the number of test attempts or the conditions under which a candidate can retest in certain U.S. jurisdictions. <h3>Purpose</h3> To examine possible corrolaries between repeat test-taker status and future practice discipline and to investigate the relationship between time-to-test and repeat testing. <h3>Methods</h3> Data were collected on nearly a quarter of a million registered nurse (RN) and practical nurse (PN) NCLEX candidates who passed the examination between 2013 and 2017. Multivariable generalized estimating equation models were used to assess the likelihood of discipline (0, 1) and the need for repeat testing (0, 1). Propensity score matching was employed to address initial group imbalance on all available covariates for models assessing discipline. <h3>Results</h3> The rate of discipline among the RN (1.0%, <i>n</i> = 2,029) and PN (1.8%, <i>n</i> = 749) samples was low. After applying propensity score matching, repeat test-taker status was found to be weakly aligned with practice discipline for RNs (<i>p</i> = .047) but was not correlated for PNs (<i>p</i> = .13). In contrast, adjusting for sex, race, ethnicity, and age, RN candidates who delayed taking the test for 60 days were 5% more likely to need to retake the NCLEX (<i>p</i> < .001). For PN candidates, a delay of 90 days was associated with a 9% increase in the likelihood of needing to retake the NCLEX (<i>p</i> < .001). <h3>Conclusion</h3> Empirical evidence supports the comparable safety profile of single– and multi–test-taker groups, underscoring the need to revaluate restrictions that limit the conditions under which candidates can retest. Furthermore, ongoing outreach to nursing programs to emphasize the importance of prompt NCLEX testing and the possible deleterious effects of delays, by even a few months, is important.

The Impact of Cheating on Score Comparability via Pool‐Based IRT Pre‐equating

AbstractFor any testing programs that administer multiple forms across multiple years, maintaining score comparability via equating is essential. With continuous testing and high‐stakes results, especially with less secure online administrations, testing programs must consider the potential for cheating on their exams. This study used empirical and simulated data to examine the impact of item exposure and prior knowledge on the estimation of item difficulty and test taker's ability via pool‐based IRT preequating. Raw‐to‐theta transformations were derived from two groups of test takers with and without possible prior knowledge of exposed items, and these were compared to a criterion raw to theta transformation. Results indicated that item exposure has a large impact on item difficulty, not only altering the difficulty of exposed items, but also altering the difficulty of unexposed items. Item exposure makes test takers with prior knowledge appear more able. Further, theta estimation bias for test takers without prior knowledge increases when more test takers with possible prior knowledge are in the calibration population. Score inflation occurs for test takers with and without prior knowledge, especially for those with lower abilities.

Assessing Differential Item Functioning (DIF) For Pearson Test of English (PTE) a study of Test Takers with Different Fields of Study

Differential Item Functioning (DIF), which is a statistical feature of an item and provides a sign of unpredicted performance of items on a test, occurs once dissimilar groups of test takers with the same level of ability show different performance on a single test. The aim of this paper was to examine DIF on the Pearson Test of English (PTE) test items. To that end, 250 intermediate EFL learners with the age range of 26 - 36 in two different fields of study (125, Engineering, and 125 Sciences) were randomly chosen for the analysis. The Item Response Theory (IRT) Likelihood Ratio (LR) approach was utilized to find items showing DIF. The scored items of 250 PTE test takers were analyzed using the IRT three-parameter model utilizing item difficulty (b parameter), item discrimination (a parameter), and pseudo-guessing (c parameter). The results of the independent samples t-test for comparison of means in two groups depicted that Science participants performed better than the Engineering ones particularly in Speaking &amp; Writing and Reading sections. It is evident that the PTE test was statistically easier for the Science students at 0.05 level. Linguistic analyses of Differential Item Functioning items also confirmed the findings of the quantitative part, indicating a far better performance on the part of Science students.

Introduction to the Special Section “Lingering Impact of COVID‐19 on Educational Measurement”

It was April of 2021. My son, Timothy, and I drove to a nearby city for him to take a college admission test. The test center closest to our home had quickly filled up due to reduced capacity (for social distancing) and increased demand. Securing a seat at a testing center half an hour away seemed to be luxury, considering the circumstances. This was a good enough reason for neither of us to complain about our lack of sleep. After dropping Timothy off, I decided to wait for him in the parking lot of the test center. I had brought a book with me and was excited about this excuse to take a brief break from the busyness of life and enjoy some quiet time while reading. I was not too far along in my reading when I looked up and saw Timothy returning to the car. “Wow. Time really flies when you're enjoying a good book,” I thought. However, I quickly realized that a college admission test would probably take longer than me reading a few pages of my book. After getting into the car, my son explained that the door to the testing center was locked and had never opened for him and a small group of other students waiting to take the test. Apparently, the test center was closed, but the students had not received any notifications about the closure. We were later told that an email about the closure had been dispersed; however, no one in this particular group of test-takers had received that email. This is just one example of how COVID-19 has had an impact on my life and the lives of those around me. Since the World Health Organization (WHO) declared a pandemic on March 11, 2020, the world it represents has been dramatically disrupted. Events have been canceled; restaurants were closed; cities were locked down; and the list goes on. The educational measurement world was not spared. For example, exams were canceled; schools were closed; and research studies were delayed. EM:IP published a special section on the impact of COVID-19 on the field of educational measurement in the Fall 2020 issue. Since then, so much has changed. The encouraging statistics of COVID-19 vaccines initially brought hope that the global disaster would end soon. The hope, however, shattered into pieces when new variants were detected one after another. What we know, now, is there are more variants in existence than the most frequently discussed Alpha, Delta, and Omicron variants. What we do not know is whether there are enough letters in the Greek alphabet to account for the present and future variants of COVID-19. When planning for this special section, I decided to design a cover graphic to complement the COVID-19 subject-matter. Initially, I was going to portray a rearview mirror to signal leaving behind the pandemic and continuing on our journey ahead. However, this idea was out of touch with the reality. Yes, the global disruption is about two years old, but we still have not seen the end of the tunnel. I finally settled on the idea of “Turning the Page.” The coronavirus may not disappear soon (or at all), but we should be ready to move forward. Now that we have more data on the lingering impact of COVID-19, the authors in this section (who are spread across three continents) speculate on different aspects of the impact on educational measurement. Some authors discuss college admissions and inequalities—hot topic issues from before 2020 that have been exacerbated by the pandemic. Others share their thoughts and experiences related to the changes to educational assessments, licensure exams, and certification exams. Still, others cover longitudinal analysis and meta-analysis. Online testing or remote assessment continues to be of great interest, as shown by several authors. Lastly, disrupted data are inevitable, especially during a pandemic. Two articles present some thoughts on decision-making using disrupted data. I hope you find this special section interesting and thought-provoking. I also hope, as Bennett's title suggests, you can look at “The Good Side of COVID-19.”

Considerations for Addressing the Needs of Culturally and Linguistically Diverse Speech-Language Pathology Praxis Test Takers.

The primary purpose of this article is to explore the speech-language pathology (SLP) Praxis test, a barrier to culturally and linguistically diverse (CLD) individuals entering the profession, by investigating first-time pass rates and mean scores by test taker race/ethnicity. Other potential barriers to licensure and certification, as well as solutions for mitigating these barriers, will also be addressed. SLP Praxis test data from two windows of time, 2008-2011 and 2014-2020, were compared for the following: (a) proportions of test taker race/ethnicity relative to U.S. demographic estimates of racial/ethnic group proportions overall, (b) proportions of racial/ethnic groups, and (c) trends in test-taker mean scores by race/ethnicity. First-attempt pass rates by racial/ethnic groups were also calculated for the 2014-2020 testing window. The percentage of some CLD SLP Praxis test-taker groups increased since the 2008-2011 testing window but is still not representative of U.S. racial/ethnic demographics. The first-attempt pass rates and overall mean scores of CLD test-taker groups remained substantially lower than White non-Hispanic/Latinx test takers. Despite the encouraging trends in SLP Praxis test-taker racial/ethnic diversity, disparities persist between the racial/ethnic makeup of SLP Praxis test takers and the demographic makeup of the United States. Consequently, these disparities have implications for the continued lack of cultural representativeness seen in our workforce.

The washback effect of TOEFL iBT and a local English Proficiency Exam on students' motivation, autonomy and language learning strategies

This study aimed to investigate the washback effect of two high stakes tests, a global language proficiency test (i.e., TOEFL iBT) and a local English Proficiency Exam (developed and administered by a state university) on students' motivation and their autonomy. The study also examined whether proficiency level moderated the potential washback effect among the two groups of test takers. Additionally, the study tried to find out the language leaning strategies used by both groups and explore the reasons behind their preferences. The study was conducted with two English language preparatory programs offered at a state university in Turkey: University Preparatory Program (UPP) and Dual Degree Program (DDP). At the end of the UPP program, the students are required to take the university's proficiency test while as for the UPP, they need to take a valid TOEFL iBT. Data were collected from 152 preparatory students (N = 65 for DDP; N = 87 for UPP) whose proficiency levels were based on the CEFR Framework ranging from A2 (upper elementary) to B1 (pre-intermediate) to B2 (intermediate). In addition to the above proficiency tests, data were gathered via motivation questionnaire, autonomous learning scale and student interviews. The results revealed no washback of TOEFL iBT exam on students' motivation regardless of their proficiency level. No washback was also observed on students' autonomy except for A2 level DDP students who had higher level of autonomy than the A2 level UPP students. Finally, the two groups used more similar than different language learning strategies while practicing the four language skills. The findings afford pedagogical implications for the use of high-stakes tests in English preparatory programs.

Anchor Point Selection: Scale Alignment Based on an Inequality Criterion.

For detecting differential item functioning (DIF) between two or more groups of test takers in the Rasch model, their item parameters need to be placed on the same scale. Typically this is done by means of choosing a set of so-called anchor items based on statistical tests or heuristics. Here the authors suggest an alternative strategy: By means of an inequality criterion from economics, the Gini Index, the item parameters are shifted to an optimal position where the item parameter estimates of the groups best overlap. Several toy examples, extensive simulation studies, and two empirical application examples are presented to illustrate the properties of the Gini Index as an anchor point selection criterion and compare its properties to those of the criterion used in the alignment approach of Asparouhov and Muthén. In particular, the authors show that—in addition to the globally optimal position for the anchor point—the criterion plot contains valuable additional information and may help discover unaccounted DIF-inducing multidimensionality. They further provide mathematical results that enable an efficient sparse grid optimization and make it feasible to extend the approach, for example, to multiple group scenarios.

Measuring financial literacy with a Situational Judgement Test: do some groups really perform worse or is it the measuring instrument?

Due to current trends in society and economy, financial literacy is often considered as an important twenty-first century skill. However, regardless of the postulated relevance, studies suggest that financial illiteracy seems to be a widespread phenomenon in the population of many nations. Some studies also show that some groups perform particularly poorly (e.g. women, persons with migration background and/or low level of education). These differences are often attributed to different individual characteristics such as abilities, dispositions or socialisation patterns. However, available research also suggests that even after controlling for them, a quite large portion of the performance differences between the various groups of test-takers remains unexplained. One explanation for performance gaps in financial literacy might be that differences in test scores could also be evoked by the test instruments itself and may thus, at least in part, be interpreted as testing bias. In this paper, we present a newly developed Situational Judgement Test, which is focused on financial competence. For this test, we examine whether differences between groups are attributable to individual differences or due to a test bias. To analyse a possible test bias, we tested one facet of financial literacy (with three factors: control of one’s financial situation, budgeting and handling of money) related to everyday money management for measuring invariance for different groups. If measuring invariance could be assumed, we analysed group differences by calculating t-tests. Results show that two factors of the test show measurement invariance for all groups considered (gender, migration and educational background, opportunities to learn). Group comparisons are thus possible and potential differences are not due to a test bias. For one factor, we can only assume measurement invariance for the group with/without migration background and with/without opportunities to learn in financial topics. When we look at group differences, we find that in contrast to the findings of many previous studies, the analysis of the mean differences does not show any systematic deficits in financial literacy for specific groups.

On the detection of speededness in data despite selective responding using factor analysis

The essay reports two studies on confirmatory factor analysis of speeded data with an effect of selective responding. This response strategy leads test takers to choose their own working order instead of completing the items along with the given order. Methods for detecting speededness despite such a deviation from the given order are proposed and investigated. The first study demonstrated that speededness was not detectable if the deviation was large and selected by all test takers but ignored in investigating the data. Adaptation to the selected working order restored the detectability. The second study assumed four groups of test takers selecting group-specific working orders. Detection of speededness required a confirmatory factor model with speed factors adapted to group-specific working orders.

Do individual differences in test-takers' appraisal of admission testing compromise measurement fairness?

Due to the increased use of cognitive tests in admission testing there have been renewed concerns that individual differences in test anxiety induces measurement bias in cognitive tests and therefore call their valid and fair use in admission testing into question. Prior studies examining measurement invariance across individual differences in test anxiety yielded mixed results. While some studies indicated measurement bias due to test anxiety, others failed to confirm this hypothesis. Some researchers hypothesized that the extent to which test anxiety induces measurement bias in cognitive tests by reducing attentional resources available to the test-taker while solving the test items, depends on their cognitive appraisal of the test-taking situation in terms of their confidence in their ability to handle the task at hand (testing problem efficacy), the relevance of the goal to do well in the admission test (goal relevance), and their perceived level of control (agency). The present study was conducted to test this hypothesis. A large sample (N = 1628) of medical school applicants was tested in the real-life admission testing situation. Using latent profile analysis, we identified four groups of test-takers differing in their appraisal of goal relevance, testing problem efficacy, worry, task-irrelevant thinking, and agency. Contrary to our predictions item response theory analyses indicated measurement invariance across the four latent profiles for all four cognitive ability tests and all four knowledge tests administered in the present study. This finding contradicts theoretical models, which postulate that individual differences in test-takers appraisal of the admission testing situation and their emotional reactions to it, compromise the measurement fairness of cognitive admission tests.

Measuring English language proficiency across subgroups: Using score equity assessment to evaluate test fairness

This study illustrates the use of score equity assessment (SEA) for evaluating the fairness of reported test scores from assessments intended for test takers from diverse cultural, linguistic, and educational backgrounds, using a workplace English proficiency test. Subgroups were defined by test-taker background characteristics that research has shown to be associated with performance on language tests. The characteristics studied included gender, age, educational background, language exposure, and previous experience with the assessment. Overall, the empirical results indicated that the statistical and psychometric methods used in producing test scores were not strongly influenced by the subgroups of test takers from which the scores were derived. This result provides evidence in support of the comparability and meaning of test scores across the various test-taker groups studied. This example may encourage language testing programs to incorporate SEA analyses to provide evidence to inform the validity and fairness of reported scores for all groups of test takers.

Is the GED a Viable Pathway to College for Adult Students? New Regression Discontinuity Evidence from Massachusetts

A mature body of research has examined the labor market returns to passing the GED, typically finding modest (or negligible) benefits for the individual. In this study, we use a regression discontinuity research design to estimate the impact of obtaining the GED on postsecondary outcomes for two self-selected groups of test-takers in Massachusetts: high school dropouts who do and do not enroll in publicly funded adult basic education (ABE) classes. In contrast with previous work, we find that earning a GED credential substantially increases enrollment and persistence in postsecondary education for ABE students who marginally pass the GED, but find no such effects for dropouts who do not participate in ABE. Specifically, our IV estimates indicate that earning a GED increases the likelihood that ABE participants ever enroll in college by 33.4-55.8 percentage points and increases enrollment for four or more quarters by 25.4-33.7 percentage points, depending on the specification. We hypothesize that although ABE students are negatively selected in terms of academic skills, they are positively selected on dimensions of non-cognitive skills and motivation relative to non-ABE test-takers. Our findings highlight a policy-relevant population of GED test-takers for whom earning a GED may be particularly beneficial and furthers our understanding of who does and does not benefit from this credential.

Examining the Impact of Covariates on Anchor Tests to Ascertain Quality Over Time in a College Admissions Test

We propose a comprehensive procedure for the implementation of a quality control process of anchor tests for a college admissions test with multiple consecutive administrations. We propose to examine the anchor tests and their items in connection with covariates to investigate if there was any unusual behavior in the anchor test results over time and if the test results differed for different groups of test takers. Descriptive statistics, ANOVA, and linear mixed effect models were used to examine the impact of covariates on the anchor test over time. Descriptive statistics and logistic regression were used to examine the quality of each anchor item over time. It is concluded that examining the impact of the test takers' covariates at different administrations can help in the process of providing fair tests for all test takers. The results are discussed with recommendations how it can be used in other large testing programs.

Hur tillförlitligt är det nationella provet i läsning i åk 9?

This article reports from a study of reliability in the Swedish national reading test in ninth grade. Based on a representative sample of 500 students, the study investigates internal consistency and power to discriminate between different levels of reading ability. Data was analyzed using both classical test theory and Rasch modelling. Results from the study suggest that neither the subscales nor the whole scale of items included in the test reach desirable levels of internal consistency. Three out of four subscales include items with strikingly low degree of internal correlation (.23–.46). Furthermore, results from the Rasch analysis suggest that the test is capable of reliable separation between two groups of test-takers only: lower performing students and higher performing students. The empirical support for categorizing student performances into six separate grade levels is therefore questionable. Implications for future development of the national test of reading are discussed.

Group differences in test-taking behaviour: an example from a high-stakes testing program

This study investigated whether different groups of test-takers vary in their reported test-taking behaviour in a high-stakes test situation. A between-group design (N = 1129) was used to examine whether high and low achievers, as well as females and males, differ in their use of test-taking strategies, and in level of reported test anxiety and motivation. The results showed differences between high and low achievers on a number of test-taking strategies, where high achievers reported using successful strategies to a higher extent. There were also gender differences: females, for example, reported using random guessing to a higher extent than males. Further, low achievers, especially females, reported significantly higher levels of test anxiety than high achievers, and high achievers reported slightly higher levels of motivation when compared to low achievers. To conclude, test-taking behaviour might bring additional variance to test scores, whether or not this is irrelevant variance is discussed.

THE BENEFITS OF BILINGUALISM: A COMPARATIVE PILOT STUDY ON THE PERFORMANCE OF BILINGUALS VERSUS MONOLINGUALS

The paper focuses on the differences and similarities in the performance of two groups of test takers taking the ECL (European Consortium for the Certificate of Attainment in Modern Languages) language exam – a proficiency-type of language exam – in Hungary and in neighbouring countries. The two groups under investigation include Hungarian test takers living in Romania, Slovakia and Serbia and native Hungarians between the ages of 14 and 19 living in Hungary. The research aims to identify whether bilingual test takers have an advantage when taking a language examination in a third language as compared to those who are monolingual. The study compares, contrasts and analyses the performance of these two distinct groups, one of which comes from a multilingual environment. It has been found out that the overall performance (achieved in each of the four components: listening, speaking, writing and reading) of bilingual test takers was significantly better than that of the monolinguals (p &lt; 0.001). Three tendencies have been identified with respect to the differences between the performance of the bilingual and monolingual test takers: first, there is a greater gap in relation to receptive skills - listening and reading – than in the case of the productive skills – speaking and writing; second, in the case of the receptive skills the gap decreases as the proficiency level increases; third, in the case of the speaking skill the gap increases as the proficiency level increases. Possible explanations are offered to explain these phenomena. DOI: http://dx.doi.org/10.7220/2335-2027.8.8

METHODS AND SOFTWARE FOR RESEARCH OF THE VALIDITY OF TEST RESULTS FOR THE TESTED GROUPS WITH CERTAIN INDIVIDUAL CHARACTERISTICS

The necessary condition for the presence of biased assessment by some test is differential item functioning in different groups of test takers. The ideas of some statistical methods for detecting Differential Item Functioning are described in the given article. They were developed in the framework of the main approaches to modeling test results: using contingency tables, regression models, multidimensional models and models of Item Response Theory. The Mantel-Haenszel procedure, logistic regression method, SIBTEST and Item Response Theory Likelihood Ratio Test are considered. The characteristics of each method and conditions of their application are specified. Overview of existing free software tools implementing these methods is carried out. Comparisons of these methods are conducted on the example of real data. Also notes that it is appropriate to use several methods simultaneously to reduce the risk of false conclusions.