Test Takers Research Articles

Challenges and opportunities for spoken English learning and instruction brought by automated speech scoring in large-scale speaking tests: a mixed-method investigation into the washback of SpeechRater in TOEFL iBT

The extensive use of automated speech scoring in large-scale speaking assessment can be revolutionary not only to test design and rating, but also to the learning and instruction of speaking based on how students and teachers perceive and react to this technology. However, its washback remained underexplored. This mixed-method study aimed to investigate the washback of TOEFL iBT Speaking’s SpeechRater on Chinese EFL learners through questionnaire and interviews, and explore its associations with test performance and its multi-levelled influential factors. The participants received a mixture of positive and negative washback, such as their motivated individual learning through personal devices, decreasing real-life communicative practices, and increasing exam-driven behaviours. Test takers’ personal understandings of automated speech scoring were found directly influential to the washback of SpeechRater that they experienced. Furthermore, their test scores of TOEFL iBT Speaking were positively correlated with the implicit washback of SpeechRater on their learning but uncorrelated with its explicit washback on their test preparation. The findings have been drawn on to propose a washback model of automated speech scoring and make suggestions to test designers, teachers and learners on how to boost its positive washback and mitigate its negative washback. This research has concluded the importance of test takers’ awareness of the integrated dimensions to evaluate spoken English in real-life use. Accordingly, instructional implications are discussed on how teachers can guide the students to utilize automated speech scoring in learning and set up comprehensive learning goals for spoken English.

The impacts of Confucianism on gender inequality in Vietnam

We quantify the influences of Confucianism on gender inequality in present-day Vietnam. We use the number (or density) of the most successful test takers in the Vietnamese imperial examinations (1075–1919) in a given district as a proxy for mastering the subject of Confucianism. Using an instrumental variable approach based on the historical expansion of Vietnamese territory and distances to the test venues, we consider the effects on labor, health, and educational outcomes for females relative to males. We find that Confucianism has long-lasting negative impacts on gender inequality in all considered aspects. However, the results also suggest that when present-day females pursue more years of schooling, they tend to perform better than males.

Young videogamers and their approach to science inquiry

BackgroundWritten instructions seldom need to be read when playing videogames. Instead, gaming often involves early information foraging and expansive exploration behaviors. We use data from the Programme for International Student Assessment (PISA) to explore whether students who regularly play videogames (daily gamers) adopt behaviors that are typical of gaming while they complete a computer-based assessment of science and if such behaviors matter for performance in procedural science knowledge.MethodsWe use item-level data from PISA 2018 from countries that administered the optional ICT questionnaire. Using a sample item and the full set of interactive science items, we develop regression models to estimate differences between daily gamers and other students in information harvesting, time to first action, and procedural science knowledge. We report average associations across countries, country-specific estimates, as well as differences between boys and girls. We report raw differences as well as differences adjusting for background characteristics. To account for the multilevel nature of the data and for the two-stage sampling design, we use replicate weights supplied with the PISA database to estimate robust standard errors using balanced repeated replication (BRR).ResultsIn 2018, 33% 15-year-olds reported playing videogames every day or almost every day. Among boys, that proportion was 49%. Daily gamers do not differ from other students in science content knowledge and in reading fluency. Nevertheless, daily gamers spend marginally less time reading instructions and display more active exploration behaviors in the assessment on items that include simulation tools.ConclusionsScience teachers and assessment developers may find inspiration in games to develop scenarios in which students can practice effective strategies for information harvesting. The extensive exploration of a problem space in order to obtain data in support of future decisions often corresponds to a positive behaviour with multiple advantages in authentic problem situations. By contrast, fast transitions into action may, in the particular situation of an assessment, be an inadequate response. Assessment developers can ensure that instructions are carefully read and understood by test takers and teachers and can guide their students to read instructions adequately.

Assessing Interactional Competence: ICE versus a Human Partner

ABSTRACT Most second language assessment researchers agree that interactional competence (IC) is an important part of the construct of oral communication. However, measurement of IC has proven challenging because at least one interlocutor is considered necessary to create an appropriate social context for test takers to demonstrate their IC. Including interlocutors in the assessment process can be impractical and may make judging test takers’ IC difficult because their performances may be impacted by the interlocutors. One potential approach to assessing oral communication that might diminish these challenges is to use a Spoken Dialogue System (SDS) as a test taker’s partner. To explore the potential of an SDS for assessing the IC, the use of an SDS and a human peer partner were compared to determine which is more appropriate for eliciting discourse for this purpose. Forty test takers completed a video-taped paired discussion task with both a human partner and Interactional Competence Elicitor (ICE), an SDS created by the researchers. Four trained raters evaluated the video-recorded performances, and results indicated that in the SDS condition raters: believed more features of IC were ratable, assigned lower scores for most IC features, and had more positive perceptions of the rating process.

Exploring the Development of an Interprofessional Situational Judgment Test

INTRODUCTION Although the prevalence of interprofessional education (IPE) is increasing, research measuring competency in IPE is underdeveloped (MacDonald et al., 2010; Shrader et al., 2017). Interprofessional assessment tools are plentiful, but most rely on self-report rather than performance data. Thus, the purpose of this study was to develop a computer-based situational judgment test to evaluate health professional student preparedness for interprofessional practice. METHODS The investigators developed the Creighton Interprofessional Situational Judgment Test (C-IPSJT) which includes nine interprofessional clinical case scenarios, each with multiple questions reflecting IPEC competencies. Subsequently, focus groups representing both IPE and health education experts were convened to assess the C-IPSJT. RESULTS Qualitative analysis revealed several themes. Participants found that the C-IPSJT was a novel assessment of interprofessional skills and case scenarios presented were reflective of current clinical practice. Challenges were found in understanding the role of the test taker, discerning correct responses, and the need for further validity testing. Additional themes included the impact team dynamics have on learner responses and how student responses could be affected by their place on the learning continuum. DISCUSSION Findings from the initial testing of the C-IPSJT demonstrate positive content validity through two independent content expert reviews. Content experts identified that the C-IPSJT could be a viable mechanism to differentiate between novice and competent health professional students. Future research includes item revision, testing over a larger sample in which psychometric properties and item analysis can be studied, and establishing content validity.

The Pass/Fail Effect: A Longitudinal Study of United States Medical Licensing Examination (USMLE) Step 1 Performance Over a Decade.

Objectives This study aimed to analyze the impact of theUnited States Medical Licensing Examination (USMLE) Step 1 transition to a pass/fail scoring system in 2022 on the performance of first-time test takers in three distinct groups: Doctor of Osteopathy (DO) and Doctor of Medicine (MD) examinees from US/Canadian schoolsand examinees from non-US/Canadian schools. The analysis spans a decade-long period from 2012 to 2022, offering insights into the implications of this pivotal change in medical education. Methods We analyzed the performance of first-time USMLE Step 1 examinees from US/Canadian MD and DO programsand non-US/Canadian schools from 2012 to 2022, including the transition year to a pass/fail scoring system. Data were obtained from USMLE performance data reports and organized into annual contingency tables. Descriptive statistics and comparative analysis were used to identify trends and differences in performance across the groups. Data visualization techniques were employed to illustrate these findings, and the results were contextualized within the broader changes in medical education. Results In 2021, first-time takers from US/Canadian MD and DO Degree programs had passrates of 96% and 94%, respectively, while non-US/Canadian schools had a pass rate of 82%. However, in 2022, these rates dropped to 93%, 89%, and 74%, respectively. The most significant relative decline was observed among non-US/Canadian Schools' first-time takers, with an 8% decrease. Repeaters consistently had lower pass rates across all groups. Conclusion The study reveals a notable decline in pass rates following the transition to pass/fail scoring, although this is based on just one year of data. This underscores the importance of students not rushing into the exam and dedicating sufficient time for preparation. The potential impact of this research could be transformative for medical education, but more years of data post-transition will be needed to confirm these initial findings. These findings serve as a reminder that the change in scoring does not diminish the rigor of the exam, prompting students to approach their studies with diligence and patienceand potentially paving the way for systemic improvements in medical education and healthcare delivery worldwide.

Prevalence and factors associated with suicidal ideation among students taking university entrance tests: revisited and a study based on Geographic Information System data.

A previous study identified students taking Bangladeshi university entrance tests as a vulnerable group at a higher risk of suicidal behaviours before the COVID-19 pandemic. However, the impact of the pandemic on the magnitude of these behaviours among this population remains unexplored. This study aimed to investigate the prevalence of suicidal ideation and associated factors among Bangladeshi university entrance test takers following the pandemic. In addition, an approach based on Geographic Information System (GIS) data was used to visualise the distribution of suicidal ideation across the country. A cross-sectional approach was used to collect data among participants taking the entrance test at Jahangirnagar University in September 2022. Using SPSS, data were analysed with chi-squared tests and binary regression, and ArcGIS was used to map the results across the nation. The study revealed a prevalence of 14.4% for past-year suicidal ideation, with 7.4% and 7.2% reporting suicide plans and attempts, respectively. Notably, repeat test-takers exhibited a higher prevalence of suicidal behaviours. Significant risk factors for suicidal ideation included urban residence, smoking, drug use, COVID-19 infection and deaths among close relations, depression, anxiety and burnout. The GIS-based distribution indicated significant variation in the prevalence of suicidal ideation across different districts, with higher rates observed in economically and infrastructurally deprived areas. Urgent measures are needed to address the high prevalence of suicidal behaviours among students taking university entrance tests students in Bangladesh, particularly in light of the COVID-19 pandemic. Enhanced mental health support, targeted prevention efforts and improved resources in economically disadvantaged regions are crucial to safeguard the well-being of these students.

Karakteristik dan Efektivitas Formula Sabun Cuci Tangan Cair Handmade dalam Menurunkan Angka Kuman

This study aims to compare the characteristics of the handmade liquid hand soap formula and its effectiveness in reducing the number of germs. The formulation used is the difference in the amount of EDTA, Texapon, and NaCl used in soap making. The characterization carried out was organoleptic, physical stability at room temperature, pH, viscosity, foam height and stability, water holding capacity, and density. Meanwhile, reduced number of germs was carried out by calculating the total plate number (ALT) of bacteria on the test taker hand before and after washing their hands. The results obtained were that the first formula (P1) which used less amounts of EDTA, Texapon, and NaCl had a bright color, lack of foam, perfect solubility,stable at room temperature, and did not cause a burning feeling in the hands. Whereas the second formulation (P2) with a higher amount of EDTA, Texapon, and NaCl hadexellence in the amoont of foam compared to the previous formula but the solubility was not perfect, less stable at room temperature and caused a burning feeling in the hands. The result of Pairef t test show that the t count is bigger than t table, so the different formulation had an effect on reducing the germ on the test taker hand based on ALT germ number parameter. P1 formulation can reduce the number of bacteria on hands by 47.64% and P2 formulation by 36.12%.

Equating of standardized science subjects tests using various methods: which is the most profitable?

A good test set can be reflected in the quality of the items. It can measure the ability of the test takers reasonably even though they are distributed in several question packages. This study uses an exploratory, descriptive method to determine the equivalence of standardized test sets in science subjects for junior high schools in Indonesia. The data were obtained from the database of Junior High School National Examination results in the subject of Natural Sciences, which consisted of 5 question packages with 40 items/package. The equating technique uses the Item Response Theory 3 PL approach with the help of R Studio Software. The research results show that the national exam questions, which consist of 5 question packages, have a good level of item difficulty and all guesses. However, the discrimination index and several items obtained unfavorable results. In addition to the results of equating the graphical method using the closeness of the test characteristic curve, the Stocking & Lord methods produce the most equivalent scores. These findings can be a reference for test developers or researchers in the field of measurement to produce better and more accurate test kits.

Item analysis of reading comprehension questions for English proficiency test using Rasch model

The need to take English as a foreign language proficiency test (known as TOEFL [Test of English Language Proficiency]) has been gaining popularity in Indonesia. The increasing demands for such a test and its expensive cost have reinforced many institutions to develop TOEFL instruments and administer the test internally. However, constructing a test instrument is a complex process that makes conducting item analysis become more challenging. Meanwhile, item analysis is crucial to assess the items’ quality. Therefore, this study reported the results of statistically analyzing 20 questions of TOEFL reading comprehension that were analyzed in terms of the test reliability, the item and person fit, and the items’ difficulty level. Thirty-eight members of the English Department Students’ Association of a state university in West Java participated in this study by taking the reading test. The data were analyzed using the Rasch model by utilizing the Quest program. The results showed that four items (36.8%) did not fulfill the ideal criteria of a valid test because they were too easy and too difficult to be given to the target test takers; thus, they needed to be discarded. Meanwhile, 16 items (63.2%) are of good quality and can be used immediately in the proficiency test, especially to measure reading comprehension skills, because they have fulfilled the standard requirements for a valid test. The findings have provided insight into the importance of item analysis in validating test instruments to improve the test quality for future administrations.

Evaluating TOEFL ITP Test: A Critical Review

The critical objective of this present study is to conduct substantial evaluation of TOEFL as a language assessment tool. Using a qualitative method, the authors of the present study evaluated various specific issues based on a set of predetermined criteria such as practicality, test reliability, validity, authenticity, and washback impact. The analysis indicates that TEOFL ITP is both feasible and dependable. The TOEFL's dependability is based on these three components. However, unless the ITP test is altered from the previously used TOEFL, the TOEFL ITP is as reliable as the TOEFL PBT. In theory, its trustworthiness is uncertain, and more investigation is needed. It is debatable whether the TOEFL ITP test items can accurately represent this task when utilized for the placement test. TOEFL ITP is a reliable English test. Knowing the TOEFL significance encourages test takers to enhance their English skills as part of the exam. More students want to take lessons to enhance their grades.

Automatic item generation for online measurement and evaluation: Turkish literature items

Developments in the field of education have significantly affected test development processes, and computer-based test applications have been started in many institutions. In our country, research on the application of measurement and evaluation tools in the computer environment for use with distance education is gaining momentum. A large pool of items is required for computer-based testing applications that provide significant advantages to practitioners and test takers. Preparing a large pool of items also requires more effort in terms of time, effort, and cost. To overcome this problem, automatic item generation has been widely used by bringing together item development subject matter experts and computer technology. In the present research, the steps for implementing automatic item generation are explained through an example. In the research, which was based on the fundamental research method, first a total of 2560 items were generated using computer technology and SMEs in field of Turkish literature. In the second stage, 60 randomly selected items were examined. As a result of the research, it was determined that a large item pool could be created to be used in online measurement and evaluation applications using automatic item generation.

INVESTIGATING KURDISH EFL UNIVERSITY STUDENTS’ AWARENESS AND PRODUCTION IN USING ENGLISH GENERIC REFERENCE IN WRITING

This article aims to discover: (1) if Kurdish EFL students are aware of English generic referential items (2) whether Kurdish EFL students are able to differentiate between English generic and specific references, (3) whether Kurdish EFL students comprehend the correct semantic interpretations of English generic reference (4) and to discover possible students’ errors in using English generic reference in writing and attribute the errors to their sources. The study draws on the assumption that Kurdish EFL students' overuse of generic reference in writing could be a contributing factor in several error sources. Therefore, a quantitative test served eighty Kurdish EFLs as the means of data collection at four distinct colleges in the Kurdistan Region for the academic year 2021–2022. The statistical analysis of the data revealed that, about half of the Kurdish EFL test takers misapplied generic reference in their writings in the areas of syntax, semantics and discourse. The reasons could be negative interlingual and intralingual transfers, lack of students’ grammatical proficiency and the learning context.

Stealth Literacy Assessments via Educational Games

Literacy assessment is essential for effective literacy instruction and training. However, traditional paper-based literacy assessments are typically decontextualized and may cause stress and anxiety for test takers. In contrast, serious games and game environments allow for the assessment of literacy in more authentic and engaging ways, which has some potential to increase the assessment’s validity and reliability. The primary objective of this study is to examine the feasibility of a novel approach for stealthily assessing literacy skills using games in an intelligent tutoring system (ITS) designed for reading comprehension strategy training. We investigated the degree to which learners’ game performance and enjoyment predicted their scores on standardized reading tests. Amazon Mechanical Turk participants (n = 211) played three games in iSTART and self-reported their level of game enjoyment after each game. Participants also completed the Gates–MacGinitie Reading Test (GMRT), which includes vocabulary knowledge and reading comprehension measures. The results indicated that participants’ performance in each game as well as the combined performance across all three games predicted their literacy skills. However, the relations between game enjoyment and literacy skills varied across games. These findings suggest the potential of leveraging serious games to assess students’ literacy skills and improve the adaptivity of game-based learning environments.

Developing and validating an online situational judgment test on the stress coping competence of nursing apprentices

Converging evidence of high stress levels in the nursing profession as well as their detrimental effects on the execution of nursing actions, quality indicators of care, and self-reported job satisfaction justify the need for test environments that assess stress coping competences in dealing with domain-specific stress factors. Reliable and valid information on the test takers’ stress coping competence can help to identify individual deficits and supportive needs and thus, to design training measures that may contribute to job retention and a skilled workforce. This article describes the development and validation of a digital assessment that ensures authenticity by means of video-based situational judgement tests. Analyses of the performance data were conducted in four steps: (1) the scaling of the data with a partial credit model in order to identify items that are subject to measurement error, (2) differential-item-function analyses for different specializations in nursing education, (3) dimensionality tests, and (4) correlations between test performance and perceived psychological work stress in order to gauge criterion validity. The total sample consisted of n = 300 German nursing apprentices. The results show that the developed test instrument can be used for a reliable and valid measurement of coping skills of (prospective) nurses, covering two essential dimensions. Limitations of the current study and related objectives of future research are discussed together with questions of practical applicability.

Hierarchical Agglomerative Clustering to Detect Test Collusion on Computer‐Based Tests

AbstractThere has been a growing interest in approaches based on machine learning (ML) for detecting test collusion as an alternative to the traditional methods. Clustering analysis under an unsupervised learning technique appears especially promising to detect group collusion. In this study, the effectiveness of hierarchical agglomerative clustering (HAC) for detecting aberrant test takers on Computer‐Based Testing (CBT) is explored. Random forest ensembles are used to evaluate the accuracy of the clustering and find the important features to classify the aberrant test takers. Testing data from a certification exam is used. The level of overlap between the exact response matches on incorrectly keyed items in the exam preparation material and HAC are compared. Integrating HAC as an investigation mean is promising in this field to improve the accuracy of classification of aberrant test takers.

Ensemble DeBERTa Models on USMLE Patient Notes Automatic Scoring using Note-based and Character-based approaches

To maximize the accuracy and efficiency of the USMLE Step 2 clinical skills examination evaluation process, we proposed an ensemble model that helps automatically score patient notes written by test takers instead of physician raters manually scoring them by appropriate features. This research used DeBERTa-base, DeBERTa-large, and DeBERTa-v3-large as three base models and ensembled them with two different approaches: Note-based and Character-based. We concluded that LSTM Note-based ensemble topped the overall performance with an F1-score of 0.81747 on the validation data, 48% higher than the F1-score of the most effective base model (DeBERTa-v3-large). Furthermore, the performance is robust when breakdown by clinical cases and folds and applied to the testing set (0.88737 accuracy). Finally, the ensemble approach to different base models (BERT-base-uncased and BERT-large-uncased) achieved a 32% F-1 score boost. We demonstrated the ensemble model has excellent potential to improve performance in general Natural Language Understanding tasks.

DEVELOPMENT ANALYSIS OF CREATIVE THINKING TEST INSTRUMENTS ON NATURAL SCIENCE MATERIALS

Creativity is an important part of improving learning achievement and readiness to face the globalization era which requires innovation. The success of learning creativity can be assessed through an appropriate holistic assessment instrument that contains indicators of creative thinking. Based on that, the purpose of this study is to describe the development of creativity assessment instruments that have been carried out by previous researchers through journal literature sources. The research method uses a systematic literature review of 13 selected articles that meet the criteria for achieving the research objectives. From the analysis of the literature review, conclusions are drawn, 1) Creative thinking instruments have been developed through the 4D research model, ADDIE and Borg & Gall with their respective stages, 2) Aspects of creative thinking used in developing tests refer to Torrance's theory (fluency, flexibility, originality, and elaboration). 3) In the technique of making a creative thinking item that refers to Torrance's theory, these four indicators must simultaneously form the basis for making questions. This means that through one creative thinking question can measuring fluency, flexibility, originality, and detail of the answers given by the test takers, 4) Testing the feasibility of creative thinking instruments includes validity, reliability, practicality, effectiveness, and item analysis (distinguishing power and level of difficulty).

The Standards Will Never Be Enough: A Racial Justice Extension

ABSTRACT Since 1952, the Standards for Educational and Psychological Testing has provided criteria for developing and evaluating educational and psychological tests and testing practice. Yet, we argue that the foundations, operations, and applications in the Standards are no longer sufficient to meet the current U.S. testing demands for fairness for all test takers. We propose racial justice extensions as principled ways to extend the Standards, through intentional actions focused on race and targeted at educational policies, processes, and outcomes in specific settings. To inform these extensions, we focus on four social-justice concepts: intersectionality derived from Black Feminist Theory; responsibility derived from moral philosophy; disparate impact derived from legal reasoning; and situatedness derived from social learning theories. We demonstrate these extensions and concepts in action by applying them to case studies of nursing licensure and placement testing.

Validation of the teacher education institution’s entrance test using the Rasch model

This study aimed to evaluate the reliability and validity of the College Entrance Test (CET) used by a teacher education institution to assess the general and specialization knowledge of prospective students in English, Mathematics, Science and Social Studies. The Rasch model was employed to analyze the data collected from the sample of 250 test takers for the general knowledge test, 122 for specialization in English, 74 for Mathematics, 122 for science and 77 for Social Studies. The measurement analysis components including person and item reliability, unidimensionality, person-item map, fit statistics, Point Measure Correlation (PTMEA) and item local dependence were used. The study findings indicated that some test components had poor person reliability and the degree of item difficulty was higher than the students' abilities. There were also concerns about the conformity to the unidimensionality criteria, suggesting an analysis of items that might form another dimension in the constructs of the tests, although each of the items is independent. Furthermore, some items were misfitting or overfitting the Rasch model. In conclusion, the CET needs improvement to ensure its quality as a reliable and valid selection tool for the college. The study's results provide significant insights into the CET's strengths and weaknesses that can guide the test developers in revising and enhancing the CET to effectively measure the general and specialization knowledge of test takers in English, Mathematics, Science and Social Studies.