Multiple-choice Items Research Articles

Increased accessibility of computer-based testing for residency application to a hospital in Brazil with item characteristics comparable to paper-based testing: a psychometric study.

With the COVID-19 pandemic, online high-stakes exams have become a viable alternative. This study evaluated the feasibility of computer-based testing (CBT) for medical residency applications in Brazil and its impacts on item quality and applicants' access compared to paper-based testing. In 2020, an online CBT was conducted in a Ribeirao Preto Clinical Hospital in Brazil. In total, 120 multiple-choice question items were constructed. Two years later, the exam was performed as paper-based testing. Item construction processes were similar for both exams. Difficulty and discrimination indexes, point-biserial coefficient, difficulty, discrimination, guessing parameters, and Cronbach's alpha coefficient were measured based on the item response and classical test theories. Internet stability for applicants was monitored. In 2020, 4,846 individuals (57.1% female, mean age of 26.64 ± 3.37 years) applied to the residency program, versus 2,196 individuals (55.2% female, mean age of 26.47 ± 3.20 years) in 2022. For CBT, there was an increase of 2,650 (120.7%) applicants, albeit with significant differences in demographic characteristics. There was a significant increase in applicants from more distant and lower-income Brazilian regions, such as the North (5.6% vs. 2.7%) and Northeast (16.9% vs. 9.0%). No significant differences were found in difficulty and discrimination indexes, point-biserial coefficients, and Cronbach's alpha coefficients between the 2 exams. Online CBT with multiple-choice questions was a viable format for a residency application exam, improving accessibility without compromising exam integrity and quality.

Psychometric properties of the Ethiopian national licensing exam in medicine: an analysis of multiple-choice questions using classical test theory

Background : The Ethiopian Ministry of Health introduced medical licensure examinations to maintain high standards in medical practice and build public trust in healthcare professionals. Studies also suggested significant issues in clinical competence among Ethiopian junior doctors as well concerns regarding unlicensed practice. Given the need to ensure safe health care, we investigated the psychometric properties of the multiple-choice items comprising the Ethiopian national licensing exam (NLE). These analyses help to provide an argument for the validity and reliability of the test scores. Method : We used a cross-sectional study design to analyze data from three cohorts of undergraduate medicine licensing examinations in Ethiopia (2020-2022, N = 2,213). Using Classical Test Theory, we assessed the psychometric properties of 600 MCQ items with 2400 single best answer choices, specifically item difficulty, item discrimination, and the number of nonfunctional distractors, and scale reliability. We provide results regarding the overall test and its sub-domains. Results : Ethiopia’s undergraduate medical licensure examination demonstrated acceptable reliability (Alpha > 0.80), with significant variability in item difficulty and examinee performance. Although these results indicate a sufficiently defensible exam, our results point to issues regarding item statistics, especially a high number of nonfunctional distractors. Conclusions : This study provides first evidence regarding the psychometric soundness of the Ethiopian NLE. However, a significant number of items should be carefully reviewed and possibly revised. As the examination is relatively new, ongoing refinement to item-development and review processes is essential to improve and ensure its quality.

The Impact of Using the Brainstorming Method on the Listening Skills of Tenth Grade Students

This study explored the impact using the brainstorming method on the listening skills of tenth grade students. To meet the study’s goals, the researchers adopted the experimental approach. They developed an instrument (i.e. listening skills test). This test consists from twenty (20) multiple choice items. The sample consists from 143 female and male 10th grade students. Those students were chosen from Marka, Amman. The researchers chose the sample through using the cluster random method. The data were collected during the first semester of the academic year (2023 / 2024). The sampled students were chosen from four sections. Two sections serve as the experimental group and two sections serve as the control group. It was found that there are significant differences between the experimental and control groups in terms of the mean score on the post-test. It was found that there are significant differences between the scores of the respondents which can be attributed to gender for the favor of males. Based on the results, the researchers recommend using the brainstorming method in curricula and the teachers’ guidebook. They recommend conducting further studies to identify the impact of the brainstorming method on other language skills and other students. Received: 14 July 2024 / Accepted: 27 October 2024 / Published: 05 November 2024

Developing Distractors for Mathematics Multiple Choice Items: A Literature Review

Abstract Introduction: This article aimed at digging deep into distractors used for mathematics multiple-choice items. The quality of distractors may be more important than their number and the stem in a multiple-choice question. Little attention is given to this aspect of item writing especially, mathematics multiple-choice questions. This article provides an engaging but succinct literature review on the development of plausible distractors for mathematics multiple choice items. Methods: A literature review was conducted on developing effective distractors for mathematics tests. The review explored potential strategies for generating distractors that effectively assess students’ understanding and problem-solving skills in mathematics. Results: Alternative sources of distractors other than students’ misconceptions are provided to aid the development of plausible distractors for mathematics multiple-choice items. Discussion: Practical guidelines for judging distractors fit for mathematics questions are provided to help teachers improve their item writing skills based on literature and experience as a mathematics teacher. Common pitfalls in distractor development were identified to enable mathematics teachers to have a clear path for their work. Limitations: This article focused on the development of plausible distractors specifically for mathematics multiple choice questions. Conclusions: Test constructors must ensure aligning the distractors with the objectives of each lesson in order to make the distractors relevant to the demands of the item.

Strengthening the Multiple-Choice Assessment: Improving Item-Writing Skills of Physician Assistant Educators.

Physician assistant (PA) educators are tasked with the challenging privilege of ensuring future PAs are knowledgeable and competent to practice medicine. Most enter academia without formal education in providing education. They learn as they go-a treacherous and steep learning curve. This study provides PA educators literature and skills to bolster the strength and validity of multiple-choice examination items to measure learner progress and identify knowledge gaps. Skills can be improved at the level of the individual educator and program or institution. Approaching item writing and item revision in a stepwise, evidence-based manner can improve the quality of the items, strengthening multiple-choice examinations. This process can yield improved assessments of students' medical knowledge and competence. This study aims to provide the PA educator with techniques to improve their item-writing skills.

The Effectiveness of Two Feedback Strategies for Teaching Vocabulary Among EFL Students

The current research aims to assess the effectiveness of two feedback strategies, (positive and corrective feedback), in enhancing the acquisition of English language vocabulary among first year students who studied English as foreign language in the English Language Department at the College of Education for women, Tikrit University, Iraq , The study is quantitative study utilizes an experimental approach, with a quasi-experimental design involving pre-test and post-test assessments, The study sample included 120 first-year students and they were divided into two experimental groups (the first included 40 students studying with corrective feedback), the second (included 40 students studying with positive feedback), and the third (control group included 40 students who did not receive any feedback). To achieve the aim of the study, a vocabulary acquisition test consisting of 60 multiple-choice items is designed, after ensuring the validity and reliability of the test by using suitable statistics methods. The test is applied pre and post to the three groups The answers are collected, and the arithmetic averages and statistical significance were calculated by using ( spss software program ) the results shows that : Both ,the corrective and positive feedback strategies have a significant impact on students' vocabulary acquisition. There were statistically significant differences at the 0.05 significance level between the students in the first experimental group, who were taught using corrective feedback, and those in the control group, who were taught using traditional methods, in favor of the experimental group. There are statistically significant differences at the 0.05 significance level between the students in the second experimental group, who were taught using positive feedback, and those in the control group, who were taught using traditional methods, in favor of the experimental group. There are also statistically significant differences at the 0.05 significance level between the students in the first experimental group, who were taught using corrective feedback, and those in the second experimental group, who were taught using the positive feedback strategy, in favor of the corrective feedback.

The Comparison of PISA 2015-2018 Mathematics Trend Items Based on Item Response Times

This study aims to explore the intricate relationship between students' response times, item characteristics, and the effort invested during the Programme for International Student Assessment (PISA) 2015 and 2018 cycles. Through the analysis of data obtained from 69 mathematics trend items administered in a computer-based format across both cycles, this research investigates the dynamics of students' response times and their implications on effort and item characteristics. Findings reveal a significant increase in students' mean response times in the 2018 cycle compared to 2015, indicating potentially heightened effort and solution behavior. Notably, item formats exerted a substantial influence on response times, with open-ended items consistently eliciting lengthier response times compared to multiple-choice items. Additionally, a correlation between response times and item difficulty emerged, suggesting that more challenging items tend to consume more time, possibly due to the complexity of involved cognitive processes. Item based effort, assessed through Response Time Fidelity (RTF) indices, highlighted that a majority of students exhibited solution behavior across both cycles to the items.. Moreover, a decrease in the proportion of students displaying rapid-guessing behavior was observed in the 2018 cycle, potentially reflecting increased engagement with the assessment. While providing insights into the interplay of response times, item characteristics, and effort, this study emphasizes the need for further exploration into the multifaceted nature of effort in educational assessments. Overall, this research contributes valuable perspectives on nuances surrounding test performance and effort evaluation within PISA mathematics assessments.

Telesimulation Use in Emergency Medicine Residency Programs: National Survey of Residency Simulation Leaders

Introduction: Coronavirus 2019 (COVID-19) accelerated the need for virtual learning including telesimulation. Many emergency medicine (EM) programs halted in-person simulation and trialed telesimulation, but specifics on its utilization and plans for future use are unknown. Telesimulation has been defined as “a process by which telecommunication and simulation resources are utilized to provide education, training, and/or assessment to learners at an off-site location.” Our objective in this study was to describe the patterns of telesimulation usage in EM residency programs during COVID-19-induced learning restrictions as well as its anticipated future utility. Methods: We identified EM simulation leaders via the EMRA Match website, institutional websites, or personal contact with residency coordinators and directors, and invited them to participate by email. Participants completed a confidential, web-based survey consisting of multiple-choice items and one free-response question, developed by our study team with consideration of survey research best practices and Messick’s validity framework. We collected data between January–February 2022. We calculated descriptive statistics for multiple-choice items and examined the free-response answers for common themes. Results: We obtained contact information for simulation leaders at 139 EM residency programs. Survey response rate was 65% (91/139). During in-person restrictions, 62% (56/91) of programs used telesimulation. Assuming all restrictions lifted, 38% (34/90) of respondents planned to continue to use telesimulation, compared to 9% (8/91) using telesimulation before COVID-19. Most respondents planned to use telesimulation for medical knowledge (26/34, 76%) and communication/teamwork-focused cases (23/34, 68%). In response to the free-response question regarding experience with and plans for use, we identified three major themes: 1) telesimulation is a valuable alternative to in-person learning; 2) telesimulation is an option for learners unable to participate in person; and 3) telesimulation is challenging for procedural education. Conclusion: Despite the relatively limited use of telesimulation in EM residencies prior to COVID-19, an increased number of programs have plans to continue incorporating telesimulation into their curricula. This plan for continued use opens opportunities for further innovation and scholarship within simulation education.

The legality and appropriateness of keeping Korean Medical Licensing Examination items confidential: a comparative analysis and review of court rulings.

This study examines the legality and appropriateness of keeping the multiple-choice question items of the Korean Medical Licensing Examination (KMLE) confidential. Through an analysis of cases from the United States, Canada, and Australia, where medical licensing exams are conducted using item banks and computer-based testing, we found that exam items are kept confidential to ensure fairness and prevent cheating. In Korea, the Korea Health Personnel Licensing Examination Institute (KHPLEI) has been disclosing KMLE questions despite concerns over exam integrity. Korean courts have consistently ruled that multiple-choice question items prepared by public institutions are non-public information under Article 9(1)(v) of the Korea Official Information Disclosure Act (KOIDA), which exempts disclosure if it significantly hinders the fairness of exams or research and development. The Constitutional Court of Korea has upheld this provision. Given the time and cost involved in developing high-quality items and the need to accurately assess examinees' abilities, there are compelling reasons to keep KMLE items confidential. As a public institution responsible for selecting qualified medical practitioners, KHPLEI should establish its disclosure policy based on a balanced assessment of public interest, without influence from specific groups. We conclude that KMLE questions qualify as non-public information under KOIDA, and KHPLEI may choose to maintain their confidentiality to ensure exam fairness and efficiency.

Improving Secondary School Students’ Achievement in Chemistry Through Interaction Pattern Approach

The study determined the effect of interaction pattern approach on secondary students’ achievement in Chemistry. The study was carried out in Awka South Local Government Area of Anambra State. The study employed a quasi-experimental research design. Specifically, the pre-test, post-test non-equivalent control group design was used. A sample size of 112 (45 male & 67 female) out of the 1,522 senior secondary two (SS2) Chemistry students was selected using purposive and simple random sampling techniques. A 50 multiple-choice objective items Chemistry Achievement Test (CAT) developed by the researchers was the instrument used for data collection. The reliability index of 0.92 was obtained for CAT using Kendall’s Coefficient of Concordance (w). Data were analysed mean and standard deviation to answer the research questions and Analysis of Covariance (ANCOVA) to test the hypotheses at 0.05 level of significance. The findings of the study revealed that interaction pattern approach (IPA) significantly improves students’ achievement in Chemistry than the conventional lecture approach (CLA). Gender has no significant influence on students’ achievement in chemistry when taught using IPA. Also, there is no significant interaction effect of instructional approaches and gender on students’ achievement in chemistry. Based on the findings, it was recommended that chemistry teachers should make use of IPA to improve students’ achievement in the subject and promote gender friendly chemistry teaching and learning.

Global cross-sectional student survey on AI in medical, dental, and veterinary education and practice at 192 faculties

BackgroundThe successful integration of artificial intelligence (AI) in healthcare depends on the global perspectives of all stakeholders. This study aims to answer the research question: What are the attitudes of medical, dental, and veterinary students towards AI in education and practice, and what are the regional differences in these perceptions?MethodsAn anonymous online survey was developed based on a literature review and expert panel discussions. The survey assessed students' AI knowledge, attitudes towards AI in healthcare, current state of AI education, and preferences for AI teaching. It consisted of 16 multiple-choice items, eight demographic queries, and one free-field comment section. Medical, dental, and veterinary students from various countries were invited to participate via faculty newsletters and courses. The survey measured technological literacy, AI knowledge, current state of AI education, preferences for AI teaching, and attitudes towards AI in healthcare using Likert scales. Data were analyzed using descriptive statistics, Mann–Whitney U-test, Kruskal–Wallis test, and Dunn-Bonferroni post hoc test.ResultsThe survey included 4313 medical, 205 dentistry, and 78 veterinary students from 192 faculties and 48 countries. Most participants were from Europe (51.1%), followed by North/South America (23.3%) and Asia (21.3%). Students reported positive attitudes towards AI in healthcare (median: 4, IQR: 3–4) and a desire for more AI teaching (median: 4, IQR: 4–5). However, they had limited AI knowledge (median: 2, IQR: 2–2), lack of AI courses (76.3%), and felt unprepared to use AI in their careers (median: 2, IQR: 1–3). Subgroup analyses revealed significant differences between the Global North and South (r = 0.025 to 0.185, all P < .001) and across continents (r = 0.301 to 0.531, all P < .001), with generally small effect sizes.ConclusionsThis large-scale international survey highlights medical, dental, and veterinary students' positive perceptions of AI in healthcare, their strong desire for AI education, and the current lack of AI teaching in medical curricula worldwide. The study identifies a need for integrating AI education into medical curricula, considering regional differences in perceptions and educational needs.Trial registrationNot applicable (no clinical trial).

Development and validation of a multiple-choice test for sustainability competence in primary school using the GreenComp framework

There is no standardized definition or measurement tool for sustainability competence, which hinders comparative research. This study addresses this gap by developing a new test based on the European reference framework GreenComp. The proposed instrument, consisting of eleven multiple-choice items, was validated through expert review and exploratory and confirmatory factor analysis, showing strong construct validity and high internal consistency reliability. Applied to students aged 8 to 12, it revealed a prevalence of basic and anthropocentric perspectives on sustainability. This promising tool can assess sustainability competence in primary education and has the potential for use across European countries.

The Effect of Presenting Geometry Items with and without Shapes on the Psychometric Properties of the Test and Students’ Test Scores

This research intended to examine the effects of presenting geometry items with and without shapes on the psychometric properties of the test and students’ test scores. The study was conducted on 480 eighth grade students. Within the scope of the study, two geometry tests were crafted, one with shapes and the other without shapes. Both tests consisted of 15 multiple-choice items. In the data collection process, a counterbalanced design was followed and the two tests were administered to the students three weeks apart. Analyses were carried out on 405 students who participated in both applications and whose test forms could be matched. The factor analysis results revealed that the factor loadings of the items and extracted variance were higher for the test with shapes compared to the shape-free form. The Cronbach’s alpha coefficient of the test containing shapes was found to be significantly higher than that calculated for the test without shapes. According to item difficulties, the questions with shapes were easier for the students than the shape-free questions. In terms of discrimination indices a difference in favor of the shape-containing test was observed in almost all items. Ferguson delta statistic, which is a measure of discrimination for the overall test, was higher in the shape-contaninig test. Correlation analysis denoted a strong positive relationship between student scores on the two tests. The paired sample t-test proved that there was a statistically significant difference between students’ scores on the shape-containing and shape-free tests. These results indicate that the geometry tests with and without shapes differ in terms of both psychometric properties and students’ test scores.

Geometric thinking program to enhance problem-solving skills of primary school teacher candidates

Geometry is a mathematics subject that has been widely considered hard. Several problems in learning geometry lead to geometric problem-solving skills. After exploring various causes, theories, and processes of geometric thinking, there should be an evaluation program to optimize problem-solving skills. This study aims to (1) develop a geometric thinking evaluation program and (2) enhance geometric problem-solving skills observed from the students’ thinking level. The research and development adopted the Analysis, Design, Development, Implementation, and Evaluation (ADDIE) model, which started from the analysis stage involving four lecturers and 306 of students. The research was conducted at Primary School Teacher Education Department, Universitas Negeri Semarang. The process continued to the design stage, at which the evaluation program was drafted, followed by the development stage to test the validity. The last two phases included the trials in which the t-test and n-gain were applied to measure the program's effectivity and the evaluation phase aimed at assessing its practicality and the student's achievement in geometry. The established evaluation program covers an instrument of 39 multiple-choice items as an early assessment to determine the thinking level and learning trajectories as a scaffolding for improving geometric thinking skills. The trajectories, in the form of worksheets, were adjusted to the students' thinking level and were effective in assisting them in elevating to the informal reasoning level as well as refining their problem-solving skills.

The effectiveness of a proposed strategy in light of anchored instruetion learning on achievement among middle school students

The research aims to identify the effectiveness of a proposed strategy in light of Anchored instruetion Learning in achieving achievement among middle school students. To achieve the aim of the research, the two researchers adopted an experimental research approach with two groups (experimental and control) with a post-test, and the null hypothesis was developed: There is no statistically significant difference at the level of significance (0.05) between the average scores of the experimental group students who studied according to the proposed strategy and the scores of the control group students who studied according to the normal method in the achievement test. The research sample consisted of (60) fifth-grade science students at Al-Suyuti Preparatory School affiliated with the General Directorate of Education of Baghdad Al-Karkh II. The sample was distributed into two groups: The experimental group consisted of (30) students who studied according to the proposed strategy. 2. The control group consisted of (30) students who studied in the usual way. The two research groups were rewarded on the variables (previous achievement, chronological age, previous knowledge, intelligence, and mathematical culture). After that, an achievement test was constructed, consisting of (30) objective, multiple-choice items. Appropriate statistical analyzes were conducted, and the psychometric properties of the test were verified. The results resulted The students in the experimental group who studied with the proposed strategy outperformed the students in the control group who studied in the usual way. In light of the results, a set of conclusions, recommendations and proposals were reached.

Using Performance Improvement Methods to Evaluate Processes for Writing Multiple-Choice Test Questions in the Postlicensure Clinical Environment: A Case Study.

This article is the last of a four-part series to guide educators on the construction and evaluation of multiple-choice test items in the post-licensure environment. Previous articles in this series described the problem and the mechanics of test item construction and evaluation. A replicable strategy for evaluating the organizational process for constructing multiple-choice test questions is provided. Steps taken to create change are described; work tools are provided. Guidance and training are needed to create multiple-choice test questions. Many educators have not had training in item construction. Educators welcomed training. Personalized mentorship resulted in improvement. Asynchronous learning alone was helpful and well received and improved self-perceived knowledge, yet fell short of achieving competence. Voluntary training may not be adequate to assure enculturation of best practices without accountability standards and monitoring. Future research is indicated to assess the situation and provide national standards for adoption within health care organizations. [J Contin Educ Nurs. 202x;5x(x):xx-xx.].

The use of ChatGPT in assessment

ChatGPT has surged interest to cause people to look for its use in different tasks. However, before allowing it to replace humans, its capabilities should be investigated. As ChatGPT has potential for use in testing and assessment, this study aims to investigate the questions generated by ChatGPT by comparing them to those written by a course instructor. To investigate this issue, this study involved 36 junior students who took a practice test including 20 multiple-choice items generated by ChatGPT and 20 others by the course instructor, resulting in a 40-item test. Results indicate that there was an acceptable degree of consistency between the ChatGPT and the course instructor. Post-hoc analyses point to consistency between the instructor and the chatbot in item difficulty, yet the chatbot’s results were weaker in item discrimination power and distractor analysis. This indicates that ChatGPT can potentially generate multiple-choice exams similar to those of the course instructor.

How Employee Well-being Mediates the Relationship between Employee Experience and Work Engagement?

The purpose of this study was to examine the impact of employee experience on work engagement. A questionnaire consisting of both open-ended and multiple-choice items was administered to 308 respondents working in seven main insurance companies operating in Turkey. Employee experience is divided into three dimensions: the perceived importance of managers, work meaningfulness, and career-focused experience. According to the results of this study, employee experience has an important effect on work engagement. Employee well-being partially mediates the relationship between employee experience and work engagement. The importance of the research is that it shows that employee experience practices and welfare policies implemented by companies will ultimately result in work engagement. We emphasize the importance of providing employees with a positive experience throughout their journey, from recruitment to post-employment, as it leads to positive outcomes at both an individual and organizational level.

Analysis of the current situation and demand for perinatal education in pregnant women in general grade A tertiary hospitals in China: a cross-sectional survey

ObjectivesThis study aimed to assess the factors influencing pregnant women’s participation in perinatal education and their demand for educational courses.DesignA cross-sectional study.SettingTertiary hospital in Suzhou, China, July–September 2022.ParticipantsPregnant women from...

Is the use of multiple-choice items and a holistically-scored paragraph translation task fair? Examining a large-scale translation subtest

AbstractThis study investigated the test fairness of the translation section of a large-scale English test in China by examining its Differential Test Functioning (DTF) and Differential Item Functioning (DIF) across gender and major. Regarding DTF, the entire translation section exhibits partial strong measurement invariance across female and male test takers, while exhibiting full measurement invariance across test takers in (1) arts &amp; humanities and social sciences (A&amp;HSS) and (2) science, technology, engineering or mathematics (STEM) majors. No major-based DIF was detected in this study. Objective test items tend to favor male test takers, while direct translation test task was more favorable to females. Combining the DIF and DTF results, there may be a cancelation effect in our case. However, the effect size of DIF is either negligible or slight to moderate, indicating minimal impact on the overall fairness of the translation test task. This study further discusses the necessity of exploring the source of DIF and the importance of combining DIF and DTF for test fairness research.