Multiple Choice Questions Items Research Articles

Docimological Quality Analysis of LLM-Generated Multiple Choice Questions in Computer Science and Medicine

Assessment is an essential part of education, both for teachers who assess their students as well as learners who may evaluate themselves. Multiple-choice questions (MCQ) are one of the most popular types of knowledge assessment, e.g., in medical education, as they can be automatically graded and can cover a wide range of learning items. However, the creation of high-quality MCQ items is a time-consuming task. The recent advent of Large Language Models (LLM), such as Generative Pre-trained Transformer (GPT), caused a new momentum for automatic question generation solutions. Still, evaluating generated questions according to the best practices for MCQ item writing is needed to ensure docimological quality. In this article, we propose an analysis of the quality of LLM-generated MCQs. We employ zero-shot approaches in two domains, namely computer science and medicine. In the former, we make use of 3 GPT-based services to generate MCQs. In the latter, we developed a plugin for the Moodle learning management system that generates MCQs based on learning material. We compare the generated MCQs against common multiple-choice item writing guidelines. Among the major challenges, we determined that while LLMs are certainly useful in generating MCQs more efficiently, they sometimes create broad items with ambiguous keys or implausible distractors. Human oversight is also necessary to ensure instructional alignment between generated items and course contents. Finally, we propose solutions for AQG developers.

Quality of multiple choice question items: item analysis

Background: There are different types of exam formats for educational assessment. Multiple choice questions (MCQs) are frequently utilized assessment tools in health education. Considering the reliability and validity in developing MCQ items is vital. Educators often face the difficulty of developing credible distractors in MCQ items. Poorly constructed MCQ items make an exam easier or too difficult to be answered correctly by students as intended learning objectives. Checking the quality of MCQ items is overlooked and too little is known about it. Therefore, this study aimed to assess the quality of MCQ items using the item response theory model. Methods: A descriptive cross-sectional study was conducted among MCQ items of public health courses administered to 2nd year nursing students at Injibara university. A total of 50 MCQ items and 200 alternatives were evaluated for statistical item analysis. The quality of MCQ items was assessed by difficulty index (DIF), discrimination index (DI), and distractor efficiency (DE) using students’ exam responses. Microsoft excel sheet and SPSS version 26 were used for data management and analysis. Results: Post-exam item analysis showed that 11 (22%) and 22 (44%) MCQs had too difficult and poor ranges for difficulty and discriminating powers respectively. The overall DE was 71.3%. About forty (20%) distractors were non-functional. Only 8 (16%) MCQs fulfilled the recommended criteria for all-DIF, DI, and DE parameters. Conclusions: The desirable criteria for quality parameters of MCQ items were satisfied only in a few items. The result implies the need for quality improvement. Continuous trainings are required to improve the instructors’ skills to construct quality educational assessment tools.

Ten tips for effective use and quality assurance of multiple-choice questions in knowledge-based assessments.

Multiple-choice questions (MCQs) are the most popular type of items used in knowledge-based assessments in undergraduate and postgraduate healthcare education. MCQs allow assessment of candidates' knowledge on a broad range of knowledge-based learning outcomes in a single assessment. Single-best-answer (SBA) MCQs are the most versatile and commonly used format. Although writing MCQs may seem straight-forward, producing decent-quality MCQs is challenging and warrants a range of quality checks before an item is deemed suitable for inclusion in an assessment. Like all assessments, MCQ-based examinations must be aligned with the learning outcomes and learning opportunities provided to the students. This paper provides evidence-based guidance on the effective use of MCQs in student assessments, not only to make decisions regarding student progression but also to build an academic environment that promotes assessment as a driver for learning. Practical tips are provided to the readers to produce authentic MCQ items, along with appropriate pre- and post-assessment reviews, the use of standard setting and psychometric evaluation of assessments based on MCQs. Institutions need to develop an academic culture that fosters transparency, openness, equality and inclusivity. In line with contemporary educational principles, teamwork amongst teaching faculty, administrators and students is essential to establish effective learning and assessment practices.

Level of Comprehension of the Material of Big Ball Game in the Online Physical Education Learning for the Fifth Grade Students of Kanisius Kalasan Elementary School

This research is conducted to determine the level of comprehension on the material of bi ball game in the online Physical Education learning for the fifth grade students of Kanisius Kalasan Elementary School. This research was a descriptive quantitative study. This study used a population as all samples were used as a population, the research population was for about 46 fifth grade students of SD Kanisius Kalasan. The data collection technique sed a written test in the form of multiple choice question items. The data analysis used the descriptive quantitative analysis with the results of the assessment of all fifth grade students were carried out with the same tool which would be summarized into 5 levels: very high, high, medium, low, and very low levels.Based on the results of the research, it can be concluded that the level of comprehension on the Big Ball Game material in the online Physical Education learning for the fifth grade students of SD Kanisius Kalasan is as follows: 35 respondents (76.1%) are in the Very High level, 2 respondents (4.34%) are in the High level, 4 respondents (8.7%) are in the Medium level, 2 respondents (4.34%) are in the Low level, and 3 respondents (6.52%) are in the Very Low level. The level of comprehension on the Big Ball Game Material in the online Physical Education learning for the fifth grade students of SD Kanisius Kalasan is very high.

Study on item analysis of multiple – choice questions amongst the undergraduate dental students

Assessments are powerful tools for bringing on the achievement of educational goals and one of the tools for bringing it rightly is “item analysis.” Multiple - Choice Questions (MCQs) are one of the most common methods of assessing the knowledge capabilities of undergraduate and postgraduate medical students. The main objective of this study was to analyze the quality of MCQs and to assess the relationships of items having good difficulty and discrimination indices with their distracter efficiency. After getting an ethical approval from Institutional Review Committee of Nepal Medical College Teaching Hospital a cross – sectional study was conducted over a period of one year from January 2022 to December 2022 in the Department of Human Anatomy of Nepal Medical College. The difficulty index (DIF I), discrimination index (DI) and distracter efficiency (DE) were calculated and analyzed. The results obtained revealed that the mean and standard deviation of DIF I, DI and DE were 56.75±22.6, 0.3±0.2 and 43.24±25.66 respectively. The items analyzed were neither too easy and nor too difficult (DIF I = 62.8%). The majority of items fulfilled the criteria of acceptable difficulty and good discrimination index and the maximum number of items were found to have 2 NFDs (DE = 33.33 %). The present study was hence undertaken as it assists to evaluate the MCQ items to discern its effectiveness in assessing the knowledge of students as well as it plays a vital role in developing a question bank having valid MCQs.

Assessment of the Quality of Multiple-Choice Questions in the Surgery Course for an Integrated Curriculum, University of Bisha College of Medicine, Saudi Arabia.

Multiple-choice questions (MCQs) have been recognized as reliable assessment tools, and incorporating clinical scenarios in MCQ stems has enhanced their effectiveness in evaluating knowledge and understanding. Item analysis is used to assess the reliability and consistency of MCQs, indicating their suitability as an assessment tool. This study aims to ensure the competence of graduates in serving the community and establish an examination bank for the surgery course. This study aims to assess the quality and acceptability of MCQs in the surgery course at the University of Bisha College of Medicine (UBCOM). A psychometric study evaluated the quality of MCQs used in surgery examinations from 2019 to 2023 at UBCOM in Saudi Arabia. The MCQs/items were analyzed and categorized for their difficulty index (DIF), discrimination index (DI), and distracter efficiency (DE) Fifth-year MBBS students undergo a rotation in the department and are assessed at the end of 12 weeks. The assessment includes 60 MCQs/items and written items. Data was collected and analyzed using SPSS version 24. A total of 189 students were examined across five test sessions, with 300 MCQ items. Student scores ranged from 28.33% to 90.0%, with an average score of 64.6%±4.35. The 300 MCQ items had a total of 900 distractors. The DIF was 75.3% for the items, and 63.3% of the items showed good discrimination. No items had negative points in terms of biserial correlation. The mean number of functional distractors per test item was 2.19±1.007, with 34% of the items having three functional distractors. The psychometric indices used to evaluate the MCQs in this study were encouraging, with acceptable DIF, distractor efficiencies, and item reliability. Providing robust faculty training and capacity-building is recommended to enhance item development skills.

ChatGPT takes on the european exam in core cardiology: an AI success story

Abstract Background ChatGPT, the trending novel artificial intelligence has triggered ongoing debate regarding its capabilities.Recently, preliminary reports showed that answered correctly in the majority of questions of the United States Medical License Examinations (USMLE). However, its ability to succeed a more precise, challenging and high-stakes post-graduate test, such as the final exam for the completion of medical residency , like the European Exam in Core Cardiology (EECC), is not known yet. Purpose We sought to evaluate the performance of ChatGPT on EECC, to test its capability on a more demanding, high-stakes post-graduate exam in Cardiology training. Methods A total of 488 publicly-available single-answer multiple choice questions (MCQs) were randomly obtained from three different MCQs sources that are traditionally used for the preparation for the EECC: 88 from the sample exam questions released since 2018 from the official ESC website, 200 from the 2022 edition of StudyPRN and 200 from the Braunwald's Heart Disease Review and Assessment (BHDRA). Questions containing audio or visual assets were excluded. After filtering, 362 MCQ items (ESC sample: 68, BHDRA:150, StudyPRN: 144) were included and considered as input source. False responses and indeterminate responses were considered as not correct. Results ChatGPT answered to 340 questions out of 362, with 22 indeterminate answers in total. The overall accuracy was 58.8% across all the question sources. More specifically, it demonstrated an accuracy for the ESC sample, BHDRA and StudyPRN of 61.7%, 52.6%, 63.8% respectively. It answered correctly 42/68 (4 indeterminate) of ESC sample questions, 79/150 (11 indeterminate) of the BHDRA and 92/144 (7 indeterminate) of the StudyPRN. Conclusion ChatGPT manages to correctly answer the majority of EECC’s questions and perform within the passing threshold range. Although it cannot yet process visual content, it can provide rational and correct answers to text-based inputs in most scenarios. The model may be able to efficiently handle a massive amount of acquired medical knowlededge, but the current approach may not substitue for critical thinkg, innovation and creativity; some of the key attibutes that doctors are expected to showcase.Performance of ChatGPT on EECCExample of MCQ input at ChatGPT

Gender differences in multiple-choice questions and the risk of losing points

I study the gender differences in performance in multiple-choice questions in a setting where wrong answers are penalized and the objective is to score as high points as possible. I exploit data from an undergraduate level microeconomics course at a Finnish university across a six-year period of 2010 and 2012-2016. The course consists of two equally weighted exams that include both multiple-choice and open-ended questions. The results show that, when controlling for the performance in the first exam, women omit more multiple-choice question items (MCQ) in the second exam than men which, in turn, translates to fewer points. Women do not do worse in open-ended questions that are similar to the MCQ, neither is the probability of them answering incorrectly to the MCQ higher. Hence, gender differences in test results might reflect differences in behavior in a very particular test setting rather than genuine differences in skills.

Korean High School Students' L1 Summary Writing as a Predictor of EFL Reading Comprehension

This study examines the role of the L1 summarization task as a predictor of English reading comprehension for Korean EFL high school students. Participants were given two different types of reading passages, along with five multiple-choice question items and one summarization task item for each passage. A rubric for summary writing was constructed for the purpose of the study and examined on its reliability and use. Correlation analysis between reading comprehension and summary writing reported a moderate to high correlation. A simple linear regression analysis of the test results indicates that L1 summary writing can be a predictor of Korean EFL learners' English reading comprehension. Despite the findings in this study, the survey results inquiring about the participants' views on summarization tasks suggest the difficulties and limitations of implementing summarization tasks in high-stakes assessments.

Feedback, fairness, and validity: effects of disclosing and reusing multiple-choice questions in medical schools

ABSTRACT Background Disclosure of items used in multiple-choice-question (MCQ) exams may decrease student anxiety and improve transparency, feedback, and test-enhanced learning but potentially compromises the reliability and fairness of exams if items are eventually reused. Evidence regarding whether disclosure and reuse of test items change item psychometrics is scarce and inconclusive. Methods We retrospectively analysed difficulty and discrimination coefficients of 10,148 MCQ items used between fall 2017 and fall 2019 in a large European medical school in which items were disclosed from fall 2017 onwards. We categorised items as ‘new’; ‘reused, not disclosed’; or ‘reused, disclosed’. For reused items, we calculated the difference from their first ever use, that is, when they were new. Differences between categories and terms were analysed with one-way analyses of variance and independent-samples t tests. Results The proportion of reused, disclosed items grew from 0% to 48.4%; mean difficulty coefficients increased from 0.70 to 0.76; that is, items became easier, P < .001, ηp 2 = 0.011. On average, reused, disclosed items were significantly easier (M = 0.83) than reused, not disclosed items (M = 0.71) and entirely new items (M = 0.66), P < .001, ηp 2 = 0.087. Mean discrimination coefficients increased from 0.21 to 0.23; that is, item became slightly more discriminating, P = .002, ηp 2 = 0.002. Conclusions Disclosing test items provides the opportunity to enhance feedback and transparency in MCQ exams but potentially at the expense of decreased item reliability. Discrimination was positively affected. Our study may help weigh advantages and disadvantages of using previously disclosed items.

The Impact of STAD Strategy on the Academic Achievement of Eighth-Grade Students in Sciences Subject and Developing their Scientific Trends

This research aims to introduce “The Impact of the STAD Strategy on the Academic Achievement of Eighth-Grade Students in Science subjects and Develop their Scientific Trends”. The researcher has constructed two null hypotheses, to reach the objectives of the research. The research's population was limited to eighth-grade basic students at (Delzyan Basic School for Girls) in Soran District the academic year's second semester (2021 - 2022). The research sample has been purposefully chosen. The research included an experimental setup with two groups: an experimental group and a control group, as well as pretest and posttest for the variable (Scientific Trends) and posttest for the variable (Academic Achievement). As a result, the STAD method was used to instruct the experimental group, while, the traditional teaching techniques were used to instruct the control group. The sample of this research consists of (41) students, distributed between two groups; the experimental group includes (21) students, and (20) students in the control group. The equality between the two research groups was tested in a variety of ways. The researcher created two research tools, the first of which consisted of (30) multiple-choice question items to assess the student's academic achievement. The second instrument, which consists of 38 items, is used to assess students' attitudes about science. The tools' authenticity and dependability have been validated. The data was analyzed and the result was obtained using statistical software (SPSS version 25). The results revealed that neither group's students had substantial achievements. &nbsp;In light of the research's findings, the researcher came to the following conclusions: STAD strategy has not affected eight-grade students’ academic achievement in the experimental group according to science subject, at the other hand, it has been of great importance in changing the trends of female students toward science positively.

Construction and Writing Flaws of the Multiple-Choice Questions in the Published Test Banks of Obstetrics and Gynecology: Adoption, Caution, or Mitigation?

Background The item-writing flaws (IWFs) in multiple-choice questions (MCQs) can affect test validity. The purpose of this study was to explore the IWFs in the published resources, estimate their frequency and pattern, rank, and compare the current study resources, and propose a possible impact for teachers and test writers. Methods This cross-sectional study was conducted from September 2017 to December 2020. MCQs from the published MCQ books in Obstetrics and Gynecology was the target resources. They were stratified into four clusters (study-book related, review books, self-assessment books, and online-shared test banks). The sample size was estimated and 2,300 out of 11,195 eligible MCQs were randomly selected. The MCQs (items) were judged on a 20-element compiled checklist that is organized under three sections as follows: (1) structural flaws (seven elements), (2) test-wiseness flaws (five elements), and (3) irrelevant difficulty flaws (eight elements). Rating was done dichotomously, 0 = violating and 1 = not violating. Item flaws were recorded and analyzed using the Excel spreadsheets and IBM SPSS. Results Twenty three percent of the items ( n = 537) were free from any violations, whereas 30% ( n = 690) contained one violation, and 47% ( n = 1073) contained more than one violation. The most commonly reported IWFs were “Options are Not in Order (61%).” The best questions with the least flaws (75th percentiles) were obtained from the self-assessment books followed by study-related MCQ books. The average scores of good-quality items in the cluster of self-assessment books were significantly higher than other book clusters. Conclusion There were variable presentations and percentages of item violations. Lower quality questions were observed in review-related MCQ books and the online-shared test banks. Using questions from these resources needs a caution or avoidance strategy. Relative higher quality questions were reported for the self-assessment followed by the study-related MCQ books. An adoption strategy may be applied with mitigation if needed.

Writing Multiple Choice Questions—Has the Student Become the Master?

Construct We compared the quality of clinician-authored and student-authored multiple choice questions (MCQs) using a formative, mock examination of clinical knowledge for medical students. Background Multiple choice questions are a popular format used in medical programs of assessment. A challenge for educators is creating high-quality items efficiently. For expediency’s sake, a standard practice is for faculties to repeat items in examinations from year to year. This study aims to compare the quality of student-authored with clinician-authored items as a potential source of new items to include in faculty item banks. Approach We invited Year IV and V medical students at the University of Adelaide to participate in a mock examination. The participants first completed an online instructional module on strategies for answering and writing MCQs, then submitted one original MCQ each for potential inclusion in the mock examination. Two 180-item mock examinations, one for each year level, were constructed. Each consisted of 90 student-authored items and 90 clinician-authored items. Participants were blinded to the author of each item. Each item was analyzed for item difficulty and discrimination, number of item-writing flaws (IWFs) and non-functioning distractors (NFDs), and cognitive skill level (using a modified version of Bloom’s taxonomy). Findings Eighty-nine and 91 students completed the Year IV and V examinations, respectively. Student-authored items, compared with clinician-authored items, tended to be written at both a lower cognitive skill and difficulty level. They contained a significantly higher rate of IWFs (2–3.5 times) and NFDs (1.18 times). However, they were equally or better discriminating items than clinician-authored items. Conclusions Students can author MCQ items with comparable discrimination to clinician-authored items, despite being inferior in other parameters. Student-authored items may be considered a potential source of material for faculty item banks; however, several barriers exist to their use in a summative setting. The overall quality of items remains suboptimal, regardless of author. This highlights the need for ongoing faculty training in item writing.

Assessing the Quality of MCQs in the Final Examination of UUM Foundation Students

The multiple-choice question (MCQ) format is commonly used to assess student knowledge as it is able to accommodate a large number of participants. The advantages of MCQs are that they are easy to handle and the results are able to be obtained quickly. To ensure that students are effectively tested, it is important to analyse the quality of MCQs by assessing the questions or items using educational measurements. The aim of this study is to assess the quality of MCQs of final exam questions of foundation students by using the Difficulty Index and Discriminant Index. The sample for this study is the Introduction to Philosophy final exam paper which is one of the courses taken by foundation students. The analysis involves 33 graded final exams and 20 MCQ items. The findings showed that there are 3 items that are ‘good’ that can be retained, 9 items that are considered as ‘fair acceptable/not acceptable’ which can be retained or revised, and 8 items that should be revised or discarded. The results of this study show that it is important to assess the quality of items in the assessment as it is able to change the selection of items in the assessment.

The effectiveness of experiential learning – STEM model with formative assessment in building students’ mastery on fluid dynamics

Fluid dynamics is related to science, technology, and daily lives, but students’ understanding of the topic is considered not optimal yet. This study aims to investigate the effectiveness of experiential learning (EL) – STEM model with formative assessment (FA) in building the mastery of fluid dynamics and explore students’ difficulties. This is a mixed-methods research with an embedded experimental design with 34 grade XI students of high school in Malang, Indonesia, as subjects, which were chosen with purposive sampling. This research used dynamics mastering test instrument in the form of 18 reasoned multiple-choice question items with a reliability of 0.65. The data were analyzed with Wilcoxon Signed Ranks test, N-gain, and the description of students’ answers. The result showed that EL-STEM with FA is effective to build the mastery of fluid dynamics with the effectiveness of the subtopic of Bernoulli’s law is higher than the subtopic of Continuity. Before learning, students had difficulties to understand almost all material of fluid dynamics. However, after learning, students only had difficulties to understand certain things, which are the relationship between physical quantities in Bernoulli’s law, the distance between the water reservoir and the water jet, the speed of air under or above the wing of an aeroplane, and the shape of aeroplane’s wing design.

Impact of a longitudinal faculty development program on the quality of multiple-choice question item writing in medical education.

Background:Like many other academic programs, medical education is incomplete without a robust assessment plan.Objective:The study aimed to evaluate the impact of longitudinal faculty development program (FDP) on the examination item quality (EIQ) from a cohort of medical college faculty members.Methods:Item analysis (IA) of multiple-choice questions (MCQs) from a cohort of medical tutors over a 3-year period (2017 [S1], 2018 [S2], and 2019 [S3]) before and following once-per-week FDP was conducted. The questions were from three randomly selected courses: man and his environment (MEV) from phase 1, central nervous system (CNS) from phase 2, and internal medicine (MED) from phase 3. Data assessed were 480 MCQs from the final exams in the courses. The parameters considered in IA were the difficulty index, index of discrimination, nonfunctional distractors (NFDs), distractor efficiency for each question item, and Cronbach's alpha (CA) for the test as a whole. Comparison over the 3 years was made using Fisher's exact test and repeated-measures ANOVA with Bonferroni test as post hoc test.Results:Overall, out of 480 MCQs, 272 had no NFD (52 [19.52%], 104 [38.24%], and 116 [42.65%] in 2017, 2018, and 2019, respectively) with a significant difference between S3, S2, and S1 (P < 0.0001). The mean CA for the exams in S1, S2, and S3, respectively, were 0.51, 0.77, and 0.84, P < 0.0001.Conclusion:There was an improvement in EIQ following the implementation of longitudinal FDP. Thus, the need for active training and retraining of the faculty for a better EIQ cannot be overemphasized.

Defining and tracking medical student self-monitoring using multiple-choice question item certainty

BackgroundSelf-monitoring is an important component of clinical practice. It is underpinned by the framework of self-efficacy which is concerned with judgments of how well one believes one can achieve or perform a task. This research aimed to develop criteria for adequate self-monitoring, then to measure patterns of self-monitoring, and to explore how these patterns relate to a student’s year in a medical course and to patterns of knowledge.MethodsAnalysis of individuals’ levels of correctness in answering assessment items and their certainty in correctness may be used to inform assessments of ability to self-monitor. Two criteria were proposed and applied to define adequate self-monitoring. Firstly, increasing proportions correct with increasing levels of certainty. Secondly, having a proportion correct for high certainty responses that was not lower than cohort levels. Student responses in progress tests comprising multiple-choice questions (MCQs) and associated certainty were analysed. Criteria for the presence of adequate self-monitoring and for adequate knowledge were applied to the results of each of four tests conducted over 2 years, and used to categorise patterns of self-monitoring and knowledge.ResultsData from 3 year group cohorts totalling 737 students were analysed. The majority (58%) of students demonstrated adequate knowledge and met both criteria for adequate self-monitoring across all four tests. The most advanced year group cohort had the highest rates of adequate knowledge and the highest rates of meeting both self-monitoring criteria. The patterns of self-monitoring were the same as the patterns of knowledge across the four tests for 454 students, but for the remaining 283 the patterns of self-monitoring and knowledge differed.ConclusionAnalysis of responses to item level certainty has informed development of a definition of adequate self-monitoring that may be applied to individual student’s responses from a single test, and to track the adequacy of a student’s self-monitoring over time. Patterns of self-monitoring tend to match patterns of knowledge, but not in all cases, suggesting the self-monitoring measure could provide additional information about student ability.

Impact Measurement on Medical Faculty for Adhering to Appropriate Guidelines in Framing Effective Multiple-Choice Questions for Item Analysis

Background: Multiple-choice questions (MCQs) are the most frequently accepted tool for the evaluation of comprehension, knowledge, and application among medical students. In single best response MCQs (items), a high order of cognition of students can be assessed. It is essential to develop valid and reliable MCQs, as flawed items will interfere with the unbiased assessment. The present paper gives an attempt to discuss the art of framing well-structured items taking kind help from the provided references. This article puts forth a practice for committed medical educators to uplift the skill of forming quality MCQs by enhanced Faculty Development programs (FDPs). Objectives: The objective of the study is also to test the quality of MCQs by item analysis. Methods: In this study, 100 MCQs of set I or set II were distributed to 200 MBBS students of Late Shri Lakhiram Agrawal Memorial Govt. Medical College Raigarh (CG) for item analysis for quality MCQs. Set I and Set II were MCQs which were formed by 60 medical faculty before and after FDP, respectively. All MCQs had a single stem with three wrong and one correct answers. The data were entered in Microsoft excel 2016 software to analyze. The difficulty index (Dif I), discrimination index (DI), and distractor efficiency (DE) were the item analysis parameters used to evaluate the impact on adhering to the guidelines for framing MCQs. Results: The mean calculated difficulty index, discrimination index, and distractor efficiency were 56.54%, 0.26, and 89.93%, respectively. Among 100 items, 14 items were of higher difficulty level (DIF I &lt; 30%), 70 were of moderate category, and 16 items were of easy level (DIF I &gt; 60%). A total of 10 items had very good DI (0.40), 32 had recommended values (0.30 - 0.39), and 25 were acceptable with changes (0.20 - 0.29). Of the 100 MCQs, there were 27 MCQs with DE of 66.66% and 11 MCQs with DE of 33.33%. Conclusions: In this study, higher cognitive-domain MCQs increased after training, recurrent-type MCQ decreased, and MCQ with item writing flaws reduced, therefore making our results much more statistically significant. We had nine MCQs that satisfied all the criteria of item analysis.

Evaluation of the Multiple-Choice Question Item Analysis of the Sixth Year Undergraduate Orthodontic Tests at the Faculty of Dentistry, King Abdulaziz University, Saudi Arabia

Aim: This study aims to assess the quality of the MCQ item analysis of dental students’ final year orthodontics examination using the difficulty index (DIF I), discrimination index (DI), and distractor efficiency (DE) parameters. Additionally, the relationship between these parameters is determined to categorize the tested MCQs into either acceptable, which would be stored in the MCQ bank, or substandard (poor), which would be eliminated or rephrased. Methods: Four MCQ examinations from the sixth-year undergraduate orthodontic course of the academic year 2018–2019 were evaluated by item analysis using the DIF I, DI, and DE. A total of 165 MCQ items (495 distractors) were collated and analyzed after 189 students answered the tests. Results: The mean and standard deviation of the DIF I and DI were 0.76 (+0.19) and 0.25 (+0.14), respectively. Distraction assessments revealed that 27.5% were non-functional distractors (NFDs) with a DE of 78.2%. There was a significant negative correlation between the DIF I and the DI (r: -0.711; p < 0.01). A significant positive correlation was found between the DIF I and NFDs (r: 0.644; p < 0.01). Of the 165 MCQ items, 153 were considered sufficiently good to be stored in the MCQ bank, and only 12 were poorly constructed and recommended for elimination or rephrasing. Conclusion: Item analysis was effective in identifying the good and poorly constructed MCQ items. Therefore, it should be used with each university MCQ examination for the development of a reliable and valid MCQ bank.

Knowledge of dental faculty in gulf cooperation council states of multiple-choice questions’ item writing flaws

ABSTRACT Multiple-Choice Questions provide an objective cost/time effective assessment. Deviation from appropriate question writing structural guidelines will most probably result in commonly ignored multiple-choice questions writing flaws, influencing the ability of the assessment to measure students’ cognitive levels thereby seriously affecting students’ academic performance outcome measures. To gauge the knowledge of multiple-choice question items writing flaws in dental faculty working at colleges in Gulf Cooperation Council (GCC) countries. A cross-sectional short online Survey MonkeyTM multiple-choice questions-based questionnaire was disseminated to dental faculty working in GCC countries during the academic year 2018/2019. The questionnaire included five test incorrect (flawed) multiple-choice questions and one correct control question. The participants were asked to identify flawed multiple-choice question items from the known 14 items writing flaws. Out of a total of 460 faculty, 216 respondents completed the questionnaires, 132 (61.1%) were from Saudi Arabia, while numbers of participants from United Arab Emirates, Kuwait and Oman were 59 (27.3), 14 (6.5%) and 11 (5.1%) respectively. Majority of participants were male (n = 141, 65.9%) compared to 73 females (34.1%). Eighty percent of the participants possessed more than five years of teaching experience. Assistant professors constituted the majority (43.3%) of the academic positions participating in this study. The overall fail rate ranged from 76.3% to 98.1% and almost 2/3rds of the participants were unable to identify one or more of the flawed item(s). No significant association was observed between the demographics (age, region, academic position and specialty) and knowledge except that of participant’s gender (p < 0.009). GCC dental faculty demonstrated below average knowledge of multiple-choice question items writing flaws. Training and workshops are needed to ensure substantial exposure to proper multiple-choice question items construction standards.