High-quality Multiple-choice Questions Research Articles

Student-generated Multiple-Choice Questions: A Java and Web-Based Tool for Students to Create Multiple Choice Tests

Student-generated questions can be an effective study technique to improve active learning, metacognitive skills, and performance on examinations. Students have shown greater success when assessed using peer-made study questions than when studying without questions. In three semesters of a kinesiology research methods course students were taught how to write high-quality multiple-choice questions that addressed course objectives and Bloom’s Taxonomy. Students were given a graded assignment to write three multiple-choice questions using Makequiz, a Java and web-based tool for helping students generate multiple choice questions. Student- generated questions that were rated as good quality (n = 169-245) were provided to the students as a study resource prior to the final exam. Of those study questions, 40 were selected each semester to be on the final exam. Students performed significantly better on student-written questions than instructor-written questions on the final exam in Class A (p < .05) and in Class C (p < .05). A majority of students felt this assignment was a worthwhile component of the course, voting to keep Makequiz in the curriculum (Class A: 52.6%, Class B: 62.3%, Class C: 58.3%) or to modify Makequiz (Class A: 25.3%, Class B: 14.5%, Class C: 18.1%). Many of the students stated it was the most valuable assignment of the course (32% Class B and Class C). Makequiz is, therefore, a recommended study tool for students. Comparisons are also made with PeerWise, an online platform for creating and sharing MCQ and feedback. Further investigation is required to measure the impact Makequiz has on learning, metacognitive skills, and anxiety levels before test-taking.

Efficacy of Faculty Development Training Workshops (FDTWs) on Writing High-Quality Multiple-Choice Questions at Northern Border University (NBU) in the Kingdom of Saudi Arabia (KSA).

A multiple-choice question (MCQ) is a frequently used assessment tool in medical education for both certification and competitive examinations. Well-constructed MCQs impact the utility of the assessment and, thus, the fate of the examinee. To analyze the basic science faculty perceptions of writing high-quality MCQs, to create awareness of item-writing flaws in constructing high-quality MCQs, and to determine the impact of faculty development training workshops (FDTWs) on MCQ writing skills. An online workshop was held over two weeks for basic science faculty to learn high-quality MCQ construction. Faculty-made MCQs were analyzed for flaws, and a questionnaire assessed the impact of the workshop on MCQ construction. Pre- and post-workshop responses were compared to evaluate the necessity of such workshops for improving faculty skills in MCQ assessments. A total of 47 (83.2%) of participating faculty believed the workshop could reduce MCQ construction errors. The participants agreed that a series of workshops were needed for lasting improvements in MCQ construction. One-day short-duration workshops, such as the current one alone, cannot achieve the objectives of training participants to write high-quality MCQs. To improve student assessment through high-quality MCQs, the faculty needs to be exposed to continuous and frequent sessions that will help them.

Tıp Eğitiminde Otomatik Soru Üretme Yöntemi Kullanılarak Oluşturulan İlk Türkçe Çoktan Seçmeli Soruların Psikometrik Analizi

Aim: Automatic item generation is "a process of using models to generate items using computer technology". The use of automatic item generation typically involves one of three primary methods: syntax-based, semantic-based, and template-based. Non-template automatic item generation approaches leverage natural language processing techniques. A study showed the potential of using template-based automatic item generation to create high-quality multiple-choice questions for assessing clinical reasoning in Turkish, marking a first in the field. However, the findings of the study were based only on expert opinions, necessitating further research to examine the psychometric qualities of Turkish items. The aim of this study was to reveal psychometric characteristics of the first Turkish case-based multiple-choice questions generated by using automatic item generation in medical education. Methods: This was a psychometric study. Three Turkish case-based multiple-choice questions generated using template-based automatic item generation on essential hypertension were included in an exam that 281 fourth-year medical students participate in. This examination was carried out in-person in classroom settings under proctor supervision. Item difficulty and item discrimination (point-biserial correlation) were calculated, and non-functioning distractors were determined. Results: All three items had acceptable levels (higher than 0.20) of point-biserial correlation (p<0.001). The item difficulty levels indicated the presence of one easy, one moderate, and one difficult question. Each item had 2-3 non-functioning options among five options. All three items had acceptable levels (higher than 0.20) of point-biserial correlation (p<0.001). The item difficulty levels indicated the presence of one easy, one moderate, and one difficult question. Each item had 2-3 non-functioning options among five options. Conclusions: The results indicated that the items successfully discriminate between high and low performers, providing validity evidence on the quality of the questions in evaluating students' comprehension of the subject. Additionally, the findings suggest that it is feasible to create multiple-choice questions with different difficulty levels in Turkish using a single automatic item generation model. This study demonstrated for the first time that automatic generation of case-based multiple-choice questions in Turkish produces acceptable psychometric characteristics in an authentic assessment setting in medical education. The ability to automatically generate effective multiple-choice questions in Turkish holds promise for enhancing the efficiency of written assessment in Turkish medical education.

Educator's blueprint: A how-to guide for developing high-quality multiple-choice questions.

Multiple-choice questions are commonly used for assessing learners' knowledge, as part of educational programs and scholarly endeavors. To ensure that questions accurately assess the learners and provide meaningful data, it is important to understand best practices in multiple-choice question design. This Educator's Blueprint paper provides 10 strategies for developing high-quality multiple-choice questions. These strategies include determining the purpose, objectives, and scope of the question; assembling a writing team; writing succinctly; asking questions that assess knowledge and comprehension rather than test-taking ability; ensuring consistent and independent answer choices; using plausible foils; avoiding grouped options; selecting the ideal response number and order; writing high-quality explanations; and gathering validity evidence before and evaluating the questions after use.

Great Question! The Art and Science of Crafting High-Quality Multiple-Choice Questions.

Assessment of medical knowledge is essential to determine the progress of an adult learner. Well-crafted multiple-choice questions are one proven method of testing a learner's understanding of a specific topic. The authors provide readers with rules that must be followed to create high-quality multiple-choice questions. Common question writing mistakes are also addressed to assist readers in improving their item-writing skills.

Assessment of Global Health Education: The Role of Multiple-Choice Questions.

Introduction: The standardization of global health education and assessment remains a significant issue among global health educators. This paper explores the role of multiple choice questions (MCQs) in global health education: whether MCQs are appropriate in written assessment of what may be perceived to be a broad curriculum packed with fewer facts than biomedical science curricula; what form the MCQs might take; what we want to test; how to select the most appropriate question format; the challenge of quality item-writing; and, which aspects of the curriculum MCQs may be used to assess.Materials and Methods: The Medical School for International Health (MSIH) global health curriculum was blue-printed by content experts and course teachers. A 30-question, 1-h examination was produced after exhaustive item writing and revision by teachers of the course. Reliability, difficulty index and discrimination were calculated and examination results were analyzed using SPSS software.Results: Twenty-nine students sat the 1-h examination. All students passed (scores above 67% - in accordance with University criteria). Twenty-three (77%) questions were found to be easy, 4 (14%) of moderate difficulty, and 3 (9%) difficult (using examinations department difficulty index calculations). Eight questions (27%) were considered discriminatory and 20 (67%) were non-discriminatory according to examinations department calculations and criteria. The reliability score was 0.27.Discussion: Our experience shows that there may be a role for single-best-option (SBO) MCQ assessment in global health education. MCQs may be written that cover the majority of the curriculum. Aspects of the curriculum may be better addressed by non-SBO format MCQs. MCQ assessment might usefully complement other forms of assessment that assess skills, attitude and behavior. Preparation of effective MCQs is an exhaustive process, but high quality MCQs in global health may serve as an important driver of learning.

Selecting student-authored questions for summative assessments

Production of high-quality multiple-choice questions (MCQs) for both formative and summative assessments is a time-consuming task requiring great skill, creativity and insight. The transition to online examinations, with the concomitant exposure of previously tried-and-tested MCQs, exacerbates the challenges of question production and highlights the need for innovative solutions. Several groups have shown that it is practical to leverage the student cohort to produce a very large number of syllabus-aligned MCQs for study banks. Although student-generated questions are well suited for formative feedback and practice activities, they are generally not thought to be suitable for high-stakes assessments. In this study, we aimed to demonstrate that training can be provided to students in a scalable fashion to generate questions of similar quality to those produced by experts and that identification of suitable questions can be achieved with minimal academic review and editing. Second-year biochemistry and molecular biology students were assigned a series of activities designed to coach them in the art of writing and critiquing MCQs. This training resulted in the production of over 1000 MCQs that were then gauged for potential by either expert academic judgement or via a data-driven approach in which the questions were trialled objectively in a low-stakes test. Questions selected by either method were then deployed in a high-stakes in-semester assessment alongside questions from two academically authored sources: textbook-derived MCQs and past paper questions. A total of 120 MCQs from these four sources were deployed in assessments attempted by over 600 students. Each question was subjected to rigorous performance analysis, including the calculation of standard metrics from classical test theory and more sophisticated item response theory (IRT) measures. The results showed that MCQs authored by students, and selected at low cost, performed as well as questions authored by academics, illustrating the potential of this strategy for the efficient creation of large numbers of high-quality MCQs for summative assessment.

Question-Based Collaborative Learning for Constructive Curricular Alignment.

IntroductionWe designed a curriculum mapping tool which enables medical students to access intended learning outcomes (ILOs) on their iPads in the workplace. Students were encouraged to use the online curriculum map in a specially planned teaching session: question-based collaborative learning (QBCL). The aim of the session was to empower medical students to constructively align their experiential learning with the learning outcomes of the undergraduate curriculum. In doing so, our session aimed to provide students with a greater understanding of the curriculum, improve their insights into assessment and their question-writing abilities.MethodsThe QBCL pre-session preparation involved reviewing a patient with a presentation that aligned to the year-specific ILOs. During a 150 minute QBCL session, students received training on how to write high quality multiple choice questions (MCQs) delivered by a faculty member of Imperial College School of Medicine. They then worked collaboratively in groups and created MCQs based on their clinical encounters. Their questions were tagged to the relevant learning objective and submitted online via the curriculum map. The student-generated MCQs were analyzed using an adjusted version of Bloom’s taxonomy. We also conducted a quantitative evaluation of the session.ResultsOne hundred and sixty-three questions were submitted, with 81% of questions being tagged to ILOs considered to show evidence of learning consistent with the “Apply” tier of Bloom’s taxonomy. The majority of students agreed that the session was interactive (80%), thought-provoking (77%) and improved their team-working skills (70%). It gave them a greater understanding of the undergraduate curriculum (65%), improved their question-writing and insight into assessments (76%), and provided an opportunity to learn from their peers (86%). Students agreed that this session covered a variety of cases (82%) and deepened their understanding of medical conditions and presentations (87%).ConclusionWe encouraged students to actively interact with the curriculum map. Students were able to achieve their own constructive alignment by writing assessment items based on real patients and linking them to the appropriate intended learning outcomes.

Can automated item generation be used to develop high quality MCQs that assess application of knowledge?

The purpose of this study was to compare the quality of multiple choice questions (MCQs) developed using automated item generation (AIG) versus traditional methods, as judged by a panel of experts. The quality of MCQs developed using two methods (i.e., AIG or traditional) was evaluated by a panel of content experts in a blinded study. Participants rated a total of 102 MCQs using six quality metrics and made a judgment regarding whether or not each item tested recall or application of knowledge. A Wilcoxon two-sample test evaluated differences in each of the six quality metrics rating scales as well as an overall cognitive domain judgment. No significant differences were found in terms of item quality or cognitive domain assessed when comparing the two item development methods. The vast majority of items (> 90%) developed using both methods were deemed to be assessing higher-order skills. When compared to traditionally developed items, MCQs developed using AIG demonstrated comparable quality. Both modalities can produce items that assess higher-order cognitive skills.

Perception of Faculty Members of Regional Medical School Toward Faculty Development Program

Introduction: Any types or process of activities that are performed by a group or an individual to enhance, promote, and improve the performance, competencies or skills of a health professional at an institute level are defined as faculty development which has many other names like staff development, academic development, and educational development. The importance of faculty development stressed by Harden is equivalent to curriculum development, which is a very difficult assignment.
 Objectives: To determine the pattern of faculty members’ perception toward their weekly faculty development program.
 Methodology: Enrolling the faculty members of the College of Medicine, University of Bisha, Kingdom of Saudi Arabia ( COM/UB,KSA). COM/UB adopting the innovative curriculum (problem-based learning, team-based learning, interactive lectures, and case-based learning and seminars. Each Tuesday, faculty members gather in the faculty development academic program (FDAP) room to discuss issues related to medical education and quality. Usually, there is a presentation on the selected topic over 25 min, followed by a discussion and workshop, and finally, a recommendation is drawn. The inclusion criterion is those studying in the college for a period of more than one year. New joiners and those studying for a period of less than one year were excluded. Faculty members were enrolled optionally and requested to fill a validated questionnaire.
 Results: The response rate was 92% of total staff. More than 89%, 87%, 86% and 74% thought that the FDAP is useful to them, helps them in addressing academic issues in the faculty, is an enriching experience, and thought that it helps them in constructing high-quality multiple choice questions (MCQs). Finally, 84% were interested in presenting topics in FDP.
 Conclusion: FDAP was positively perceived by the faculty members of the University Bisha/College of Medicine, since it enriches their experience and satisfies their academic job.
 Keywords: faculty, development, academic, program, MCQs

Student-Written Multiple-Choice Questions-a Practical and Educational Approach.

The purpose of our student-led project was to fulfill junior medical students' demand for instructive, curriculum-specific practice questions while providing a learning experience and teaching opportunity for participating senior students. Eleven second-year students were taught how to write high-quality multiple-choice questions through an interactive workshop. Subsequently, they were instructed to write questions with detailed explanations fortheir assigned lecture topics. Thirty-four student-written and faculty-reviewed questions were combined with 16 purely faculty-written questions to create a 50-question exam. No significant difference was found in question difficulty between the student-written (79.5%) and faculty-written (84.0%) questions (p = 0.37). The discrimination index and point biserial correlation were higher for student-written (0.29, 0.32) vs. faculty-written (0.17, 0.25) questions (p < .01, < .05). The test-takers learned key course topics, while the test-writers reviewed key first-year objectives and refined their test-taking strategies. The project provided a model for feasibly developing comprehensive, high-quality, and curriculum-specific questions.

Are multiple-choice questions a good tool for the assessment of clinical competence in Internal Medicine?

There are many feasible tools for the assessment of clinical practice, but there is a wide consensus on the fact that the simultaneous use of several different methods could be strategic for a comprehensive overall judgment of clinical competence. Multiple-choice questions (MCQs) are a well-established reliable method of assessing knowledge. Constructing effective MCQ tests and items requires scrupulous care in the design, review and validation stages. Creating high-quality multiple-choice questions requires a very deep experience, knowledge and large amount of time. Hereby, after reviewing their construction, strengths and limitations, we debate their completeness for the assessment of professional competence.

Building an Exam Through Rigorous Exam Quality Improvement

Assessment drives learning; therefore, valid and robust assessments are important. To achieve trustworthy assessments with high-quality multiple choice questions, we established a rigorous system of exam quality improvement (QI). Our exam QI process helps ensure implementation of best practices and evidence-based approaches to student assessment, as well as identify opportunities for process modifications that ultimately lead to systematic improvements. We provide a step-by-step summary of our QI process for consideration by other institutions. The model outlined here may serve as a template for other institutions who hope to improve their exams through exam QI processes.

Will a Short Training Session Improve Multiple‐Choice Item‐Writing Quality by Dental School Faculty? A Pilot Study

Faculty members are expected to write high-quality multiple-choice questions (MCQs) in order to accurately assess dental students' achievement. However, most dental school faculty members are not trained to write MCQs. Extensive faculty development programs have been used to help educators write better test items. The aim of this pilot study was to determine if a short workshop would result in improved MCQ item-writing by dental school faculty at one U.S. dental school. A total of 24 dental school faculty members who had previously written MCQs were randomized into a no-intervention group and an intervention group in 2015. Six previously written MCQs were randomly selected from each of the faculty members and given an item quality score. The intervention group participated in a training session of one-hour duration that focused on reviewing standard item-writing guidelines to improve in-house MCQs. The no-intervention group did not receive any training but did receive encouragement and an explanation of why good MCQ writing was important. The faculty members were then asked to revise their previously written questions, and these were given an item quality score. The item quality scores for each faculty member were averaged, and the difference from pre-training to post-training scores was evaluated. The results showed a significant difference between pre-training and post-training MCQ difference scores for the intervention group (p=0.04). This pilot study provides evidence that the training session of short duration was effective in improving the quality of in-house MCQs.

Quality of multiple choice questions in undergraduate pharmacology assessments in a teaching hospital of Kerala, India: an item analysis

Background: Carefully constructed, high quality multiple choice questions can serve as effective tools to improve standard of teaching. This item analysis was performed to find the difficulty index, discrimination index and number of non functional distractors in single best response type questions.Methods: 40 single best response type questions with four options, each carrying one mark for the correct response, was taken for item analysis. There was no negative marking. The maximum marks was 40. Based on the scores, the evaluated answer scripts were arranged with the highest score on top and the least score at the bottom. Only the upper third and lower third were included. The response to each item was entered in Microsoft excel 2010. Difficulty index, Discrimination index and number of non functional distractors per item were calculated.Results: 40 multiple choice questions and 120 distractors were analysed in this study. 72.5% items were good with a difficulty index between 30%-70%. 25% items were difficult and 2.5% items were easy. 27.5% items showed excellent discrimination between high scoring and low scoring students. One item had a negative discrimination index (-0.1). There were 9 items with non functional distractors.Conclusions: This study emphasises the need for improving the quality of multiple choice questions. Hence repeated evaluation by item analysis and modification of non functional distractors may be performed to enhance standard of teaching in Pharmacology.

Comparison in the quality of distractors in three and four options type of multiple choice questions.

IntroductionThe number of distractors needed for high quality multiple choice questions (MCQs) will be determined by many factors. These include firstly whether English language is their mother tongue or a foreign language; secondly whether the instructors who construct the questions are experts or not; thirdly the time spent on constructing the options is also an important factor. It has been observed by Tarrant et al that more time is often spent on constructing questions than on tailoring sound, reliable, and valid distractors.ObjectivesFirstly, to investigate the effects of reducing the number of options on psychometric properties of the item. Secondly, to determine the frequency of functioning distractors among three or four options in the MCQs examination of the dermatology course in University of Bahri, College of Medicine.Materials and methodsThis is an experimental study which was performed by means of a dermatology exam, MCQs type. Forty MCQs, with one correct answer for each question were constructed. Two sets of this exam paper were prepared: in the first one, four options were given, including one key answer and three distractors. In the second set, one of the three distractors was deleted randomly, and the sequence of the questions was kept in the same order. Any distracter chosen by less than 5% of the students was regarded as non-functioning. Kuder-Richardson Formula 20 (Kr-20) measures the internal consistency and reliability of an examination with an acceptable range 0.8–1.0. Chi square test was used to compare the distractors in the two exams.ResultsA significant difference was observed in discrimination and difficulty indexes for both sets of MCQs. More distractors were non-functional for set one (of four options), but slightly more reliable. The reliability (Kr-20) was slightly higher for set one (of four options). The average marks in option three and four were 34.163 and 33.140, respectively.ConclusionCompared to set 1 (four options), set 2 (of three options) was more discriminating and associated with low difficulty index but its reliability was low.

Using cognitive models to develop quality multiple-choice questions

With the recent interest in competency-based education, educators are being challenged to develop more assessment opportunities. As such, there is increased demand for exam content development, which can be a very labor-intense process. An innovative solution to this challenge has been the use of automatic item generation (AIG) to develop multiple-choice questions (MCQs). In AIG, computer technology is used to generate test items from cognitive models (i.e. representations of the knowledge and skills that are required to solve a problem). The main advantage yielded by AIG is the efficiency in generating items. Although technology for AIG relies on a linear programming approach, the same principles can also be used to improve traditional committee-based processes used in the development of MCQs. Using this approach, content experts deconstruct their clinical reasoning process to develop a cognitive model which, in turn, is used to create MCQs. This approach is appealing because it: (1) is efficient; (2) has been shown to produce items with psychometric properties comparable to those generated using a traditional approach; and (3) can be used to assess higher order skills (i.e. application of knowledge). The purpose of this article is to provide a novel framework for the development of high-quality MCQs using cognitive models.

Can We Share Multiple Choice Questions across Borders? Validation of the Dutch Knowledge Assessment in Family Medicine in Flanders

Background. One of the methods to test knowledge of Family Medicine trainees is a written exam composed of multiple choice questions. Creating high-quality multiple choice questions requires a lot of experience, knowledge, and time. This study explores the opportunity to run the Dutch knowledge assessment in Flanders as well, the use of this test for formative purposes. Methods. The study test was performed in a Flemish sample of postgraduate Family Medicine (FM) trainees and FM trainers. The Dutch test, adjusted to the Flemish context, was analyzed according to the classical test theory: difficulty factor and discriminating power of the items and reliability of the test. Results. 82 of the 154 items well divided the group into two equal parts of correct and incorrect responders. The distribution of the discrimination index, of the items with an acceptable difficulty factor, was [−0.012–0.530]. The item-test-correlation shows that 52 items do not fit, and 87 items need revision in varying degrees. The test reliability was 0.917. Conclusion. The test was highly reliable, but many MC questions appeared to be too easy and poorly discriminative. Therefore, we question the test validity and recommend reconsideration of the items based on difficulty before it is applied and used as a mandatory formative test.

Ärztliche Fachzeitschriften als implizite Vorbilder für gute Multiple-Choice-Fragen – in welchem Maß erfüllen die Tests der continous medical education inzwischen formale Gütekriterien?

Many clinicians, particularly those from university hospitals and associated teaching hospitals, use multiple-choice questions in different professional roles: as examiners for student assessment and as life-long learners in respect to the CME programmes of medical journals. For this group of physicians it might thus be desirable that medical journals serve as implicit role models for formulating high-quality multiple-choice questions. 370 multiple-choice questions taken from the 2011 volumes of three different medical journals were reviewed. A checklist consisting of 11 formal quality criteria, 5 quality categories and one global criterion, taken from the literature and from the German Item Management System, was used. High-quality questions were found in the categories "language", "structure", and "signposting the question type." However, "application to cases or problems" and "analysis, interpretation, problem solving" were realised in less than 20%, and "cues" were detected in almost half of the questions. A subgroup analysis showed different quality levels between the journals. To serve as role models for high-quality multiple-choice questions (MCQ) German medical journals should deliver formal quality criteria and good writing examples to their MCQ authors and ensure strict adherence to these criteria. The development of journal-independent formal standards might prove worthwhile.

The role of difficulty and discrimination in constructing multiple-choice examinations: With guidelines for practical application

Abstract A task common to all accounting professors is that of constructing examinations; frequently containing multiple-choice questions. This paper explores some of the concepts from testing and measurement theory that underly the development of high quality multiple-choice questions. First, the examination is conceptualized as a measurement device to assess the degree of student learning. Then, the concepts of difficulty and discrimination are developed and applied to the testing process. Here convergent and discriminant validity are discussed. Finally, specific guidelines are developed for use in generating high-quality multiple-choice questions.