Multiple Choice Questions Items Research Articles

Overview of medical student assessment: Why, what, who, and how

تناقش الورقة مختلف القضايا المتعلقة بالتقييم الطبي في سياق العالم العربي مع التركيز على شكلين من أشكال التقييم: الشكل الأول يتضمن أسئلة الاختيار من متعدد والشكل الثاني للتقييم يتضمن الطريقة الموضوعية المنظمة للامتحانات السريرية. ويتم تحديد نظام التقييم المناسب حسب المحتوى وطريقة التدريس والمعلومات والمهارات المفروض اكتسابها من قبل الممارس الصحي. يحفز التقييم الطلاب على الدراسة الجادة, كما أنه يستخدم لنقل الطلاب لمستوى تدريسي أعلى. إن التقييم الجيد لا يمكن استيراده بل يجب أن ينبع من البيئة المحلية مع مراعاة الخلفية الثقافية واللغوية والتعليمية للطلاب. في المقابل فإن التقييم المركزي الذي لا يخضع لسيطرة مباشرة من المعلمين المتصلين بطلابهم يناسب الامتحانات الموحدة مثل فحص التراخيص الطبية بالولايات المتحدة, لكنه يرتبط بالعديد من التحديات في الامتحانات الداخلية داخل المؤسسة التعليمية: فهو يُعقِّد إدارة الامتحان، مع تهميش لدور المعلمين، مع وجود ظلم في معاملة غير المتساويين كالمتساويين من الطلاب. ويغطي التقييم عنصري الطب: العلم النظري والعملي. بحيث تقيم أسئلة الاختيار من متعدد المعرفة وتطبيقاتها. بينما تقيم الطريقة الموضوعية المنظمة للامتحانات السريرية المهارات العملية. في المقابل تعتبر كتابة أسئلة الاختيار من متعدد بطريقة جيدة ومتقنة عملية صعبة فهي تأخذ الكثير من الجهد والوقت للتحضير لكنها سهلة التقديم للطلاب وسهلة التصحيح. وتعتمد الطريقة الموضوعية المنظمة للامتحانات السريرية على محاكاة المرضى مستبدلة الطريقة التقليدية للحالات الحقيقية السريرية الطويلة والقصيرة. لكن هذه الطريقة تحرم الطلاب المتميزين من طرح أسئلة على حالات محاكاة المرضى خارج النص المقرر. لذا اقترح استخدام محاكاة المرضى باستعمال أشخاص لديهم تجربة شخصية سابقة للحالة المرضية التي يجري الامتحان عليها. كما أقترح أيضا التركيز على تقييم المعلومات المهمة والحاسمة لدى الطلاب وتخصيص نسبة كبيرة من الدرجات لها. وينبغي أن يحكم على فشل الطلاب في كامل الامتحان إذا أخفقوا في بعض من هذه العناصر المهمة The paper discusses various issues related to assessment in the context of the Arab world and focusing on 2 forms of assessment: multiple choice questions (MCQs) and objective structured clinical examinations (OSCEs). The appropriate assessment system is determined by the content and method of teaching as well as the expected knowledge and skills of the final product, the health professional. Assessment motivates students to study hard. It is also used to make decisions on promotion of students. A good assessment system cannot be imported; it must be home grown taking into consideration the cultural, linguistic, and educational background of the students. Centralized assessment not under the immediate control of teachers in contact with the students is appropriate for standardized examinations like the United States Medical Licensure Examination (USMLE) but is associated with many challenges in internal examinations within the teaching institution: complicated logistics, marginalization of the teachers, and the injustice of treating un-equals as equals. Assessment covers the two components of medicine: the science and the art (practical). The MCQ format assesses knowledge and its applications. The OSCE format assesses practical skills. Writing good MCQ items takes a lot of effort and time to review but is easy to administer and score. The OSCE based on simulated patients (SP) has ably replaced the traditional long and short clinical cases but penalizes the advanced candidate who asks the SP questions off the script. I propose using SPs who actually had personal experience of the condition being tested. I also propose some items in the OSCE that are of critical knowledge for professionals and which should have higher scores assigned to them. Students should be failed in the whole examination if they do not know some of these critical items.

Examining the Multiple Choice Educational Examinations of College Students

Evaluation of students' academic achievement is an unavoidable task for all faculty members of universities. The majority of academic institutions use multiple choice exams to evaluate their students. The question items that make up these exams need to possess certain psychometric properties to measure the student achievements. The purpose of this research was to examine the item properties of multiple choice exams used in the college of health of Kashan University of Medical Sciences. In this cross sectional descriptive study, a number of multiple choice exams used by the instructors were randomly selected. The difficulty index, discrimination index, and Cronbach alpha was calculated for every exam by using LERTAP 5.0 software prepared by the Assessment Systems Corporation of the United States to perform an item analysis. A total of 1343 multiple choice question items presented to the students at the college of health for evaluating 37 different subjects was analyzed. The results of analysis showed that the mean values of difficulty index and discriminating index were0.68 and 0.20, respectively. The average value of difficulty )0.68( and discrimination index )0.21( were relatively within the recommended values. However, some of the tests had value less than the recommended range and need careful reexamination. It was concluded that a feedback delivered to the instructors may improve these indexes. Further research is needed to examine this subject.

Peer review improves the quality of MCQ examinations

The aim of this study was to assess the effect of the introduction of peer review processes on the quality of multiple-choice examinations in the first three years of an Australian medical course. The impact of the peer review process and overall quality assurance (QA) processes were evaluated by comparing the examination data generated in earlier years (2008) with those held under the new QA regime (2009 and 2010) from the same blueprint. Statistical analysis and comparisons of overall examination performance were made by year. Regarding multiple-choice questions (MCQs), item analysis was used to compare the proportion of difficult and discriminating items and functional distractors on summative examinations in 2008 (pre-implementation of peer review) and 2009 and 2010 (post-implementation). The impact of peer review processes resulted in a decrease in the number of items with negative discrimination; increases in reliability, appropriate item difficulty, and numbers of items with significant discrimination. There was an associated improvement in the effectiveness of distractors for the MCQ items. The trend of overall improvement in the quality of MCQ items continued in 2009 and 2010. The introduction of QA processes, specifically peer review of MCQ items has resulted in a sustained improvement in the quality of MCQ items within our examinations.

Developing a web-based multiple-choice question item bank

Developing a web-based multiple-choice question item bank

A practical discussion to avoid common pitfalls when constructing multiple choice questions items

This paper is an attempt to produce a guide for improving the quality of Multiple Choice Questions (MCQs) used in undergraduate and postgraduate assessment. Multiple Choice Questions type is the most frequently used type of assessment worldwide. Well constructed, context rich MCQs have a high reliability per hour of testing. Avoidance of technical items flaws is essential to improve the validity evidence of MCQs. Technical item flaws are essentially of two types (i) related to testwiseness, (ii) related to irrelevant difficulty. A list of such flaws is presented together with discussion of each flaw and examples to facilitate learning of this paper and to make it learner friendly. This paper was designed to be interactive with self-assessment exercises followed by the key answer with explanations.

Impact of item-writing flaws in multiple-choice questions on student achievement in high-stakes nursing assessments

Multiple-choice questions (MCQs) are frequently used to assess students in health science disciplines. However, few educators have formal instruction in writing MCQs and MCQ items often have item-writing flaws. The purpose of this study was to examine the impact of item-writing flaws on student achievement in high-stakes assessments in a nursing programme in an English-language university in Hong Kong. From a larger sample, we selected 10 summative test papers that were administered to undergraduate nursing students in 1 nursing department. All test items were reviewed for item-writing flaws by a 4-person consensus panel. Items were classified as 'flawed' if they contained > or = 1 flaw. Items not containing item-writing violations were classified as 'standard'. For each paper, 2 separate scales were computed: a total scale which reflected the characteristics of the assessment as administered and a standard scale which reflected the characteristics of a hypothetical assessment including only unflawed items. The proportion of flawed items on the 10 test papers ranged from 28-75%; 47.3% of all items were flawed. Fewer examinees passed the standard scale than the total scale (748 [90.6%] versus 779 [94.3%]). Conversely, the proportion of examinees obtaining a score > or = 80% was higher on the standard scale than the total scale (173 [20.9%] versus 120 [14.5%]). Flawed MCQ items were common in high-stakes nursing assessments but did not disadvantage borderline students, as has been previously demonstrated. Conversely, high-achieving students were more likely than borderline students to be penalised by flawed items.

Psychometric Characteristics and Response Times for Content-Parallel Extended-Matching and One-Best-Answer Items in Relation to Number of Options

This study investigated the impact of item format and number of options on the psychometric characteristics (p values and biserials) and response times for multiple-choice questions (MCQs) appearing on Step 2 of the United States Medical Licensing Examination. In all, 192 MCQ items were used in the study. Each item was presented in two formats: in a two-item extended-matching set and as an independent item. For the extended matching format, there were two versions: a base version that included all options (10 to 26) and an 8-option version. For the independent-item format, there were three versions: a base version that included all options, and 8-option and 5-option versions created by a group of physicians that selected options without information about examinee performance. All items were embedded in unscored sections of the 2005-06 Step 2 test forms. Versions of items with more options were harder and required more testing time; no differences in item discrimination were observed. Mean response times for items presented in the extended-matching format were lower than for those presented as independent items, primarily because of shorter response times for the second item presented in a set. Use of the extended-matching format and smaller numbers of options per item (and more items) should result in more efficient use of testing time and greater score precision per unit of testing time.

Relationship Between Item Difficulty and Discrimination Indices in True/False-Type Multiple Choice Questions of a Para-clinical Multidisciplinary Paper

This paper reports the relationship between the difficulty level and the discrimination power of true/false-type multiple-choice questions (MCQs) in a multidisciplinary paper for the para-clinical year of an undergraduate medical programme. MCQ items in papers taken from Year II Parts A, B and C examinations for Sessions 2001/02, and Part B examinations for 2002/03 and 2003/04, were analysed to obtain their difficulty indices and discrimination indices. Each paper consisted of 250 true/false items (50 questions of 5 items each) on topics drawn from different disciplines. The questions were first constructed and vetted by the individual departments before being submitted to a central committee, where the final selection of the MCQs was made, based purely on the academic judgement of the committee. There was a wide distribution of item difficulty indices in all the MCQ papers analysed. Furthermore, the relationship between the difficulty index (P) and discrimination index (D) of the MCQ items in a paper was not linear, but more dome-shaped. Maximal discrimination (D = 51% to 71%) occurred with moderately easy/difficult items (P = 40% to 74%). On average, about 38% of the MCQ items in each paper were "very easy" (P > or =75%), while about 9% were "very difficult" (P <25%). About two-thirds of these very easy/difficult items had "very poor" or even negative discrimination (D < or =20%). MCQ items that demonstrate good discriminating potential tend to be moderately difficult items, and the moderately-to-very difficult items are more likely to show negative discrimination. There is a need to evaluate the effectiveness of our MCQ items.

A solution to the cueing effects of multiple choice questions: the Un-Q format.

Although the cueing effects inherent in conventional multiple choice questions (MCQs) present serious limitations, this format continues to dominate testing programmes. The present study was undertaken to estimate the effects of cueing when MCQs are used to test medical students, and to evaluate the reliability, validity and feasibility of an alternative testing format. Equivalent items in both MCQ and open-ended, or uncued (Un-Q), formats were administered to 34 third- and fourth-year medical students. The students' mean % correct score on the MCQs was 11 percentage points higher than their mean level of performance on equivalent Un-Qs. When a second set of more difficult items was administered to 16 of these students, their mean performance on the MCQ items was 22 percentage points higher than their performance on equivalent Un-Qs. The results support the feasibility of large group administration of tests constructed in an open-ended format that can be scored by computer. Not only is this format equally reliable and economical when compared with the MCQ, but it also provides important advantages that strengthen its face validity. The Un-Q format can be used to test either simple recall or certain higher level problem-solving skills that cannot be tested by MCQs. Even more important, the results also suggest that the Un-Q format may be a more effective discriminator of academically marginal examinees.

A reappraisal of the use of the phi-coefficient in multiple choice examinations.

The phi-coefficient of an item in a multiple choice question (MCQ) examination is often used to determine whether that item is suitable for re-use in future examinations. In order to be of value in this regard, the coefficient must be shown to be an objective and consistent index of the discriminating power of an MCQ item. The behaviour of the phi-coefficient (phi) was investigated in two one-from-five MCQ and two true/false examinations. It is shown that the magnitude of the phi-coefficient for the items in any examination is a function not only of the discriminating power of the items, but also of the magnitude of the countermark for incorrect responses, of the proportion of 'don't know' responses in the examination, and of how the 'don't know' responses are handled in the analysis. It is further shown that the reproducibility of the phi, when calculated for any pair of randomly selected portions of a class of students, is very poor. We conclude that the error of estimation of the phi is of the same order of magnitude as the phi's normal operative range.