Multiple Choice Question Assessment Research Articles

Effectiveness of an audience response system on orthodontic knowledge retention of undergraduate dental students--a randomised control trial.

Objective:: To determine the effect of an audience response system (ARS) on knowledge retention of dental students and to gauge student perceptions of using the ARS. Design:: Randomised control study. Setting:: School of Dentistry, Cardiff University. Participants:: Seventy four second-year dental students were stratified by gender and randomised anonymously to one of two groups. Methods:: One group received a lecture on orthodontic terminology and diagnosis in a traditional didactic format and the other received the same lecture integrated with ARS slides. Students completed an assessment of multiple-choice questions (MCQs) scored out of 20, before and immediately after the lecture. Students were also asked to complete a self-reported questionnaire on their perceptions of ARS. Results:: Both groups had statistically significant increases in MCQ scores post-lecture (ARS mean increase 3.6 SD2.0, 95% CI 2.2–3.5 and Didactic mean increase 2.9 SD2.3, 95% CI 2.8–4.3). A mixed-design analysis of variance showed that ARS led to an improved MCQ score (by 0.8 or 25%) compared to the didactic group, although this effect was not significant (P = 0.15). The effect of gender at baseline (P = 0.49), post-lecture (P = 0.73) and increase in MCQ score split by group (P = 0.46) was also not significant. Students reported that the ARS was easy to use, helped them engage with the lecture and encouraged them to work harder. Conclusion:: The ARS did not lead to a significant increase in short-term orthodontic knowledge recall of students compared with didactic teaching. However, the use of ARS within orthodontic teaching could make lectures more interactive and engaging.

Online MCQ Assessment Anxiety Amongst 1st Year Psychology Students

First year undergraduate students studying psychology online with the University of the Highlands and Islands experienced anxiety while taking the first of two online multiple choice question (MCQ) assessments. The literature reveals that test anxiety, computer anxiety and technostress have been recognised as issues for some time. As a distributed University that relies heavily on online technology, these issues are potentially significant for UHI and other higher education institutions utilising online learning platforms. Feedback from students after the first MCQ assessment revealed that test anxiety, computer anxiety and technostress were all experienced in some form during the deployment of the assessment. Consequentially, steps were taken to mitigate against these reported issues. Detailed technical advice and guidance was issued ahead of the second online MCQ assessment. End of module feedback from students indicated that anxiety-related issues were reduced by the advice issued to them. For subsequent cohorts of students, formative assessments and detailed technical advice and guidance were issued. MCQ tests were reduced from two to one. An examination of assessment pass rates and feedback for this second cohort of students indicated that the use of a formative assessment and detailed technical advice and guidance deployed prior to the summative assessment were effective at ameliorating against online test anxiety. There are a number of factors that might interact in a complex way to affect student performance in an online MCQ test setting. It is important that wherever possible, we deploy assessments that are a valid test of students’ knowledge and understanding of subject material rather than their ability to deal with situations that engender anxiety and stress.

An Enhancement-focused framework for developing high quality single best answer multiple choice questions.

The primary goal of any assessment of students is to provide valid and reliable evaluations of students' knowledge and skills as well as provision of accurate feedback to students about their performance. Contrary to best practice guidelines for development of multiple choice questions (MCQs), however, items used within medical schools are often flawed. This disappoints students and discourages examiners from using in-house MCQ databases. Vetting and reviewing items can improve the quality of MCQs. In this paper, we describe our approach to standardize the format used for MCQ assessment and provide recommendations for quality enhancement of high-stakes assessment. A collaborative enhancement-focused vetting and review approach to development of high quality single best answer MCQs has been described. Implementation of a collaborative strategy to blueprint, vet, review and standard set MCQ items for high stakes examinations can effectively contribute to assessment quality assurance. Similarly, shared responsibility for post examination analyses of items may reveal the psychometric properties of items in need of improvement and contribute to closure of the assessment outcomes feedback loop. Devolving responsibility for implementation of assessment processes as an integral part of educational practices and values can maximize reliability and standards of assessment processes. We contend that while logistics and time constraints are of concern to busy faculty members, judicious utilization of resources to develop well-written MCQ items are well worth the effort to produce reliable and valid examinee scores. An enhancement-focused approach can be institutionally rewarding and lead to improved quality of high stakes assessments.

Developing an Array Binary Code Assessment Rubric for Multiple-Choice Questions Using Item Arrays and Binary-Coded Responses

The multiple-choice question (MCQ) is the foundation of knowledge assessment in K–12, higher education, and standardized entrance exams (including the GRE, MCAT, and DAT). However, standard MCQ exams are limited with respect to the types of questions that can be asked when there are only five choices. MCQs offering additional choices more extensively test an examinee’s knowledge of complex concepts and thus an examiner can get a more accurate assessment of the examinee’s comprehension. We introduce an assessment paradigm in which choices of the MCQs are items in an array encoded with a number that can be expressed via binary notation in the standard five-choice MCQ answer sheet. Numbers can be represented by combinations of zeroes (unfilled bubbles) and ones (filled bubbles) at each of the five numerical place-holder positions. A scanner that recognizes five filled/unfilled “bubbles” per horizontal line of the answer sheet makes it possible to represent 31 options as possible answers. The rubric provides a more comprehensive assessment of a student’s knowledge than a standard five-choice MCQ. It has the potential to revolutionize MCQ testing. We present examples of MCQ assessment of chemical concepts at all levels of Bloom’s taxonomy, employing item arrays and binary-coded choices.

Pediatric emergency medicine asynchronous e-learning: a multicenter randomized controlled Solomon four-group study.

Asynchronous e-learning allows for targeted teaching, particularly advantageous when bedside and didactic education is insufficient. An asynchronous e-learning curriculum has not been studied across multiple centers in the context of a clinical rotation. We hypothesize that an asynchronous e-learning curriculum during the pediatric emergency medicine (EM) rotation improves medical knowledge among residents and students across multiple participating centers. Trainees on pediatric EM rotations at four large pediatric centers from 2012 to 2013 were randomized in a Solomon four-group design. The experimental arms received an asynchronous e-learning curriculum consisting of nine Web-based, interactive, peer-reviewed Flash/HTML5 modules. Postrotation testing and in-training examination (ITE) scores quantified improvements in knowledge. A 2 × 2 analysis of covariance (ANCOVA) tested interaction and main effects, and Pearson's correlation tested associations between module usage, scores, and ITE scores. A total of 256 of 458 participants completed all study elements; 104 had access to asynchronous e-learning modules, and 152 were controls who used the current education standards. No pretest sensitization was found (p = 0.75). Use of asynchronous e-learning modules was associated with an improvement in posttest scores (p < 0.001), from a mean score of 18.45 (95% confidence interval [CI] = 17.92 to 18.98) to 21.30 (95% CI = 20.69 to 21.91), a large effect (partial η(2) = 0.19). Posttest scores correlated with ITE scores (r(2) = 0.14, p < 0.001) among pediatric residents. Asynchronous e-learning is an effective educational tool to improve knowledge in a clinical rotation. Web-based asynchronous e-learning is a promising modality to standardize education among multiple institutions with common curricula, particularly in clinical rotations where scheduling difficulties, seasonality, and variable experiences limit in-hospital learning.

To what extent is mathematical ability predictive of performance in a methodology and statistics course? Can an action research approach be used to understand the relevance of mathematical ability in psychology undergraduates?

Research methods and statistical analysis is typically the least liked and most anxiety provoking aspect of a psychology undergraduate degree, in large part due to the mathematical component of the content. In this first cycle of a piece of action research, students’ mathematical ability is examined in relation to their performance across different assessments. A maths test, including only com onents relevant to psychological research and analysis, was designed and subsequently completed by 427 students. Factor analysis revealed three distinct facets: understanding of mathematical procedures; interpretation of findings; and understanding the semantics of mathematics. Only the procedural and interpretative factors were predictive of overall course performance. Higher scores on both factors predicted better performance on multiple choice questions assessment and an unseen exam, whereas only the interpretation factor predicted performance on a critical thinking assignment and a lab report. The findings are considered with a view to developing another cycle of action research that more actively involves students.

Using computer-based technology to improve feedback to staff and students on MCQ assessments

The massification of higher education (HE) has led to an unprecedented increase in the number of students in the classrooms, resulting in increased workload for teaching staff, sometimes leading to a great reliance on Multiple Choice Questions (MCQs) examinations with limited feedback provided to students. The central role of feedback in student learning is well recognised; however, it is often one of the poorest scoring items in Student Evaluation of Teaching and Learning (SETL) surveys. Teaching staff also need feedback on the quality of the examinations they set. In response to these key teaching and learning issues, this study developed a computer-aided system for providing meaningful and customised feedback on performance to students and the quality of MCQ assessment items to teaching staff. Furthermore, student and staff perceptions of the effectiveness and value of the feedback to learning were evaluated. Student experiences indicated that the feedback was timely, well presented, easy to understand and would aid revision for further study. In addition, the efficient provision of automatically generated item performance information to teaching staff allowed convenient quality assurance (QA) monitoring, informed staff of the progress of students’ learning, and enabled modification of teaching to better support student learning.

Does student confidence on multiple‐choice question assessments provide useful information?

Feedback from multiple-choice question (MCQ) assessments is typically limited to a percentage correct score, from which estimates of student competence are inferred. The students' confidence in their answers and the potential impact of incorrect answers on clinical care are seldom recorded. Our purpose was to evaluate student confidence in incorrect responses and to establish how confidence was influenced by the potential clinical impact of answers, question type and gender. This was an exploratory, cross-sectional study conducted using a convenience sample of 104 Year3 dental students completing 20 MCQs on implant dentistry. Students were asked to select the most correct response and to indicate their confidence in it for each question. Identifying both correctness and confidence allowed the designation of uninformed (incorrect and not confident) or misinformed (incorrect but confident) responses. In addition to recording correct/incorrect responses and student confidence, faculty staff designated incorrect responses as benign, inappropriate or potentially harmful if applied to clinical care. Question type was identified as factual or complex. Logistic regression was used to evaluate relationships between student confidence, and question type and gender. Students were misinformed more often than uninformed (22% versus 8%), and misinformed responses were more common with complex than factual questions (p<0.05). Students were significantly more likely to be confident of correct than incorrect benign, incorrect inappropriate or incorrect harmful answers (p<0.001), but, contrary to expectations, confidence did not decrease as answers became more harmful. Recording student confidence was helpful in identifying uninformed versus misinformed responses, which may allow for targeted remediation strategies. Making errors of calibration (confidence and accuracy) more visible may be relevant in feedback for professional development.

Improving assessment of paediatric acute breathing difficulties in medical education: a cluster randomized controlled trial

Impact assessment of an adjunct to standard teaching on knowledge gain and self-assessed confidence for students undertaking their paediatric attachment and evaluation of an acute breathing difficulties (ABDs) learning package. Pragmatic cluster randomized controlled trial involving 248 UK medical undergraduates within six paediatric clinical rotations. Intervention groups received an online ABD learning package consisting of symptom-based decision-making pathways underpinned by a ratified ABD evidence-based guideline. Control groups received an online dummy package. Outcome measures were a pre- and post-intervention multiple choice question assessment and confidence questionnaire utilising 5-point Likert scales. Significant knowledge gain in the intervention group (mean 6.84, 95% CI 5.56-8.12) versus no such difference in the control group, significant improvement in self-assessed confidence in intervention group versus no such difference in control group and significant confidence differences in ABD-specific statements were noted. In the satisfaction questionnaire, the learning content was noted to be new to only 16% participants. Supplementation of standard teaching with an interactive delivery method resulted in cognitive gain and self-assessed confidence improvement in interaction with children with ABDs. This indicates that a well-designed tool can help prepare students for direct interaction with sick children as a junior clinician. ISRCTN27499282.

Negatively-Marked MCQ Assessments That Reward Partial Knowledge Do Not Introduce Gender Bias Yet Increase Student Performance and Satisfaction and Reduce Anxiety

Multiple-choice question (MCQ) examinations are increasingly used as the assessment method of theoretical knowledge in large class-size modules in many life science degrees. MCQ-tests can be used to objectively measure factual knowledge, ability and high-level learning outcomes, but may also introduce gender bias in performance dependent on topic, instruction, scoring and difficulty. The ‘Single Answer’ (SA) test is often used in which students choose one correct answer, in which they are unable to demonstrate partial knowledge. Negatively marking eliminates the chance element of guessing but may be considered unfair. Elimination testing (ET) is an alternative form of MCQ, which discriminates between all levels of knowledge, while rewarding demonstration of partial knowledge. Comparisons of performance and gender bias in negatively marked SA and ET tests have not yet been performed in the life sciences. Our results show that life science students were significantly advantaged by answering the MCQ test in elimination format compared to single answer format under negative marking conditions by rewarding partial knowledge of topics. Importantly, we found no significant difference in performance between genders in either cohort for either MCQ test under negative marking conditions. Surveys showed that students generally preferred ET-style MCQ testing over SA-style testing. Students reported feeling more relaxed taking ET MCQ and more stressed when sitting SA tests, while disagreeing with being distracted by thinking about best tactics for scoring high. Students agreed ET testing improved their critical thinking skills. We conclude that appropriately-designed MCQ tests do not systematically discriminate between genders. We recommend careful consideration in choosing the type of MCQ test, and propose to apply negative scoring conditions to each test type to avoid the introduction of gender bias. The student experience could be improved through the incorporation of the elimination answering methods in MCQ tests via rewarding partial and full knowledge.

Computerized Summative Assessment of Multiple-choice Questions: Exploring Possibilities with the Zimbabwe School Examination Council Grade 7 Assessments

The purpose of this study was to develop educational software for online assessment of multiple choice responses (MCQs). An automated assessment software program, duly developed in this study can display assessment items, record candidates' answers, and mark and provide instant reporting of candidates' performance scores. Field tests of the software were conducted on four primary schools located in Bindura town using a previous year summative Grade 7 assessment set by the Zimbabwe School Examination Council (ZIMSEC). Results were that computerized assessment in mathematics has the potential to enhance the quality of assessment standards and can drastically reduce material costs to the examination board. The paper exposes test mode benefits inherent in computer-based assessments, such as one-item display and ease of candidates selecting/changing optional answers. It also informs the ongoing debate on possible enhancement of candidates' performance on a computer-based assessment relative to the traditional pen-and-paper assessment format. The need for the development of diagnostic instructional software to compliment computerized assessments is one of the recommendations of the study.

The trials and tribulations of designing and utilising MCQs in HE and for assessing forensic practitioner competency

Multiple Choice Questions (MCQs) are a very well known, traditional and accepted method of assessment. The use of MCQs for testing students has produced numerous debates amongst academics concerning their effectiveness as they are viewed as practical and efficient but also perceived as possibly „too easy‟ and potentially unable to appropriately test the higher order cognitive skills that essay questions can assess. The use of MCQs in a forensic science context is currently being investigated, not only for use within forensic science education, but also for the testing of competency of qualified forensic practitioners. This paper describes a Higher Education Academy funded project that is investigating the design and the implementation of MCQs for testing forensic practitioners and the lessons that have been learnt so far, that will assist academics in the development of robust MCQ assessments within forensic science degrees to promote and assess deep learning.

The trials and tribulations of designing and utilising MCQs in HE and for assessing forensic practitioner competency

Multiple Choice Questions (MCQs) are a very well known, traditional and accepted method of assessment. The use of MCQs for testing students has produced numerous debates amongst academics concerning their effectiveness as they are viewed as practical and efficient but also perceived as possibly „too easy‟ and potentially unable to appropriately test the higher order cognitive skills that essay questions can assess.The use of MCQs in a forensic science context is currently being investigated, not only for use within forensic science education, but also for the testing of competency of qualified forensic practitioners. This paper describes a Higher Education Academy funded project that is investigating the design and the implementation of MCQs for testing forensic practitioners and the lessons that have been learnt so far, that will assist academics in the development of robust MCQ assessments within forensic science degrees to promote and assess deep learning.

Comparative assessment of multiple choice questions versus short essay questions in pharmacology examinations

This retrospective study compared the performance of medical students in multiple choice questions (MCQs) and short essay questions (SEQs). During the 3 year analysis, 533 students had an average score of 51.34% (SD 9.9) in the SEQ and 64.71%(SD 9.9)in the MCQs. Regression analysis showed a significant correlation(r=0.64, P<0.01) between MCQs and SEQs. When student performance was grouped by final course grade, a statistically significant correlation between MCQs and SEQs scores existed only for the 405 students who received a passing grade (r=0.21, P<0.01). The MCQ and SEQ scores were not correlated for the 128 students who failed (r=0.11, P=0.08) or for 70 students who achieved distinctions (r=-0.27, P=0.13). MCQ scores were significantly higher (P<0.01) than SEQ for each of the groups when analyzed by the two-way ANOVA test. The result of this study suggests that for most students, the strong correlation between MCQ and SEQ indicates that student performance was independent of testing format. For students at either end of the performance spectrum, the lack of correlation suggests that the performance in one of the testing format had a strong influence on the final course grade. In addition, those students who failed the course were likely to be weak in both testing modalities, whereas students in all grade groups were more likely to perform better in the MCQs than SEQs.

Comparison of traditional and interactive teaching methods in a UK emergency department

Didactic teaching remains a core component of undergraduate education, but developing computer assisted learning (CAL) packages may provide useful alternatives. We compared the effectiveness of interactive multimedia-based tutorials with traditional, lecture-based models for teaching arterial blood gas interpretation to fourth year medical students. Participants were randomized to complete a tutorial in either lecture or multimedia format containing identical content. Upon completion, students answered five multiple choice questions assessing post-tutorial knowledge, and provided feedback on their allocated learning method. Marks revealed no significant difference between either group. All lecture candidates rated their teaching as good, compared with 89% of the CAL group. All CAL users found multiple choice questions assessment useful, compared with 83% of lecture participants. Both groups highlighted the importance of interaction. CAL complements other teaching methods, but should be seen as an adjunct to, rather than a replacement for, traditional methods, thus offering students a blended learning environment.

Systematic Approach for the Propagation of Postgraduate Dental Implant Education in Developing Countries

To analyze the outcome of the formative feedback and summative assessment in 2 different groups of dentists after a postgraduate dental implant course. This course was conducted with the hope of developing a systematic approach to implant dentistry training. A 6-day mini-residency postgraduate dental implant course which was approved for 75 credit hours by the international congress of oral implantologists and approved for fellowship after a satisfactory grade at the end of course examination was conducted at the Faculty of Dentistry of the University of Ibadan in West Africa. This course was organized by the Caribbean affiliate of the International Congress of Oral Implantologists utilizing the syllabus which has been instrumental in training Caribbean Dentist since 2004. The postgraduate course in implantology was delivered to 2 groups of participants: group I comprising mainly of faculty staff from the major dental schools in Nigeria with previous postgraduate qualifications and group II were general dental practitioners in private practice in Nigeria.The outcome of the assessment by the 2 different groups was compared with the outcome of implant postgraduate courses in dentistry that are presently documented in Medline/Pubmed. There was no significant difference in the reason for attending the course when the 2 groups are compared (P value ranged from 0.82 to 0.93). Both groups expressed good and similar response to the overall quality of the presentation (P value = 0.19), comprehensiveness of topics covered (P value = 0.34), organization of presentation (P value = 0.34). The multiple choice question assessment at the end of the course was very difficult for both groups and the performance was below average for both groups. The 6-days mini-residency course combined with the 1-month distance learning component was adequate for the sustainability stage and this lead to certification of 72% of the delegates via the examination route.

Web-MCQ: A set of methods and freely available open source code for administering online multiple choice question assessments

E-learning approaches have received increasing attention in recent years. Accordingly, a number of tools have become available to assist the nonexpert computer user in constructing and managing virtual learning environments, and implementing computer-based and/or online procedures to support pedagogy. Both commercial and free packages are now available, with new developments emerging periodically. Commercial products have the advantage of being comprehensive and reliable, but tend to require substantial financial investment and are not always transparent to use. They may also restrict pedagogical choices due to their predetermined ranges of functionality. With these issues in mind, several authors have argued for the pedagogical benefits of developing freely available, open source e-learning resources, which can be shared and further developed within a community of educational practitioners. The present paper supports this objective by presenting a set of methods, along with supporting freely available, downloadable, open source programming code, to allow administration of online multiple choice question assessments to students.

Comparing Online and Offline Administration of Multiple Choice Question Assessments to Psychology Undergraduates: Do Assessment Modality or Computer Attitudes Influence Performance?

Recent interest in and use of online assessments across a range of disciplines has raised a number of issues relating to student perceptions and performance when using this medium. The validity of such online tests is a crucial consideration, especially if they are to be used for summative as well as formative assessment. The current study examined higher education students' performance on online assessments which they were required to take as part of an undergraduate psychology course. The marks obtained by students required to take the same multiple choice question (MCQ) assessment online and offline (in pen-and-paper format) were compared, and relationships between performance and computer anxiety and computer engagement measures were explored. The results indicate minimal influence of assessment modality, computer anxiety and computer engagement on MCQ test scores, with only very small and non significant effect sizes being observed overall. No evidence of any significant relationships between gender and age and computer attitudes was observed. To conclude, the results provide promising initial support for the use of online summative assessments in contexts similar to the one used here, allaying some concerns about disadvantaging certain groups of students. However, further research is needed to explore possible performance differences across different contexts, assessment types, and student cohorts.

Assessment of multiple-choice questions in selected test banks accompanying text books used in nursing education.

The purpose of this study was to assess multiple-choice questions used in test-banks accompanying selected nursing textbooks. A random sample of 2,913 questions was selected from a convenience sample of 17 test banks. Questions were evaluated on (a) adherence to generally accepted guidelines for writing multiple-choice questions; (b) cognitive level as defined by Bloom's (1961) taxonomy; and (c) distribution of correct answers as A, B, C, or D. The results were 2,233 violations of item-writing guidelines, most of which were minor but some were serious. A large number of questions (47.3%) were written at the knowledge level and only 6.5% were written at the analysis level. The correct answers were evenly distributed: c2s ranged from 0.00 to 4.84; chi square value needed to reach .05 probability was 26.30. Faculty are encouraged to evaluate multiple-choice questions from test banks carefully before using them for exams.