Multiple Choice Questions Items Research Articles

Knowledge of dental faculty in gulf cooperation council states of multiple-choice questions’ item writing flaws

ABSTRACT Multiple-Choice Questions provide an objective cost/time effective assessment. Deviation from appropriate question writing structural guidelines will most probably result in commonly ignored multiple-choice questions writing flaws, influencing the ability of the assessment to measure students’ cognitive levels thereby seriously affecting students’ academic performance outcome measures. To gauge the knowledge of multiple-choice question items writing flaws in dental faculty working at colleges in Gulf Cooperation Council (GCC) countries. A cross-sectional short online Survey MonkeyTM multiple-choice questions-based questionnaire was disseminated to dental faculty working in GCC countries during the academic year 2018/2019. The questionnaire included five test incorrect (flawed) multiple-choice questions and one correct control question. The participants were asked to identify flawed multiple-choice question items from the known 14 items writing flaws. Out of a total of 460 faculty, 216 respondents completed the questionnaires, 132 (61.1%) were from Saudi Arabia, while numbers of participants from United Arab Emirates, Kuwait and Oman were 59 (27.3), 14 (6.5%) and 11 (5.1%) respectively. Majority of participants were male (n = 141, 65.9%) compared to 73 females (34.1%). Eighty percent of the participants possessed more than five years of teaching experience. Assistant professors constituted the majority (43.3%) of the academic positions participating in this study. The overall fail rate ranged from 76.3% to 98.1% and almost 2/3rds of the participants were unable to identify one or more of the flawed item(s). No significant association was observed between the demographics (age, region, academic position and specialty) and knowledge except that of participant’s gender (p < 0.009). GCC dental faculty demonstrated below average knowledge of multiple-choice question items writing flaws. Training and workshops are needed to ensure substantial exposure to proper multiple-choice question items construction standards.

Analysis of One Best MCQS in Five Preuniversity Physiology Examinations

Objectives&#x0D; One-best MCQs (Multiple Choice Questions) is a popular form of assessment where the student selects the best possible answer from the options provided. The objective of this study was to analyze the MCQs used in preliminary examinations in terms of Difficulty Index, Differentiation Index and Distracter efficiency.&#x0D; Method&#x0D; Total 100 MCQ items with one correct option and three distracters were taken from past 5 years preuniversity examinations and analyzed for level of difficulty and power of discrimination. A total of 149, 156, 154, 156 and 148 students appeared for the exam in each year. The Difficulty index was calculated by the formula P = R/T. High and low groups consisting of upper and lower 27% students, respectively, were taken after arranging the scores in descending order for calculating the Discrimination Index as follows: DI = 2 x [(H-L)/N]. Distracter efficiency was determined on the basis of the number of Nonfunctioning Distracters (NF-Ds). An ideal item should have a DFI between 0.30–70.0, a DI of &gt;0.24 and a DE of 100%.&#x0D; Results&#x0D; The Difficulty indices for the ten papers of 10 MCQ`s each were (Mean±S. D) 0.50±0.24, 0.49±0.25, 0.52±0.23, 0.55±0.19, 0.73±0.22, 0.68±0.18, 0.53±0.24, 0.65±0.18, 0.54±0.3, 0.47±0.26 respectively. Differentiation indices were (Mean±S. D)0.37±0.12, 0.34±0.17, 0.35±0.14, 0.43±0.15, 0.40±0.15, 0.43±0.17, 0.40±0.20, 0.46±0.11, 0.40±0.16, 0.40±0.20. Thus as a whole, the papers were of average difficulty and were able to differentiate between the students. Thirtyfive MCQs were ideal MCQs. In individual analysis, total 37 MCQs were very easy (Difficulty Index&gt;0.70) and 18 were found to be very tough (Difficulty Index&lt;0.30). In Distracter analysis, 20% of the distracters were found to be nonfunctional (NFDs). 30% of the MCQ`s had One NFD, while 9% and 4% had two and three NFD`s respectively.&#x0D; Conclusion&#x0D; Overall, the papers had very good difficulty and differentiation indices which were quite consistent over the five years but there are a few MCQs and distracters which need to be modified to further enhance accurate assessment of the students.

The future of online testing and assessment: question quality in MOOCs

If MOOCs are to play a role in the future of higher education it is imperative that we critically examine how they are currently functioning. In particular, questions persist about the role MOOCs will play in the future of formal accredited learning. As the focus turns from informal and free to formal, accredited and paid, greater scrutiny will be brought to bear on the quality of the courses themselves. Although there have been some empirical studies into the quality of MOOCs, a notable gap exists in that such research has not examined Multiple Choice Questions (MCQs) which are a key component of much MOOC assessment and testing. Previous research suggests that flawed MCQ items may compromise the reliability and validity of these assessments, potentially leading to inconsistent outcomes for students. This study was hence designed to examine MCQ quality in MOOCs. 204 MCQs were analysed, from a selection of 18 MOOCs, sampling the domains of computing, social science and health sciences. Over 50% of MCQs (112) contained at least one item flaw; 57 MCQs contained multiple flaws. A large proportion of MOOC MCQs violated item-writing guidelines, which is comparable with previous studies examining the prevalence of flaws in assessments in more traditional educational contexts. The problem of low quality MCQs can be ameliorated by appropriate faculty training and pre- and post-test quality checks. These activities are essential if MOOCs are to become a force that can enable enhanced and improved pedagogies in the future of higher education, instead of simply proceeding to replicate existing poor practices at scale.

Examining the impact of specific types of item-writing flaws on student performance and psychometric properties of the multiple choice question.

This article was migrated. The article was marked as recommended. Background: Item-writing flaws (IWFs) are common in multiple choice questions (MCQs) despite item-writing guidelines. Previous studies have shown that IWFs impact validity as observed through student performance, item difficulty, and discrimination. Most previous studies have examined IWFs collectively and have shown that they have a diverse impact. The aim of the study was to determine if the effects of individual types of IWFs are systematic and predictable. Method: A cross-over study design was used. 100 pairs of MCQ items (with and without an IWF) were constructed to test 10 types of IWFs. Medical students were invited to participate in a mock examination. Paper A consisted of 50 flawed followed by 50 unflawed items. Paper B consisted of 50 unflawed followed by 50 flawed items. The effect of each of the IWFs on mean item scores, item difficulty and discrimination were examined. Results: The hypothesised effect of IWFs on mean item scores was confirmed in only 4 out of 10 cases. ' Longest choice is correct', 'Clues to the right answer (Eponymous terms)' and ' Implausible distractors' positively impacted, while 'Central idea in choices rather than stem' negatively impacted mean item scores. Other flaws had either the opposite or no statistically significant effect. IWFs did not impact item difficulty or discrimination. Conclusion: The effect of IWFs is neither systematic nor predictable. Unpredictability in assessment produces error and thus loss of validity. Therefore, IWFs should be avoided. Faculties should be encouraged to invest in item-writing workshops in order to improve MCQs. However, the cost of doing so should be carefully weighed against the benefits of developing programmes of assessment.

Are multiple-choice questions a good tool for the assessment of clinical competence in Internal Medicine?

There are many feasible tools for the assessment of clinical practice, but there is a wide consensus on the fact that the simultaneous use of several different methods could be strategic for a comprehensive overall judgment of clinical competence. Multiple-choice questions (MCQs) are a well-established reliable method of assessing knowledge. Constructing effective MCQ tests and items requires scrupulous care in the design, review and validation stages. Creating high-quality multiple-choice questions requires a very deep experience, knowledge and large amount of time. Hereby, after reviewing their construction, strengths and limitations, we debate their completeness for the assessment of professional competence.

Item Analysis of Multiple Choice Questions at the Department of Paediatrics, Arabian Gulf University, Manama, Bahrain.

The current study aimed to carry out a post-validation item analysis of multiple choice questions (MCQs) in medical examinations in order to evaluate correlations between item difficulty, item discrimination and distraction effectiveness so as to determine whether questions should be included, modified or discarded. In addition, the optimal number of options per MCQ was analysed. This cross-sectional study was performed in the Department of Paediatrics, Arabian Gulf University, Manama, Bahrain. A total of 800 MCQs and 4,000 distractors were analysed between November 2013 and June 2016. The mean difficulty index ranged from 36.70-73.14%. The mean discrimination index ranged from 0.20-0.34. The mean distractor efficiency ranged from 66.50-90.00%. Of the items, 48.4%, 35.3%, 11.4%, 3.9% and 1.1% had zero, one, two, three and four nonfunctional distractors (NFDs), respectively. Using three or four rather than five options in each MCQ resulted in 95% or 83.6% of items having zero NFDs, respectively. The distractor efficiency was 91.87%, 85.83% and 64.13% for difficult, acceptable and easy items, respectively (P <0.005). Distractor efficiency was 83.33%, 83.24% and 77.56% for items with excellent, acceptable and poor discrimination, respectively (P <0.005). The average Kuder-Richardson formula 20 reliability coefficient was 0.76. A considerable number of the MCQ items were within acceptable ranges. However, some items needed to be discarded or revised. Using three or four rather than five options in MCQs is recommended to reduce the number of NFDs and improve the overall quality of the examination.

Effectiveness of longitudinal faculty development programs on MCQs items writing skills: A follow-up study.

This study examines the long-term impact of the faculty development programs on the multiple choice question (MCQ) items’ quality leading to study its effect on the students’ overall competency level during their yearly academic assessment. A series of longitudinal highly constructed faculty development workshops were conducted to improve the quality of the MCQs items writing skills. A total of 2207 MCQs were constructed by 58 participants for the assessment of 882 students’ cognitive competency level during the academic years 2012–2015. The MCQs were analyzed for the difficulty index (P-value), discriminating index (DI), presence/absence of item writing flaws (IWFs), and non-functioning distractors (NFDs), Bloom’s taxonomy cognitive levels, test reliability, and the rate of students’ scoring. Significant improvement in the difficulty index and DI were noticed during each successive academic year. Easy and poor discriminating questions, NFDs and IWFs were decreased significantly, whereas distractor efficiency (DE) mean score and high cognitive level (K2) questions were increased substantially during the each successive academic year. Improved MCQs’ quality leaded to increased competency level of the borderline students. Overall, the longitudinal faculty development workshops help in improving the quality of the MCQs items writing skills of the faculty that leads to students’ high competency levels.

Exploring examinee behaviours as validity evidence for multiple-choice question examinations.

Clinical-vignette multiple choice question (MCQ) examinations are used widely in medical education. Standardised MCQ examinations are used by licensure and certification bodies to award credentials that are meant to assure stakeholders as to the quality of physicians. Such uses are based on the interpretation of MCQ examination performance as giving meaningful information about the quality of clinical reasoning. There are several assumptions foundational to these interpretations and uses of standardised MCQ examinations. This study explores the implicit assumption that cognitive processes elicited by clinical-vignette MCQ items are like the processes thought to occur with 'real-world' clinical reasoning as theorised by dual-process theory. Fourteen participants (three medical students, five residents and six staff physicians) completed three sets of five timed MCQ items (total 15) from the Medical Knowledge Self-Assessment Program (MKSAP). Upon answering a set of MCQs, each participant completed a retrospective think aloud (TA) protocol. Using constant comparative analysis (CCA) methods sensitised by dual-process theory, we performed a qualitative thematic analysis. Examinee behaviours fell into three categories: clinical reasoning behaviours, test-taking behaviours and reactions to the MCQ. Consistent with dual-process theory, statements about clinical reasoning behaviours were divided into two sub-categories: analytical reasoning and non-analytical reasoning. Each of these categories included several themes. Our study provides some validity evidence that test-takers' descriptions of their cognitive processes during completion of high-quality clinical-vignette MCQs align with processes expected in real-world clinical reasoning. This supports one of the assumptions important for interpretations of MCQ examination scores as meaningful measures of clinical reasoning. Our observations also suggest that MCQs elicit other cognitive processes, including certain test-taking behaviours, that seem 'inauthentic' to real-world clinical reasoning. Further research is needed to explore if similar themes arise in other contexts (e.g. simulated patient encounters) and how observed behaviours relate to performance on MCQ-based assessments.

Evaluation of multiple choice questions using item analysis tool: a study from a medical institute of Ahmedabad, Gujarat

Background: Multiple choice question (MCQ) assessments are becoming popular means to assess knowledge for many screening examinations among several fields including Medicine. The single best answer MCQs may also test higher-order thinking skills. Hence, MCQs remain useful assessment gadget. Objectives: 1) To evaluate Multiple Choice Questions for testing their quality. 2) To explore the association between difficulty index (p-value) and discrimination indices (DI) with distractor efficiency (DE). 3) To study the occurrence of functioning distractors for MCQs. Methods: Total five MCQ test sessions were conducted among interns of a medical institute of Ahmedabad city Gujarat, between April 2016 to March 2017, as part of their compulsory rotating postings in the department. The average participation in each of the sessions was 17 interns, thus a total of 85 interns getting enrolled. For each test session, the questionnaire consisted of forty MCQs having 4 options including a single best answer. The MCQs were analyzed for difficulty index (DIF-I, p-value), discrimination index (DI), and distractor efficiency (DE). Results: Total 85 interns attended the tests consisting of total 200 MCQ items (questions) from four major medical disciplines namely - Medicine, Surgery, Obstetrics &amp; Gynecology and Community Medicine. Mean test scores of each test ranged from 36.0% to 45.8%.The reliability of the tests, the Kuder Richardson (KR) 20, ranged from 0.29 to 0.52. The standard error of Measurement ranged from 2.59 to 2.79.Out of total 200 MCQs, seventy nine (n=79) had Discrimination index (DI) &lt;0.15 (poor), and 61 had DI ≥0.35 (excellent). Easy items having average DE of all tests was 20.1%. Conclusions: Items having average difficulty and high discrimination with functioning distractors should be incorporated into tests to improve the validity of the assessment.

Evaluating the long-term impact of faculty development programs on MCQ item analysis

Purpose: Evaluating the long-term impact of faculty development programs (FDPs) can help monitor the effectiveness of the program and identify areas for development. This study examined long-term differences in confidence, knowledge, behaviors, and policies of faculty members who attended FDPs on multiple choice question (MCQ) item analysis and faculty members who did not attend the FDPs.Methods: A cross-sectional study design was used, by administering a 24-item survey to a representative sample (simple random selection) of 61 faculty members at King Abdulaziz University Faculty of Medicine.Results: Among respondents, 34% did not attend FDPs; 53% attended 1–3 FDPs; and 13% attended more than 3 FDPs on MCQ item analysis. Results showed that faculty knowledge on elements of MCQ item analysis was significantly greater (p = .01) for members who attended the FDPs. Faculty who attended FDPs on MCQ item analysis were twice more likely to conduct item analysis in general (p = .020) and four times more likely to conduct item analysis for more than 70% of module examinations (p = .005).Conclusion: FDPs focused on MCQ item analysis can yield systematic changes on faculty confidence, knowledge, and behaviors. Moreover, FDPs also need support from the department and need sustained strategic support to ensure continued effectiveness.

North Korean refugee doctors' preliminary examination scores.

PurposeAlthough there have been studies emphasizing the re-education of North Korean (NK) doctors for post-unification of the Korean Peninsula, study on the content and scope of such re-education has yet to be conducted. Researchers intended to set the content and scope of re-education by a comparative analysis for the scores of the preliminary examination, which is comparable to the Korean Medical Licensing Examination (KMLE).MethodsThe scores of the first and second preliminary exams were analyzed by subject using the Wilcoxon signed rank test. The passing status of the group of NK doctors for KMLE in recent 3 years were investigated. The multiple-choice-question (MCQ) items of which difficulty indexes of NK doctors were lower than those of South Korean (SK) medical students by two times of the standard deviation of the scores of SK medical students were selected to investigate the relevant reasons.ResultsThe average scores of nearly all subjects were improved in the second exam compared with the first exam. The passing rate of the group of NK doctors was 75%. The number of MCQ items of which difficulty indexes of NK doctors were lower than those of SK medical students was 51 (6.38%). NK doctors’ lack of understandings for Diagnostic Techniques and Procedures, Therapeutics, Prenatal Care, and Managed Care Programs was suggested as the possible reason.ConclusionThe education of integrated courses focusing on Diagnostic Techniques and Procedures and Therapeutics, and apprenticeship-style training for clinical practice of core subjects are needed. Special lectures on the Preventive Medicine are likely to be required also.

A comparison of clinical-scenario (case cluster) versus stand-alone multiple choice questions in a problem-based learning environment in undergraduate medicine.

ObjectivesTo compare stand-alone multiple choice questions (MCQs) and integrated clinical-scenario (case cluster) multiple choice questions (CS-MCQs) in a problem-based learning (PBL) environment.MethodsA retrospective descriptive analysis of MCQ examinations was conducted in a course that integrates the subspecialties of anatomical pathology, chemical pathology, hematology, immunology, microbiology and pharmacology. The MCQ items were analyzed for their reliability (Kuder–Richardson-20, KR-20), level of difficulty (Pi), discrimination index (Di), item distractors and student performances. The statistical analysis of the results was extracted from the integrity online item-analysis programme. The results of the standard stand-alone and CS multiple choice questions were compared.ResultsKR-20 for the CS-MCQs and stand-alone MCQs was consistently high. KR-20 and Pi were higher for the CS-MCQs. There was no significant difference between the CS-MCQs and stand-alone MCQs in Pi and Di. A range of difficulty levels was found based on Bloom's taxonomy. The mean scores for the class were higher for the CS-MCQ examination. The compilation of the CS-MCQ examination was more challenging.ConclusionsCS-MCQs compare favorably to stand-alone MCQs and provide opportunities for the integration of sub-specialties and assessment in keeping with PBL. They assess students' cognitive skills and are reliable and practical. Different levels of item difficulty promote multi-logical and critical thinking. Students' scores were higher for the CS-MCQ examination, which may suggest better understanding of the material and/or better question clarity. The scenarios have to flow logically. Increasing the number of scenarios ensures the examination of more course content.

Application of a utility analysis to evaluate a novel assessment tool for clinically oriented physiology and pharmacology.

Multiple-choice questions are a gold-standard tool in medical school for assessment of knowledge and are the mainstay of licensing examinations. However, multiple-choice questions items can be criticized for lacking the ability to test higher-order learning or integrative thinking across multiple disciplines. Our objective was to develop a novel assessment that would address understanding of pathophysiology and pharmacology, evaluate learning at the levels of application, evaluation and synthesis, and allow students to demonstrate clinical reasoning. The rubric assesses student writeups of clinical case problems. The method is based on the physician's traditional postencounter Subjective, Objective, Assessment and Plan note. Students were required to correctly identify subjective and objective findings in authentic clinical case problems, to ascribe pathophysiological as well as pharmacological mechanisms to these findings, and to justify a list of differential diagnoses. A utility analysis was undertaken to evaluate the new assessment tool by appraising its reliability, validity, feasibility, cost effectiveness, acceptability, and educational impact using a mixed-method approach. The Subjective, Objective, Assessment and Plan assessment tool scored highly in terms of validity and educational impact and had acceptable levels of statistical reliability but was limited in terms of acceptance, feasibility, and cost effectiveness due to high time demands on expert graders and workload concerns from students. We conclude by making suggestions for improving the tool and recommend deployment of the instrument for low-stakes summative assessment or formative assessment.

MCQ tests in Advanced Trauma Life Support (ATLS©): Development and revision

BackgroundIn Advanced Trauma Life Support (ATLS©) courses, multiple choice question (MCQ) tests are used to assess student's post course knowledge. As part of the ninth Edition Revision Process, existing MCQ tests were reviewed and revised by an International MCQ Revision group. The aim of this study was to evaluate the revision procedure and its effects. MethodsBased on psychometric data and evidence based guidelines for adequate MCQ item and test development, a detailed stepwise approach was determined and followed to evaluate the existing MCQs, and to guide test item revision or replacement. ResultsThe MCQ Revision group composed three new draft test versions comprising of 40 MCQs each. These were beta-tested among ATLS Instructors in various countries involved in ATLS. Psychometric analysis demonstrated that a minority of MCQ items required revision to create three equally balanced tests. After these final adjustments, a new set of three validated MCQ tests was available for use in 9th edition ATLS provider courses. Beta testing was performed using instructors but not students. The failure rate amongst students of ATLS provider courses increased significantly after introduction of the new MCQ tests. ConclusionATLS tests were revised and updated using current evidence based guidelines and psychometric analysis. Difficulty of the tests was not initially beta-tested on students. Increasing test item discrimination and quality resulted in lower test scores by students.

It is time to progress beyond the System 1 versus System 2 dichotomy.

None

Faculty development programs improve the quality of Multiple Choice Questions items' writing.

The aim of this study was to assess the utility of long term faculty development programs (FDPs) in order to improve the quality of multiple choice questions (MCQs) items' writing. This was a quasi-experimental study, conducted with newly joined faculty members. The MCQ items were analyzed for difficulty index, discriminating index, reliability, Bloom's cognitive levels, item writing flaws (IWFs) and MCQs' nonfunctioning distractors (NFDs) based test courses of respiratory, cardiovascular and renal blocks. Significant improvement was found in the difficulty index values of pre- to post-training (p = 0.003). MCQs with moderate difficulty and higher discrimination were found to be more in the post-training tests in all three courses. Easy questions were decreased from 36.7 to 22.5%. Significant improvement was also reported in the discriminating indices from 92.1 to 95.4% after training (p = 0.132). More number of higher cognitive level of Bloom's taxonomy was reported in the post-training test items (p<0.0001). Also, NFDs and IWFs were reported less in the post-training items (p<0.02). The MCQs written by the faculties without participating in FDPs are usually of low quality. This study suggests that newly joined faculties need active participation in FDPs as these programs are supportive in improving the quality of MCQs' items writing.

An Enhancement-focused framework for developing high quality single best answer multiple choice questions.

The primary goal of any assessment of students is to provide valid and reliable evaluations of students' knowledge and skills as well as provision of accurate feedback to students about their performance. Contrary to best practice guidelines for development of multiple choice questions (MCQs), however, items used within medical schools are often flawed. This disappoints students and discourages examiners from using in-house MCQ databases. Vetting and reviewing items can improve the quality of MCQs. In this paper, we describe our approach to standardize the format used for MCQ assessment and provide recommendations for quality enhancement of high-stakes assessment. A collaborative enhancement-focused vetting and review approach to development of high quality single best answer MCQs has been described. Implementation of a collaborative strategy to blueprint, vet, review and standard set MCQ items for high stakes examinations can effectively contribute to assessment quality assurance. Similarly, shared responsibility for post examination analyses of items may reveal the psychometric properties of items in need of improvement and contribute to closure of the assessment outcomes feedback loop. Devolving responsibility for implementation of assessment processes as an integral part of educational practices and values can maximize reliability and standards of assessment processes. We contend that while logistics and time constraints are of concern to busy faculty members, judicious utilization of resources to develop well-written MCQ items are well worth the effort to produce reliable and valid examinee scores. An enhancement-focused approach can be institutionally rewarding and lead to improved quality of high stakes assessments.

Development of knowledge tests for multi-disciplinary emergency training: a review and an example.

The literature is sparse on written test development in a post-graduate multi-disciplinary setting. Developing and evaluating knowledge tests for use in multi-disciplinary post-graduate training is challenging. The objective of this study was to describe the process of developing and evaluating a multiple-choice question (MCQ) test for use in a multi-disciplinary training program in obstetric-anesthesia emergencies. A multi-disciplinary working committee with 12 members representing six professional healthcare groups and another 28 participants were involved. Recurrent revisions of the MCQ items were undertaken followed by a statistical analysis. The MCQ items were developed stepwise, including decisions on aims and content, followed by testing for face and content validity, construct validity, item-total correlation, and reliability. To obtain acceptable content validity, 40 out of originally 50 items were included in the final MCQ test. The MCQ test was able to distinguish between levels of competence, and good construct validity was indicated by a significant difference in the mean score between consultants and first-year trainees, as well as between first-year trainees and medical and midwifery students. Evaluation of the item-total correlation analysis in the 40 items set revealed that 11 items needed re-evaluation, four of which addressed content issues in local clinical guidelines. A Cronbach's alpha of 0.83 for reliability was found, which is acceptable. Content and construct validity and reliability were acceptable. The presented template for the development of this MCQ test could be useful to others when developing knowledge tests and may enhance the overall quality of test development.

Major Reforms and Issues of the Medical Licensing Examination Systems in Korea

Since the establishment of the national medical licensing examination board in 1992, the medical licensing examination system has changed enormously and this has had a number of impacts on examination services. All those reforms were aimed at improving the relevance and reliability of the test. Several attempts of the testing system have appeared in the new examination service, and which have also brought about the changes in the medical school curriculum such as introducing integrated courses instead of traditional subjects, using test scores as a reference to the post-graduation selection test. Some examples of changes in the examination system are as follows: 1) choosing three integrated test subjects and outlines of their reference content instead of 15 academic subjects, 2) adjusting the ratio of multiple choice question items to focus more on the problem solving level, 3) introduction of ‘one-best answer’ single set and ‘extended matching type items, 4) item construction based on real clinical cases and real clinical materials. Recently, a clinical skill test system has been introduced to measure examinees’ basic clinical skills competencies. Despite continuing efforts, the examination system still has many issues remaining to be solved. These problems include the differential weighting of test items, appropriate threshold for passing, and practicality of pre-testing to stabilize the passing rate and avoid the hazards of newness and undesirably difficult test items.

Automatic generation of multiple choice questions using dependency-based semantic relations

In this paper, we present an unsupervised dependency-based approach to extract semantic relations to be applied in the context of automatic generation of multiple choice questions (MCQs). MCQs also known as multiple choice tests provide a popular solution for large-scale assessments as they make it much easier for test-takers to take tests and for examiners to interpret their results. Manual generation of MCQs is a very expensive and time-consuming task and yet they often need to be produced on a large scale and within short iterative cycles. We approach the problem of automated MCQ generation with the help of unsupervised relation extraction, a technique used in a number of related natural language processing problems. The goal of Unsupervised relation extraction is to identify the most important named entities and terminology in a document and then recognise semantic relations between them, without any prior knowledge as to the semantic types of the relations or their specific linguistic realisation. We use these techniques to process instructional texts and identify those facts (terminology, entities, and semantic relations between them) that are likely to be important for assessing test-takers' familiarity with the instructional material. We investigate an approach to learn semantic relations between named entities by employing a dependency tree model. Our findings show that an optimised configuration of our MCQ generation system is capable of attaining high precision rates, which are much more important than recall in the automatic generation of MCQs. We also carried out a user-centric evaluation of the system, where subject domain experts evaluated automatically generated MCQ items in terms of readability, usefulness of semantic relations, relevance, acceptability of questions and distractors and overall MCQ usability. The results of this evaluation make it possible for us to draw conclusions about the utility of the approach in practical e-Learning applications.