Traditional Multiple-choice Tests Research Articles

Stepping into the virtual realm: Examining the utility of non-immersive VR on medical residents' self-efficacy, performance, and satisfaction

Central Venous Catheterization (CVC) is conducted in more than 5 million patients annually in the United States. Currently, CVC theoretical knowledge is typically taught with didactic lectures with minimal assessment of knowledge before simulation training and clinical environment. To assess CVC theoretical knowledge, virtual reality (VR) can develop residents’ surgical skills and can provide feedback on performance. While all types of VR can be used to assess improvements in residents’ knowledge prior to simulation training, few studies have compared the impact of non-immersive VR on improving resident knowledge. The current study was developed to identify the utility of a non-immersive CathSim VR on improvements in resident CVC theoretical knowledge, self-efficacy, and satisfaction when compared to a traditional multiple-choice assessment This was completed through an empirical investigation with 34 medical residents. The results showed that when compared to a traditional multiple-choice assessment, the non-immersive CathSim VR assessment led to improvements in self-efficacy and potentially better learning experience. Meanwhile, results also showed that there were no differences in assessment score, number of attempts, and overall satisfaction. Thus, we can conclude that non-immersive VR has the potential to be more effective than traditional multiple-choice test in improving self-efficacy and is as effective for residents’ assessment score, number of attempts, and overall satisfaction.

Workplace-based assessment for anesthesia residents: efficacy and competence issues

Background. Workplace-based assessment (WPBA) is a new technology for evaluating medical residents that is believed to be more effective than traditional multiple-choice tests or exam-based assessment methods. The aim of the study was to investigate the WPBA efficacy in final-year medical residents: multi-source feedback (360°) assessment, direct procedure assessment (Anaesthesia-Clinical Evaluation Exercise (A-CEX)) and clinical case-based assessment, and to identify difficult competencies, which graduate residents acquire the least successfully. Materials and methods. From April 1 to June 15, 2023, 36 residents were evaluated at the workplace: the multi-source feedback 360°-evaluation form was filled out by three representatives of the medical team where the resident worked during the on-site part of the residency (1 anesthesiologist, 1 nurse and 1 other specialist); the form of direct procedure assessment was filled out by the teacher on face-to-face bases directly during the resident’s performance of the general anesthesia in low-risk patients (American Society of Anesthesiologists (ASA) I or II), preoperative examination of low-risk patients (ASA I or II) and neuraxial anesthesia in low-risk patients (ASA I or II) with the following feedback; assessment based on a clinical case was carried out by teachers during the conference of residents, where each of them presented a clinical case of anesthetic management or treatment of a critically ill patient. Results. According to the results of the multi-source feedback (360°) evaluation, most of the residents have received 7 or more points out of 10 possible. Communication with the medical team (odds ratio (OR) 1.9 [1.05–3.5], p = 0.048) and management in a critical situation (OR 2 [1.14–3.83], p = 0.024) were most difficult competencies during the multi-source feedback (360°) assessment. During direct procedure of general anesthesia, the most difficult competencies were: 1) management in a critical situation — knowledge/recognition of risks and how to avoid and treat them (OR 5.29 [1.9–14.4], p = 0.001 compared to documentation); 2) solving problems, making decisions (OR 12.6 [1.5–10,4], p = 0.007 compared to documentation); 3) interaction in the team (OR 2.7 [1.1–6.5], p = 0.049). No problematic competencies were identified during the assessment of neuraxial anesthesia competencies and preoperative examination. Conclusions. WPBA techniques such as multi-source feedback (360°) assessment, direct procedure evaluation (A-CEX) and clinical case-based assessment are effective in final year residents, improving their communication skills, readiness to work independently, help identify skill deficiencies. Communication with the medical team and management in a critical situation were difficult competencies during the multi-source feedback (360°) evaluation. Difficult competences during the direct evaluation of general anesthesia turned out to be the management in a critical situation; solving problems, making decisions; interaction in the team. No problematic competencies were identified during the assessment of neuraxial anesthesia competencies and preoperative examination.

Pregraduation online curriculum delivery of internal medicine course during the COVID-19 pandemic: Experience of a medical college in Saudi Arabia

Background: The COVID-19 pandemic has challenged medical education and resulted in a rapid and unexpected global shift toward online education. The study presents the experience of the College of Medicine of King Saud University in delivering a comprehensive internal medicine course for final-year medical students, which includes the implementation of novel methods of assessment. Methods: A 2-week revision of the internal medicine course was conducted for 70 pregraduation medical students. Lectures and focused case-based tutorials were conducted online. Novel assessment methods included brief written essays summarizing the tutorials, open-book short-answer question (SAQ) examination, and a virtual standardized oral examination (vSOE). A questionnaire was sent to the students (n = 70) on the contents of the course, clarity of objectives, duration, and online platform. Results: The response rate was 99.3%. Students scored higher in the SAQs compared with traditional multiple-choice tests (mean score: 92.3% vs. 69.8%). The vSOE was efficient, and grades were comparable to traditional objective structured clinical examination. The majority of students (93%) reported the scientific content as good or excellent, whereas approximately 30% found the course goals lacked a clear description. The majority (76%) found the course length appropriate, and 94% reported the online platform installation and operation easy or very easy. Conclusions: The comprehensive online internal medicine course was successfully delivered to pregraduation students. Innovative online examination methods can be a reasonable alternative in the case of lockdown. This modality of teaching and assessment was considered useful and met the expectations of students.

Timed versus untimed recognition of L2 collocations: Does estimated proficiency modulate congruency effects?

AbstractCongruency (the availability of a direct first language translation) and level of proficiency have been reported among the most important determinants of second language collocation processing. However, only very few studies looked at the interaction between the two determinants, and none of these directly compared untimed collocation recognition assessed through traditional tests to timed recognition evident in psycholinguistic tasks. The current study administered both types of form recognition measures to 228 female Saudi English as a foreign language learners in two separate experiments: a traditional multiple-choice test (Experiment 1) and a timed acceptability judgment task (Experiment 2). Experiment 2 also tested 37 native speakers of English as a baseline for comparison. Congruency, estimated proficiency (vocabulary test scores), and the interaction between the two were evaluated as predictors of untimed and timed recognition through mixed-effects modeling. Results showed that congruency and estimated proficiency had a clear effect on untimed and timed recognition. More interesting, the effect of proficiency was clearer on timed recognition with a gradual decrease in the first language effect as proficiency increased getting closer to nativelike collocation processing. Results have implications for second language collocation learning and testing.

Addressing the shortcomings of traditional multiple-choice tests: subset selection without mark deductions

This article presents the results of an empirical study of “subset selection” tests, which are a generalisation of traditional multiple-choice tests in which test takers are able to express partial knowledge. Similar previous studies have mostly been supportive of subset selection, but the deduction of marks for incorrect responses has been a cause for concern. For the present study, a novel marking scheme based on Akeroyd’s “dual response system” was used instead. In Akeroyd’s system, which assumes that every question has four answer options, test takers are able to split their single 100% bet on one answer option into two 50% bets by selecting two options, or into four 25% bets by selecting no options. To achieve full subset selection, this idea was extended so that test takers could also split their 100% bet equally between three options. The results indicate increased test reliability (in the sense of measurement consistency), and also increased satisfaction on the part of the test takers. Furthermore, since the novel marking scheme does not in principle lead to either inflated or deflated marks, this makes it easy for educators who currently use traditional multiple-choice tests to switch to using subset selection tests.

An Experimental Validation of Sequential Multiple-Choice Tests

Testwiseness may introduce construct-irrelevant variance to multiple-choice test scores. Presenting response options sequentially has been proposed as a potential solution to this problem. In an experimental validation, we determined the psychometric properties of a test based on the sequential presentation of response options. We created a strong validity criterion by providing participants with different levels of information on a domain about which they had no prior knowledge. Test takers’ capability of guessing the correct answer was strongly reduced by the sequential test format, whereas sequential test scores were as valid and reliable as multiple-choice test scores. We concluded that the sequential presentation of response options should be investigated more closely as a viable alternative to the traditional multiple-choice test format.

Partial Credit in Answer-Until-Correct Multiple-Choice Tests Deployed in a Classroom Setting

ABSTRACT The answer-until-correct (AUC) method of multiple-choice (MC) testing involves test respondents making selections until the keyed answer is identified. Despite attendant benefits that include improved learning, broad student adoption, and facile administration of partial credit, the use of AUC methods for classroom testing has been extremely limited. This study presents scoring properties and item analysis for 26 AUC university course examinations, administered using a commercial scratch-card response system. Here, we show that beyond the traditional pedagogical advantages of AUC, the availability of partial credit adds psychometric advantages by boosting both the mean item discrimination and overall test-score reliability, when compared to tests scored dichotomously upon initial response. Furthermore we also find a strong correlation between students’ initial-response successes and the likelihood that they would obtain partial credit when they make incorrect initial responses. Thus, partial credit is being granted based on partial knowledge that remains latent in traditional MC tests. The fact that these advantages are realized in real-life classroom tests may motivate further expansion of the use of AUC MC tests in higher education.

The Study of Portfolio Assessment on EFL Students’ English Reading

In Taiwan, English teachers usually use the traditional assessment to test the students' level of reading comprehension such as the traditional multiple-choice tests have been mainly adopted to evaluate students' reading ability and students are usually asked to answer the questions with "standard answers”, rather than think over the questions critically and give responses rationally (Jasmine, 1992 & Yeh, 2000). A scholar pointed out portfolio is a tool of documenting students' academic accomplishments for examining students' knowledge (Meyer, 1992). Moreover, the portfolio assessment is almost used in English writing class (Tsai, 2005), but the studies of portfolio assessment on English reading are lacked. Teachers might notice that the procedure of students' learning and understand what problems or difficulties students could have about English learning. Some scholars indicated that one of advantages of portfolio assessment is to assist students to realize their own strengths and weaknesses in reading process (Cunningham, 1999). Therefore, the purpose of this study aimed to investigate the effects of portfolio assessment on students' English reading ability. The quantitative and qualitative research designs were applied in this study. The questionnaire focused on three aspects that are reflection of reading competence and reading portfolio assessment, the comparison between portfolio assessment and traditional assessment and a review of portfolio assessment. The participants of this study were 71 sophomores who are all English majors. Besides, ten students from 71 participants, were to be interviewed voluntarily. Instructional implications for EFL educator to apply effectively the portfolio assessment using in reading class are presented.

The Development of Two-Tier Multiple Choice Test to Assess Students’ Conceptual Understanding about Light and Optical Instruments

This study aimed to develop a two-tier multiple choice test about the concept of “light and optical instruments” in 8th grade of Indonesian science curriculum. The test development procedure had three general steps: defining the content area of the test, identification on students’ conceptions, and the development of two-tier multiple choice test. The final version of two-tier multiple choice test consisted of 25 items question. This test was administered to 95 junior high school students. The students had completed a unit on light and optical instruments. The reliability of the test was 0.76. Based on the data analysis, twenty-two alternative conceptions were identified. The results of the study showed that the two-tier multiple choice test was effective in determining the students’ misconceptions and also it might be used as an alternative to the traditional multiple choice test. In conclusion, two-tier multiple choice test can be used to assess students’ conceptual understanding as well as students’ misconceptions on light and optical instruments concept.

An Automatic Quantification of the Randomness of Answering Correctly in Taking Traditional Multiple-choice Tests

In taking traditional multiple-choice tests, random guessing is unavoidable yet nonnegligible. To uncover the “unfairness” caused by random guessing, this paper designed a Microsoft Excel template with the use of relevant functions to automatically quantify the probability of answering correctly at random, eventually figuring out the least scores a testee should get to pass a traditional multiple-choice test with different probabilities of answering correctly at random and the “luckiness” for passing it. This paper concludes that, although random guessing is nonnegligible, it is unnecessary to remove traditional multiple-choice items from all testing activities, because it can be controlled through changing the passing score and the number of options or reducing its percentage in a test.

Validity and Reliability of the Standardized Orthopedic Assessment Tool (SOAT): A Variation of the Traditional Objective Structured Clinical Examination

Health care professions have replaced traditional multiple choice tests or essays with structured and practical, performance-based examinations with the hope of eliminating rater bias and measuring clinical competence. To establish the validity and reliability of the Standardized Orthopedic Assessment Tool (SOAT) as a measure of clinical competence of orthopaedic injury evaluation. Descriptive laboratory study. University. A total of 60 undergraduate students and 11 raters from 3 Canadian universities and 1 standardized patient. Students were required to complete a 30-minute musculoskeletal evaluation in 1 of 2 randomly assigned mock scenarios involving the knee (second-degree medial collateral ligament sprain) or the shoulder (third-degree supraspinatus muscle strain). We measured interreliability with an intraclass correlation coefficient (ICC) (2,k) and stability of the tool with standard error of measurement and confidence intervals. Agreement was measured using Bland-Altman plots. Concurrent validity was measured using a Pearson product moment correlation coefficient whereby the raters' global rating of a student was matched to the cumulative mean grade score. The ICCs were 0.75 and 0.82 for the shoulder and knee cases, respectively. Bland-Altman plots indicated no systematic bias between raters. In addition, Pearson product moment correlation analysis demonstrated a strong relationship between the overall cumulative mean grade score and the global rating score of the examinees' performances. This study demonstrated good interrater reliability of the SOAT with a standard error of measurement that indicated very modest stability, strong agreement between raters, and correlation indicative of concurrent validity.

Conceptually and Methodologically Vexing Issues in Teacher Knowledge Assessment

The purpose of this article is to describe the vexing issues that arise as researchers try to define and assess the knowledge teachers need to teach reading and writing effectively in the primary grades. Over the past several years, the authors of this article have developed and tested an assessment system of teacher knowledge under a grant titled The Primary Grade Reading & Writing Teacher Knowledge Project. The research team developed and tested 2 subscales: a traditional multiple-choice test that measures the inert or “in the head” knowledge teachers possess about reading and writing instruction, and an accompanying classroom observation instrument that measures the enacted or “classroom teaching” knowledge teachers use to teach reading and writing to young students. This article reports on 6 conceptually and methodologically vexing issues that arose as the authors developed and tested the assessment system. Issues revolved around the specific knowledge to measure, the evidence needed by various stakeholders as convincing evidence of that knowledge, potential concerns related to the use of measures of teachers’ knowledge, queries about the best way to measure that knowledge, special problems the use of classroom observations present for measuring teacher knowledge, and the predictive validity of such knowledge measures.

Development of two-tier diagnostic instrument and assess students understanding in chemistry

The objective of this study was to develop a two-tier test about the subject of “Separation of Matter” in 9th grade chemistry curriculum and to argue the efficiency of this test on students’ achievement. First, we interviewed 21 students to develop multiple-choice items. Secondly, 24 students were asked to select the most appropriate answer to each question and then they gave explanations for their choices to develop a two-tier test. At the third phase of this study, a two-tier test including 15 questions was developed and applied to141 students for collecting data. The results of the study showed that the two-tier test was effective on determining the students’ misconceptions and also it might be used as an alternative to the traditional multiple choice test for of assessment and evaluation of alternative students’ achievement. Key words: Two-tier test, chemistry, misconception, traditional multiple choice test, achievement.

Science Education and Student Diversity: Synthesis and Research Agenda – By Okhee Lee and Aurolyn Luykx

Science Education and Student Diversity: Synthesis and Research Agenda – By Okhee Lee and Aurolyn Luykx

Manipulating interactivity with thematically hyperlinked news texts: a media learning experiment

This article concerns the influence of news presentation on the web. The study had two primary goals. The first was to test the effects of interactivity on learning, and the second was to explore the role of motivation in learning from interactive media. Hypotheses were tested using an experimental design. Study participants were assigned one of four web news stories with structures that encouraged varying degrees of interactivity. The results of hypothesis testing were heavily dependent on which measure of learning was employed. A traditional multiple-choice test of recognition verified an effect of motivation but not of interactive behaviour. A comprehension measure of learning, supported by the cognitive constructivism theory of learning employed in this study, supported an effect of interactive behaviour but not of motivation. Explanations for the findings are proposed and implications for mass communicators discussed.

An examination of factors contributing to a reduction in subgroup differences on a constructed-response paper-and-pencil test of scholastic achievement.

The authors investigated subgroup differences on a multiple-choice and constructed-response test of scholastic achievement in a sample of 197 African American and 258 White test takers. Although both groups had lower mean scores on the constructed-response test, the results showed a 39% reduction in subgroup differences compared with the multiple-choice test. The results demonstrate that the lower subgroup differences were explained by more favorable test perceptions for African Americans on the constructed-response test. In addition, the two test formats displayed comparable levels of criterion-related validity. The results suggest that the constructed-response test format may be a viable alternative to the traditional multiple-choice test format in efforts to simultaneously use valid predictors of performance and minimize subgroup differences in high-stakes testing.

Comparison between computer‐aided testing and traditional multiple choice: an equivalence study

Computer-aided learning (CAL) is a viable alternative to traditional teaching methods. While CAL is used for teaching only, computer-aided testing (CAT) can be applied as a computer-based examination method. The aim of the present study was to compare the performance of undergraduate students in computer-interactive tests and traditional multiple choice tests and to evaluate the influence of question length and language skills on test results. Fifty-two native German-speaking students and five non-native speaking students were randomly assigned to two groups. They were required to answer 20 conventional multiple choice questions and 20 questions using the interactive MobiTed system. An analysis of variance and t-tests (alpha = 0.05) were used to compare scores. Non-native speakers scored significantly lower on the interactive test compared with the native speaking students (P = 0.048 for long questions and P = 0.003 for short questions). There were no significant group differences on the conventional multiple choice test (P = 0.052 for long questions and P = 0.089 for short questions). Based on the results of the native speaking students neither test method nor question length had an influence on overall performance. Interactive multiple choice tests may provide an alternative to written tests, especially when considering time expenditure. Language skills and question length should be taken into account when setting time limits for examination.

A Comparison of Multiple‐Choice and Open‐Ended Response Formats for the Assessment of Listening Proficiency in English

Abstract: Listening comprehension involving dynamic cognitive mental processes is invisible and can only be measured by the listeners' reconstruction of the message they have heard. The form and type of response might influence the test takers' listening performance. This study examines whether the subjects' listening performance varied when the types of responses were different. Traditional multiple‐choice (MC), multiple‐choice cloze (MCC), and open‐ended (OE) questions were employed to evaluate the subjects' listening comprehension. One hundred fifty‐nine technical college students whose L1 was Mandarin Chinese served as subjects and received all three response formats in three balanced treatments. The results indicated that different types of responses did have a significant effect on the subjects' performance. Subjects performed best on the MCC test, followed by the traditional MC test. They scored lowest on the OE test. A posttest survey revealed that 97% of the test takers preferred the MC formats because the given alternative answers facilitated comprehension of spoken stimuli and greater accuracy in guessing.

Gender and Racial/Ethnic Differences on Performance Assessments in Science

We examined whether the differences in mean scores among gender and racial/ethnic groups on science performance assessments are comparable to the differences that are typically found among these groups on traditional multiple-choice tests. To do this, several hands-on science performance assessments and other measures were administered to over 2,000 students in grades five, six, and nine as part of a field test of California’s statewide testing program. Girls tended to have higher overall mean scores than boys on the performance measures, but boys tended to score higher than girls on certain types of questions within a performance task. In contrast, differences in mean scores among racial/ethnic groups on one type of test (or question) were comparable to the differences among these groups on the other measures studied. Overall, the results suggest that the type of science test used is unlikely to have much effect on gender or racial/ethnic differences in scores.

Gender and Racial/Ethnic Differences on Performance Assessments in Science

We examined whether the differences in mean scores among gender and racial/ethnic groups on science performance assessments are comparable to the differences that are typically found among these groups on traditional multiple-choice tests. To do this, several hands-on science performance assessments and other measures were administered to over 2,000 students in grades five, six, and nine as part of a field test of California’s statewide testing program. Girls tended to have higher overall mean scores than boys on the performance measures, but boys tended to score higher than girls on certain types of questions within a performance task. In contrast, differences in mean scores among racial/ethnic groups on one type of test (or question) were comparable to the differences among these groups on the other measures studied. Overall, the results suggest that the type of science test used is unlikely to have much effect on gender or racial/ethnic differences in scores.