Exam Performance Research Articles

Generative artificial intelligence vs. law students: an empirical study on criminal law exam performance

ABSTRACT Claims that GPT-4 can outperform more than 90% of human test-takers in the US Uniform Bar Examination have sparked heated debates about the impact of Generative AI (GenAI) on legal education, academic integrity, and the future of legal practice. Yet GenAI’s capabilities in broader legal examination contexts – including in jurisdictions outside the US – are unclear. This study addresses this gap by evaluating GenAI’s performance against students who took the ‘Criminal Law’ final exam at an Australian law school in Spring 2023. Various AI models and prompt engineering techniques were used to generate 10 distinct answers to the exam question. Five criminal law tutors, unaware of AI involvement, graded a mix of AI-generated and student responses. Then, the tutors were briefed on the AI-generated papers they marked and engaged in reflective semi-structured interviews. The study found that GenAI performed below the student average in questions that required detailed legal and critical analysis. However, all GenAI papers performed better than students in open-ended questions and essay writing tasks. These results provide a benchmark for the capabilities and limitations of GenAI in higher education and provide insights into the potential implications of its application to legal assessments and education, curriculum development, and the future workforce.

Association Between Resident In-Service Exam Scores by Postgraduate Year and Subsequent Board Qualifying Exam.

Prior work notes the AUA In-Service Exam (ISE) percentile ranking of chief residents correlates with the American Board of Urology Qualifying Exam (QE) performance. We present a 5-year analysis of resident performance on the ISE and subsequent QE to determine if earlier time points in training may identify those needing additional educational support. Participant ISE scores over a 5-year period from 2014 to 2018 and subsequent QE scores in 2019 were recorded. Pearson's correlation coefficient measured the association between percentage questions correct for each ISE year and QE. Youden Index calculated the optimal cut point for yearly ISE percentage correct that would predict scoring greater than the lowest quartile and decile on the QE. Median percent questions correct on ISE increased over postgraduate year (PGY) 1 (47%), PGY2 (56.5%), and PGY3 (70%) but remained stable thereafter (PGY4-PGY5) at approximately 70%. Median QE percent correct in 2019 was 66% (SD 7.6%). Correlation of percent questions correct between ISE and QE improved from 0.31 to 0.53 over training duration. The lowest decile and quartile percent correct scores on the QE were 56% and 60%, respectively. Percent correct ISE score predicting performance above the lowest decile 2019 QE score increased from 38% in PGY1 to 57% in PGY2 and leveled off after PGY3 (∼70%). Similar observations were noted with lowest quartile QE score. Scoring approximately 70% of questions correct on the ISE during PGY3 and later years was associated with a low risk of failing the QE. Such information provides benchmarks for residency programs to offer targeted educational content for at-risk candidates.

First-Generation Low-Income Individuals in Medicine: A Scoping Review.

This comprehensive scoping review of the medical literature on first-generation low-income (FGLI) individuals in medicine aimed to synthesize the highest levels of evidence to inform medical education stakeholders. Database searches were conducted in Academic Search Premier, Education Research Premier, ERIC, Ovid MEDLINE, Ovid Embase, Professional Development Collection, PubMed, Scopus, Google Scholar, and Web of Science Core Collection from database inception through March 15, 2023. English-language articles on first-generation or low-income individuals in medicine from U.S. medical schools were included. Articles were evaluated for level of evidence and themes chosen. Database searches resulted in 27,075 citations, 247 of which qualified for data extraction. The articles were classified by evidence level: level I (n = 2), level II (n = 17), level III (n = 90), level IV (n = 78), and level V (n = 60). Publications reported on 9 major outcomes: exam performance (n = 108), medical school performance (n = 63), residency and fellowship performance (n = 7), honor society status (n = 12), leave of absence (n = 9), withdrawal, dismissal, and attrition (n = 20), medical education graduation (n = 37), career choice (n = 109), and intent to practice in disadvantaged and rural communities (n = 60). Compared with their peers, FGLI individuals had lower medical school and standardized exam scores, enrollment in national medical honor societies and multiple degree programs, and graduate medical education performance and higher rates of leaves of absence, incompletion of medical education, pursuing primary care and family medicine specialties, and intent to practice in underserved communities. Despite an increase in the number of FGLI individuals in medicine, there remains significant opportunity to improve their inclusion and support. Multi-institutional, prospective, risk-adjusted, observational studies are required to determine how to best support FGLI individuals through all medical career stages.

Differences in Algebra I End-of-Course exam performance by the at-risk status of Hispanic high school girls: A Texas multiyear analysis

Addressed in this investigation was a comparison of Algebra I End-of-Course exam performance between Hispanic girls who were at-risk and Hispanic girls who were not at-risk in the 2016–2017, 2017–2018, and 2018–2019 school years. The dependent variables investigated in this study were Hispanic girls performance in the Approaches Grade Level standard, the Meets Grade Level standard, the Masters Grade Level standard, and raw score on the Algebra I End-of-Course exam. Inferential statistical procedures yielded statistically significant differences in all three school years for Hispanic girls who were at-risk. In all three school years, Hispanic girls who were at-risk failed to meet each of the three grade level standards at statistically significantly lower rates than Hispanic girls who were not at-risk. In raw score comparison, Hispanic girls answered more than 11 items, on average, less correctly than Hispanic girls who were not at-risk. Implications and recommendations for future research were made.

Impact of the Timing of Content Introduction on Student Exam Performance in General Chemistry

Impact of the Timing of Content Introduction on Student Exam Performance in General Chemistry

Increased Student Employment Is Associated with Inferior Biology Exam and Course Performance

Increased Student Employment Is Associated with Inferior Biology Exam and Course Performance

Exploring teachers' experiences in teaching learners with Dissociative identity disorder at rural schools in the Amajuba district KwaZulu-Natal

This study aims to investigate how Dissociative Identity Disorder (DID) affects learners’ academic achievements, engagements, and overall school experience. The objectives were to identify the prevalence of DID among learners in rural KwaZulu-Natal schools, examine the impact of DID on their academic performance, and explore interventions that could support affected learners. The study gained qualitative insight from interviews and focus groups involving learners, teachers, and psychologists. This comprehensive method facilitated a nuanced understanding of DID's effects on learning and academic outcomes. Results indicated a significant correlation between learners with DID and lower academic performance when compared to their peers without DID. Qualitative findings revealed that dissociative episodes, memory lapse, and the psychological stress associated with Dissociative Identity DID adversely affected learners’ participation in class, homework completion, and exam performance. The study further found a general lack of awareness and resources in rural schools to effectively support learners with DID, compounding the challenges they face. The study concludes with several key findings: the need for targeted training for educators in identifying and supporting learners with DID, the importance of integrating psychological support within the school system, and the potential benefits of personalized learning plans for affected learners. The focus should not merely be on academic performance but on creating a holistic experience where every learner, regardless of their challenges, feels included and valued. The study calls for further research and strategies to identify and address dissociative symptoms in learners.

Optional Exam Retakes Reduce Anxiety but may Exacerbate Score Disparities Between Students with Different Social Identities.

Use of high-stakes exams in a course has been associated with gender, racial, and socioeconomic inequities. We investigated whether offering students the opportunity to retake an exam makes high-stakes exams more equitable. Following the control value theory of achievement emotions, we hypothesized that exam retakes would increase students' perceived control over their performance and decrease the value of a single exam attempt, thereby maximizing exam performance. We collected data on exam scores and experiences with retakes from three large introductory biology courses and assessed the effect of optional exam retakes on gender, racial/ethnic, and socioeconomic disparities in exam scores. We found that Black/African American students and those who worked more than 20 h a week were less likely to retake exams. While exam retakes significantly improved student scores, they slightly increased racial/ethnic and socioeconomic disparities in scores partly because of these differences in participation rates. Most students reported that retake opportunities reduced their anxiety on the initial exam attempt. Together our results suggest that optional exam retakes could be a useful tool to improve student performance and reduce anxiety associated with high-stakes exams. However, barriers to participation must be examined and reduced for retakes to reduce disparities in scores.

The correlation between service quality, students’ satisfaction, and board exam performance

The study examines how service quality and students’ satisfaction affect the performance of graduates in the Licensure Examination conducted by the Philippine Regulation Commission, scrutinizes the quality of service, and assesses the level of satisfaction of the graduates with the rendered services. The researchers made use of a descriptive-correlational, adopted-modified, and validated survey questionnaire as tools of the study. Mann-Whitney, Kruskal-Wallis, Pearson-r correlation, and mean analysis were used as the statistical procedures of the study. Results showed that the perceptions of the graduates in the quality of service of the University that falls in the dimension of tangibility and responsiveness have a positive correlation with their performance in LET. Likewise, the satisfaction of the graduates with the quality of service in the dimension of reliability, responsiveness, and empathy has a positive correlation to the latter. Further study in another discipline is recommended as the study is limited only to science major graduates. Keywords: Board exam performance; let performance; service quality; students’ satisfaction

Is everything everywhere? A hands-on activity to engage undergraduates with key concepts in quantitative microbial biogeography.

The ubiquity and ease with which microbial cells disperse over space is a key concept in microbiology, especially in microbial ecology. The phenomenon prompted Baas Becking's famous "everything is everywhere" statement that now acts as the null hypothesis in studies that test the dispersal limitation of microbial taxa. Despite covering the content in lectures, exam performance indicated that the concepts of dispersal and biogeography challenged undergraduate students in an upper-level Microbial Ecology course. Therefore, we iteratively designed a hands-on classroom activity to supplement the lecture content and reinforce fundamental microbial dispersal and biogeography concepts while also building quantitative reasoning and teamwork skills. In a class period soon after the lecture, the students formed three-to-five-person teams to engage in the activity, which included a hands-on dispersal simulation and worksheet to guide discussion. The simulation involved stepwise neutral immigration or emigration and then environmental selection on a random community of microbial taxa represented by craft poms. The students recorded the results at each step as microbial community data. A field guide was provided to identify the taxonomy based on the pom phenotype and a reference to each taxon's preferred environmental niches. The worksheet guided a reflection of student observations during the simulation. It also sharpened quantitative thinking by prompting the students to summarize and visualize their and other teams' microbial community data and then to compare the observed community distributions to the idealized expectation given only selection without dispersal. We found that the activity improved student performance on exam questions and general student satisfaction and comfort with the biogeography concepts. Activity instructions and a list of needed materials are included for instructors to reproduce for their classrooms.

Using systems modeling to facilitate undergraduate physiology student learning and retention of difficult concepts.

Physiology concepts, such as acid-base balance, may be difficult for students to understand. Systems modeling, a cognitive tool, allows students to visualize their mental model of the problem space to enhance learning and retention. Methods: We performed a within-subjects three-period randomized control comparison of systems modeling versus written discussion activities in an undergraduate asynchronous online Anatomy and Physiology II course. Participants (n=108) were randomized to groups with differing treatment orders across three course units: endocrine, immune, and acid-base balance. Participants demonstrated content understanding either through constructing systems modeling diagrams (M) or written discussion posts (W) in a MWM, MMW, or WMM sequence. Results: For each of three units, student performance was assessed on six standardized multiple-choice questions embedded within a 45-question exam. The same six questions per unit, eighteen questions total, was again assessed on the 75-question final exam. The groups demonstrated no significant difference in performance in the endocrine unit exam (mean difference, MD=-0.036). However, the modeling group outperformed the writing group in the immune unit exam (MD=0.209) and widened the gap in the acid-base balance unit exam (MD=0.243). On the final exam, performance was again higher for the modeling group on acid-base balance content as mean difference increased to 0.306 despite the final exam content for acid-base balance being significantly more difficult compared to other units (modeling: F(2)=29.882, p<.001; writing: F(2)=25.450, p<.001). Conclusions: These results provide initial evidence that participation in systems modeling activities enhance student learning of difficult physiology content as evidenced by improved multiple-choice question performance.

When can we identify the students at risk of failure in the national medical licensure examination in Japan using the predictive pass rate?

ContextFailure of students to pass the National Medical Licensure Examination (NMLE) is a major problem for universities and the health system in Japan. To assist students at risk for NMLE failure as early as possible after admission, this study investigated the time points (from the time of admission to graduation) at which predictive pass rate (PPR) can be used to identify students at risk of failing the NMLE.MethodsSeven consecutive cohorts of medical students between 2012 and 2018 (n = 637) at the Gifu University Graduate School of Medicine were investigated. Using 7 variables before admission to medical school and 10 variables after admission, a prediction model to obtain the PPR for the NMLE was developed using logistic regression analysis at five time points, i.e., at admission and the end of the 1st, 2nd, 4th, and 6th grades. All students were divided into high (PPR < 95%) and low (PPR ≥ 95%) risk groups for failing the NMLE at the five time points, respectively, and the movement between the groups during 6 years in school was simulated.ResultsMedical students who passed the NMLE had statistically significant factors at each of the 5 time points, and the number of significant variables increased as their grade in school advanced. In addition, two factors extracted at admission were also selected as significant variables at all other time points. Especially, age at entry had a consistent and significant effect during medical school.ConclusionsRisk analysis based on multiple variables, such as PPR, can inform more effective intervention compared to a single variable, such as performance in the mock exam. A longer prospective study is required to confirm the validity of PPR.

Time spent on active learning activities does not necessarily correlate with student exam performance: a controlled case study.

Active learning, including student thinking and discussion in class, has been shown to increase student learning gains. However, it is less clear how instructor-level variation in the implementation and timing of active learning activities affects student gains. Our study aims to investigate the extent to which the time spent on individual episodes of active learning activities influences student performance. We hypothesized that instructors who let students spend more time on peer discussion and individual thinking on practice problems associated with particular learning objectives would have better student exam scores on exam questions addressing those objectives. To test this hypothesis, we obtained a large data set of classroom recordings and student exam scores from an introductory biology course at a large 4-year university, where three instructors shared identical teaching materials and exams for different course offerings. Contrary to our hypothesis, although the three instructors spent significantly different amounts of time on episodes of thinking and peer discussion, there was no correlation between the total time spent on active learning activities and student performance on exam questions. Linear mixed-effects modeling of the effect of the length of episodes of student thinking and discussion on exam score found that in the context of shared instructional materials, the amount of course time spent on active learning activities did not reliably predict student performance on associated exam questions. This result held true even when only considering learning objectives with high variations in performance between offerings, difficult exam questions, or exam questions requiring higher-order thinking skills. Although our study was only conducted in one course, our results imply that time spent per individual episode of student thinking or peer discussion may not be the primary factor explaining the positive effects of active learning and that it may be worthwhile to explore other factors.

Anki flashcards: Spaced repetition learning in the undergraduate medical pharmacology curriculum.

Teaching clinical pharmacology is often a challenge for medical schools. The benefits and popularity of active recall and spaced repetition through Anki flashcards are well-established and can offer a solution for teaching complex topics, but educators are often unfamiliar with this resource. We implemented 501 faculty-generated pharmacology flashcards in five modules across the medical preclinical curriculum, available to 104 first-year students. At the end of each module, students were surveyed on the usefulness of this novel resource. The data from the cohort who had access to flashcards was compared with the previous cohort, without access, to analyse whether student use of Anki flashcards changed students' perceptions of the pharmacology curriculum and whether there were changes in pharmacology exam performance. Seventy-five percent of the respondents rated the Anki pharmacology flashcards as 'very useful' or 'somewhat useful'. Eight hundred and seventy-five responses were analysed with a natural language processing algorithm, showing that fewer students mentioned pharmacology as a difficult topic in the cardiovascular and renal modules, compared with the cohort who did not use Anki flashcards. There was not a statistically significant difference in test scores between the cohorts. Anki flashcards were well-received by medical students, which might have impacted their perception of the curriculum, as evidenced by the decrease in mentions of pharmacology being a difficult topic, maintaining consistency in academic performance. Educators should consider providing flashcards to offer spaced repetition opportunities in the curriculum; an additional benefit could be increasing information equality in medical schools.

Influence of Model Evolution and System Roles on ChatGPT's Performance in Chinese Medical Licensing Exams: Comparative Study.

With the increasing application of large language models like ChatGPT in various industries, its potential in the medical domain, especially in standardized examinations, has become a focal point of research. The aim of this study is to assess the clinical performance of ChatGPT, focusing on its accuracy and reliability in the Chinese National Medical Licensing Examination (CNMLE). The CNMLE 2022 question set, consisting of 500 single-answer multiple choices questions, were reclassified into 15 medical subspecialties. Each question was tested 8 to 12 times in Chinese on the OpenAI platform from April 24 to May 15, 2023. Three key factors were considered: the version of GPT-3.5 and 4.0, the prompt's designation of system roles tailored to medical subspecialties, and repetition for coherence. A passing accuracy threshold was established as 60%. The χ2 tests and κ values were employed to evaluate the model's accuracy and consistency. GPT-4.0 achieved a passing accuracy of 72.7%, which was significantly higher than that of GPT-3.5 (54%; P<.001). The variability rate of repeated responses from GPT-4.0 was lower than that of GPT-3.5 (9% vs 19.5%; P<.001). However, both models showed relatively good response coherence, with κ values of 0.778 and 0.610, respectively. System roles numerically increased accuracy for both GPT-4.0 (0.3%-3.7%) and GPT-3.5 (1.3%-4.5%), and reduced variability by 1.7% and 1.8%, respectively (P>.05). In subgroup analysis, ChatGPT achieved comparable accuracy among different question types (P>.05). GPT-4.0 surpassed the accuracy threshold in 14 of 15 subspecialties, while GPT-3.5 did so in 7 of 15 on the first response. GPT-4.0 passed the CNMLE and outperformed GPT-3.5 in key areas such as accuracy, consistency, and medical subspecialty expertise. Adding a system role insignificantly enhanced the model's reliability and answer coherence. GPT-4.0 showed promising potential in medical education and clinical practice, meriting further study.

Creating educational podcasts as an expository methodology for active learning and its relationship with satisfaction: A comparative study with live oral presentations

ABSTRACT The educational podcast is a didactic resource requiring specific planning to create a digital audio file with educational content. These podcasts promote meaningful learning and stimulate cognitive skills. This study aimed to analyze the effectiveness of educational podcasts in increasing students’ satisfaction and performance in content acquisition compared to live oral presentations. An experimental methodology was used, involving 148 students from the University of Zaragoza (2022-2023). Students were divided into two groups: one used podcasts, and the others gave live oral presentations. They assessed both types of presentations using an evaluation rubric and collaborated on content development. Data on academic performance, satisfaction, ratings, gender, age, and admission cut-off marks were collected for statistical analysis. Both groups reported high satisfaction, with in-class presentations deemed slightly more appropriate. Exam performance was similar across groups. Collaborative, student-centered learning increased satisfaction, but intra-group conflicts prevented clear distinctions between the methodologies.

The Effects of Implementing Gamification in the Hepatology Curriculum for Medical Students.

Background Gamification, the incorporation of game theory into the curriculum, has been correlated with improved knowledge retention compared to standard didactics. Objective To determine the impact of gamified hepatology modules on medical student knowledge retention and exam performance. Methods We created three web-based, gamified hepatology modules with 15-question pre- and post-tests. Differences in each module's pre- and post-test scores were compared using paired t-tests. Medical school exam scores (total score and hepatology-specific score) in module users versus non-users were compared using independent two-sample t-tests. Results Module completion yielded significant increases in pre- to post-test scores for the jaundice (p=0.002) and anatomy modules (p<0.001). Module users scored 1.2 points higher on the total exam score (p=0.4) and 2 points higher on the hepatology exam score (p=0.31). Post-module survey results revealed higher knowledge retention in hepatology topics, expanded interest in hepatology, and an increased inclination to use web-based learning platforms for future learning experiences. Conclusion Module use led to improved post-test scores in the modules. Module users also had higher hepatology exam and total exam scores, yet, this difference was not statistically significant. Overall, this study suggests that gamification may be beneficial in aiding hepatology knowledge recall.

Evaluating the competency of ChatGPT in MRCP Part 1 and a systematic literature review of its capabilities in postgraduate medical assessments.

As a large language model (LLM) trained on a large data set, ChatGPT can perform a wide array of tasks without additional training. We evaluated the performance of ChatGPT on postgraduate UK medical examinations through a systematic literature review of ChatGPT's performance in UK postgraduate medical assessments and its performance on Member of Royal College of Physicians (MRCP) Part 1 examination. Medline, Embase and Cochrane databases were searched. Articles discussing the performance of ChatGPT in UK postgraduate medical examinations were included in the systematic review. Information was extracted on exam performance including percentage scores and pass/fail rates. MRCP UK Part 1 sample paper questions were inserted into ChatGPT-3.5 and -4 four times each and the scores marked against the correct answers provided. 12 studies were ultimately included in the systematic literature review. ChatGPT-3.5 scored 66.4% and ChatGPT-4 scored 84.8% on MRCP Part 1 sample paper, which is 4.4% and 22.8% above the historical pass mark respectively. Both ChatGPT-3.5 and -4 performance was significantly above the historical pass mark for MRCP Part 1, indicating they would likely pass this examination. ChatGPT-3.5 failed eight out of nine postgraduate exams it performed with an average percentage of 5.0% below the pass mark. ChatGPT-4 passed nine out of eleven postgraduate exams it performed with an average percentage of 13.56% above the pass mark. ChatGPT-4 performance was significantly better than ChatGPT-3.5 in all examinations that both models were tested on. ChatGPT-4 performed at above passing level for the majority of UK postgraduate medical examinations it was tested on. ChatGPT is prone to hallucinations, fabrications and reduced explanation accuracy which could limit its potential as a learning tool. The potential for these errors is an inherent part of LLMs and may always be a limitation for medical applications of ChatGPT.

Correlating Advanced Pharmacy Practice Experience Exam Scores and Pharmacist Licensure Status in a Single Program

ObjectivePharmacist licensure exam first-time pass rates have declined for several years. Predictors of licensure exam performance, including the Pharmacy College Admission Test, are no longer required, necessitating alternative strategies. Our aim is to assess the relationship between numerical scores on advanced pharmacy practice experience (APPE) exams and pharmacist licensure exams first-time pass rates. MethodsWe conducted a single-center, retrospective, observational analysis. APPE examination results between May 2020 and November 2023 were analyzed for association with licensure information from the Texas State Board of Pharmacy. Exam scores were collected for student cohorts graduating from 2021 to 2022 as study cohort data. Correlation between exam scores and probability of successful licensure was assessed using logistic regression. A classification and regression tree analysis identified the most significant threshold. Predictive ability of the best-fit model was prospectively validated using the 2023 graduate cohort. ResultsThe overall licensure success rate was 80.6% for the study cohort (2021–2022, n = 206). Exam scores were well correlated to the likelihood of licensing success. The most significant threshold was 77.8%. The success rate was 82.5% vs 16.7% for those scoring above and below the threshold, respectively. The observed and predicted licensure success rates were 91.0% and 88.8%, respectively, in the validation cohort (2023, n = 89). The positive and negative prediction values were 94.9% and 40.0%, respectively. ConclusionThe performances on APPE exams were reasonable in predicting the first-time licensure success rate for a graduating class. Our assessment appears promising as a risk-stratification tool for students in gaining successful pharmacist licensure.

Implementing active learning in a teaching–learning sequence on rolling motion for mechanical engineers

In this work, we describe a teaching–learning sequence (TLS) on rolling motion, designed and implemented within a Physics 1 course for Mechanical Engineers. The TLS was grounded in Physics Education Research findings and employed active learning strategies, particularly simulation-based exercises that included both conceptual questions and quantitative tasks. We detail and reflect on the development process of the TLS, focussing in particular on the alignment between the identified learning goals and the proposed instructional activities. We present data on students’ attainment of these learning goals, utilizing a post-instruction conceptual survey and an analysis of students’ performance in the final exam. Our research aims to contribute to the implementation of research-based, active learning sequences in large-enrollment contexts, specifically in those courses where both conceptual understanding and formal problem solving are valued.