Exam Questions Research Articles

Using active student participation to cultivate critical thinking in examinations

ABSTRACT Exams can be a stressful time in a student’s life where they are required to review notes and memorize facts and figures. Many students leave these exams and forget the material relatively quickly. An alternative pedagogical exam style where students create their own exam questions is a method that engages students through active participation and fosters critical thinking. The current paper discusses a student-led exam approach in a criminal justice course where students work together to create short answer exam questions that focus on the key course content. The questions require students to synthesize what they have learned and apply this information to suggest solutions for issues presented throughout the class. The answers must provide justification for their solutions and move beyond regurgitation of their notes. Students in this course have praised the exam style, explaining how the method has helped them learn, apply, and retain the information. The exam style can be applied to a multitude of courses and will likely benefit students following their graduation.

Arabic Question Generation Using Transformers

After the increased reliance on online education, online assessment became an essential tool for educators to remotely monitor and evaluate students’ understanding in order to assist them properly. However, the laborious process of creating exam questions is a challenge for most teachers. Thus, automated Question Generation aims to assist teachers by generating questions from given data. Limited research has been conducted to tackle this issue in the Arabic Language due to the complexity of the language and the limited amount of available Arabic data. This paper explores different implementations of the transformer models, that demonstrated their superiority in natural language processing. Three approaches were introduced to tackle this problem with Arabic data using an Arabic-based transformer, an English-based transformer, and a multilingual-based transformer. Each of the fine-tuned models was trained using ARCD, XGLUE, DialectBench, and ArabicQA data sets and evaluated on automatic and manual metrics. Two of the proposed models achieve state-of-the-art results on the Arabic question generation task. The English transformer obtained a ROUGE score of 0.59 on XGLUE, while the Arabic transformer model achieves 0.49 on ARCD. Both of these models demonstrate excellent quality of questions through human-conducted evaluations by achieving low WER and high GC, U, and A scores.

Performance evaluation of ChatGPT 4.0 on cardiovascular questions from the medical knowledge self-assessment program

Abstract Background Medical trainees are increasingly using online chat-based artificial intelligence (AI) platforms as supplementary resources for board exam preparation and clinical decision support. Prior studies have evaluated the performance of AI chatbots like ChatGPT on various general standardized tests such as the United States Medical Licensing Examination (USMLE), but little is known about their performance on subspecialty-focused exam questions, particularly related to clinical management and treatment. Objective This study aims to evaluate the performance of ChatGPT version 4.0 on the cardiovascular questions from the Medical Knowledge Self-Assessment Program (MKSAP) 19, a widely used resource for board exam preparation in the United States. Methods We submitted all cardiovascular questions from MKSAP 19 to ChatGPT 4.0, covering a broad range of cardiology topics in a multiple-choice format. Performance was gauged against both the official MKSAP answer key and average trainee scores obtained from the MKSAP website. Out of 129 questions, 4 were invalidated due to post-publication data, and 18 were excluded due to reliance on visual aids, leaving 107 questions for analysis. Results ChatGPT 4.0 correctly answered 93 out of 107 questions, reflecting an 87% accuracy rate, compared to a 60% accuracy rate averaged among all human users for the same questions (p<0.0001). ChatGPT accuracy rates for each question category (e.g., heart failure, electrophysiology, etc.) are provided in the figure. On the 14 questions that ChatGPT answered incorrectly, human users averaged a 47% accuracy rate. All 14 incorrectly answered questions were related to clinical management and treatment (as opposed to diagnosis or epidemiology), and the majority (10/14) involved choosing treatment medications or choosing between medications versus interventions. Conclusion ChatGPT 4.0 surpassed the typical 70% accuracy threshold required for passing the internal medicine board certification but performed less well in answering questions about clinical management and treatment. This performance suggests that ChatGPT 4.0 could be an effective adjunct tool for cardiology education, but raises some concerns about its use in clinical decision support for trainees.

Performance Assessment of GPT 4.0 on the Japanese Medical Licensing Examination.

To evaluate the accuracy and parsing ability of GPT 4.0 for Japanese medical practitioner qualification examinations in a multidimensional way to investigate its response accuracy and comprehensiveness to medical knowledge. We evaluated the performance of the GPT 4.0 on Japanese Medical Licensing Examination (JMLE) questions (2021-2023). Questions are categorized by difficulty and type, with distinctions between general and clinical parts, as well as between single-choice (MCQ1) and multiple-choice (MCQ2) questions. Difficulty levels were determined on the basis of correct rates provided by the JMLE Preparatory School. The accuracy and quality of the GPT 4.0 responses were analyzed via an improved Global Qualily Scale (GQS) scores, considering both the chosen options and the accompanying analysis. Descriptive statistics and Pearson Chi-square tests were used to examine performance across exam years, question difficulty, type, and choice. GPT 4.0 ability was evaluated via the GQS, with comparisons made via the Mann-Whitney U or Kruskal-Wallis test. The correct response rate and parsing ability of the GPT4.0 to the JMLE questions reached the qualification level (80.4%). In terms of the accuracy of the GPT4.0 response to the JMLE, we found significant differences in accuracy across both difficulty levels and option types. According to the GQS scores for the GPT 4.0 responses to all the JMLE questions, the performance of the questionnaire varied according to year and choice type. GTP4.0 performs well in providing basic support in medical education and medical research, but it also needs to input a large amount of medical-related data to train its model and improve the accuracy of its medical knowledge output. Further integration of ChatGPT with the medical field could open new opportunities for medicine.

The association between family socio-demographic factors, parental mediation and adolescents’ digital literacy: a cross-sectional study

BackgroundDigital technology has a two-sided impact whether it is used appropriately or not. It is crucial to promote the benefits of digital technology and reduce its harms by bolstering adolescents’ digital literacy. The appropriateness of the strategies parents use to manage their children’s digital lives plays a key role. This study aims to investigate the separate effects of family socio-demographic factors and varied dimensions of parental mediation on adolescents’ digital literacy separately and simultaneously to explore the appropriate parental mediation strategies even in the absence of favorable family status.MethodsA cross-sectional survey was utilized with a sample of 1894 Chinese adolescents in junior and senior high schools (mean age = 14.01, 48.94% female), using a set of exam questions to test children’s objective digital literacy online. Family socio-demographic factors (parental marital status, economic status, and parental education levels) and varied dimensions of parental mediation were included as predictors of digital literacy. Hierarchical regression analysis was used to estimate the relation between the variables.ResultsHigher parental education levels were positively associated with adolescents’ parental mediation and digital literacy while better family economic status and parental divorce were negatively associated. Among the dimensions of parental mediation, active mediation, restrictive mediation and supervision were significantly associated with adolescents’ digital literacy, but not for co-use. Active mediation and supervision showed significant positive associations with adolescents’ digital literacy, whereas, co-use showed a significant negative association when all parental mediation dimensions were integrated into the model, The significant association between restrictive mediation and digital literacy disappeared. Parental mediation has diluted the impact of family socio-demographic factors on adolescents’ digital literacy.ConclusionsEffective interventions are needed to promote parental active mediation and supervision to help adolescents develop digital literacy and navigate the digital world, especially for adolescents who do not have highly educated parents or whose parents are divorced.

Evaluation and mitigation of cognitive biases in medical language models.

Increasing interest in applying large language models (LLMs) to medicine is due in part to their impressive performance on medical exam questions. However, these exams do not capture the complexity of real patient-doctor interactions because of factors like patient compliance, experience, and cognitive bias. We hypothesized that LLMs would produce less accurate responses when faced with clinically biased questions as compared to unbiased ones. To test this, we developed the BiasMedQA dataset, which consists of 1273 USMLE questions modified to replicate common clinically relevant cognitive biases. We assessed six LLMs on BiasMedQA and found that GPT-4 stood out for its resilience to bias, in contrast to Llama 2 70B-chat and PMC Llama 13B, which showed large drops in performance. Additionally, we introduced three bias mitigation strategies, which improved but did not fully restore accuracy. Our findings highlight the need to improve LLMs' robustness to cognitive biases, in order to achieve more reliable applications of LLMs in healthcare.

Evaluating the Performance of ChatGPT, Gemini, and Bing Compared with Resident Surgeons in the Otorhinolaryngology In-service Training Examination.

Large language models (LLMs) are used in various fields for their ability to produce human-like text. They are particularly useful in medical education, aiding clinical management skills and exam preparation for residents. To evaluate and compare the performance of ChatGPT (GPT-4), Gemini, and Bing with each other and with otorhinolaryngology residents in answering in-service training exam questions and provide insights into the usefulness of these models in medical education and healthcare. Eight otorhinolaryngology in-service training exams were used for comparison. 316 questions were prepared from the Resident Training Textbook of the Turkish Society of Otorhinolaryngology Head and Neck Surgery. These questions were presented to the three artificial intelligence models. The exam results were evaluated to determine the accuracy of both models and residents. GPT-4 achieved the highest accuracy among the LLMs at 54.75% (GPT-4 vs. Gemini p=0.002, GPT-4 vs. Bing p<0.001), followed by Gemini at 40.50% and Bing at 37.00% (Gemini vs. Bing p=0.327). However, senior residents outperformed all LLMs and other residents with an accuracy rate of 75.5% (p<0.001). The LLMs could only compete with junior residents. GPT- 4 and Gemini performed similarly to juniors, whose accuracy level was 46.90% (p=0.058 and p=0.120, respectively). However, juniors still outperformed Bing (p=0.019). The LLMs currently have limitations in achieving the same medical accuracy as senior and mid-level residents. However, they outperform in specific subspecialties, indicating the potential usefulness in certain medical fields.

Enhancing exam question quality in medical education through bootstrapping.

Medical schools are required to assess and evaluate their curricula and to develop exam questions with strong reliability and validity evidence, often based on data derived from statistically small samples of medical students. Achieving a large enough sample to reliably and validly evaluate courses, assessments, and exam questions would require extensive data collection over many years, which is inefficient, especially in the fast-changing educational environment of medical schools. This article demonstrates how advanced quantitative methods, such as bootstrapping, can provide reliable data by resampling a single dataset to create many simulated samples. This economic approach, among others, allows for the creation of confidence intervals and, consequently, the accurate evaluation of exam questions as well as broader course and curriculum assessments. Bootstrapping offers a robust alternative to traditional methods, improving the psychometric quality of exam questions, and contributing to fair and valid assessments in medical education.

Implementation of HOTS-based Instructions in Pesantren: A Case Study in Pondok Modern Tazakka Batang

The development of Higher Order Thinking Skills (HOTS) in students at Islamic boarding schools (pesantren) is crucial due to their numerous advantages, such as the capacity to tackle social, scientific, and practical problems with effectiveness. Whether or not Pondok Modern Tazakka successfully integrated the use of HOTS questions in exams is the goal of this study. Through the application of descriptive phenomenological research methods and data collection techniques, including the tabulation of 132 test sheets in 44 subjects tested on KMI Class 6 students and interviews with 62 teachers in 8 subjects, it was discovered that 8 subjects exhibited no HOTS questions at all (19%), 6 subjects exhibited a decreasing number of HOTS questions (14%), 19 subjects exhibited a sharp increase in HOTS questions (45%), and 9 subjects exhibited a constant number of HOTS questions. The study's findings also demonstrate that, as long as supervisors (senior teachers) take a more proactive approach to giving junior teachers comprehensive training and ensure that all exam questions in each subject are HOTS exam questions, it is possible to guarantee that all exam questions will contain HOTS.

An analysis of calculus application questions in grade 12 mathematics exams in South Africa

This study employed a quantitative survey methodology to analyze the 2021-2023 grade 12 calculus application exam questions in the context of South Africa. A total of 11 grade 12 teachers and 220 grade 12 students participated in the study. Analysis of the survey data and students’ test results points to the view that examiners sometimes misjudge the level of difficulty of the exam questions and include questions that are very difficult, which fail to discriminate between strong and weak candidates. It was noticed that calculus applications questions that most students could not correctly answer included multiple mathematical skills that were compacted in a short statement. This paper recommends that mathematics examiners should continuously reflect on their own work and find ways to improve their skills in setting future exam papers. The inclusion of item difficulty and item discrimination indices in post-exam analysis of results could be part of the examiners’ reflective practice. Feedback from the grade 12 mathematics teachers on the standard of each exam question is just as important to the examiners as the examiners’ reports are to the teachers.

The Analysis of the Characters of Wording for High School Classical Chinese

Classical Chinese carries the splendid culture of China. It is the important learning content for Chinese in high school. At the same time, the scoring rate is not good in the corresponding examination. Reading and comprehending classical Chinese in high school is the hardest part, especially for students who are not good at language learning. It is meaningful to analyze the characters of wording for classical Chinese. In this paper, NLP (Natural Language Processing) technology is applied to analyze the word usage for classical Chinese in high school. Firstly, ancient poems and literature in teaching materials in high school and classical Chinese exam questions of the past 15 years are collected. Secondly, raw data is preprocessed, cut words, and calculated word frequencies. Finally, top N words are generated and the experimental data is analyzed. According to the experimental results, the methods proposed in this paper can be used to analyze words for classical Chinese in high school. The Top N words are calculated and analyzed. It provides a good reference for high school students to learn classical Chinese.

7870 Exploring The Role Of ChatGPT In Pediatric Endocrinology Education; Exam Preparation And Question Generation

Abstract Disclosure: J. Tarkoff: None. A.G. Martinez Sanchez: None. Large language models (LLMs) hold substantial promise for improving physician knowledge and expertise. Their role in medical education and the generation of diverse diagnoses can be crucial, potentially leading to positive effects on clinical outcomes. The New England Journal of Medicine pediatric case challenges have examined ChatGPT [1]. Yet, its performance in specialized areas like Pediatric Endocrinology remains unexplored. This study assesses the effectiveness of ChatGPT 4 in responding to questions from the Pediatric Endocrine Self-Assessment Program (PESAP). It also examines if the AI is suitable for creating an educational quiz for residents and fellows. Methods: ChatGPT 4 underwent testing with questions from the 2021-2022 version of PESAP, utilizing the prompt: “Can you assist from the perspective of a pediatric endocrinologist with the following patient case”. Responses were evaluated for initial correctness, and performance was analyzed across various competency categories corresponding to the 7 “umbrella sections” of the tool (Adrenal, Bone, Carbohydrate and Lipid Metabolism/Obesity, Growth, Pituitary, Reproductive System, and Thyroid). Subsequently, we personalized the model by incorporating the questions and detailed responses from the PESAP. This customization resulted in a model that was then utilized to create a 10-question proof-of-concept quiz for four board-certified pediatric endocrinologists. The quiz included a scoring system designed to measure the extent and depth of ChatGPT knowledge. Results: ChatGPT 4 accurately answered 52% of PESAP questions, demonstrating varying performance across specific categories, ranging from 30% (Adrenal) to 78% (Reproductive System). In 16 questions, ChatGPT 4 did not provide an initial answer, requiring a specific request for a response. For questions related to thyroid cancer, explicit prompts were necessary to instruct responses based on the American Thyroid Association 2015 guidelines. In the endocrinologist quiz, the average score was 80%, ranging from 60% to 100%. Discussion: Currently, using ChatGPT 4 as the final diagnostic tool, especially in pediatric endocrinology, should be approached with caution, based on our assessment. However, when incorporating distinct Pediatric Endocrinology case studies, ChatGPT 4 successfully generated valuable educational questions, reinforcing fundamental concepts in the field. We anticipate that as LLMs advance and receive direct medical training, the out-of-the-box diagnostic accuracy will improve, leading to a transformative impact on medical education through this technology.

Performance of ChatGPT on Hand Surgery Board-Style Examination Questions

Background The performance of large-language models, such as ChatGPT, on medical and sub-specialty examinations has been preliminarily explored in fields such as radiology, obstetrics and gynecology, and orthopedic surgery. However, no literature assessing ChatGPT’s ability to answer hand surgery exam questions exists. This study’s purpose was to evaluate ChatGPT’s performance on hand surgery board-style examination questions. Methods All questions from the American Society for Surgery of the Hand (ASSH) Hand 100 Exam, Beginner, and Intermediate Assessment tools were entered into ChatGPT-3.5. Responses were regenerated two times to identify inconsistencies. Duplicate questions, questions with figures and/or videos, and questions that ChatGPT refused to provide a response to were excluded. ChatGPT’s correct response rate, answer modifications, and human accuracy were recorded. Results 117 questions from the 3 assessment tools were analyzed: 49 from the ASSH Hand 100, 32 from the Beginner, and 36 from the Intermediate Assessment tools. On ChatGPT’s initial attempt, 40.82% (20/49), 50.0% (16/32), 38.89% (14/36) of questions were correctly answered, respectively. Overall, ChatGPT correctly answered 50/117 (42.7%) of questions on the first try. ChatGPT excelled in topics (&gt;60% correct) of mass/tumor, nerve, wrist, and performed poorly (&lt;40% correct) on topics regarding anatomy/basic science/imaging, brachial plexus, congenital, elbow, tendon, and vascular disorders, trauma. On the Beginner and Intermediate Exams, humans correctly answered 56.64% and 62.73% of questions, respectively. Conclusions ChatGPT can correctly answer simpler hand surgery questions but performed poorly when compared to humans on higher-difficulty questions.

Experimental assessment of the performance of artificial intelligence in solving multiple-choice board exams in cardiology.

The aim of the present study was to evaluate the performance of various artificial intelligence (AI)-powered chatbots (commercially available in Switzerland up to June 2023) in solving a theoretical cardiology board exam and to compare their accuracy with that of human cardiology fellows. For the study, a set of 88 multiple-choice cardiology exam questions was used. The participating cardiology fellows and selected chatbots were presented with these questions. The evaluation metrics included Top-1 and Top-2 accuracy, assessing the ability of chatbots and fellows to select the correct answer. Among the cardiology fellows, all 36 participants successfully passed the exam with a median accuracy of 98% (IQR 91-99%, range from 78% to 100%). However, the performance of the chatbots varied. Only one chatbot, Jasper quality, achieved the minimum pass rate of 73% correct answers. Most chatbots demonstrated a median Top-1 accuracy of 47% (IQR 44-53%, range from 42% to 73%), while Top-2 accuracy provided a modest improvement, resulting in a median accuracy of 67% (IQR 65-72%, range from 61% to 82%). Even with this advantage, only two chatbots, Jasper quality and ChatGPT plus 4.0, would have passed the exam. Similar results were observed when picture-based questions were excluded from the dataset. Overall, the study suggests that most current language-based chatbots have limitations in accurately solving theoretical medical board exams. In general, currently widely available chatbots fell short of achieving a passing score in a theoretical cardiology board exam. Nevertheless, a few showed promising results. Further improvements in artificial intelligence language models may lead to better performance in medical knowledge applications in the future.

A Study on Calculus Examination in Japanese Top University Entrance Exams and Educational Enlightenment for Gifted Students in Mathematics

Calculus is the cornerstone of many branches of modern mathematics and a crucial component of mathematics education. This article conducts in-depth qualitative and quantitative research and analysis on the calculus questions in the entrance exams of three top universities in Japan (University of Tokyo, Kyoto University, and Tokyo Institute of Technology). It is found that the calculus questions in the entrance exams of the three universities cover a wide range of knowledge and are highly comprehensive, with comparable difficulty in testing calculus knowledge. At the same time, the examination of calculus emphasizes the characteristics of students' comprehensive mathematical literacy.

Generative artificial intelligence vs. law students: an empirical study on criminal law exam performance

ABSTRACT Claims that GPT-4 can outperform more than 90% of human test-takers in the US Uniform Bar Examination have sparked heated debates about the impact of Generative AI (GenAI) on legal education, academic integrity, and the future of legal practice. Yet GenAI’s capabilities in broader legal examination contexts – including in jurisdictions outside the US – are unclear. This study addresses this gap by evaluating GenAI’s performance against students who took the ‘Criminal Law’ final exam at an Australian law school in Spring 2023. Various AI models and prompt engineering techniques were used to generate 10 distinct answers to the exam question. Five criminal law tutors, unaware of AI involvement, graded a mix of AI-generated and student responses. Then, the tutors were briefed on the AI-generated papers they marked and engaged in reflective semi-structured interviews. The study found that GenAI performed below the student average in questions that required detailed legal and critical analysis. However, all GenAI papers performed better than students in open-ended questions and essay writing tasks. These results provide a benchmark for the capabilities and limitations of GenAI in higher education and provide insights into the potential implications of its application to legal assessments and education, curriculum development, and the future workforce.

Capabilities of GPT-4 in ophthalmology: an analysis of model entropy and progress towards human-level medical question answering

BackgroundEvidence on the performance of Generative Pre-trained Transformer 4 (GPT-4), a large language model (LLM), in the ophthalmology question-answering domain is needed.MethodsWe tested GPT-4 on two 260-question multiple choice question...

A Strategic Approach to Succeed on Clinical Case-Based Multiple-Choice Exams

Despite the importance of case-based multiple-choice question (CBMCQ) exams, medical educators rarely discuss strategies to systematically approach these questions, and literature on the topic is limited. Through trial-and-error, many students discover or develop more refined and efficient approaches to answering CBMCQs that help them maximize the application of their knowledge base. In this article we present a structured approach to answering CBMCQs, including meta-cognitive strategies to understand the reasons for errors. We provide strategies for success on CBMCQ-based exams and approaches to challenging question types. We also present tips for helping neurodiverse students and an approach for medical educators to teach these skills to their learners.

Closing the Gap? The Ability of Adaptive Learning Courseware to Close Outcome Gaps in Principles of Microeconomics

Research shows that students who identify as low-income, first-generation, and/or racially diverse disproportionately underperform in college and earn fewer degrees than other students. This study explores the integration of adaptive learning courseware assignments as a tool to help close these outcome gaps and to ensure more equitable learning across diverse student groups. Adaptive learning courseware is an educational technology that requires students to master the same learning objectives but, for each student, the courseware determines the order and timing of content based on how that student interacts with the courseware, thus enabling an individualized learning path for each student. Adaptive learning assignments were implemented in five sections of a highly-enrolled Principles of Microeconomics course at a medium-sized state university in the United States. This study draws from student data (n=581), which includes adaptive learning assignment completion data, detailed exam and final grade data, and institutional demographic data. Descriptive statistics and regression analyses are used to explore if the completion of adaptive learning assignments disproportionately benefited low-income, first-generation, or racially diverse students, thus helping close the gap between students from different backgrounds. Findings include significant evidence that adaptive learning assignment completion was correlated with more exam questions answered correctly by all students, with this correlation being disproportionately stronger for students who identify as being from a minority background and for foundational (easy) exam questions.

Two-Phase Individual Assessments: A Second-Chance Assessment Strategy With Individualized Feedback to Promote Assessment for Learning.

High-stakes multiple-choice question (MCQ) exams in medical education typically focus on assessment of learning at a single point without providing feedback for improvement. Educators can achieve a more balanced approach to MCQ exams by combining efficient assessment of learning with the feedback and improvement opportunities of assessment for learning. As part of a curriculum renewal at Baylor College of Medicine's MD program, the Two-Phase Individual Assessment (TPIA) model was launched within a 4-week preclinical Foundations of Medicine course in August 2023. The TPIA model featured weekly assessments, each consisting of 2 MCQ exams given on the same day with a 4-hour study period in between. Exams were paired, consisting of an equal number of items that addressed the same learning objectives. After the initial exam, students received an individualized feedback report indicating correctly and incorrectly answered objectives. Students applied individualized feedback reports to self-identify and remediate deficiencies in preparation for the second-chance exam. Only the highest score counted toward the final grade. Among 230 medical student participants, significant performance improvements between morning and afternoon exams were observed across the first 3 weekly TPIAs, with mean score increases of 4.93, 5.06, and 10.86. Mean change in performance in week 4 was not significant. Student end-of-course survey responses indicated a strong preference for the TPIA format. Responses highlighted the value of offering individualized feedback, providing unstructured time to address knowledge gaps, and ensuring a corresponding opportunity for improvement. Future research will extend TPIA's implementation in more clinical-oriented courses to validate TPIA's effectiveness and explore its effect on long-term knowledge retention through more cumulative examinations. Integrating TPIA-based approaches may advance educational systems toward developing a culture of feedback and embracing second chances to support learning and professional growth, thereby preparing medical professionals for the demands of 21st-century medicine.