Exam Questions Research Articles

Large Language Models Take on Cardiothoracic Surgery: A Comparative Analysis of the Performance of Four Models on American Board of Thoracic Surgery Exam Questions in 2023.

Objectives Large language models (LLMs), for example, ChatGPT, have performed exceptionally well in various fields. Of note, their success in answering postgraduate medical examination questions has been previously reported, indicating their possible utility in surgical education and training. This study evaluated the performance of four different LLMs on the American Board of Thoracic Surgery's(ABTS) Self-Education and Self-Assessment in Thoracic Surgery (SESATS) XIII question bank to investigate the potential applications of these LLMs in the education and training of future surgeons. Methods The dataset in this study comprised 400 best-of-four questions from the SESATS XIII exam. This included 220 adult cardiac surgery questions, 140 general thoracic surgery questions, 20 congenital cardiac surgery questions, and 20 cardiothoracic critical care questions. The GPT-3.5 (OpenAI, San Francisco, CA) and GPT-4 (OpenAI) models were evaluated, as well as Med-PaLM 2 (Google Inc., Mountain View, CA) and Claude 2 (Anthropic Inc., San Francisco, CA), and their respective performances were compared. The subspecialties included were adult cardiac, general thoracic, congenital cardiac, and critical care. Questions requiring visual information, such as clinical images or radiology, were excluded. Results GPT-4 demonstrated a significant improvement over GPT-3.5 overall (87.0% vs. 51.8% of questions answered correctly, p < 0.0001). GPT-4 also exhibited consistently improved performance across all subspecialties, with accuracy rates ranging from 70.0% to 90.0%, compared to 35.0% to 60.0% for GPT-3.5. When using the GPT-4 model, ChatGPT performed significantly better on the adult cardiac and general thoracic subspecialties (p < 0.0001). Conclusions Large language models, such as ChatGPT with the GPT-4 model, demonstrate impressive skill in understanding complex cardiothoracic surgical clinical information, achieving an overall accuracy rate of nearly 90.0% on the SESATS question bank. Our study shows significant improvement between successive GPT iterations. As LLM technology continues to evolve, its potential use in surgical education, training, and continuous medical education is anticipated to enhance patient outcomes and safety in the future.

Pelatihan Bahasa Inggris Reading Comprehension “It’s All About Reading Skill” Di SMAN Karya Sakti Muara Kelingi Musi Rawas

SMAN Karya Sakti is one of the educational units at high school level in Karya Sakti, Kec. Muara Kelingi, Kab. Musi Rawas, South Sumatra. In carrying out its activities, SMAN Karya Sakti is under the auspices of the Ministry of Education and Culture with B accreditation. SMAN Karya Sakti is located at Jalan Tapak Libok Blok A, Karya Sakti, Kec. Muara Kelingi, Kab. Musi Rawas, South Sumatra. The aim of this Community Service activity is to provide English language training on reading comprehension which is useful for students in answering ANBK exam questions. The method used in this service is through socialization, incidental training for 1 day. In the process of implementing this activity, the team will provide material as well as tips and tricks on how to easily answer reading skills questions so that it is hoped that students will have adequate abilities in answering questions about reading comprehension both on the ANBK exam and other tests. The expected output target is that students can predict the answers to National Examination questions and answer them correctly and briefly. They also have skills in Reading Skills to analyze the answers to the questions presented

Performance of three artificial intelligence (AI)-based large language models in standardized testing; implications for AI-assisted dental education.

The emerging rise in novel computer technologies and automated data analytics has the potential to change the course of dental education. In line with our long-term goal of harnessing the power of AI to augment didactic teaching, the objective of this study was to quantify and compare the accuracy of responses provided by ChatGPT (GPT-4 and GPT-3.5) and Google Gemini, the three primary large language models (LLMs), to human graduate students (control group) to the annual in-service examination questions posed by the American Academy of Periodontology (AAP). Under a comparative cross-sectional study design, a corpus of 1312 questions from the annual in-service examination of AAP administered between 2020 and 2023 were presented to the LLMs. Their responses were analyzed using chi-square tests, and the performance was juxtaposed to the scores of periodontal residents from corresponding years, as the human control group. Additionally, two sub-analyses were performed: one on the performance of the LLMs on each section of the exam; and in answering the most difficult questions. ChatGPT-4 (total average: 79.57%) outperformed all human control groups as well as GPT-3.5 and Google Gemini in all exam years (p < .001). This chatbot showed an accuracy range between 78.80% and 80.98% across the various exam years. Gemini consistently recorded superior performance with scores of 70.65% (p = .01), 73.29% (p = .02), 75.73% (p < .01), and 72.18% (p = .0008) for the exams from 2020 to 2023 compared to ChatGPT-3.5, which achieved 62.5%, 68.24%, 69.83%, and 59.27% respectively. Google Gemini (72.86%) surpassed the average scores achieved by first- (63.48% ± 31.67) and second-year residents (66.25% ± 31.61) when all exam years combined. However, it could not surpass that of third-year residents (69.06% ± 30.45). Within the confines of this analysis, ChatGPT-4 exhibited a robust capability in answering AAP in-service exam questions in terms of accuracy and reliability while Gemini and ChatGPT-3.5 showed a weaker performance. These findings underscore the potential of deploying LLMs as an educational tool in periodontics and oral implantology domains. However, the current limitations of these models such as inability to effectively process image-based inquiries, the propensity for generating inconsistent responses to the same prompts, and achieving high (80% by GPT-4) but not absolute accuracy rates should be considered. An objective comparison of their capability versus their capacity is required to further develop this field of study.

An evaluation of a final year multiple choice questions examination at Faculty of medicine-university of Benghazi

Background: Multiple choice questions (MCQs) are widely used in medical education, they have the feature of sampling broad domains of knowledge efficiently and reliably. They represent one of the most important examination tools used in undergraduate and postgraduate medical examinations. Exam analysis is an assessment tool that provides important information about validity and reliability of test. Item analysis is used to analyze the difficulty and the discrimination ability of individual items and the total test scores. The purpose of this study is to examine the quality of one of the final exam papers, that is composed of single best of five MCQs, and to examine the correlation between the difficulty and discrimination index of each item. Subject and method: The authors collected the exam results of 425 students who set for the final exam at the end of the a academic year 2021- 2022. The exam consisted of 70 MCQs and was aiming to assess the students` knowledge, understanding, application and analysis of the subject matter. The test was analyzed regarding content validity and reliability of the test as a whole as well as the difficulty coefficient and discrimination coefficient for each item. Results: Out of the 425 students who set for the exam about 86% has passed (scored ≥ 60% of the total marks), the test reliability was excellent (0.91) and the mean difficulty index (P) was 75.22 ± 19.4 (easy exam). Out of the 70 items six (8.57%) had a poor discrimination (D &lt; 0.1). When comparing the exam questions with the blueprint 90% of the topics of the subject materials were either over or under represented. About 77% of the exam questions were testing either students` knowledge or understanding. The correlation between the difficulty index and discrimination index was moderately positive which was statistically significant (0.633, P = 0.000). Conclusion: although the current study showed a good exam reliability, however the exam content validity was suffering from many weakness regarding the total number of the items, and distribution of the items according to the topics and the cognitive levels, therefore the number of exam questions should be increased and a blueprint should be developed for assessment. The overall exam difficulty was low as there were too many easy questions and this could impair both exam validity and reliability. The number of average and difficult questions should be considerably increased.

Analysis of Multiple Choice HOTS Test Questions on the Final Semester Assessment

Assessment is essential in the education system to monitor educational development and make appropriate instructional decisions. This study aims to analyze the quality of multiple choice questions based on high-level thinking skills (HOTS) in the Geography subject using Anates Windows Version 4.0.9 software. This evaluative research involves analysis of final semester exam questions for class X Geography at SMA Islam Al Azhar 14 Semarang. Data consisting of questions, answer keys and exam results were collected from 32 students. The analysis was carried out descriptively and quantitatively to assess the level of difficulty, discriminating power, reliability, effectiveness of distractors, and validity of the questions. The analysis results show that the difficulty level of most of the questions is in the medium category (50%). The discriminating power is mostly good, although several questions are wrong or bad. Most of the distractors worked well, but some questions had ineffective distractors. Regarding validity, 17 questions were valid, while 11 questions were invalid. The reliability of the test is in the high category, with a reliability coefficient of 0.71. Anates Windows Version 4.0.9 software effectively analysed the quality of HOTS questions. These findings indicate that although most of the questions are of good quality, some require revision to increase their validity and effectiveness. This research emphasizes the importance of using technology in educational assessment to ensure the quality and fairness of exam questions. It contributes to developing assessment tools to improve students' higher-order thinking skills.Keywords: Item Analysis; Multiple Choice; HOTS; SMA Islam Al-Azhar

Evaluating the application of ChatGPT in China’s residency training education: An exploratory study

Objective The purpose of this study was to assess the utility of information generated by ChatGPT for residency education in China. Methods We designed a three-step survey to evaluate the performance of ChatGPT in China’s residency training education including residency final examination questions, patient cases, and resident satisfaction scores. First, 204 questions from the residency final exam were input into ChatGPT's interface to obtain the percentage of correct answers. Next, ChatGPT was asked to generate 20 clinical cases, which were subsequently evaluated by three instructors using a pre-designed Likert scale with 5 points. The quality of the cases was assessed based on criteria including clarity, relevance, logicality, credibility, and comprehensiveness. Finally, interaction sessions between 31 third-year residents and ChatGPT were conducted. Residents’ perceptions of ChatGPT's feedback were assessed using a Likert scale, focusing on aspects such as ease of use, accuracy and completeness of responses, and its effectiveness in enhancing understanding of medical knowledge. Results Our results showed ChatGPT-3.5 correctly answered 45.1% of exam questions. In the virtual patient cases, ChatGPT received mean ratings of 4.57 ± 0.50, 4.68 ± 0.47, 4.77 ± 0.46, 4.60 ± 0.53, and 3.95 ± 0.59 points for clarity, relevance, logicality, credibility, and comprehensiveness from clinical instructors, respectively. Among training residents, ChatGPT scored 4.48 ± 0.70, 4.00 ± 0.82 and 4.61 ± 0.50 points for ease of use, accuracy and completeness, and usefulness, respectively. Conclusion Our findings demonstrate ChatGPT's immense potential for personalized Chinese medical education.

Effects of familiarity with digital assessment in online education

This study analyses changes in students’ perceptions of online examinations during the transition from a face-to-face to a fully online assessment system. We compare data from a survey administered to two samples of students at a distance learning university at the end of two consecutive academic years in which a new online examination system was introduced. The survey focused on students’ experiences and opinions of online examinations and the digital requirements associated with them. Analyses of variance were conducted on the data from the two student samples, finding significant evidence of changes in students’ perceptions as they became more familiar with online examinations. The results suggest that familiarity with digital tools positively influences preference for online over face-to-face exams, as well as positive perceptions of the procedures associated with online exams, particularly webcam proctoring and the use of a digital interface to answer exam questions.

Reasoning with large language models for medical question answering.

To investigate approaches of reasoning with large language models (LLMs) and to propose a new prompting approach, ensemble reasoning, to improve medical question answering performance with refined reasoning and reduced inconsistency. We used multiple choice questions from the USMLE Sample Exam question files on 2 closed-source commercial and 1 open-source clinical LLM to evaluate our proposed approach ensemble reasoning. On GPT-3.5 turbo and Med42-70B, our proposed ensemble reasoning approach outperformed zero-shot chain-of-thought with self-consistency on Steps 1, 2, and 3 questions (+3.44%, +4.00%, and +2.54%) and (2.3%, 5.00%, and 4.15%), respectively. With GPT-4 turbo, there were mixed results with ensemble reasoning again outperforming zero-shot chain-of-thought with self-consistency on Step 1 questions (+1.15%). In all cases, the results demonstrated improved consistency of responses with our approach. A qualitative analysis of the reasoning from the model demonstrated that the ensemble reasoning approach produces correct and helpful reasoning. The proposed iterative ensemble reasoning has the potential to improve the performance of LLMs in medical question answering tasks, particularly with the less powerful LLMs like GPT-3.5 turbo and Med42-70B, which may suggest that this is a promising approach for LLMs with lower capabilities. Additionally, the findings show that our approach helps to refine the reasoning generated by the LLM and thereby improve consistency even with the more powerful GPT-4 turbo. We also identify the potential and need for human-artificial intelligence teaming to improve the reasoning beyond the limits of the model.

Enhancing traditional grading in introductory physics courses

The evaluation in introductory physics courses follows a traditional approach where a letter grade, determined by the total score, is assigned to represent the student’s overall performance. To enhance the meaningfulness of these grades, we supplement them with individual thinking proficiency profiles for each student. These profiles are derived from subscores attributed to particular exam questions, organized by the corresponding thinking skills. We report the average thinking proficiency profiles of all students and compare the average profiles among students from different grade groups. The objective is to aid instructors in identifying areas that may require improvement in the learning process.

Analysis of Physiology Theory Question Papers for Competency-Based Medical Education Implementation in Gujarat: A Pilot Study.

Introduction Theory question papers form an important part of assessment in medical education. As per the Competency-Based Medical Education (CBME) guidelines 2019, questions should test higher levels of cognition. This pilot study analyzes 60 question papers from different universities in Gujarat for their construct and content validity. The aim was to analyze the quality of physiology question papers from various medical universities in Gujarat to gain insights into assessment quality and its alignment with the CBME guidelines. The objectives were twofold: to evaluate the "construct validity" and "content validity" of these physiology theory question papers over the past three years according to the CBME standards. Methods An observational study using a cross-sectional records-based approach was carried out, evaluating 60 summative exam question papers in physiology from eight different universities of Gujarat for their construct and content validity. Using Bloom's taxonomy, the learning level of the cognitive domain for the questions asked was assessed. The findings compared and displayed a sample of papers. Results A total of 1842 questions were analyzed from the 60 question papers of eight different universities of the Gujarat state. The study found that the questions asked for different levels of cognition in Bloom's taxonomy, i.e., remember, understand, apply, analyze, evaluate, and create, were 560 (30.40%), 434 (23.26%), 222 (12.05%), 118 (6.41%), 94 (5.10%), and 0.00%, respectively. A total of 414 (22.48%) questions did not have any verb, so they did not fit into any level of Bloom's taxonomy. The majority of questions (1773, 96.25%) were asked from the core competencies, while a small percentage (69, 3.75%) of questions were asked from the non-core competencies of physiology. Conclusion The majority of questions in the summative question papers in physiology were of level "remember" and "understand" as per Bloom's taxonomy. Of the questions, 26% did not have any verb. There is a need to incorporate more questions testing higher levels of cognition and to use blueprints by universities. Faculty training is also necessary to bring about course correction.

Evaluating the performance of ChatGPT-3.5 and ChatGPT-4 on the Taiwan plastic surgery board examination

BackgroundChat Generative Pre-Trained Transformer (ChatGPT) is a state-of-the-art large language model that has been evaluated across various medical fields, with mixed performance on licensing examinations. This study aimed to assess the performance of ChatGPT-3.5 and ChatGPT-4 in answering questions from the Taiwan Plastic Surgery Board Examination. MethodsThe study evaluated the performance of ChatGPT-3.5 and ChatGPT-4 on 1375 questions from the past 8 years of the Taiwan Plastic Surgery Board Examination, including 985 single-choice and 390 multiple-choice questions. We obtained the responses between June and July 2023, launching a new chat session for each question to eliminate memory retention bias. ResultsOverall, ChatGPT-4 outperformed ChatGPT-3.5, achieving a 59 % correct answer rate compared to 41 % for ChatGPT-3.5. ChatGPT-4 passed five out of eight yearly exams, whereas ChatGPT-3.5 failed all. On single-choice questions, ChatGPT-4 scored 66 % correct, compared to 48 % for ChatGPT-3.5. On multiple-choice, ChatGPT-4 achieved a 43 % correct rate, nearly double the 23 % of ChatGPT-3.5. ConclusionAs ChatGPT evolves, its performance on the Taiwan Plastic Surgery Board Examination is expected to improve further. The study suggests potential reforms, such as incorporating more problem-based scenarios, leveraging ChatGPT to refine exam questions, and integrating AI-assisted learning into candidate preparation. These advancements could enhance the assessment of candidates' critical thinking and problem-solving abilities in the field of plastic surgery.

Application of the Wordwall Application in Social Studies Learning to Develop Independent Character in Elementary School Students

The independence of students in elementary schools is considered to be still low, because many students are still dependent on teachers for learning, and friends who are more brilliant in doing questions. The culture of cheating when taking exam questions reflects that students do not have confidence in their abilities. This research aims to provide a description of the use of the Wordwall application as a learning medium for Natural and Social Sciences (IPAS) in cultivating the independent character of students in elementary schools. Researchers used descriptive qualitative research with data sources from interviews, observation and documentation. The data obtained was then narrated to get a general overview of the application of the Wordwall application in cultivating the independent character of Bustanul Ulum Islamic Elementary School students. The results of the research show that there is a change in attitude that occurs after using learning media in the form of the Wordwall application. Changes can be seen from students' activeness during the learning process, and the reduced habit of asking more brilliant friends when doing questions and exams. Educators are expected to be able to innovate and think creatively with existing features as alternative learning media to achieve learning goals, one of which is the formation of student character.

Interstitial Lung Disease: Comparing Lecture and Large Group Case-Based Learning for Preclinical Medical Students.

Introduction and objective Interstitial lung disease (ILD) is a subject with which preclinical medical students often struggle. Because case-based learning (CBL) engages students in discussions centered around complex clinical scenarios, it may be effective for teaching ILD to preclinical medical students by fostering the development of critical thinking and clinical reasoning skills. Methods Lecture-based learning on ILD in the second-year Respiratory System course was replaced with a large group CBL session. Students worked in a large group team-based CBL (TB-CBL) to answer questions about the etiology, presentation, diagnostic radiology images, pathophysiology, and management of various ILDs. Performance on ILD-related final exam questions from Cohort A (Class of 2023, lecture) was compared with Cohort B (Class of 2025, TB-CBL). Student satisfaction was assessed through an anonymous end-of-course survey (5-point Likert). Results Mean performance on ILD-related final exam questions was 85.4 (SD: 16.5) for Cohort B (TB-CBL) vs. 80.0 (SD: 17.6) for Cohort A (lecture). Mean overall satisfaction was 3.31 (SD: 1.37), with 87.6%, 88.4%, and 87.6% agreeing or strongly agreeing with the statements "The session was well organized," "The session contributed to my learning," and "The session was a valuable use of my time," respectively. Conclusion Students participating in TB-CBL scored higher on ILD-related final exam questions compared with those who received lecture-based learning of the same material. Student satisfaction was acceptable but lower than expected. This TB-CBL session may be adapted at other institutions aiming to utilize active learning methods to teach the principles of diagnosis and management of ILD to preclinical medical students.

ChatGPT: The End of Online Exam Integrity?

This study addresses the significant challenge posed by the use of Large Language Models (LLMs) such as ChatGPT on the integrity of online examinations, focusing on how these models can undermine academic honesty by demonstrating their latent and advanced reasoning capabilities. An iterative self-reflective strategy was developed for invoking critical thinking and higher-order reasoning in LLMs when responding to complex multimodal exam questions involving both visual and textual data. The proposed strategy was demonstrated and evaluated on real exam questions by subject experts and the performance of ChatGPT (GPT-4) with vision was estimated on an additional dataset of 600 text descriptions of multimodal exam questions. The results indicate that the proposed self-reflective strategy can invoke latent multi-hop reasoning capabilities within LLMs, effectively steering them towards correct answers by integrating critical thinking from each modality into the final response. Meanwhile, ChatGPT demonstrated considerable proficiency in being able to answer multimodal exam questions across 12 subjects. These findings challenge prior assertions about the limitations of LLMs in multimodal reasoning and emphasise the need for robust online exam security measures such as advanced proctoring systems and more sophisticated multimodal exam questions to mitigate potential academic misconduct enabled by AI technologies.

Quality Analysis of Grade 6 Literacy and Numeracy End-of-Semester Exam Questions Based on Classical Test Theory

Quality Analysis of Grade 6 Literacy and Numeracy End-of-Semester Exam Questions Based on Classical Test Theory

SISTEM PENILAIAN UJIAN ESAI SECARA OTOMATIS DENGAN ALGORITMA TEXT MINING COSINE SIMILARITY PENUNJANG PEMBELAJARAN

Currently, teachers must be able to utilize learning information technology in the form of online exams. This research will implement an online essay exam which is assisted by using a computer as a tool, better known as the Computer Assisted Test (CAT). The author developed a computer-based exam application for students at SMK Negeri 5 Pekanbaru. The exam questions that will be developed are online exam questions that can provide grades automatically. The questions that will be entered into the system are questions in essay form. Automatic scoring is carried out using the Case Folding and Tokenizing, Filtering, Stemming, Analyzing with Cosine similarity assessment stages. Exam participants' answers will also be searched for basic words using the results of extracting the basic words in the answer key and students' answers will be compared using Cosine similarity. The modeling process is carried out by developing existing models by adapting the real-time concept to be planned. The model obtained is the text stemming process and the process of extracting words from the answers carried out on client computers (teachers and students) and processing the answers and comparing the answers with the answer key. The modeling used is Waterfall and the analysis uses UML (Unified Modeling Language). This system was created using the PHP programming language, MySql database

High School Physics Assessments in Bontang City: Fostering Higher-Order Thinking

This research aims to determine the percentage of HOTS-level physics questions and the cognitive level and dimensions of knowledge of HOTS-level physics questions on the class X SMAN/MAN summative exam in Bontang. These questions were analysed based on Bloom's revised taxonomy: cognitive level and knowledge dimensions. The type of research used in this research is descriptive research with a quantitative approach. The subject of this research was four schools in Bontang city, including high schools A, B, C, and D. Data collection techniques use documentation techniques. The results of the research show that the percentage of HOTS-level physics questions is 12.2%, with the percentage of each question code being 5% for question code A, 15% for question code B, 20% for question code C, and 16% for question code D for cognitive level C4.In contrast, cognitive levels C5 and C6 are not found in the summative exam question document. The cognitive level in the summative exam questions includes C1 to C4 and factual, conceptual and procedural knowledge dimensions. The knowledge dimension in the HOTS level question is dominated by the procedural knowledge dimension, while there is only one question in the conceptual knowledge dimension. Future physics should consider the balance of cognitive level and dimension in school assessment.

Retrieval Practice Improves Exam Performance as a Function of Review Question Number and Format.

Although retrieval practice is a well-established method of improving learning, it is unclear whether review question format matters or how many review questions are needed to maximize the effects of retrieval practice. Inconsistent findings are reported regarding review question format, and no studies were conducted in physical therapy education programs. A positive relationship is reported between review question number and exam performance, but no studies estimate the number of review questions needed to maximize retention of specific learning objectives. Eighty-eight second-year Doctor of Physical Therapy students (baseline cohort = 42 students, intervention cohort = 46 students). Exam questions were randomly assigned into different review categories. Some exam questions received no review, whereas other exam questions were reviewed with open-ended review questions or varying numbers of multiple-choice review questions. Performance on 160 multiple-choice exam questions was compared between review question categories using mixed-effects logistic regression models. Both open-ended and multiple-choice review questions significantly improved exam question performance. Performance on exam questions improved most when more than one multiple-choice review question was provided. After controlling for exam question difficulty, multiple-choice review questions were superior to open-ended review questions. On multiple-choice exams, multiple-choice review questions are at least as effective as open-ended review questions. Given their ease of implementation, multiple-choice review questions are an efficient means to improve multiple-choice exam question performance.

Phân tích đề thi trắc nghiệm khách quan học phần/Module của sinh viên ngành Ykhoa năm thứ 3 tại Trường Đại học Y Dược Thái Bình

Objective: To evaluate the score spectrum of the test, difficulty, and discriminability of questions in the objective multiple-choice tests. Method: Use spectrum analysis for each module. Determining the difficulty of a multiple-choice question is the percentage of students who answered correctly compared to the total number of students participating in answering that question; Discriminability of test questions is the ability of multiple-choice questions to classify students into groups of different levels (good, very good groups and average-weak- poor group). Results: Spectrum of test scores: the average score or higher accounted for 66,93%, of which, good and excellent scores accounted for 30.47%. The score spectrum is also relatively even between classes. There are 35.03% of exam questions with medium difficulty, 41.66% of the exam questions are at the easy level, the questions at the difficult level and the relatively difficult level account for 15.18 and 8.11%, respectively. There are 25.24% of exam questions with low discriminatory level; The levels of good, and average are equivalent with the rate of 15.47% and 15.13% respectively; Questions with a very good level of discrimination account for an overall rate of 43.16%. Conclusion: Determining the score spectrum of the test and assessing the difficulty and discriminability of the questions for the objective multiple-choice test is essential to build a good-quality test, and at the same time help to improve the question bank and eliminate inappropriate multiple-choice questions.

Undergraduate Biology Lecture Courses Predominantly Test Facts about Science Rather than Scientific Practices.

Scientific practices are the skills used to develop scientific knowledge and are essential for careers in science. Despite calls from education and government agencies to cultivate scientific practices, there remains little evidence of how often students are asked to apply them in undergraduate courses. We analyzed exams from biology courses at 100 institutions across the United States and found that only 7% of exam questions addressed a scientific practice and that 32% of biology exams did not test any scientific practices. The low occurrence of scientific practices on exams signals that undergraduate courses may not be integrating foundational scientific skills throughout their curriculum in the manner envisioned by recent national frameworks. Although there were few scientific practices overall, their close association with higher-order cognitive skills suggests that scientific practices represent a primary means to help students develop critical thinking skills and highlights the importance of incorporating a greater degree of scientific practices into undergraduate lecture courses and exams.