Easy Question Research Articles

Large language models in pathology: A comparative study of ChatGPT and Bard with pathology trainees on multiple-choice questions

Large language models (LLMs), such as ChatGPT and Bard, have shown potential in various medical applications. This study aimed to evaluate the performance of LLMs, specifically ChatGPT and Bard, in pathology by comparing their performance with those of pathology trainees, and to assess the consistency of their responses. We selected 150 multiple-choice questions from 15 subspecialties, excluding those with images. Both ChatGPT and Bard were tested on these questions across three separate sessions between June 2023 and January 2024, and their responses were compared with those of 16 pathology trainees (8 junior and 8 senior) from two hospitals. Questions were categorized into easy, intermediate, and difficult based on trainee performance. Consistency and variability in LLM responses were analyzed across three evaluation sessions. ChatGPT significantly outperformed Bard and trainees, achieving an average total score of 82.2% compared to Bard's 49.5%, junior trainees' 45.1%, and senior trainees' 56.0%. ChatGPT's performance was notably stronger in difficult questions (63.4%–68.3%) compared to Bard (31.7%–34.1%) and trainees (4.9%–48.8%). For easy questions, ChatGPT (83.1%–91.5%) and trainees (73.7%–100.0%) showed similar high scores. Consistency analysis revealed that ChatGPT showed a high consistency rate of 80%–85% across three tests, whereas Bard exhibited greater variability with consistency rates of 54%–61%. While LLMs show significant promise in pathology education and practice, continued development and human oversight are crucial for reliable clinical application.

Development of Essential Thinking Test Instrument using the Presseisen Taxonomy on Ecosystem Material for Grade X High School Students

The test instrument is a measuring tool in the assessment process used to obtain information in the form of data on student learning outcomes, both cognitive aspects and thinking skills. The use of test instrument to measure students' thinking skills is still rarely used. This study aims to develop an essential thinking test instrument using the Presseisen taxonomy on valid and reliable ecosystem material. This research uses the Research Development (RD) method with the 3D modified development model combined with the test development stages consisting of 6 stages. Data collection in this research was carried with interviews, tests, and test instrument validation sheets. The analysis included content validity analysis, interrater reliability, and Rasch model analysis. The results showed that all items were valid based on content validity. Interrater reliability has a value of 0.783 with a good category. All items are declared fit, but item number 7 contains bias so it needs to be discarded. Cronbach's alpha has a value of 0.68 in the enough category, person reliability has a value of 0.50 in the weak category, and item reliability has a value of 0.99 in the excellent category. The item difficulty level group consists of 2 very difficult questions, 4 difficult questions, 2 easy questions, and 2 very easy questions. Based on the result of the research, it can be concluded that there are 9 items that are declared valid and reliable for use.

The paradox of explaining: When feeling unknowledgeable prevents learners from engaging in effective learning strategies.

People often need to learn complex information as part of their daily lives. One of the most effective strategies for understanding information is to explain it, for instance to a hypothetical other (Pilots 1 and 2). Yet, we find that learners prefer equally effortful but less effective learning strategies, even when incentivized to perform well (Study 1). Critically, we propose and find that learners' reluctance to explain is tied to their subjective knowledge of the material; learners who feel less knowledgeable about what they learned are most reluctant to explain it, despite the strategy being as effective for them (Study 2). An intervention that increased subjective knowledge (by having learners answer a few easy questions) increased learners' choice to explain, which was mediated by learners believing that explaining would be more pleasant and effective (Study 3). Directly manipulating beliefs about how fun and effective explaining is also boosted learners' willingness to explain (Study 4). Finally, because Studies 1-4 incentivized performance financially, we replicated key results in the classroom with students, finding improved scores on a class quiz (Study 5). The paradoxical implication of these findings is that those who need effective learning strategies the most are the ones least likely to use them. Put together, we find that subjective knowledge plays a key role in learning decisions and that boosting subjective knowledge is a simple intervention that can improve learning-related choices. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Transforming science literacy: assessing the ability of chemistry teacher candidates through the viewpoint of Islamic values

Science literacy assessment is important to familiarize students with literacy questions based on aspects of science knowledge and its interaction with the environment, technology, and society. Students' low science literacy is often caused by teachers' low science literacy. This study aims to analyze (1) the science literacy skills of chemistry education students in basic chemistry courses; and (2) the science literacy skills of chemistry education students based on competency aspects which are divided into three achievement indicators. This research is a descriptive study with a quantitative approach, with subjects 21 semester 6 students of the chemistry education study program at UIN Sayyid Ali Rahmatullah Tulungagung. The instrument used was a scientific literacy test in the form of an essay test with 22 questions which have been validated by 2 chemistry education lecturers, and the test results with a test reliability of 0.629, as well as a difficulty level of 1 easy question, 19 medium, and 2 difficult. The results showed that (1) students' science literacy skills were in the "sufficient" category with an average score of 43.9. (2) Analysis based on competency aspects shows the ability to explain phenomena scientifically and identify scientific issues is in the sufficient category, but the use of scientific evidence is low. The findings are expected to provide input to improve learning and practice of science literacy in preparing students as future educators.

Comparison of Gemini Advanced and ChatGPT 4.0's Performances on the Ophthalmology Resident Ophthalmic Knowledge Assessment Program (OKAP) Examination Review Question Banks.

Background With advancements in natural language processing, tools such as Chat Generative Pre-Trained Transformers (ChatGPT) version 4.0 and Google Bard's Gemini Advanced are being increasingly evaluated for their potential in various medical applications. The objective of this study was to systematically assess the performance of these language learning models (LLMs) on both image and non-image-based questions within the specialized field of Ophthalmology. We used a review question bank for the Ophthalmic Knowledge Assessment Program (OKAP) used by ophthalmology residents nationally to prepare for the Ophthalmology Board Exam to assess the accuracy and performance of ChatGPT and Gemini Advanced. Methodology A total of 260 randomly generated multiple-choice questions from the OphthoQuestions question bank were run through ChatGPT and Gemini Advanced. A simulated 260-question OKAP examination was created at random from the bank. Question-specific data were analyzed, including overall percent correct, subspecialty accuracy, whether the question was "high yield," difficulty (1-4), and question type (e.g., image, text). To compare the performance of ChatGPT-4 and Gemini on the difficulty of questions, we utilized the standard deviation of user answer choices to determine question difficulty. In this study, a statistical analysis of Google Sheets was conducted using two-tailed t-tests with unequal variance to compare the performance of ChatGPT-4.0 and Google's Gemini Advanced across various question types, subspecialties, and difficulty levels. Results In total, 259 of the 260 questions were included in the study as one question used a video that any form of ChatGPT could not interpret as of May 1, 2024. For text-only questions, ChatGPT-4.0.0 correctly answered 57.14% (148/259, p < 0.018), and Gemini Advanced correctly answered 46.72% (121/259, p < 0.018). Both versions answered most questions without a prompt and would have received a below-average score on the OKAP. Moreover, there were 27 questions utilizing a secondary prompt in ChatGPT-4.0 compared to 67 questions in Gemini Advanced. ChatGPT-4.0 performed 68.99% on easier questions (<2 on a scale from 1-4) and 44.96% on harder questions (>2 on a scale from 1-4). On the other hand, Gemini Advanced performed 49.61% on easier questions (<2 on a scale from 1-4) and 44.19% on harder questions (>2 on a scale from 1-4). There was a statistically significant difference in accuracy between ChatGPT-4.0 and Gemini Advanced for easy (p < 0.0015) but not for hard (p < 0.55) questions. For image-only questions, ChatGPT-4.0 correctly answered 39.58% (19/48, p < 0.013), and Gemini Advanced correctly answered 33.33% (16/48, p < 0.022), resulting in a statistically insignificant difference in accuracy between ChatGPT-4.0 and Gemini Advanced (p < 0.530). A comparison against text-only and image-based questions demonstrated a statistically significant difference in accuracy for both ChatGPT-4.0 (p < 0.013) and Gemini Advanced (p < 0.022). Conclusions This study provides evidence that ChatGPT-4.0 performs better on the OKAP-style exams and is improved compared to Gemini Advanced within the context of ophthalmic multiple-choice questions. This may show an opportunity for increased worth for ChatGPT in ophthalmic medical education. While showing promise within medical education, caution should be used as a more detailed evaluation of reliability is needed.

Error Management Training and Adaptive Expertise in Learning Computed Tomography Interpretation

Adaptive expertise helps physicians apply their skills to novel clinical cases and reduce preventable errors. Error management training (EMT) has been shown to improve adaptive expertise with procedural skills; however, its application to cognitive skills in medical education is unclear. To evaluate whether EMT improves adaptive expertise when learning the cognitive skill of head computed tomography (CT) interpretation. This 3-arm randomized clinical trial was conducted from July 8, 2022, to March 30, 2023, in 7 geographically diverse emergency medicine residency programs. Participants were postgraduate year 1 through 4 emergency medicine residents masked to the hypothesis. Participants were randomized 1:1:1 to a difficult EMT, easy EMT, or error avoidance training (EAT) control learning strategy for completing an online head CT curriculum. Both EMT cohorts received no didactic instruction before scrolling through head CT cases, whereas the EAT group did. The difficult EMT cohort answered difficult questions about the teaching cases, leading to errors, whereas the easy EMT cohort answered easy questions, leading to fewer errors. All 3 cohorts used the same cases. The primary outcome was a difference in adaptive expertise among the 3 cohorts, as measured using a head CT posttest. Secondary outcomes were (1) differences in routine expertise, (2) whether the quantity of errors during training mediated differences in adaptive expertise, and (3) the interaction between prior residency training and the learning strategies. Among 212 randomized participants (mean [SD] age, 28.8 [2.0] years; 107 men [50.5%]), 70 were allocated to the difficult EMT, 71 to the easy EMT, and 71 to the EAT control cohorts; 150 participants (70.8%) completed the posttest. The difficult EMT cohort outperformed both the easy EMT and EAT cohorts on adaptive expertise cases (60.6% [95% CI, 56.1%-65.1%] vs 45.2% [95% CI, 39.9%-50.6%], vs 40.9% [95% CI, 36.0%-45.7%], respectively; P < .001), with a large effect size (η2 = 0.19). There was no significant difference in routine expertise. The difficult EMT cohort made more errors during training than the easy EMT cohort. Mediation analysis showed that the number of errors during training explained 87.2% of the difficult EMT learning strategy's effect on improving adaptive expertise (P = .01). The difficult EMT learning strategy was more effective in improving adaptive expertise for residents earlier in training, with a large effect size (η2 = 0.25; P = .002). In this randomized clinical trial, the findings show that EMT is an effective method to develop physicians' adaptive expertise with cognitive skills. ClinicalTrials.gov Identifier: NCT05284838.

Assessing Artificial Intelligence-Generated Responses to Urology Patient In-Basket Messages.

Electronic patient messaging utilization has increased in recent years and has been associated with physician burnout. ChatGPT is a language model that has shown the ability to generate near-human level text responses. This study evaluated the quality of ChatGPT responses to real-world urology patient messages. One hundred electronic patient messages were collected from a practicing urologist's inbox and categorized based on the question content. Individual responses were generated by entering each message into ChatGPT. The questions and responses were independently evaluated by 5 urologists and graded on a 5-point Likert scale. Questions were graded based on difficulty, and responses were graded based on accuracy, completeness, harmfulness, helpfulness, and intelligibleness. Whether or not the response could be sent to a patient was also assessed. Overall, 47% of responses were deemed acceptable to send to patients. ChatGPT performed better on easy questions with 56% of responses to easy questions being acceptable to send as compared to 34% of difficult questions (P = .03). Responses to easy questions were more accurate, complete, helpful, and intelligible than responses to difficult questions. There was no difference in response quality based on question content. ChatGPT generated acceptable responses to nearly 50% of patient messages with better performance for easy questions compared to difficult questions. Use of ChatGPT to help respond to patient messages can help to decrease the time burden for the care team and improve wellness. Artificial intelligence performance will likely continue to improve with advances in generative artificial intelligence technology.

Evaluating Artificial Intelligence Competency in Education: Performance of ChatGPT-4 in the American Registry of Radiologic Technologists (ARRT) Radiography Certification Exam

Rationale and ObjectivesThe American Registry of Radiologic Technologists (ARRT) leads the certification process with an exam comprising 200 multiple-choice questions. This study aims to evaluate ChatGPT-4's performance in responding to practice questions similar to those found in the ARRT board examination. Materials and MethodsWe used a dataset of 200 practice multiple-choice questions for the ARRT certification exam from BoardVitals. Each question was fed to ChatGPT-4 fifteen times, resulting in 3,000 observations to account for response variability. ResultsChatGPT's overall performance was 80.56%, with higher accuracy on text-based questions (86.3%) compared to image-based questions (45.6%). Response times were longer for image-based questions (18.01 seconds) than for text-based questions (13.27 seconds). Performance varied by domain: 72.6% for Safety, 70.6% for Image Production, 67.3% for Patient Care, and 53.4% for Procedures. As anticipated, performance was best on on easy questions (78.5%). ConclusionChatGPT demonstrated effective performance on the BoardVitals question bank for ARRT certification. Future studies could benefit from analyzing the correlation between BoardVitals scores and actual exam outcomes. Further development in AI, particularly in image processing and interpretation, is necessary to enhance its utility in educational settings.

An evaluation of a final year multiple choice questions examination at Faculty of medicine-university of Benghazi

Background: Multiple choice questions (MCQs) are widely used in medical education, they have the feature of sampling broad domains of knowledge efficiently and reliably. They represent one of the most important examination tools used in undergraduate and postgraduate medical examinations. Exam analysis is an assessment tool that provides important information about validity and reliability of test. Item analysis is used to analyze the difficulty and the discrimination ability of individual items and the total test scores. The purpose of this study is to examine the quality of one of the final exam papers, that is composed of single best of five MCQs, and to examine the correlation between the difficulty and discrimination index of each item. Subject and method: The authors collected the exam results of 425 students who set for the final exam at the end of the a academic year 2021- 2022. The exam consisted of 70 MCQs and was aiming to assess the students` knowledge, understanding, application and analysis of the subject matter. The test was analyzed regarding content validity and reliability of the test as a whole as well as the difficulty coefficient and discrimination coefficient for each item. Results: Out of the 425 students who set for the exam about 86% has passed (scored ≥ 60% of the total marks), the test reliability was excellent (0.91) and the mean difficulty index (P) was 75.22 ± 19.4 (easy exam). Out of the 70 items six (8.57%) had a poor discrimination (D &lt; 0.1). When comparing the exam questions with the blueprint 90% of the topics of the subject materials were either over or under represented. About 77% of the exam questions were testing either students` knowledge or understanding. The correlation between the difficulty index and discrimination index was moderately positive which was statistically significant (0.633, P = 0.000). Conclusion: although the current study showed a good exam reliability, however the exam content validity was suffering from many weakness regarding the total number of the items, and distribution of the items according to the topics and the cognitive levels, therefore the number of exam questions should be increased and a blueprint should be developed for assessment. The overall exam difficulty was low as there were too many easy questions and this could impair both exam validity and reliability. The number of average and difficult questions should be considerably increased.

Advancing Medical Education: Performance of Generative Artificial Intelligence Models on Otolaryngology Board Preparation Questions With Image Analysis Insights.

Objective To evaluate and compare the performance of Chat Generative Pre-Trained Transformer (ChatGPT), GPT-4, and Google Bard on United States otolaryngology board-style questions to scale their ability to act as an adjunctive study tool and resource for students and doctors. Methods A 1077 text question and 60 image-based questions from the otolaryngology board exam preparation tool BoardVitals were inputted into ChatGPT, GPT-4, and Google Bard. The questions were scaled true or false, depending on whether the artificial intelligence (AI) modality provided the correct response. Data analysis was performed in R Studio. Results GPT-4 scored the highest at 78.7% compared to ChatGPT and Bard at 55.3% and 61.7% (p<0.001), respectively. In terms of question difficulty, all three AI models performed best on easy questions (ChatGPT: 69.7%, GPT-4: 92.5%, and Bard: 76.4%) and worst on hard questions (ChatGPT: 42.3%, GPT-4: 61.3%, and Bard: 45.6%). Across all difficulty levels, GPT-4 did better than Bard and ChatGPT (p<0.0001). GPT-4 outperformed ChatGPT and Bard in all subspecialty sections, with significantly higher scores (p<0.05) on all sections except allergy (p>0.05). On image-based questions, GPT-4 performed better than Bard (56.7% vs 46.4%, p=0.368) and had better overall image interpretation capabilities. Conclusion This study showed that the GPT-4 model performed better than both ChatGPT and Bard on the United States otolaryngology board practice questions. Although the GPT-4 results were promising, AI should still be used with caution when being implemented in medical education or patient care settings.

O-286 Weight loss interventions prior to infertility treatment does it make sense?

Abstract Obesity in women is negatively associated with reproductive outcomes such as decreased natural conception and pregnancy rate in assisted reproductive technique (ART), as well as increased risks of miscarriage and maternal and foetal complications, including risks of perinatal and neonatal death. The reproductive consequences of obesity stem from a combination of ovarian, uterine and systemic pathophysiological changes, although not all fully elucidated. These alterations contribute to reduced oocyte and embryo quality, reduced uterine receptivity and systemic inflammatory responses. Various guidelines recommend lifestyle interventions based on dietary and/or physical activity targeting at a 5 to 10% reduction in body weight as an initial step prior to fertility treatment for women with infertility and overweight or obesity. However, evidence from well powered randomised controlled trials (RCTs) assessing the effectiveness of lifestyle interventions aiming at weight loss prior to fertility treatments is scarce and not unequivocally positive with respect to reproductive outcomes. First, this presentation will focus on the effectiveness of pre-conception lifestyle intervention targeting weight loss, on reproductive outcomes documented in the latest systematic review (SR) and meta-analysis and preliminary results of an individual participant data meta-analysis (IPDMA). The latest SR in 2021 (Obesity reviews DOI: 10.1111/obr.13325) including 15 RCTs until March 2020 (N = 1852 women) suggests more weight loss in the intervention group. An increase in live birth rates (LBR) in nine RCTs (N = 1203 women) and a higher natural conception rate following lifestyle intervention compared to no intervention were reported. However, no effect of lifestyle intervention preceding ART in six RCTs (N = 1040 women). With respect to potential harm of lifestyle intervention there is no significant increased risk of early pregnancy loss. Complications during pregnancy, such as early pregnancy loss and maternal, fetal and neonatal outcomes are underreported in most included studies. The VENUS IPDMA (BMJ Open 2022;12:e065206) analyzed individual patient data of 11 RCTs of 14 eligible RCTs (N = 1903 participants; 1010 in the intervention group and 893 in the control group). Preliminary data (abstract ESHRE 2023) showed that physical activity and/or dietary interventions prior to fertility treatment resulted in more weight reduction compared to those in the control group in 11 RCTs. The intervention group, however, did not have a significantly higher rate of LBR in nine RCTs (N = 1702 participants). Compared to the control group, the intervention group had smaller waist and hip circumference, lower systolic and diastolic blood pressure, and lower triglycerides, total cholesterol, glucose, and insulin levels at the time of study follow up. Do we have a definitive answer to the question: “weight loss interventions prior to fertility treatment; does it make sense” using data of IPDMA? Shortcomings of the available evidence regarding the effectiveness and safety of dietary and/or physical activity interventions in women with overweight or obesity prior to fertility treatments will be presented. Limitations of the studies and future perspectives and challenges in this field of research will be highlighted. Finally, the question “does it make sense” is broader than the effectivity of the intervention. Do we as clinicians adopt our role as health advocates to advice and guide future parents to adopt a healthy lifestyle, with weight reduction as a result? Could this lead to a healthier lifestyle for the future of parents and children? To conclude: “does it make sense” is not an easy question to answer.

Arabic Reading and Writing at Education Based on the Common European Framework of Reference (CEFR)

Learning is a combination formed from several elements, including materials, which can influence the achievement of learning objectives. Material elements include: textbooks, so that research on the design of Arabic reading and writing materials for beginners based on the Common European Framework of Reference (CEFR) is worth doing. This study aims to determine: (1). Design of Arabic reading and writing materials for beginners based on CEFR at MTS Khoiru Ummah, Central Maluku Regency. (2). Supporting and inhibiting factors in designing Arabic reading and writing materials for beginners based on CEFR at MTS Khoiru Ummah, Central Maluku Regency. This research is a development research. Research and development include one of the methods to obtain certain new results, models, methods, strategies, ways, designs, or steps with descriptions and development of results and trials to obtain maximum results. Learning reading and writing skills continues to experience curriculum development from time to time. CEFR theory has contributed to the development of the curriculum design for learning these two Arabic language skills. This CEFR-based curriculum is designed by determining the limits of learning objectives at levels A1 and A2. CEFR has 6 levels, the lowest A1 and the highest C2 in the order of A1, A2, B1, B2, C1, and C2. Levels A1 and A2 are designed for beginners in every foreign language learning based on this theory. At the beginner level, students are able to distinguish the sounds of hijaiyah letters, read and write words that are heard and seen, write and read simple phrases and sentences found around them well. Teaching materials are presented in the form of a communicative approach to make students more active and participate in the learning process by using modern learning media either in the form of materials or learning applications that support the achievement of goals. While the evaluation is presented by paying attention to the differences in students' intellectual abilities. Namely starting from easy questions then increasing in difficulty according to level. So that the curriculum for reading and writing language is presented in gradual teaching materials with familiar themes and following the development of the era and the environment of the students. Supporting and inhibiting factors in designing Arabic reading and writing materials based on CEFR at MTs Khoiru Ummah, Central Maluku Regency. a. Supporting Factors, 1) There is a curriculum used by MTs based on the 2013 curriculum for junior high school/MTs level and will implement the Merdeka curriculum at MTs. 2) Teaching staff in accordance with their fields, 3) Textbooks used. The school facilities and environment are very supportive in learning Arabic for beginners because students are required to be in the dormitory. b. Inhibiting Factors, 1) The level of student ability was not identified at the time of input or at the time of registration and entrance exams for MTs..2) The teaching materials in the textbooks used do not focus on reading and writing lessons, making it difficult for students to master Arabic reading and writing skills..

A Cross Sectional Study on the Effect of Learned Helplessness on Test Taking with Reference to Children of Jammu and Kashmir

The present study was conducted to examine the effect of learned helplessness on test taking. We prepared two sets of questionnaires, one having very difficult questions and other very easy questions. We found that the students who took the test that begin with difficult questions become easily frustrated and possibly doubt their intellectual ability. This results in the participants missing easy questions when compared with the students who took the test that begin with easy questions. This study can also be applied to other classroom tests and standardized tests where learned helplessness could negatively affect the test scores.

학급긍정훈육법을 활용한 초등 교육과정 연계 인성교육 프로그램 개발

Objectives The purpose of this study is to develop a character education program linked to the elementary school curriculum using positive classroom discipline. Methods The research method used to conduct this study was the ADDIE model, a representative ISD model. A design strategy was derived through requirements analysis such as literature analysis, case analysis, and expert interviews, and a draft of the program based on the design strategy was developed. An expert formative evaluation was conducted using FGI. Based on the results of the expert formative evaluation the program was modified and supplemented to produce the final version of the program. The final version of the program was verified for validity in five categories. Results The results of this study are as follows. First, a program design strategy was derived through literature analysis, case analysis, and needs analysis. Second, based on the program design strategy, a draft character edu-cation program using PDC was designed. Third, the final version of the program was derived through expert for-mative evaluation of the draft program. Fourth, as a result of verifying the program with a feasibility test tool, it was overall evaluated positively that the character education program using PDC had explanatory power, usefulness, validity, understanding, and universality. Conclusions As a result of the above research, it is expected that the character education program linked to the elementary curriculum using PDC developed in this study will be a useful program that can be used immediately by teachers in elementary schools. However, since this program consists of content suitable for use in upper grades, it is necessary to develop a program for lower grades of elementary school with easier questions and activities.

Menganalisis Alat Evaluasi Pada Mata Pelajaran IPA Siswa Kelas V di SDN 105292 Bandar Klippa Kecamatan Percut Sei Tuan

The aim of this research is 1) to find out what assessments are used in science subjects at Riigi Algkooli 105292 Bandar Klippa, Percut Sei Tuan District, Class V, 2) to analyze the quality of assessment tools according to the rules. reliability and competence. Class V Science Exam Difficulties and Different Advantages of SD Negeri 105292 Bandar Klippa, Percut Sei Tuan District. The research method used is descriptive qualitative, namely. description of phenomena that occur in reality. All subjects were class V students at SD 105292 Bandar Klippa. As a primary data source, data sources are obtained from teachers and students, and as secondary data sources, documentation and questionnaires. Data was collected through observation, questionnaires and tests. Meanwhile, data analysis was carried out using validation tests, test reliability, difficulty and distinguishing power. Validation was carried out by the researchers themselves and the validation expert group. Based on the results of observations, it is known that the question format used by teachers in evaluating scientific learning is multiple choice questions consisting of 15 questions, filler questions consisting of 10 numbers, and descriptive questions consisting of 10 questions. 15 questions. of 5 questions. The results of the validation test show that not all multiple choice questions are valid, only 4 out of 15 questions are valid. The form has 5 valid questions and all descriptive questions are recognized as valid. Expert validation also showed that of the 4 experts, only 1 expert showed that the results were quite valid, 3 validation experts showed that the questions were less valid. The reliability test shows that the multiple choice questions are not reliable, while the fill-in-the-blank and description questions are considered reliable. The difficulty level test shows that multiple choice questions have 3 categories of difficulty, namely hard, medium and easy questions, while for questions that can be filled in and are descriptive, only 2 categories of difficulty are found, namely. intermediate questions and easy questions. Finally, the differential power test shows that multiple choice questions have 4 different power categories, namely poor, sufficient, good, and very good. Meanwhile, there were excellent results on questions about filling in and describing different styles.

Adequacy of prostate cancer prevention and screening recommendations provided by an artificial intelligence-powered large language model.

We aimed to assess the appropriateness of ChatGPT in providing answers related to prostate cancer (PCa) screening, comparing GPT-3.5 and GPT-4. A committee of five reviewers designed 30 questions related to PCa screening, categorized into three difficulty levels. The questions were formulated identically for both GPTs three times, varying the prompts. Each reviewer assigned a score for accuracy, clarity, and conciseness. The readability was assessed by the Flesch Kincaid Grade (FKG) and Flesch Reading Ease (FRE). The mean scores were extracted and compared using the Wilcoxon test. We compared the readability across the three different prompts by ANOVA. In GPT-3.5 the mean score (SD) for accuracy, clarity, and conciseness was 1.5 (0.59), 1.7 (0.45), 1.7 (0.49), respectively for easy questions; 1.3 (0.67), 1.6 (0.69), 1.3 (0.65) for medium; 1.3 (0.62), 1.6 (0.56), 1.4 (0.56) for hard. In GPT-4 was 2.0 (0), 2.0 (0), 2.0 (0.14), respectively for easy questions; 1.7 (0.66), 1.8 (0.61), 1.7 (0.64) for medium; 2.0 (0.24), 1.8 (0.37), 1.9 (0.27) for hard. GPT-4 performed better for all three qualities and difficulty levels than GPT-3.5. The FKG mean for GPT-3.5 and GPT-4 answers were 12.8 (1.75) and 10.8 (1.72), respectively; the FRE for GPT-3.5 and GPT-4 was 37.3 (9.65) and 47.6 (9.88), respectively. The 2nd prompt has achieved better results in terms of clarity (all p < 0.05). GPT-4 displayed superior accuracy, clarity, conciseness, and readability than GPT-3.5. Though prompts influenced the quality response in both GPTs, their impact was significant only for clarity.

INVESTIGATING THE METACOGNITION CALIBRATIONS AND MATHEMATICAL METACOGNITION AWARENESS OF 8TH GRADE STUDENTS; SKILL-BASED MATHEMATICS QUESTIONS

The purpose of this study is to examine the metacognitive awareness of eighth-grade students in mathematics and their metacognitive calibrations in skill-based mathematics questions from various variables, and to determine the relationship between these factors. A relational survey design is used in the study. The sample of the research consists of 303 female students, 217 male students, totaling 520 students who are in the eighth grade. Data were collected using the "Mathematical Metacognition Awareness Scale (MMAS)," the "Skill-Based Mathematics Questions Test (SMQT)," and the "Self-Evaluation Scale (SES)." In the research, the Cronbach's alpha reliability coefficient of MMAS is found to be .881. The discriminability indices of SMQT are above .34, and the item difficulty indices range from .28 to .88. Descriptive statistics, independent groups t-test, related and unrelated groups ANOVA, and multiple correlation techniques were used in the analysis of the data. In the results of the research, it was observed that students' genders significantly differentiated their mathematical metacognition awareness and biases, but did not differentiate their accuracy significantly. The socio-economic levels (SEL) of students were found to significantly affect their mathematical metacognition awareness but did not differentiate their metacognitive calibrations. The students' mathematics grades were found to significantly differentiate their mathematical metacognition awareness and metacognitive calibrations. There was no significant relationship between students' mathematical metacognition awareness and metacognitive calibrations. Another finding obtained from the study is that students tend to be better calibrated in medium and difficult mathematics questions compared to easy questions, and in difficult and easy questions, they tend to be excessively confident compared to medium difficulty questions.

(033) Artificial Intelligence ChatGPT and GPT4 Performance on Male and Female Sexual Dysfunction, Sexually Transmitted Infection, and Male Factor Infertility in the 2019 to 2023 American Urological Association Self-Assessment Study Programs

Abstract Introduction Large Language Model (LLM) applications in Medicine are increasing. Chatbots ChatGPT and GPT4 were tested against medical exams. The vetting of these tests as a source of information for both the patient and physician is needed. A credible source of medical information may be used as the gold standard against which the LLM can be tested. Objective We assess the performance of ChatGPT and GPT4 to answer the American Urological Association (AUA) Self-Assessment Study Program (SASP) questions on male sexual dysfunction (MSD), female sexual dysfunction (FSD), sexually transmitted infection (STI), and male factor infertility (MFI). We aim to find out how credible this LLM is as a source of medical advice. Methods Four registered users of the SASP identified the questions using open book mode in tests from 2019 to 2023, spanning their subscriptions. The questions were ranked for difficulty on a five-point Likert scale. OpenAI ChatGPT 3.5 and GPT4 were used to answer the questions. The GPT program was set to turn off chat history. No plug-ins or feedback were permitted. Prompts were generated from question stems masking question ID and adding the phrase "I am a urologist preparing for my board exam. Please answer the following question:" Images were deleted from questions and substituted with a brief description. All questions were answered separately, first using GPT-4 and then ChatGPT. Three consecutive responses were generated for each prompt, and the consensus answer was tallied. Answers were compared between GPT and provided answer key of SASP. Descriptive statistics, chi-Pearson Chi-square, and Fisher's exact tests were applied. Results We identified 115 questions in the domains of sexual dysfunction, STI, and MFI. Only one question had an associated image. GPT4 performed better than ChatGPT in all domains, providing correct answers of 60% versus 40% (p=0.000). A total of 89.9% of correct answers were three times unanimously regenerated by GPT4 (p=0.007) and 67.4% by ChatGPT (p=0.244). Within each domain, there was no significant difference in the correct answers for each chatbot (Table-1) or test year (p=0.682). The SASP source reference answers were Campbell's Urology only in 38.3 %, AUA guidelines only in 17.4%, AUA core curriculum only in 10.4%, and combinations with other sources in 33.9%. There was no significant association of source reference with the correct answers of GPT4 (p=0.058) or ChatGPT (p=0.451). Only 19.1% were open-access sources, 20% were partially open-access, and 60.9% were restricted to subscribers. The availability of the source did not significantly affect the correct answers of GPT4 (p=0.272) or ChatGPT (p=0.231). Both chatbots' correct answers were associated significantly with easier questions (Table-2). Conclusions The LLM tested has an average accuracy as a source of credible medical information on Sexual dysfunction, STI, and MFI. GPT4 performs better than ChatGPT, especially when there is a unanimous regenerated response. The development of a better model to serve a broader group of physicians and patients will require training of the chatbot on credible urology literature that includes the currently 81% restricted sources. Disclosure No.

Helmholtz and the Conservation of Energy: Contexts of Creation and Reception

Helmholtz and the Conservation of Energy: Contexts of Creation and Reception

Helmholtz and the Conservation of Energy: Contexts of Creation and Reception

Helmholtz and the Conservation of Energy: Contexts of Creation and Reception