Instructional Prompts Research Articles

Automatic generation of physics items with Large Language Models (LLMs)

High-quality items are essential for producing reliable and valid assessments, offering valuable insights for decision-making processes. As the demand for items with strong psychometric properties increases for both summative and formative assessments, automatic item generation (AIG) has gained prominence. Research highlights the potential of large language models (LLMs) in the AIG process, noting the positive impact of generative AI tools like ChatGPT on educational assessments, recognized for their ability to generate various item types across different languages and subjects. This study fills a research gap by exploring how AI-generated items in secondary/high school physics aligned with educational taxonomy. It utilizes Bloom's taxonomy, a well-known framework for designing and categorizing assessment items across various cognitive levels, from low to high. It focuses on a preliminary assessment of LLMs ability to generate physics items that match the Bloom’s taxonomy application level. Two leading LLMs, ChatGPT (GPT-4) and Gemini, were chosen for their strong performance in creating high-quality educational content. The research utilized various prompts to generate items at different cognitive levels based on Bloom's taxonomy. These items were assessed using multiple criteria: clarity, accuracy, absence of misleading content, appropriate complexity, correct language use, alignment with the intended level of Bloom's taxonomy, solvability, and assurance of a single correct answer. The findings indicated that both ChatGPT and Gemini were skilled at generating physics assessment items, though their effectiveness varied based on the prompting methods used. Instructional prompts, particularly, resulted in excellent outputs from both models, producing items that were clear, precise, and consistently aligned with the Application level of Bloom's taxonomy.

Comparison of the Quality of Discharge Letters Written by Large Language Models and Junior Clinicians: Single-Blinded Study.

Discharge letters are a critical component in the continuity of care between specialists and primary care providers. However, these letters are time-consuming to write, underprioritized in comparison to direct clinical care, and are often tasked to junior doctors. Prior studies assessing the quality of discharge summaries written for inpatient hospital admissions show inadequacies in many domains. Large language models such as GPT have the ability to summarize large volumes of unstructured free text such as electronic medical records and have the potential to automate such tasks, providing time savings and consistency in quality. The aim of this study was to assess the performance of GPT-4 in generating discharge letters written from urology specialist outpatient clinics to primary care providers and to compare their quality against letters written by junior clinicians. Fictional electronic records were written by physicians simulating 5 common urology outpatient cases with long-term follow-up. Records comprised simulated consultation notes, referral letters and replies, and relevant discharge summaries from inpatient admissions. GPT-4 was tasked to write discharge letters for these cases with a specified target audience of primary care providers who would be continuing the patient's care. Prompts were written for safety, content, and style. Concurrently, junior clinicians were provided with the same case records and instructional prompts. GPT-4 output was assessed for instances of hallucination. A blinded panel of primary care physicians then evaluated the letters using a standardized questionnaire tool. GPT-4 outperformed human counterparts in information provision (mean 4.32, SD 0.95 vs 3.70, SD 1.27; P=.03) and had no instances of hallucination. There were no statistically significant differences in the mean clarity (4.16, SD 0.95 vs 3.68, SD 1.24; P=.12), collegiality (4.36, SD 1.00 vs 3.84, SD 1.22; P=.05), conciseness (3.60, SD 1.12 vs 3.64, SD 1.27; P=.71), follow-up recommendations (4.16, SD 1.03 vs 3.72, SD 1.13; P=.08), and overall satisfaction (3.96, SD 1.14 vs 3.62, SD 1.34; P=.36) between the letters generated by GPT-4 and humans, respectively. Discharge letters written by GPT-4 had equivalent quality to those written by junior clinicians, without any hallucinations. This study provides a proof of concept that large language models can be useful and safe tools in clinical documentation.

Evaluating and Enhancing Large Language Models' Performance in Domain-Specific Medicine: Development and Usability Study With DocOA.

The efficacy of large language models (LLMs) in domain-specific medicine, particularly for managing complex diseases such as osteoarthritis (OA), remains largely unexplored. This study focused on evaluating and enhancing the clinical capabilities and explainability of LLMs in specific domains, using OA management as a case study. A domain-specific benchmark framework was developed to evaluate LLMs across a spectrum from domain-specific knowledge to clinical applications in real-world clinical scenarios. DocOA, a specialized LLM designed for OA management integrating retrieval-augmented generation and instructional prompts, was developed. It can identify the clinical evidence upon which its answers are based through retrieval-augmented generation, thereby demonstrating the explainability of those answers. The study compared the performance of GPT-3.5, GPT-4, and a specialized assistant, DocOA, using objective and human evaluations. Results showed that general LLMs such as GPT-3.5 and GPT-4 were less effective in the specialized domain of OA management, particularly in providing personalized treatment recommendations. However, DocOA showed significant improvements. This study introduces a novel benchmark framework that assesses the domain-specific abilities of LLMs in multiple aspects, highlights the limitations of generalized LLMs in clinical contexts, and demonstrates the potential of tailored approaches for developing domain-specific medical LLMs.

Contextualized and Personalized Math Word Problem Generation in Authentic Contexts Using Generative Pre-trained Transformer and Its Influences on Geometry Learning

Recently, automatic question generation (AQG) has been researched extensively for educational purposes. Existing approaches generally lack relevant information on the authentic context and problem diversity with various difficulty levels, so we proposed a new AQG system for generating contextualized and personalized mathematic word problems (MWP) in authentic contexts using the Generative Pre-trained Transformers (GPT). Our proposed system comprises (1) authentic contextual information acquisition through image recognition by TensorFlow and augmented reality (AR) measurement by AR Core, (2) a personalized mechanism based on instructional prompts to generate three different difficulty levels for learners’ different needs, and (3) MWP generation through GPT with authentic contextual information and personalized needs. We conducted a quasi-experiment with the participation of 52 students to evaluate the effectiveness of the proposed system on geometry learning performance. Further, the learning behaviors were analyzed in the aspects of authentic context, mathematics, and reflective behavior. The findings showed better results in geometry learning performances from students who learned with contextualized and personalized MWPs than those who were taught without contextualization and personalization on MWPs. Moreover, it was found that student’s ability to comprehend the practical situation or scenario presented in a problem (problem context understanding) and students’ ability to recognize relevant information from the problem context (identifying contextual information) significantly improved their learning performance. Moreover, students’ ability to apply math concepts and solve medium-level MWP also contributes to the improvement of learning performance. Meanwhile, learners showed positive perceptions toward the proposed system in facilitating geometry learning. Therefore, it is useful to promote an authentic context setting for mathematical problem-solving.

An Investigation into the Utility of Large Language Models in Geotechnical Education and Problem Solving

The study explores the capabilities of large language models (LLMs), particularly GPT-4, in understanding and solving geotechnical problems, a specialised area that has not been extensively examined in previous research. Employing a question bank obtained from a commonly used textbook in geotechnical engineering, the research assesses GPT-4’s performance across various topics and cognitive complexity levels, utilising different prompting strategies like zero-shot learning, chain-of-thought (CoT) prompting, and custom instructional prompting. The study reveals that while GPT-4 demonstrates significant potential in addressing fundamental geotechnical concepts and problems, its effectiveness varies with specific topics, the complexity of the task, and the prompting strategies employed. The paper categorises errors encountered by GPT-4 into conceptual, grounding, calculation, and model inherent deficiencies related to the interpretation of visual information. Custom instructional prompts, specifically tailored to address GPT-4’s shortcomings, significantly enhance its performance. The study reveals that GPT-4 achieved an overall problem-solving accuracy of 67% with custom instructional prompting, significantly higher than the 28.9% with zero-shot learning and 34% with CoT. However, the study underscores the importance of human oversight in interpreting and verifying GPT-4’s outputs, especially in complex, higher-order cognitive tasks. The findings contribute to understanding the potential and limitations of current LLMs in specialised educational fields, providing insights for educators and researchers in integrating AI tools like GPT-4 into their teaching and problem-solving approaches. The study advocates for a balanced integration of AI in education to enrich educational delivery and experience while emphasising the indispensable role of human expertise alongside technological advancements.

Promoting pre-service teachers’ knowledge integration from multiple text sources across domains with instructional prompts

Multiple document comprehension and knowledge integration across domains are particularly important for pre-service teachers, as integrated professional knowledge forms the basis for teaching expertise and competence. This study examines the effects of instructional prompts and relevance prompts embedded in pre-service teachers’ learning processes on the quality their knowledge integration in multiple document comprehension across domains. 109 pre-service teachers participated in an experimental study. They read four texts on “competencies” from different knowledge domains and wrote a text on a given scenario. Experimental group 1 was aided with instructional and relevance prompts, while experimental group 2 received only relevance prompts. The control group received no prompting. Perceived relevance of knowledge integration was assessed in a pre-post-test. Pre-service teachers’ separative and integrative learning, epistemological beliefs, metacognition, study-specific self-concept, and post-experimental motivation were assessed as control variables. Participants’ texts were analyzed concerning knowledge integration by raters and with computer linguistic measures. A key finding is that combined complex prompting enhances pre-service teachers perceived relevance of knowledge integration. This study found effects of prompting types on the pre-service teachers’ semantic knowledge structures. Implications for transfer are discussed.

Bridging domains: examining the effects of relevance instructions and guiding questions on pre-service teachers’ first- and second-order knowledge integration

The conceptualization of pre-service teachers’ knowledge integration typically involves the distinction of two types: first-order knowledge integration, which includes merging domain-specific knowledge entities into a common knowledge base, and second-order knowledge integration, which refers to the integrated (simultaneous) application of knowledge from diverse domains. This study investigates the effect of instructional prompts in the form of (a) relevance instructions and (b) guiding questions on promoting pre-service teachers’ first-order knowledge integration in a reading- and writing-based learning setting with three domain-specific study texts: one text each referring to content knowledge (CK), general pedagogical knowledge (PK), or pedagogical-content knowledge (PCK). Furthermore, the study explores whether pre-service teachers’ second-order knowledge integration depends on the degree to which they engaged in first-order knowledge integration when reading and writing about different domain-specific learning contents. The study applied a three parallel group experimental design. An analysis of essays written by N = 83 German language pre-service teachers indicated positive effects of both prompts on first-order knowledge integration. Moreover, a mediation analysis showed that pre-service teachers’ second-order knowledge integration is mediated by their first-order knowledge integration. The results are discussed and integrated into the existing body of research, practical implications are presented, and limitations of the study are explained.

Open modelling problems: cognitive barriers and instructional prompts

Open mathematical modelling problems that can be solved with multiple methods and have multiple possible results are an important part of school curricula in mathematics and science. Solving open modelling problems in school should prepare students to apply their mathematical knowledge in their current and future lives. One characteristic of these problems is that information that is essential for solving the problems is missing. In the present study, we aimed to analyze students’ cognitive barriers while they solved open modelling problems, and we evaluated the effects of instructional prompts on their success in solving such problems. A quantitative experimental study (N = 263) and a qualitative study (N = 4) with secondary school students indicated that identifying unknown quantities and making numerical assumptions about these quantities are important cognitive barriers to solving open modelling problems. Task-specific instructional prompts helped students overcome these barriers and improved their solution rates. Students who were given instructional prompts included numerical assumptions in their solutions more often than students who were not given such prompts. These findings contribute to theories about solving open modelling problems by uncovering cognitive barriers and describing students’ cognitive processes as they solve these problems. In addition, the findings contribute to improving teaching practice by indicating the potential and limitations of task-specific instructional prompts that can be used to support students’ solution processes in the classroom.

A new way to co-play with digital media: Evaluating the role of instructional prompts on parent–child interaction quality.

A new way to co-play with digital media: Evaluating the role of instructional prompts on parent–child interaction quality.

Seductive details hamper learning even when they do not disrupt

Previous research often revealed detrimental effects of seductive details on learning with multimedia instruction, but there are mixed findings regarding how to best explain these detrimental effects. We investigated whether the detrimental effects of seductive details are mainly mediated by the cognitive processes of diversion (deeper processing of seductive details rather than pertinent content) or disruption (unsuccessful attempts to integrate seductive details with pertinent content) by assessing the effects of instructional prompts. In an online learning experiment, participants (N = 247) learned either without seductive details (control condition) or with seductive details in one of three conditions: Participants received either a prompt informing them about the irrelevance of seductive details (irrelevance-prompt), a prompt to process seductive details and pertinent content separately (separation-prompt), or no prompt within their task instruction. We assessed recall and transfer of knowledge as dependent variables. Supporting the diversion hypothesis, participants in the no-prompt condition regarded seductive details as more relevant and consequently spent more time processing them compared to participants in the irrelevance-prompt condition, which negatively influenced their recall performance. Against the disruption hypothesis, participants in the no-prompt condition reported lower integration avoidance between seductive details and pertinent content compared to participants in the separation-prompt condition, but this led to better rather than worse transfer performance. Our results thus suggest diversion, and not disruption, to be the main process driving the seductive details effect. Reducing the details’ diverting potential seems a good way to deal with seductive details in instruction.

Why and How Electronic Job Coaches Improve Employment for People with Disabilities

Even though the 2014 Workforce Innovation and Opportunities Act strengthens federal and state commitments to employing people with disabilities, there remain millions of Americans with mild to moderate intellectual disabilities (ID) who are still unemployed. To succeed in competitive employment, workers with ID require help with memory, task sequence and planning, and time management; skills that are often referred to as executive functioning. Typically, employers do not have the training to manage people with ID with these challenges. This task falls to job coaches, job developers or employment specialists (e.g., providers) and the agencies that provide such services. Without effective and efficient tools to support the executive functioning of individuals with ID, policies that aim to affect their competitive integrated employment can result in an unmanageable burden on the provider. Today’s jobs are more complex. Providers need tools that can help them effectively and efficiently build, record and adjust prompts and instructions as changes occur in real-time. However, they may not have the time and/or the skills required to make clear, accurate instructional prompts for multiple workers with ID in different settings. To identify the needs of employers, providers, and workers with ID, a NIDILRR-funded project collected data from supervisors, employers and providers of employment services to workers with ID. These interviews were used to determine the requirements of an ideal system that would help them better instruct, coach, track progress and manage teams of employees with ID. This brief describes their findings and development of a cloud-based interactive electronic job coaching app that allows emplyment services providers to efficiently convert work assignments into the essential details required by the mobile worker with ID, and then wirelessly transfer the appropriate instructions to the worker’s electronic coach.

Socratic Video Understanding on Unmanned Aerial Vehicles

In this work, we propose a system for video understanding through zero-shot reading comprehension using Socratic Models. Specifically, we create a language-based world-state history of events and objects present in a scene captured by an Unmanned Aerial Vehicle (UAV). To achieve this, video footage from RYZE Tello microdrones is transmitted to a ground computer for further processing. The semantically rich information offered by Large Language Models (LLMs) enables open-ended reasoning, such as event forecasting with minimal human intervention, in a cost-effective robotic system. BLIP-2 is employed to answer a given set of instructional prompts, creating a log-state of objects, humans, and hazards that can be searched. Simultaneously, it suggests probable actions in the scene and can assist the human controller with an estimated best command. The BLIP-2 instructional prompts are then combined with OpenAI's da-vinci-003/gpt-3.5-turbo to generate comprehensive video descriptions and summarize likely actions. The LLM-enhanced generated texts achieve a GUNNING Fog median grade level in the range of 7-12.

Supplemental Material for A New Way to Co-Play With Digital Media: Evaluating the Role of Instructional Prompts on Parent–Child Interaction Quality

Supplemental Material for A New Way to Co-Play With Digital Media: Evaluating the Role of Instructional Prompts on Parent–Child Interaction Quality

Measuring adaptive teaching in classroom discourse: Effects on student learning in elementary science education

Adaptive teaching is considered fundamental to teaching quality and student learning. It describes teachers’ practices of adjusting their instruction to students’ diverse needs and levels of understanding. Adaptive teaching on a micro level has also been labeled as contingent support and has been shown to be effective in one-to-one and small-group settings. In the literature, the interplay of teachers’ diagnostic strategies and instructional prompts aiming at tailored support are emphasized. Our study adds to this research by presenting a reliable measurement approach to adaptive classroom discourse in elementary science which includes a global index and the single indices of diagnostic strategies, instructional support, and student understanding. Applying this coding scheme, we investigate whether N = 17 teachers’ adaptive classroom discourse predicts N = 341 elementary school students’ conceptual understanding of “floating and sinking” on two posttests. In multilevel regression analyses, adaptive classroom discourse was shown to be effective for long-term student learning in the final posttest, while no significant effects were found for the intermediate posttest. Further, the single index of diagnostic strategies in classroom discourse contributed to long-term conceptual restructuring. Overall, teachers rarely acted adaptively which points to the relevance of teacher professional development.

Our Infrastructural Loves

Embracing a critical pedagogy of care and support in the teaching and learning environment of the architectural design studio, this essay discusses ways in which architecture can be reimagined via infrastructural love. By exploring infrastructural support systems we seek other ways of situating architectural design amidst contemporary environmental, social, and political crises. We have undertaken this work in collaboration with architectural design students by deploying a feminist ethics of care that combines interdisciplinary theories and practices. Our essay unfolds as an infrastructural rhythm of instructional prompts. In conclusion we present a concept that emerged from our studio: poetics pragmatics.

Contrastive Video Examples in Teacher Education: A Matter of Sequence and Prompts

Everyday teaching requires teachers to deal with a variety of pedagogical issues, such as classroom disruptions. Against the background of on-going calls for an evidence-informed practice, teachers should ground their pedagogical decisions not only on subjective theories or experience-based knowledge but also on educational theories and empirical findings. However, research suggests that pre- and in-service teachers rather refer to experiential knowledge than to educational knowledge when addressing practical, pedagogical issues. One reason for the infrequent use of educational knowledge is that acquired knowledge has remained inert and cannot be applied to complex situations in practice. Therefore, implementing learning with contrastive (i.e., functional and dysfunctional) video examples in teacher education seems promising to promote pre-service teachers’ acquisition of educational knowledge. The 2×2-intervention study (N= 220) investigated the effects of the video sequence (dysfunctional-functional/functional-dysfunctional) and of video analysis prompts (with/without) on learning outcomes (concept knowledge, application knowledge) and on learning processes (written video analyses). Results revealed that the sequence dysfunctional-functional led to higher application knowledge in the post-test. There was no sequencing effect on concept knowledge. Prompted groups showed higher concept knowledge and application knowledge in the post-test. Furthermore, both experimental factors affected learning processes, which resulted in higher learning outcomes. In conclusion, learning with contrastive video examples in teacher education seems to be more effective if the video examples are presented in the sequence dysfunctional-functional and if instructional prompts guide the video analysis. The results substantiate the relevance of instructional guidance in learning with video examples and broaden the scope of validity of the concept of learning from errors.

Understanding self-regulated learning in Singapore’s social studies classrooms

ABSTRACT Self-regulated learning, reflective thinking, and meta-cognition have been given greater emphasis in Singapore’s education system as central to the development of 21st century competencies and lifelong learning capacities. This study explores students’ perceptions of their Self-Regulated Learning (SRL) strategies in Secondary Social Studies after engaging in a Computer-Based Learning Environment (CBLE) lesson with online information sources. Semi-structured group interviews were used to tease out the SRL strategies utilised by students from two different schools. SRL strategies were embedded in the CBLE as instructional prompts created for the purpose of assisting students during task completion. Students were then asked to reflect and comment on their work to identify SRL strategies they utilised in their learning. Findings demonstrate a range of SRL behaviours conveyed by students in the areas of planning for learning tasks, monitoring their learning, and reflection on their learning. The study highlights the importance of SRL strategies in Social Studies education to improve student learning.

How drawing prompts can increase cognitive engagement in an active learning engineering course

AbstractBackgroundRecent engineering education research has found improved learning outcomes when instructors engage students actively (e.g., through practice problems) rather than passively (e.g., in lectures). As more instructors shift toward active learning, research needs to identify how different types of activities affect students' cognitive engagement with concepts in the classroom. In this study, we investigate the effects of prompting novice students to draw when solving problems, a professional practice of engineers.PurposeWe investigate whether implementing instructional prompts to draw in an active learning classroom (a) increases students' use and value of drawing as a problem‐solving strategy and (b) enhances students' problem‐solving performance.MethodWe compared survey data and exam scores collected in one undergraduate class that received prompts to draw in video lectures and in‐class problems (drawing condition) and one class that received no drawing prompts (control condition).ResultsAfter drawing prompts were implemented, students' use and value of drawing increased, and these effects persisted to the end of the semester. Students were more likely to draw when provided drawing prompts. Furthermore, students who received prompts outperformed students who did not on exam questions that target conceptual understanding.ConclusionsOur findings reveal how implementing drawing prompts in an active learning classroom may help students engage in drawing and solve problems conceptually. This study contributes to our understanding of what types of active learning activities can improve instructional practices in engineering education. Particularly, we show how prompts that foster authentic engineering practices can increase cognitive engagement in introductory‐level engineering courses.

Evaluating the Effects of Instructional Prompts and Strategic Incremental Rehearsal on the Letter Identification Mastery of Two Typically Developing Kindergarteners.

Letter identification impacts the developmental progression of future reading skills. In this study, we evaluated the effects of instructional prompts on letter identification in a single-subject design for 2 kindergarten students who experienced difficulties with letter identification skills. In Phase 1, we evaluated the number of letters participants correctly identified without instructional prompts, and in Phase 2, we evaluated the effects of 2 different instructional prompts and rewards. Phase 3 evaluated the combined effects of strategic incremental rehearsal (SIR) and the most effective instructional prompt identified in Phase 2. Results showed that an effective, yet different instructional prompt was identified for each participant and that SIR plus an instructional prompt was effective for further increasing both participants' correct letter identification. The results of this study demonstrated a useful and effective method for identifying instructional prompts, strategies, or other interventions that improve students' academic responding.

Designing a scaffold for mechanistic reasoning in organic chemistry

Abstract Designing problems and learning activities is a key factor to initiating students’ engagement with the course material and influencing their reasoning processes. Although tasks and problems are a central part of teaching and assessments in the chemistry classroom, they may not engage students in deep reasoning or in a way that is intended through a task. Some problems may cause an algorithmic or a surface approach. Even with designing clever problems, students may not use a larger variety of chemistry ideas and connect them in meaningful ways. Here the idea of scaffolding students’ answering process comes into play. Structuring students’ reasoning process through instructional prompts or structured worksheets supports students in activating and connecting knowledge pieces in a more meaningful way and positively slows down their fast decision-making process. This paper will discuss the importance of asking questions in chemistry teaching and highlights the idea of contrasting cases, drawn from cognitive psychology, as a task design principle. In addition to having contrasting cases as a good problem format, the idea of scaffolding students’ reasoning while solving contrasting cases through the use of instructional prompts that scaffold the reasoning process will be exemplarily showcased for mechanistic reasoning in organic chemistry.