Automated Feedback System Research Articles

Lessons learned from designing an open-source automated feedback system for STEM education

AbstractAs distance learning becomes increasingly important and artificial intelligence tools continue to advance, automated systems for individual learning have attracted significant attention. However, the scarcity of open-source online tools that are capable of providing personalized feedback has restricted the widespread implementation of research-based feedback systems. In this work, we present RATsApp, an open-source automated feedback system (AFS) that incorporates research-based features such as formative feedback. The system focuses on core STEM competencies such as mathematical competence, representational competence, and data literacy. It also allows lecturers to monitor students’ progress. RATsApp can be used at different levels of STEM education or research, as it allows the creation and customization of the educational content. We present a specific case of its implementation in higher education, where we report the results of a usability survey (N=64), using the technology acceptance model 2 (TAM2), to evaluate the user experience of undergraduate students. Our findings confirm the applicability of the TAM2 framework, revealing that factors such as the relevance to the course of study, output quality, and ease of use significantly influence the perceived usefulness. We also found a linear relation between the perceived usefulness and the intention to use, which in turn is a significant predictor of the frequency of use. Moreover, the formative feedback feature of RATsApp received positive feedback, indicating its potential as an educational tool. Furthermore, as an open-source platform, RATsApp encourages public contributions to its ongoing development, fostering a collaborative approach to improve educational tools.

Application of entertainment interactive robot based on speech recognition in English artificial intelligence teaching evaluation and automatic feedback

With the development of intelligent voice and interactive robot technology, new technologies have built a virtual E-learning learning environment that can provide students with an immersive learning experience, making this new learning mode more entertaining. This article investigates the application of entertainment interactive robots based on speech recognition in English artificial intelligence teaching evaluation and automatic feedback. The system has constructed an oral evaluation model based on deep reinforcement learning, which learns the optimal behavioral strategies through interaction with the environment. The model will train through oral conversations with learners to learn how to accurately evaluate oral proficiency and provide relevant feedback. After the construction of the system is completed, the accuracy and efficiency of the system are improved by adjusting the parameters of the model, increasing the diversity of training data, and improving the user interface and interaction mode based on user feedback, making it more friendly and easy to use. The experimental results show that the English oral evaluation and automatic feedback system designed in this paper based on deep reinforcement learning and speech recognition algorithms has high accuracy and efficiency. The system can accurately evaluate learners’ oral proficiency and provide personalized learning suggestions based on individual differences.

AI 기반 피드백의 교육적 활용 가능성 탐색

This study examines the potential of utilizing natural language processing technology to provide objective and efficient feedback in Korean language classrooms through an automated feedback system. To validate the effectiveness of AI-based automated feedback, the study compared corrections made by human annotators with those generated by the AI model, GPT4, using a corpus of beginner Korean language learners. The results indicated high concordance rates between human and GPT4 corrections, with Level 1 accuracy at 85.71% and Level 2 accuracy at 84.07%. Corrections primarily focused on grammatical corrections, averaging 61%, while lexical corrections comprised about 39%. Corrections across various parts of speech, including nouns, verbs, and adjectives, were performed at similar rates by both humans and GPT4. However, GPT4 showed limitations in appropriately correcting errors that required contextual understanding. Despite these limitations, the AI-based feedback system can be an effective supplementary tool in educational settings. Teachers can reduce their workload and focus more on content and structural feedback, while learners benefit from real-time, personalized feedback that encourages self-review and improves language proficiency. This study underscores the importance of AI-based educational tools and emphasizes the need for an integrated approach involving teacher collaboration to continually enhance the AI-based feedback system.

Histological evaluation of monopolar and bipolar radiofrequency microneedling treatment in a porcine model.

Fractional radiofrequency microneedling (FRM) is widely used as an option for skin rejuvenation, however there is a lack of histological evidence for the various energy delivery systems available. The objective was to assess thermal denaturation of tissue and the wound healing response in monopolar mode versus bipolar mode. Histological analysis was performed to demonstrate the efficacy of automatic impedance feedback system in monopolar mode. In this study, the acute thermal effects caused by monopolar FRM treatment to the dorsal skin of pigs were assessed histologically by hematoxylin & eosin (H&E) staining. Then, one session of either monopolar or bipolar FRM was used to treat one or the other side of the pig using varying power levels and pulse widths. The acute and chronic tissue reactions were assessed using H&E, immunofluorescence, and western blot analysis at 0, 14, 30, and 90 days after treatment. The efficacy of the impedance feedback system was also monitored histologically. High-energy FRM treatment produced tissue loss and necrosis. The power level and pulse duration significantly affected the coagulation amount. Histopathology at 0, 14, 30, and 90 days showed that the skin tissue reaction was more pronounced for bipolar compared to monopolar FRM. Immunofluorescence showed the expression of TGF-β, Ki67, MMP3, and elastin increased dramatically with both modes, but were higher in the bipolar FRM treated side. The automatic impedance feedback system could effectively adjust the output energy. We found that bipolar FRM produced greater thermal effects, more collagen coagulation, and more pronounced molecular changes compared with monopolar mode in a porcine animal model.

AI in teaching English writing: automatic scoring and feedback system

Abstract Artificial intelligence technology, with its excellent data processing and analyzing ability, has become a new method for scientifically evaluating the quality of students’ writing. In this paper, we construct an automatic scoring model for English writing using a hybrid neural network, then use a random forest algorithm to categorize the assignment utterances and design an automatic writing feedback system. The results of the teaching case show that the mean posttest writing score of the experimental class is 16.832, and the mean value of the posttest writing error rate is 7.278, both of which perform significantly better than the control class. Moreover, the assignments from the experimental class demonstrated a higher level of word performance. As the experiment progressed, the students’ perceptions of this paper’s system grew more positive. This underscores the significance of the research conducted in this paper.

PCIT에 참여하는 부모의 상호작용 기술 증진을 위한 인공지능 기반 자동 피드백 시스템 예비연구

PCIT에 참여하는 부모의 상호작용 기술 증진을 위한 인공지능 기반 자동 피드백 시스템 예비연구

In Search of Alignment between Learning Analytics and Learning Design: A Multiple Case Study in a Higher Education Institution

Learning design (LD) has increasingly been recognized as a significant contextual element for the interpretation and adoption of learning analytics (LA). Yet, few studies have explored how instructors integrate LA feedback into their learning designs, especially within open automated feedback (AF) systems. This research presents a multiple-case study at one higher education institution to unveil instructors’ pilot efforts in using an open AF system to align LA and LD within their unique contexts, with the goal of delivering personalized feedback and tailored support. A notable finding from these cases is that instructors successfully aligned LA with LD for personalized feedback through checkpoint analytics in highly structured courses. Moreover, they relied on checkpoint analytics as an evaluation mechanism for evaluating impact. Importantly, students perceived a stronger sense of instructors’ support, reinforcing previous findings on the effectiveness of personalized feedback. This study contributes essential empirical insights to the intersection of learning analytics and learning design, shedding light on practical ways educators align LA and LD for personalized feedback and support.

AI-facilitated reflective practice in physical education: an auto-assessment and feedback approach

ABSTRACT To improve students’ sports skills performance, it is important to engage them in reflective practice. However, in physical classes, a teacher generally needs to face a number of students, and hence it is almost impossible to provide detailed guidance or feedback to individual students. Scholars have been trying to use Artificial Intelligence (AI) technologies to provide personalized support to individual students in diverse courses; however, in terms of promoting physical skills, there is a lack of sufficient research or practices. The present study aimed to develop a yoga automatic assessment and feedback system using AI technology to provide personalized feedback to engage individual students in reflective practice. To examine the learning effectiveness of the developed system, a total of 96 students were assigned to be the experimental group and adopted the yoga automatic assessment and feedback system for learning, while a total of 91 students were in the control group and adopted the general online learning system. The results showed that adopting the yoga automatic assessment and feedback system for learning could significantly increase students’ yoga skills performance. In addition, it had positive effects on students’ skills learning, which was conducive to “promoting reflection,” “enhancing learning motivation,” and “obtaining feedback.”

Automatic Feedback System for X-ray Flux at BL08U1A Soft X-ray Spectromicroscopy Beamline of Shanghai Synchrotron Radiation Facility

An online automatic feedback system has been successfully installed and commissioned at the BL08U1A Soft X-ray Spectromicroscopy Beamline of Shanghai Synchrotron Radiation Facility, which can monitor the incident X-ray beam in real time by measuring the blade-edge signals of the exit slit and automatically adjust the elliptical cylindrical mirror parameters to achieve beam calibration and maintain the optimal X-ray flux of the sample. This work provides a comprehensive description of the hardware composition, system implementation, feedback logic, function and software design, system optimization and commission, as well as the online experimental results supported by the system. The experimental results demonstrated that the online automatic feedback system is capable of effectively maintaining the optimal X-ray beam flux for X-ray absorption spectroscopy experiments. Its success can provide valuable technique assistance for the design, construction and optimization of similar systems at various beamlines in synchrotron sources in the future.

Implementation of Automated Feedback System for Japanese Essays in Intermediate Education

Traditional Automated Essay Scoring (AES) only provides students with a holistic score, unable to provide meaningful feedback on students writing. Holistic, structure, style, word, and readability are chosen from the 6+1 writing-trait theory to create an Automated Essay Feedback (AEF) for Japanese L1 students. By combining these rule-based traits with a data-driven model, wecreated a hybrid system that can automatically grade and give feedback to students. The system automatically identifies parts of student writing that need improvement, then recommends corrective and suggestive feedback. Our contributions are twofold: design a 5-writing-trait AEF for Japanese L1 students and implement the holistic corrective writing-trait.

Involving psychological therapy stakeholders in responsible research to develop an automated feedback tool: Learnings from the ExTRAPPOLATE project

Understanding stakeholders’ views on novel autonomous systems in healthcare is essential to ensure these are not abandoned after substantial investment has been made. The ExTRAPPOLATE project applied the principles of Responsible Research and Innovation (RRI) in the development of an automated feedback system for psychological therapists, ‘AutoCICS’. A Patient and Practitioner Reference Group (PPRG) was convened over three online workshops to inform the system's development. Iterative workshops allowed proposed changes to the system (based on stakeholder comments) to be scrutinized. The PPRG reference group provided valuable insights, differentiated by role, including concerns and suggestions related to the applicability and acceptability of the system to different patients, as well as ethical considerations. The RRI approach enabled the anticipation of barriers to use, reflection on stakeholders’ views, effective engagement with stakeholders, and action to revise the design and proposed use of the system prior to testing in future planned feasibility and effectiveness studies. Many best practices and learnings can be taken from the application of RRI in the development of the AutoCICS system.

Teacher Training Effectiveness in Self-Regulation in Virtual Environments.

Higher education in the 21st century faces the challenge of changing the way in which knowledge is conveyed and how teachers and students interact in the teaching-learning process. The current pandemic caused by SARS-CoV-2 has hastened the need to face up to this challenge and has furthered the need to approach the issue from the perspective of digitalisation. To achieve this, it is necessary to design training programmes geared towards teaching staff and which address both the use of technology and instructional design aimed at promoting the development of self-regulated learning (SRL) and automatic feedback systems. In this study, work was carried out with 23 teachers (8 inexperienced and 15 experienced teachers) in a training programme conducted through Moodle. The aims were: (1) to test whether there were any significant differences between the behaviour patterns of new teachers compared to experienced teachers, (2) to determine whether clusters of behaviour patterns corresponded to the type of teacher and (3) to ascertain whether the level of teacher satisfaction with the training activity in digital teaching will depend on the type of teacher. A quantitative as well as a qualitative design was applied. Differences were found in the behaviour patterns in the training activities for the development of rubrics and use of learning analytics systems in virtual learning environments. It was also found that the type of teacher did not correspond exactly to the behaviour cluster in the learning platform. In addition, no significant differences were found in the level of satisfaction between the two kinds of teacher. The main contribution this study makes is to provide a detailed description of the training stage as well as the materials required for its repetition. Further analytical studies are required on teacher perception of training programmes in digital teaching in order to provide personalised training proposals that lead to an effective use of teaching in digital environments.

SG-ACCORD: Designing Virtual Agents for Soft Skills Training in the School Context

This study focuses on the behavior of interactive agents and how to ground their implementation in psychology and cognitive modeling to capture fundamental psycho-pedagogical elements to deliver an effective competency-based training and assessment tool for teachers, trainers, and educators working in the school context. The study describes the design and implementation of a Serious Game, “Attain Cultural Integration through COnflict Resolution skill Development” (SG-ACCORD), to develop and assess soft skills that are pivotal in educational working contexts (i.e., negotiation, effective communication, intercultural conflict management). The SG-ACCORD has been developed by applying a tailored methodology, namely the Educational Technology-enhanced Role-Playing, addressing two main facets: psycho-pedagogical (theoretical frameworks) and technological (production of artificial worlds based on computer-simulated models). The game is a single-player virtual role-play offering a series of scenarios reflecting the real-life experiences of teachers in which the role of interactions and human factors is crucial. It focuses on the simulation of a dialogue between two characters (an avatar controlled by a human player and a BOT computer-controlled counterpart), enacting interethnic conflicts between teacher and students within realistic school context scenarios. The SG-ACCORD represents a free, easily manageable, and flexible educational tool that can be used as a stand-alone resource and envisages an automatic feedback and debriefing system.

Deep Learning Assisted Detection of Abdominal Free Fluid in Morison's Pouch During Focused Assessment With Sonography in Trauma

Background: The use of focused assessment with sonography in trauma (FAST) enables clinicians to rapidly screen for injury at the bedsides of patients. Pre-hospital FAST improves diagnostic accuracy and streamlines patient care, leading to dispositions to appropriate treatment centers. In this study, we determine the accuracy of artificial intelligence model-assisted free-fluid detection in FAST examinations, and subsequently establish an automated feedback system, which can help inexperienced sonographers improve their interpretation ability and image acquisition skills.Methods: This is a single-center study of patients admitted to the emergency room from January 2020 to March 2021. We collected 324 patient records for the training model, 36 patient records for validation, and another 36 patient records for testing. We balanced positive and negative Morison's pouch free-fluid detection groups in a 1:1 ratio. The deep learning (DL) model Residual Networks 50-Version 2 (ResNet50-V2) was used for training and validation.Results: The accuracy, sensitivity, and specificity of the model performance for ascites prediction were 0.961, 0.976, and 0.947, respectively, in the validation set and 0.967, 0.985, and 0.913, respectively, in the test set. Regarding feedback prediction, the model correctly classified qualified and non-qualified images with an accuracy of 0.941 in both the validation and test sets.Conclusions: The DL algorithm in ResNet50-V2 is able to detect free fluid in Morison's pouch with high accuracy. The automated feedback and instruction system could help inexperienced sonographers improve their interpretation ability and image acquisition skills.

Machine Learning-Enabled Automated Feedback: Supporting Students’ Revision of Scientific Arguments Based on Data Drawn from Simulation

A design study was conducted to test a machine learning (ML)-enabled automated feedback system developed to support students’ revision of scientific arguments using data from published sources and simulations. This paper focuses on three simulation-based scientific argumentation tasks called Trap, Aquifer, and Supply. These tasks were part of an online science curriculum module addressing groundwater systems for secondary school students. ML was used to develop automated scoring models for students’ argumentation texts as well as to explore emerging patterns between students’ simulation interactions and argumentation scores. The study occurred as we were developing the first version of simulation feedback to augment the existing argument feedback. We studied two cohorts of students who used argument only (AO) feedback (n = 164) versus argument and simulation (AS) feedback (n = 179). We investigated how AO and AS students interacted with simulations and wrote and revised their scientific arguments before and after receiving their respective feedback. Overall, the same percentages of students (49% each) revised their arguments after feedback, and their revised arguments received significantly higher scores for both feedback conditions, p < 0.001. Significantly greater numbers of AS students (36% across three tasks) reran the simulations after feedback as compared with the AO students (5%), p < 0.001. For AS students who reran the simulation, their simulation scores increased for the Trap task, p < .001, and for the Aquifer task, p < 0.01. AO students who did not receive simulation feedback but reran the simulations increased simulation scores only for the Trap task, p < .05. For the Trap and Aquifer tasks, students who increased simulation scores were more likely to increase argument scores in their revisions than those who did not increase simulation scores or did not revisit simulations at all after simulation feedback was provided. This pattern was not found for the Supply task. Based on these findings, we discuss strengths and weaknesses of the current automated feedback design, in particular the use of ML.

Automatic feedback in online learning environments: A systematic literature review

Feedback is an essential component of scaffolding for learning. Feedback provides insights into the assistance of learners in terms of achieving learning goals and improving self-regulated skills. In online courses, feedback becomes even more critical since instructors and students are separated geographically and physically. In this context, feedback allows the instructor to customize learning content according to the students' needs. However, giving feedback is a challenging task for instructors, especially in contexts of large cohorts. As a result, several automatic feedback systems have been proposed to reduce the workload on the part of the instructor. Although these systems have started gaining research attention, there have been limited studies that systematically analyze the progress achieved so far as reported in the literature. Thus, this article presents a systematic literature review on automatic feedback generation in learning management systems. The main findings of this review are: (1) 65.07% of the studies demonstrate that automatic feedback increases student performance in activities; (2) 46.03% of the studies demonstrated that there is no evidence that automatic feedback eases instructors’ workload; (3) 82.53% of the studies showed that there is no evidence that manual feedback is more efficient than automatic feedback; and (4) the main method used for automatic feedback provision is the comparison with a desired answer in some subject (such as logic circuits or programming).

Hemodialysis crossover study using a relative blood volume change-guided ultrafiltration control compared with standard hemodialysis: the BV-UFC study

BackgroundIt has been difficult to sufficiently achieve body-fluid management using blood volume (BV) monitor during hemodialysis (HD) with constant ultrafiltration (UF) rate. Recently, a relative BV change-guided UF control (BV-UFC) system was developed by combining the concepts of an automatic feedback system that could control the UF rate and profile with real- time monitoring of relative changes in BV (%ΔBV). However, this system has limited application in the clinical setting. Therefore, in this study, we aimed to perform the crossover study on HD with BV-UFC compared to standard HD in terms of hemodynamic stability during HD.MethodsForty-eight patients entered an 8-week crossover period of standard HD or HD with BV-UFC. Prevalence of intradialytic hypotension (IDH) as a primary outcome and changes in blood pressure (BP), differences in %ΔBV, and achievement of the target ultrafiltration volume as secondary outcomes were compared. IDH was defined as a reduction in systolic BP ≥20 mmHg from the baseline value at 10 min after HD initiation.ResultsNo significant differences were found in the prevalence of IDH, frequency of intervention for symptomatic IDH, and achievement of the target ultrafiltration volume between the groups. The %ΔBV was significantly fewer (-12.1 ± 4.8% vs. -14.4 ± 5.2%, p <0.001) in the HD with BV-UFC than that in the standard HD.ConclusionsHD with BV-UFC did not reduce the prevalence of IDH compared with standard HD. The relief of a relative BV reduction at the end of HD may be beneficial in patients undergoing HD with BV-UFC.Trial RegistrationUMIN, UMIN000024670. Registered on December 1, 2016.

System providing automated feedback improves task learning outcomes during child restraint system (CRS) installations

Objectives The objective was to build and test an automated, interactive educational system to teach adults how to install a child restraint system (CRS) into a vehicle seat. Methods The automated feedback system (AFS) consisted of a mockup vehicle fixture, convertible CRS, and doll. Sensors were implemented into the equipment so that forward-facing (FF) CRS installation errors could be detected. An interactive display monitor guided users through the CRS installation process and alerted them when steps were done incorrectly. Sixty adult volunteers were recruited and randomized into either the treatment group or the control group. The treatment group used the AFS to guide them through a practice installation. The control group also completed a practice installation using the same equipment fixture without the feedback feature turned on; they only had standard printed instruction manuals to guide their tasks. Then, participants from both groups completed a second CRS installation in a real vehicle with standard instruction manuals only. The frequencies and types of errors in all the installations were evaluated by a Child Passenger Safety Technician (CPST). Error rates were compared between the treatment and control groups using lower-tailed t-tests and Pearson’s chi-square tests. Error rates were evaluated considering minor and serious errors together and also considering serious errors alone. Results Compared to the control group, participants who trained with the AFS exhibited fewer overall errors (minor and serious) in their fixture installations (p < 0.0001) as well as their follow-up vehicle installations (p < 0.0001). Specifically, participants in the treatment group had fewer errors in choosing an installation method, locking the seat belt (SB), tightening the SB or lower anchors (LA), and tightening the harness (p = 0.0002, p = 0.0003, p = 0.0084, and p = 0.0098, respectively, compared to control group during follow-up vehicle installations). The treatment group also performed significantly better than the control group when only serious errors were considered. Conclusions An automated feedback system is an effective way to teach basic CRS installation skills to users.

Acoustic activity of Litopenaeus vannamei fed pelleted and extruded diets

Passive acoustics monitoring (PAM) is an exceptional tool to analyze the feeding behavior of shrimp, as they emit a “click” sound by the mandibular occlusion. Hence, acoustic demand-feeding systems based on the click signals energy have been applied to improve feeding management in shrimp farming. These acoustic signals may be affected by the texture properties of the foods pellets, which are modified by the manufacturing process. The aim of the present study was to evaluate the acoustic activity of Litopenaeus vannamei fed pelleted and extruded diets. Two commercial diets were used, manufactured by extrusion and pelleting processing with similar formulation (36% crude protein) and pellet diameter. Acoustic evaluation was performed with these diets offered “as is” dry and previously soaked in water for one and two hours in a 2 × 3 factorial arrangement. The pellet hardness and moisture content were also measured. The acoustic trial was performed using omnidirectional hydrophones inside acoustic foam anechoic chambers. Three fastened shrimp (10 g) were placed inside acoustic chambers and audio recordings started (30 min) when food (0.5 g) was offered. The texture of diets was closely related to the acoustic intensity (Root Mean Square Amplitude - RMS) produced by L. vannamei during the feeding activity. The greater hardness of the extruded diet resulted in higher RMS than pelleted food when offered dry, which could be an advantage for the sound detection of feeding activity. The lower RMS observed when both diets were soaked suggests that softer texture pellets can be eventually ingested without the need of mandibles occlusion, and consequently without clicking sound emission. Additionally, unusual clicks train signals were first reported when shrimp fed soaked diets, probably due to mandibles rubbing. The relationship between textural properties and acoustical responses of foods should gain more attention, especially considering the potential applications on automatic acoustic feedback systems in shrimp farming.

Using spoken language technology for generating feedback to prepare for the TOEFL iBT® test: a user perception study

ABSTRACT Recent technology advancements have increased the prospects for automated spoken language technology to provide feedback on speaking performance. In this study we examined user perceptions of using an automated feedback system for preparing for the TOEFL iBT® test. Test takers and language teachers evaluated three types of machine-generated feedback: domain sub-scores, task scores, and scores based on individual linguistic features. Results showed that the vast majority of the participants perceived all feedback types to be useful, although the teachers showed slightly more scepticism than the students. We believe that these findings provide initial support for making such feedback available to those preparing for the TOEFL iBT test or other similar tests of spoken language proficiency. Study outcomes also inform our understanding of the role of technology in test preparation and the extent to which the use of technology changes preparation behaviours and preferences.