Automated Scoring System Research Articles

NIMG-63. LEVERAGING LLMS FOR ACCURATE DIFFERENTIATION OF RADIATION NECROSIS AND TUMOR PROGRESSION IN BRAIN MRI REPORTS: A STUDY ON AUTOMATED SCORING AND CLINICAL IMPLICATIONS

Abstract BACKGROUND Accurate differentiation between radiation necrosis and tumor progression or recurrence in patients treated with stereotactic radiosurgery for brain metastases is critical for guiding clinical management. This study leverages the advanced natural language processing capabilities of Meta Llama3, an artificial intelligence (AI) large language model (LLM), combined with prompt engineering, to rapidly categorize brain magnetic resonance imaging (MRI) radiology reports. Our objective was to develop an automated scoring system to classify concern for radiation necrosis, tumor progression, equivocal findings, or stable exams. METHODS Using a comprehensive dataset of reports annotated by expert radiologists, we ran inference on a 70-billion parameter Llama3 model (temperature 0.2, top_p 0.9) with specific prompts designed to capture the nuanced language and diagnostic criteria related to radiation necrosis or tumor progression. RESULTS The first pass was performed on a training dataset of 107 reports and did not predefine the clinical conditions. This demonstrated 43.4% accuracy in scoring when compared to a human user’s classification of each report. Agreement between the human reader and Llama3 was assessed using the Gwet agreement coefficient, AC1=0.411 (99%CI 0.396-0.426). Multiple iterations of targeted prompt engineering were then employed to narrow the definition of radiation necrosis and tumor progression, with specific examples and nuanced language used to achieve a higher degree of accuracy at 72.0%, AC1=0.719 (99%CI 0.717-0.722). DISCUSSION/FUTURE DIRECTIONS This surpasses recent demonstration of lower human-LLM agreement in radiographic score assignment, with further room for calibration on a dataset of several thousand reports. Next, we will correlate automated interpretations with actual clinical management decisions for radiation necrosis (e.g., initiation of steroids, bevacizumab, Laser Interstitial Thermal Therapy, and/or repeat imaging). This automated scoring system holds significant potential for LLMs in clinical applications. Future work will focus on integrating this model into clinical workflows and expanding its capabilities to include longitudinal monitoring of patient outcomes.

Exploring feasibility of vision-based automated evaluation during laboratory courses in manufacturing

Manual assessment of laboratory exercises and assignments by human instructors is subjective and time-consuming, introducing errors, inconsistencies, and biases. A preliminary study developing an automated evaluation system utilizing computer vision has been proposed in this article to address these challenges. It utilizes computer vision to assess the accuracy and quality of components machined by students enrolled during the manufacturing laboratory course. The system includes image acquisition hardware, algorithms for objective decisions, and an interface for evaluating student performance. The system is implemented during one of the laboratory classes, and a comparative assessment is carried out with manual evaluations followed by student feedback. It has been shown that the system can address subjectivity concerns, reducing the workload on course instructors and teaching assistants. The research also broadens the utility of computer vision in manufacturing education and creates interest among enrolled students to appreciate the role of newer technologies in a core discipline.

A Computer Vision System for an Automated Scoring of a Hand-drawn Geometric Figure

Visual Reproduction is a condition to measure Visual Spatial Memory as one of the cognitive domains commonly used to measure visuo-spatial memory. Geometric figures serve as stimulus material, and probands have to reproduce the figures from memory through a hand drawing. The scoring of the drawing has subjective elements. This study aims to evaluate the scoring criteria for the Figural Reproduction Test (FRT), part of the Indonesian Neuropsychological Test Battery, and to develop and evaluate an automated scoring system based on computer vision technology (FRT-CVAS). Scoring evaluation conducted by Cohen Kappa analysis, accuracy, sensitivity, and specificity. The analyzes of the three criteria of the manual confirmed a subjective element in the scoring of the shape of triangles by a moderate (0.74) inter-rater agreement; this agreement could be improved to 0.84 by a slight modification of its criteria. FRT-CVAS, based on computer vision’s identification of the different elements of the hand drawing, was developed and trained using 290 drawings. The system was additionally tested by comparing its scoring with the scoring of two independent raters on 120 drawings from a second data set. FRT-CVAS recognized all elements, and its comparison between human raters showed a high accuracy and sensitivity (minimally 0.91), while the specificity was 0.80 for one of the three criteria. FRT-CVAS offers a highly standardized, consistent, precise, and objective output from the first card in the FRT. This approach is advantageous to data-hungry alternatives such as deep learning when applied to the automated scoring of hand drawings with relatively little data available for training.

NLP and OCR based Automatic Answer Script Evaluation System

NLP and OCR based Automatic Answer Script Evaluation System

Analysis and comparison of retinal vascular parameters under different glucose metabolic status based on deep learning.

To develop a deep learning-based model for automatic retinal vascular segmentation, analyzing and comparing parameters under diverse glucose metabolic status (normal, prediabetes, diabetes) and to assess the potential of artificial intelligence (AI) in image segmentation and retinal vascular parameters for predicting prediabetes and diabetes. Retinal fundus photos from 200 normal individuals, 200 prediabetic patients, and 200 diabetic patients (600 eyes in total) were used. The U-Net network served as the foundational architecture for retinal artery-vein segmentation. An automatic segmentation and evaluation system for retinal vascular parameters was trained, encompassing 26 parameters. Significant differences were found in retinal vascular parameters across normal, prediabetes, and diabetes groups, including artery diameter (P=0.008), fractal dimension (P=0.000), vein curvature (P=0.003), C-zone artery branching vessel count (P=0.049), C-zone vein branching vessel count (P=0.041), artery branching angle (P=0.005), vein branching angle (P=0.001), artery angle asymmetry degree (P=0.003), vessel length density (P=0.000), and vessel area density (P=0.000), totaling 10 parameters. The deep learning-based model facilitates retinal vascular parameter identification and quantification, revealing significant differences. These parameters exhibit potential as biomarkers for prediabetes and diabetes.

Tree narrative and AI interaction: Optimization and narrative bias handling prospects

This paper reviews the current research on interactive narratives, particularly the tree narrative structure, analyzing the methods of player-AI interaction, the design of narrative branches, and the mechanisms by which player decisions influence the story's progression. It explores the definitions, classifications, and applications of tree narratives in gaming, emphasizing the significance of player-AI interaction and its impact on narrative outcomes. The study identifies and analyzes potential narrative deviations resulting from player-AI interactions within tree structures and introduces an AI-assisted automatic scoring system based on hypothesis testing and Bayesian analysis to effectively identify narrative biases.

52246 Application of the Deep learning-based automated scoring system to predict eyelid Monopolar Radiofrequency treatment outcome: A Retrospective, Multicenter, Preliminary study

52246 Application of the Deep learning-based automated scoring system to predict eyelid Monopolar Radiofrequency treatment outcome: A Retrospective, Multicenter, Preliminary study

Retraction Note: Data monitoring in student psychological automatic evaluation system by using dynamic acquisition algorithm

Retraction Note: Data monitoring in student psychological automatic evaluation system by using dynamic acquisition algorithm

P-204 Automated embryo assessment: a simple web application correlates scores with embryo morphology, developmental velocity, ploidy, and clinical outcomes

Abstract Study question How does the novel automated scoring system in our laboratory correlate with treatment success in different subpopulations? Summary answer The web app automates embryo assessment, linking scores to morphology, development speed, ploidy, and clinical outcomes in treatments with both patient and donor oocytes. What is known already In the realm of assisted reproductive technologies, numerous platforms employing artificial intelligence (AI) have emerged for embryo assessment. Substantial research has been dedicated to evaluating the efficacy of AI in predicting pregnancy outcomes, particularly following blastocyst transfer. However, not all available options have undergone external validation, a crucial step deemed necessary before investing in such tools. This project endeavors to validate the utility of EMBRYOAID v1.0, a straightforward web application capable of assessing embryos through photos and/or videos. Study design, size, duration This is a retrospective cohort study including 485 patients who underwent IVF treatments with embryos (n = 5,710) cultured in EmbryoScope® time-lapse systems. Senior embryologists routinely assessed blastocysts using ASEBIR morphological criteria (A-D). Then, time-lapse videos were scored by the EMBRYOAID algorithm (0-10). We investigated the relationship between automated scoring and: (I) conventional morphology, (II) morphokinetic parameters, (III) euploidy rate, and (IV) implantation in cycles with own oocytes, egg donation, and preimplantation genetic testing for aneuploidies (PGT-A). Participants/materials, setting, methods The association with morphology and morphokinetics was investigated in 5,710 embryos. Ploidy correlation involved 258 blastocysts undergoing trophectoderm biopsy and next-generation sequencing analysis. Implantation association was studied in 381 single blastocyst transfers. Finally, a multivariable analysis considered oocyte age, body mass index, type of transfer (fresh-frozen) and day of transfer (5-6) to quantify the relevance of automated scoring in different treatment subpopulations. Model performance was assessed through ROC curves and area under the curve calculations. Main results and the role of chance The results that involved grouping were quartiled based on sample size, calculated by the statistical software. This deep-learning score was related to conventional morphology (7.9±1 for A, n = 295; 6.9±1.3 for B n = 1168; 5.9±1.3 for C, n = 830; and 3.1±1.8 for D, n = 3,417)*. Embryos with faster development in early stages received higher scores, considering division times at 2, 3, 4, and 5 cells*. Euploidy rate increased as the embryo score was higher: 39.1% for ≤5.6 (n = 54), 45.5% for 5.6-6.2 (n = 61), 48.5% for 6.2-6.9 (n = 65) and 58.2% for >6.9 (n = 78). Similarly, the implantation rate was also higher in accordance with the score: 42.7% for ≤6.2 (n = 73), 50.6% for 6.2-7.4 (n = 87), 60.2% for 7.4-8.2 (n = 103) and 69.4% for >8.2 (n = 118). The stepwise multivariate analysis showed that embryo score was related to the odds of implantation in conventional treatments with patient oocytes (OR = 1.4; 95% CI [1.1-1.8]*) and in the oocyte donation program (OR = 1.2; 95% CI [1–1.4]*). The AUCs demonstrated the following performance: 0.643 (0.559-0.727) for patient oocytes and 0.677 (0,624-0,731) for the oocyte donation program. There was no significant association of embryo score with implantation in PGT-A treatments. Limitations, reasons for caution The retrospective nature of the study, despite serving as a thorough external validation, introduces inherent limitations. Notably, in PGT-A treatments, embryos underwent assisted hatching on day 3, potentially influencing the model’s performance. Wider implications of the findings The implications of our findings validate a user-friendly tool for automated embryo scoring in the laboratory. The achieved performance is comparable or superior to other tested models, particularly in egg donation programs, potentially encouraging clinics to adopt this tool routinely. Trial registration number Not applicable

P-157 Comparative analysis of iDAScore and KIDScore in predicting live birth following single frozen day 5 blastocyst transfer

Abstract Study question How do iDAScore® and KIDScoreTM perform in predicting live birth outcomes after single frozen Day 5 blastocyst transfers? Summary answer In a euploid major cohort, both KIDScoreTM and iDAScore® demonstrated comparable predictive ability for live birth. However, iDAScore® offers the advantage of being fully automated. What is known already In the field of in vitro fertilization (IVF), the escalating use of artificial intelligence (AI) for predicting clinical outcomes is a burgeoning trend. One key contributor to this trend is time-lapse incubation, which captures images of developing images, generating a wealth of data. This data has facilitated the creation of various predictive models for implantation. KIDScoreTM is one such model that uses morphokinetic data to predict the probability of implantation after a Day 5 embryo transfer. Similarly, iDAScore® is an AI-driven automated scoring system specifically designed to automatically identify embryos with the greatest likelihood of implantation. Study design, size, duration This retrospective study included 779 patients (maternal age range: 19-44, mean age: 33.38) who underwent single frozen day 5 blastocyst transfer cycle from January 2021 to January 2023. Of these, 621 patients transferred euploid blastocysts, 48 transferred mosaic blastocysts, and 110 transferred non-PGT-A tested blastocysts. This study comprised of 12.1% (94/779) fair-to-good and fair grade (below AB and BA), 51.5% (401/779) good grade (AB and BA), and 36.4% (284/779) excellent grade (AA) blastocysts. Participants/materials, setting, methods After PIEZO-ICSI, embryos were cultured in EmbryoScopeTM Plus (Vitrolife, Denmark) until they reached the blastocyst stage. Blastocysts graded at least as fair according to Gardner (1999), were either biopsied for PGT-A and vitrified (Cryotec, Japan), or vitrified without biopsy. Clinical pregnancy and the number of gestational sacs were determined using ultrasound and successful pregnancies were tracked until delivery. KIDScoreTM (KIDScoreTM Day 5 v3.2) and iDAScore® (iDAScore® v2.0) were retrospectively generated for each transferred blastocyst. Main results and the role of chance Box plot showed that both KIDScoreTM and iDAScore® have positive correlation with morphology (Excellent vs. Good vs. Fair-to-Good and Fair; KIDScoreTM median: 8.8 vs. 6.8 vs. 5.7; iDAScore® median: 8.57 vs. 7.68 vs. 6.615), ICM (A vs. B; KIDScoreTM median: 7.5 vs. 6.5; iDAScore® median: 8.14 vs. 7.26) and TE (A vs. B vs. C; KIDScoreTM median: 8.7 vs. 6.5 vs. 4.65; iDAScore® median: 8.41 vs. 7.515 vs. 5.2). In this cohort, the overall implantation rate was 67.91% (529/779), and the live birth rate was 59.31% (462/779). Further statistical analysis revealed interesting correlations. A linear regression analysis indicated a statistically significant positive correlation between KIDScoreTM and iDAScore® (F(1,777)=297.6, p < 0.001, adjusted R2=0.276). Wilcoxon rank sum tests showed that KIDScoreTM has significant higher mean scores in blastocysts resulting in live birth compared to those that did not (7.44 vs. 7.22, w = 66577, p < 0.05). iDAScore® also had higher mean scores in blastocysts resulting in live birth but without statistically significant difference (7.56 vs. 7.48, w = 71328, p > 0.05). ROC curve analysis showed that AUC for KIDScoreTM was 54.4 and AUC for iDAScore® was 51.9, indicating no significant difference in their predictive ability for live birth (p = 0.2455). Limitations, reasons for caution The study’s cohort predominantly consisted of blastocysts with good morphology and were euploid, potentially skewing the results. Therefore, the findings may not be representative of all IVF patients and should be interpreted with caution. Additionally, further study should investigate the capability of iDAScore® 2.0 to predict euploidy. Wider implications of the findings While KIDScoreTM had a higher AUC than iDAScore®, there was no statistically significant difference. The major advantage of iDAScore® lies in its automation, eliminating the need for manual embryo evaluation. This can greatly enhance consistency in embryo grading and selection. Trial registration number NOT APPLICABLE

Artificial Intelligence-based Segmentation of Residual Pancreatic Cancer in Resection Specimens Following Neoadjuvant Treatment (ISGPP-2): International Improvement and Validation Study.

Neoadjuvant therapy (NAT) has become routine in patients with borderline resectable pancreatic cancer. Pathologists examine pancreatic cancer resection specimens to evaluate the effect of NAT. However, an automated scoring system to objectively quantify residual pancreatic cancer (RPC) is currently lacking. Herein, we developed and validated the first automated segmentation model using artificial intelligence techniques to objectively quantify RPC. Digitized histopathological tissue slides were included from resected pancreatic cancer specimens from 14 centers in 7 countries in Europe, North America, Australia, and Asia. Four different scanner types were used: Philips (56%), Hamamatsu (27%), 3DHistech (10%), and Leica (7%). Regions of interest were annotated and classified as cancer, non-neoplastic pancreatic ducts, and others. A U-Net model was trained to detect RPC. Validation consisted of by-scanner internal-external cross-validation. Overall, 528 unique hematoxylin and eosin (H & E) slides from 528 patients were included. In the individual Philips, Hamamatsu, 3DHistech, and Leica scanner cross-validations, mean F1 scores of 0.81 (95% CI, 0.77-0.84), 0.80 (0.78-0.83), 0.76 (0.65-0.78), and 0.71 (0.65-0.78) were achieved, respectively. In the meta-analysis of the cross-validations, the mean F1 score was 0.78 (0.71-0.84). A final model was trained on the entire data set. This ISGPP model is the first segmentation model using artificial intelligence techniques to objectively quantify RPC following NAT. The internally-externally cross-validated model in this study demonstrated robust performance in detecting RPC in specimens. The ISGPP model, now made publically available, enables automated RPC segmentation and forms the basis for objective NAT response evaluation in pancreatic cancer.

Exploring Pre-Service Teachers’ Cognitive Processes and Calibration with an Unsupervised Learning-Based Automated Evaluation System

In the context of formative assessment in classrooms, the incorporation of automated evaluation (AE) systems and teachers’ interactions with them hold significant importance. This study aimed to investigate the cognitive processes of pre-service teachers as they engaged with an AE system. We developed an unsupervised learning-based AE system, the Scoring Assistant using Artificial Intelligence (SAAI). SAAI calculates scores without relying on predefined labels and generates scientific keywords from student responses to provide constructive feedback. We collected a substantial number of constructed responses from students, and four pre-service teachers evaluated these responses initially without any external assistance and then re-evaluated them using SAAI scores as a reference point. Employing a mixed-methods approach, this study demonstrated a strong level of consistency between human raters and SAAI scores. Pre-service teachers also reflectively recalibrated their assessments and adjusted their rubrics to identify students’ learning more accurately. This study highlights the practical application of AE in real classroom settings and demonstrates how AE can enhance efficiency and accuracy in K-12 science assessments, thus supporting teachers.

Developing an Effective Off-the-job Training Model and an Automated Evaluation System for Thoracoscopic Esophageal Atresia Surgery

BackgroundPediatric minimally invasive surgery requires advanced technical skills. Off-the-job training (OJT), especially when using disease-specific models, is an effective method of acquiring surgical skills. To achieve effective OJT, it is necessary to provide objective and appropriate skill assessment feedback to trainees. We aimed to construct a system that automatically evaluates surgical skills based on forceps movement using deep learning (DL). MethodsUsing our original esophageal atresia OJT model, participants were tasked with performing esophageal anastomosis. All tasks were recorded for image analysis. Based on manual objective skill assessments, each participant's surgical skills were categorized into two groups: good and poor. The motion of the forceps in both groups was used as training data. Employing this training data, we constructed an automated system that recognized the movement of forceps and determined the quality of the surgical technique. ResultsThirteen participants were assigned to the good skill group and 32 to the poor skill group. These cases were validated using an automated skill assessment system. This system showed a precision of 75%, a specificity of 94%, and an area under the receiver operating characteristic curve of 0.81. ConclusionsWe constructed a system that automatically evaluated the quality of surgical techniques based on the movement of forceps using DL. Artificial intelligence diagnostics further revealed the procedures important for suture manipulation. Levels of EvidenceLevel IV

An Online Assessment : Survey Panel

Abstract : The Electronic Test System (ETS) is crafted to streamline the assessment process, allowing examiners to effortlessly create and administer exams. Examiners can incorporate a variety of question types, including multiple-choice and open-ended text responses. The system utilizes advanced techniques to automate the evaluation of assessments, drawing from a database of questions curated by administrators. The ETS boasts intelligent capabilities for evaluating written responses, recognizing not only exact matches but also synonymous terms within the answers. Additionally, the system offers a manual override feature, enabling examiners to manually review and adjust exam results when necessary. With a user-friendly interface, the ETS accommodates examiners, candidates, and super administrators. Users can assume roles as either examiners or candidates. Upon logging in, candidates can view and complete their scheduled assessments from any location with internet access, removing the need for traditional tools like pen and paper. The ETS aims to provide a comprehensive and efficient assessment solution. This fully automated system saves time, delivers quick and accurate results, and allows candidates the flexibility to complete exams conveniently from any location. Keywords: Assessment, Evaluator, Participant, Administrator, Manual Review, Automated Evaluation System

A multi-label dataset and its evaluation for automated scoring system for cleanliness assessment in video capsule endoscopy.

An automated scoring system for cleanliness assessment during video capsule endoscopy (VCE) is presently lacking. The present study focused on developing an approach to automatically assess the cleanliness in VCE frames as per the latest scoring i.e., Korea-Canada (KODA). Initially, an easy-to-use mobile application called artificial intelligence-KODA (AI-KODA) score was developed to collect a multi-label image dataset of twenty-eight patient capsule videos. Three readers (gastroenterology fellows), who had been trained in reading VCE, rated this dataset in a duplicate manner. The labels were saved automatically in real-time. Inter-rater and intra-rater reliability were checked. The developed dataset was then randomly split into train:validate:test ratio of 70:20:10 and 60:20:20. It was followed by a comprehensive benchmarking and evaluation of three multi-label classification tasks using ten machine learning and two deep learning algorithms. Reliability estimation was found to be overall good among the three readers. Overall, random forest classifier achieved the best evaluation metrics, followed by Adaboost, KNeighbours, and Gaussian naive bayes in the machine learning-based classification tasks. Deep learning algorithms outperformed the machine learning-based classification tasks for only VM labels. Thorough analysis indicates that the proposed approach has the potential to save time in cleanliness assessment and is user-friendly for research and clinical use. Further research is required for the improvement of intra-rater reliability of KODA, and the development of automated multi-task classification in this field.

Deep learning-based classification of erosion, synovitis and osteitis in hand MRI of patients with inflammatory arthritis

ObjectivesTo train, test and validate the performance of a convolutional neural network (CNN)-based approach for the automated assessment of bone erosions, osteitis and synovitis in hand MRI of patients with...

A Comprehensive Study on a Deep-Learning-Based Electrocardiography Analysis for Estimating the Apnea-Hypopnea Index.

This study introduces a deep-learning-based automatic sleep scoring system to detect sleep apnea using a single-lead electrocardiography (ECG) signal, focusing on accurately estimating the apnea-hypopnea index (AHI). Unlike other research, this work emphasizes AHI estimation, crucial for the diagnosis and severity evaluation of sleep apnea. The suggested model, trained on 1465 ECG recordings, combines the deep-shallow fusion network for sleep apnea detection network (DSF-SANet) and gated recurrent units (GRUs) to analyze ECG signals at 1-min intervals, capturing sleep-related respiratory disturbances. Achieving a 0.87 correlation coefficient with actual AHI values, an accuracy of 0.82, an F1 score of 0.71, and an area under the receiver operating characteristic curve of 0.88 for per-segment classification, our model was effective in identifying sleep-breathing events and estimating the AHI, offering a promising tool for medical professionals.

Analyzing novice and competent programmers' problem-solving behaviors using an automated evaluation system

Background and ContextIn today's tech-driven world, programming courses are crucial. Yet, teaching programming is challenging, leading to high student failure rates. Understanding student learning patterns is key, but there's a lack of research utilizing tools to automatically collect and analyze interaction data for insights into student performance and behaviors. ObjectivesStudy aims to compare problem-solving behaviors of novice and competent programmers during coding tests, identifying patterns and exploring relationships with program correctness. MethodWe built an online system with programming challenges to collect behavior data from novice and competent programmers. Our system analyzed data using various metrics to explore behavior-program correctness relationships. FindingsAnalysis showed distinct problem-solving behavior patterns. Competent programmers had fewer syntax errors, spent less time fixing bugs, and had higher program correctness. Novices made more syntax errors and spent more time fixing coding errors. Both groups used tabs for code structure, but competent programmers introduced unfamiliar variables more often and commented on them afterward. Emphasizing iterative revisions and active engagement enhances problem-solving skills and programming proficiency. Radar charts are effective for identifying improvement areas in teaching programming. The relationship between behavior and program correctness was positively correlated for competent programmers but not novices. ImplicationsStudy findings have implications for programming education. Radar charts help teachers identify course improvement areas. Novices can learn from competent programmers' behavior. Instructors should encourage continuous skill improvement through revisions and engagement. Identified unfamiliar programming aspects offer insights for targeted learning.

Comprehensive evaluation of a new automatic scoring system for cleanliness assessment in video capsule endoscopy

AbstractA reliable, quick‐to‐assess, and automatic scoring system for cleanliness assessment in video capsule endoscopy (VCE) is presently not available. The present study proposes an approach to automatically assess the cleanliness in VCE frames as per the latest scoring system, that is, Korea‐Canada (KODA). First, a new multi‐label frame dataset containing medical scores of 28 VCE videos was generated through the proposed mobile‐based application called Artificial Intelligence‐KODA (AI‐KODA) score. The scores were saved automatically in real‐time through the application. The generated dataset was transformed into three datasets based on the scores, and each of the dataset was then randomly split into train:validate:test ratio of 60:20:20. Second, a comprehensive evaluation, interpretation, and benchmarking of the three classification tasks were performed with the help of eight transfer learning algorithms on NVIDIA RTX A5000 workstation. Thorough analysis indicates that AI‐KODA utilized with AI is reliable, quick‐to‐access, and free from observer bias. It promotes automatic scoring system for cleanliness assessment in VCE. The meta‐data is available here (link).

Tapping into the Pedagogical Potential of infinigoChatIC: Evidence from iWrite Scoring and Comments and Lu & Ai’s Linguistic Complexity Analyzer

ChatGPT has drawn rapid interest among linguists and language teachers, with academics and industries rushing to tap into its potential. Yet the pedagogical potential of ChatGPT for L2 English writing remains to be further explored academically. This study aims to examine the effects of ChatGPT, or one of its Chinese versions called infinigoChatIC, in improving the quality of L2 writing and in facilitating the development of L2 proficiency among some postgraduate students learning an English reading and writing course in a prestigious university in Southwest China. Specifically, the main research questions focus on how infinigoChatIC-polished L2 English writings perform in comparison to L2 learner writings, and how the improvement of English writing is reflected in the changes of lexical and syntactic complexity, as evidenced by measured data and its pedagogical significance. A set of 24 English writings are collected as part of regular homework assignments, then this set of English writing is polished by the teacher researcher in the infinigoChatIC dialogue box with the prompt “please correct grammatical errors in the following English text so that it is grammatically accurate.” These two sets of writings are scored in iWrite automatic scoring system and then analyzed in Lu & Ai’s web-interfaced linguistic complexity analyzer. The study finds that infinigoChatIC can effectively improve the quality of L2 English writing, with lexical complexity vastly improved and yet surprisingly overall syntactic complexity slightly reduced. Pedagogical implications are discussed.