Automatic Scoring Research Articles

Abstract 4138217: A Machine Learning-Based Novel Risk Score Model for Spontaneous Coronary Artery Dissection

Background: Spontaneous coronary artery dissection (SCAD) remains a poorly understood differential diagnosis of acute coronary syndrome (ACS). We aimed to develop a machine learning-based risk score model to predict in-hospital mortality in patients with SCAD. Methods: The National Inpatient Sample (NIS) 2016-2020 database was used to identify all adult patients (≥18 years of age) with SCAD, identified using ICD-10 code I25.42. Relevant clinical characteristics and outcomes were extracted. The primary endpoint was in-hospital all-cause mortality. The dataset was randomly split into 3 subsets: training, validation, and testing, with a ratio of 0.7, 0.2, and 0.1, respectively. The risk score was generated using R software’s Autoscore package, which is a machine learning-based automatic clinical score generator, and performance was evaluated using the area under receiver-operative characteristics curve (AUC) with 95% confidence intervals (95%CI). Results: Among the 8,260 SCAD patients identified [age 60.9±14.4 years, female 4,708 (57%)], there were 480 (5.8%) primary outcome events. Out of the top 20 covariates seen in the Parsimony plot, we used six for our risk score (0–100): age, comorbidity burden, cardiogenic shock, balloon pump, extracorporeal membrane oxygenation, and race (Figure 1A). The AUC for the derivation and validation cohorts was 0.862 (0.817–0.907) and 0.833 (0.788–0.879), respectively (Figures 1B and 1C). Conclusion: We developed and internally validated a machine learning-based risk score model to predict outcomes in patients with SCAD with a high AUC of 0.833 (0.788–0.879). Risk score models guide decision-making, and future studies are needed for external validation of our findings.

Exploring the potential of DistilBERT architecture for automatic essay scoring task

Exploring the potential of DistilBERT architecture for automatic essay scoring task

Analytical Features Assisted Hierarchical Classification for Automatic Sleep Stage Scoring

Analytical Features Assisted Hierarchical Classification for Automatic Sleep Stage Scoring

Vertical back movement of cows during locomotion: detecting lameness with a simple image processing technique.

This research paper proposes a simple image processing technique for automatic lameness detection in dairy cows under farm conditions. Seventy-five cows were selected from a dairy farm and visually assessed for a reference/real lameness score (RLS) as they left the milking parlor, while simultaneously being video-captured. The method employed a designated walking path and video recordings processed through image analysis to derive a new computerized automatic lameness score (ALDS) based on calculated factors from back arch posture. The proposed automatic lameness detection system was calibrated using 12 cows, and the remaining 63 were used to evaluate the diagnostic characteristics of the ALDS. The agreement and correlation between ALDS and RLS were investigated. ALDS demonstrated high diagnostic accuracy with 100% sensitivity and specificity and was found to be 100% accurate with a perfect agreement (ρc = 1) and strong correlation (r = 1, P < 0.001) for lameness detection in binary scores (lame/non-lame). Moreover, the ALDS had a strong agreement (ρc = 0.885) and was highly correlated (r = 0.840; 0.796-1.000 95% confidence interval, P < 0.001) with RLS in ordinal scores (lameness severity; LS1 to LS5). Our findings suggest that the proposed method has the potential to compete with vision-based lameness detection methods in dairy cows in farm conditions.

C anton MT: Investigating Back-Translation and Model-Switch Mechanisms for Cantonese-English Neural Machine Translation

This paper investigates the development and evaluation of machine translation models from Cantonese to English (and backward), where we propose a novel approach to tackle low-resource language translations. Despite recent improvements in Neural Machine Translation (NMT) models with Transformer-based architectures, Cantonese, a language with over 80 million native speakers, has below-par State-of-the-art commercial translation models due to a lack of resources. The main objectives of the study are to develop a model that can effectively translate Cantonese to English and evaluate it against state-of-the-art commercial models. To achieve this, a new parallel corpus has been created by combining different available corpora online with preprocessing and cleaning. In addition, a monolingual Cantonese dataset has been created through web scraping to aid the synthetic parallel corpus generation. Following the data collection process, several approaches, including fine-tuning models, back-translation , and model switch , have been used. The translation quality of models has been evaluated with multiple quality metrics, including lexicon-based metrics (SacreBLEU and hLEPOR) and embedding-space metrics (COMET and BERTscore). Based on the automatic metrics, the best model is selected and compared against the 2 best commercial translators using a new human evaluation framework HOPES . The best model proposed in this investigation (NLLB-mBART) with model switch mechanisms has reached comparable and even better automatic evaluation scores against State-of-the-art commercial models (Bing and Baidu Translators), with a SacreBLEU score of 16.8 on our test set. Furthermore, an open-source web application has been developed to allow users to translate between Cantonese and English, with the different trained models available for effective comparisons between models from this investigation and users. CantonMT is available at https://github.com/kenrickkung/CantoneseTranslation

External validation of a fully automated evaluation tool: a retrospective analysis on 68,471 scored embryos.

To externally validate a fully automated embryo classification in in vitro fertilization (IVF) treatments. Retrospective cohort study SUBJECTS: A total of 6,434 patients undergoing 7,352 IVF treatments contributed 70,456 embryos. Embryos were evaluated by conventional morphology and retrospectively scored using a fully automated deep learning-based algorithm across conventional IVF, oocyte donation, and PGT-A cycles. The primary outcomes were implantation and live birth including odds ratios (ORs) from generalized estimating equation (GEE) models. Secondary outcomes were embryo morphology, euploidy and miscarriage. Exploratory outcomes included comparison between conventional methodology and artificial intelligence (AI) algorithm with areas under the ROC curves (AUCs), agreement degree between AI and embryologists, Cohen's Kappa coefficient and relative risk (RR). Implantation and live birth rates increased as the automatic embryo score rose. The GEE model, controlling for confounders, showed the automatic score was associated with an OR of 1.31 (95%CI[1.25-1.36]) for implantation in treatments using oocytes from patients, and an OR of 1.17 (95%CI[1.14-1.20]) in the oocyte donation program, with no significant association in PGT-A treatments. For live birth, the ORs were 1.27 (95%CI[1.21-1.33]) for patients, 1.16 (95%CI[1.13-1.19]) for donors, and 1.05 (95%CI[1-1.10]) for PGT-A cycles. The average score was higher in embryos with better morphology, in euploid embryos compared to aneuploid embryos, and in embryos that resulted in a full-term pregnancy compared to those that miscarried. Concordance between the highest-scoring embryo and the embryo with the best conventional morphology was 71.4%(95%CI[67.7%-75.0%]) in treatments with patient oocytes and 61.0%(95%CI[58.6%-63.4%]) in the oocyte donation program. Overall, the Cohen's Kappa coefficient was 0.63. The automatic embryo score showed similar AUCs to conventional morphology, although implantation was higher when the transferred embryo matched the highest-scoring embryo from each cohort (57.36% vs. 49.98%). RR indicated a 1.14-fold increase in implantation likelihood when the top-ranked embryo was transferred. Fully automated embryo scoring effectively ranked embryos based on their potential for implantation and live birth. The performance of the conventional methodology was comparable to that of the artificial intelligence-based technology; however, better clinical outcomes were observed when the highest-scoring embryo in the cohort was transferred.

A benchmark for Rey-Osterrieth complex figure test automatic scoring

The Rey–Osterrieth complex figure (ROCF) test is a neuropsychological task that can be useful for early detection of cognitive decline in the elderly population. Several computer vision systems have been proposed to automate this complex analysis task, but the lack of public benchmarks does not allow a fair comparison of these systems. To advance in that direction, we present a benchmarking framework for the automatic scoring of the ROCF test that provides: the ROCFD528 dataset, which is the first open dataset of ROCF line drawings; and experimental results obtained by several modern deep learning models, which can be used as a baseline for comparing new proposals. We evaluate different state-of-the-art convolutional neural networks (CNNs) under traditional and transfer learning paradigms. Experimental quantitative results (MAE = 3.448) indicate that a CNN specifically designed for sketches outperforms other state of the art CNN architectures when the number of examples available is limited. This benchmark can also be a paradigmatic example within the broad field of machine learning for the development of efficient and robust models for analyzing line drawings and sketches not only in classification but also in regression tasks.

A Pertinence Score for Political Discourse Analysis: The Case of 2018 Colombian Elections

This study proposes a quantitative method to assess the pertinence of political language on national issues, addressing the complexity of analyzing political discourse and its relevance to citizens’ concerns. Using word embeddings and linguistic models trained on Wikipedia, a “pertinence score” was developed to measure the relevance of political discourse in contexts such as the economy and health. The method was applied to the 2018 Colombian presidential election, revealing significant differences in thematic pertinence between candidates. Survey validation confirmed the correlation between automatic and human scores, highlighting the model’s ability to discriminate ideological positions through lexical analysis.

An automatic method for scoring poultry footpad dermatitis with deep learning and thermal imaging

Footpad dermatitis (FPD) is a common poultry condition that can negatively influence chickens’ production, welfare, and health. However, no automated tool for monitoring FPD in live chickens is currently available. The objective of this study was to develop and optimize deep learning models to monitor hens’ FPD scores (i.e., 0–2 scale with higher scores indicating poorer footpad conditions). A total of 700 Hy-Line W-36 hens were raised in four cage-free housing systems integrated with Electrostatic Particle Ionization and various bedding materials. A GoPro camera with an upward lens was placed inside a transparent box. Individual laying hens were placed on the top surface of the box to acquire RGB images. In addition, a thermal camera was used to record RGB and thermal images of footpads, and the images were manually scored to assess their footpad conditions. Preprocessing techniques (e.g., filtration, separation, and augmentation) were deployed to enhance dataset quality and size. Moreover, YOLOv8 models (YOLOv8n, YOLOv8s, YOLOv8m, YOLOv8l, and YOLOv8x) and YOLOv7 models (YOLOv7 and YOLOv7x) were comparatively evaluated for predicting FPD scores. The results show that the YOLOv8l outperformed other models, with higher recall (96.6 %), mAP@0.50 (97.0 %), and F1-score (95.0 %). Additionally, the YOLOv8l-FPD model exhibited a high mAP@0.50 for score 0 (98.0 %), score 1 (95.0 %), and score 2 (97.9 %) and F1-score (95.0 %) for all FPD scores. Notably, using thermal images could result in faster convergence of model training and slightly better FPD score prediction performance than RGB images. The proposed technique can be useful for non-invasive automatic FPD scoring and further improve automation levels and animal welfare in the egg industry.

An optimized BERT fine-tuned model using an artificial bee colony algorithm for automatic essay score prediction

Background The Automatic Essay Score (AES) prediction system is essential in education applications. The AES system uses various textural and grammatical features to investigate the exact score value for AES. The derived features are processed by various linear regressions and classifiers that require the learning pattern to improve the overall score. Issues Moreover, the classifiers face catastrophic forgetting problems, which maximizes computation complexity and reduce prediction accuracy. The forgetting problem can be resolved using the freezing mechanism; however, the mechanism can cause prediction errors. Method Therefore, this research proposes an optimized Bi-directional Encoder Representation from Transformation (BERT) by applying the Artificial Bee Colony algorithm (ABC) and Fine-Tuned Model (ABC-BERT-FTM) to solve the forgetting problem, which leads to higher prediction accuracy. Therefore, the ABC algorithm reduces the forgetting problem by selecting optimized network parameters. Results Two AES datasets, ASAP and ETS, were used to evaluate the performance of the optimized BERT of the AES system, and a high accuracy of up to 98.5% was achieved. Thus, based on the result, we can conclude that optimizing the BERT with a suitable meta-heuristic algorithm, such as the ABC algorithm, can resolve the forgetting problem, eventually increasing the AES system’s prediction accuracy.

Learning to Score: A Coding System for Constructed Response Items via Interactive Clustering

Constructed response items that require the student to give more detailed and elaborate responses are widely applied in large-scale assessments. However, the hand-craft scoring with a rubric for massive responses is labor-intensive and impractical due to rater subjectivity and answer variability. The automatic response coding method, such as the automatic scoring of short answers, has become a critical component of the learning and assessment system. In this paper, we propose an interactive coding system called ASSIST to efficiently score student responses with expert knowledge and then generate an automatic score classifier. First, the ungraded responses are clustered to generate specific codes, representative responses, and indicator words. The constraint set based on feedback from experts is taken as training data in metric learning to compensate for machine bias. Meanwhile, the classifier from responses to code is trained according to the clustering results. Second, the experts review each coded cluster with the representative responses and indicator words to score a rating. The coded cluster and score pairs will be validated to ensure inter-rater reliability. Finally, the classifier is available for scoring a new response with out-of-distribution detection, which is based on the similarity between response representation and class proxy, i.e., the weight of class in the last linear layer of the classifier. The originality of the system developed stems from the interactive response clustering procedure, which involves expert feedback and an adaptive automatic classifier that can identify new response classes. The proposed system is evaluated on our real-world assessment dataset. The results of the experiments demonstrate the effectiveness of the proposed system in saving human effort and improving scoring performance. The average improvements in clustering quality and scoring accuracy are 14.48% and 18.94%, respectively. Additionally, we reported the inter-rater reliability, out-of-distribution rate, and cluster statistics, before and after interaction.

AI-Powered Telemedicine for Automatic Scoring of Neuromuscular Examinations.

Telemedicine is now being used more frequently to evaluate patients with myasthenia gravis (MG). Assessing this condition involves clinical outcome measures, such as the standardized MG-ADL scale or the more complex MG-CE score obtained during clinical exams. However, human subjectivity limits the reliability of these examinations. We propose a set of AI-powered digital tools to improve scoring efficiency and quality using computer vision, deep learning, and natural language processing. This paper focuses on automating a standard telemedicine video by segmenting it into clips corresponding to the MG-CE assessment. This AI-powered solution offers a quantitative assessment of neurological deficits, improving upon subjective evaluations prone to examiner variability. It has the potential to enhance efficiency, patient participation in MG clinical trials, and broader applicability to various neurological diseases.

Abutment Tooth Formation Simulator for Naked-Eye Stereoscopy

Virtual reality is considered to be useful in improving procedural skills in dental education, but systems using wearable devices such as head-mounted displays (HMDs) have many problems in terms of long-term use and hygiene, and the accuracy of stereoscopic viewing at close ranges is inadequate. We developed an abutment tooth formation simulator that utilizes a display (spatial reality display—SRD) to precisely reproduce 3D space with naked-eye stereoscopic viewing at close range. The purpose of this was to develop and validate the usefulness of an abutment tooth formation simulator using an SRD. A 3D-CG (three-dimensional computer graphics) dental model that can be cut in real time was output to the SRD, and an automatic quantitative scoring function was also implemented by comparing the cutting results with exemplars. Dentists in the department of fixed prosthodontics performed cutting operations on both a 2D display-based simulator and an SRD-based simulator and conducted a 5-point rating feedback survey. Compared to the simulator that used a 2D display, the measurements of the simulator using an SRD were significantly more accurate. The SRD-based abutment tooth formation simulator received a positive technical evaluation and high dentist satisfaction (4.37), suggesting its usefulness and raising expectations regarding its future application in dental education.

Tree narrative and AI interaction: Optimization and narrative bias handling prospects

This paper reviews the current research on interactive narratives, particularly the tree narrative structure, analyzing the methods of player-AI interaction, the design of narrative branches, and the mechanisms by which player decisions influence the story's progression. It explores the definitions, classifications, and applications of tree narratives in gaming, emphasizing the significance of player-AI interaction and its impact on narrative outcomes. The study identifies and analyzes potential narrative deviations resulting from player-AI interactions within tree structures and introduces an AI-assisted automatic scoring system based on hypothesis testing and Bayesian analysis to effectively identify narrative biases.

Controlled voice quality modifications

Within-speaker variability, which results from the plasticity of speech production, is an inherent feature of speaker comparison. This study examines targeted modifications of voice quality, both phonatory and articulatory, in Czech. Fifteen speakers were instructed to read a text in 15 different versions (e.g. palatalised voice, denasalised voice, breathy phonation, or a combination of open jaw and creaky voice). Acoustic analyses revealed that F3 is relatively stable across various voice quality settings, while harmonicity and spectral slope indicators are sensitive to the phonatory modifications. A perceptual test, administered online to 120 participants, showed that palatalised, pharyngealised, creaky, and pressed voice were regarded as most different from the speakers’ habitual voices. Finally, automatic speaker recognition scores were very good with the targeted voice quality modifications, with LLR between 3 and 10. Pressed phonation turned out to have the greatest effect on all three types of analysis.

Effects of repeated unihemispheric concurrent dual-site tDCS and virtual reality games on motor coordination of sedentary adolescent girls

BackgroundThis study investigated the effects of repetitive unihemispheric concurrent dual-site anodal transcranial direct current stimulation (a-tDCSUHCDS) associated with the use of virtual reality games (VR) on the motor coordination of sedentary adolescent girls.MethodsThirty-six inactive adolescent girls were randomly assigned into 3 groups (n = 12 per group): (1) VR + a-tDCSUHCDS, (2) VR + sham-tDCSUHCDS, and (3) Control. The VR + a-tDCSUHCDS and VR + s-tDCSUHCDS groups received the intervention three times a week for four weeks. In each experimental session, participants first received either 20 min of a-tDCSUHCDS (2 mA at each anodal electrode) targeting the primary motor cortex (M1) and the left dorsolateral prefrontal cortex (DLPFC) or sham and then performed VR for 1 h. The control group received no intervention. Eye-hand coordination (EHC) and bimanual coordination (BC) were measured at baseline, post-intervention, and two weeks later (retention test) using the automatic scoring mirror tracer and continuous two-arm coordination test, respectively.ResultsResults showed that the EHC was significantly higher in the VR + a-tDCS and VR + s-tDCS groups at post-intervention (all ps< 0.001) and the retention test (all ps< 0.001) compared to the control group. Moreover, the EHC was significantly higher in the VR + a-tDCS group compared to the VR + s-tDCS group (p = 0.024) at the retention. Similarly, VR + a-tDCS and VR + s-tDCS improved BC compared to the control group at post-intervention (all ps< 0.001) and retention test (all ps< 0.001). In addition, higher BC was observed in the VR + a-tDCS group compared to the VR + s-tDCS group (p< 0.001) at the retention test.ConclusionsOur results suggest that adding a-tDCSUHCDS to VR over 12 sessions may have an additional effect on VR training for improving and retaining motor coordination in sedentary adolescent girls.

Automatic rating method based on deep transfer learning for machine translation considering contextual semantic awareness

With the acceleration of globalization, machine translation (MT) plays an increasingly prominent role in cross-language communication. However, how to evaluate the quality of machine translation, especially considering the nuances in different semantic contexts, remains a challenge. This paper proposes an automatic scoring model for machine translation quality based on deep transfer learning, which aims to accurately perceive and evaluate the quality of translated texts in different semantic contexts. Firstly, a pre-trained deep neural network model is used to extract the semantic feature representation of the sentences in the source language and the target language, so as to capture the semantic information of the sentences. Then, deep transfer learning is used to map the semantic features of source language and target language into the shared feature space. By sharing feature space, an effective relationship is established between the semantic representations of two languages, so as to achieve cross-language quality evaluation. The experimental results show that the model has made significant progress in evaluating the quality of machine translation. Compared with traditional methods, this model can evaluate translation quality more accurately and has better generalization ability. The automatic scoring model of machine translation quality based on deep transfer learning not only improves the accuracy and efficiency of machine translation quality assessment, but also provides new ideas and methods for future research.

Comparison of the impact of dimensionality reduction and data splitting on classification performance in credit risk assessment

Credit risk assessment (CRA) plays an important role in credit decision-making process of financial institutions. Today, developing big data analysis and machine learning methods have marked a new era in credit risk estimation. In recent years, using machine learning methods in credit risk estimation has emerged as an alternative method for financial institutions. The past demographic and financial data of the person whose CRA will be performed is important for creating an automatic artificial intelligence credit score prediction model based on machine learning. It is also necessary to use features correctly to create accurate machine learning models. This article aims to investigate the effects of dimensionality reduction and data splitting steps on the performance of classification algorithms widely used in the literature. In our study, dimensionality reduction was performed using Principal Component Analysis (PCA). Random Forest (RF), Logistic Regression (LR), Decision Tree (DT), Naive Bayes (NB) algorithms were chosen for classification. Percentage splitting (PER, 66–34%) and k-fold (k = 10) cross-validation techniques were used when dividing the data set into training and test data. The results obtained were evaluated with accuracy, recall, F1 score, precision, and AUC metrics. German data set was used in this study. The effect of data splitting and dimension reduction on the classification of CRA systems was examined. The highest ACC in PER and CV data splitting was obtained with the RF algorithm. Using data splitting methods and PCA, the highest accuracy was observed with RF and the highest AUC with NB, with 13 PCs in which 80% of the variance was obtained. As a result, the data set consisting of a total of 20 features, expressed by 13 PCs, achieved similar or higher success than the results obtained from the original data set.

Research on the Design of a System based on Machine Learning Algorithms for Automatic Scoring of English Writing Ability

Research on the Design of a System based on Machine Learning Algorithms for Automatic Scoring of English Writing Ability

Exploring the Possibility of Automatic Scoring for Arrow Responses of Simple Pendulum Items Using a Image Contours

Exploring the Possibility of Automatic Scoring for Arrow Responses of Simple Pendulum Items Using a Image Contours