Automatic Evaluation Research Articles

New stellar age estimates using SPInS based on Gaia DR3 photometry and LAMOST DR8 abundances

Reliable stellar age estimates are fundamental for testing several problems in modern astrophysics, in particular since they set the timescales of Galactic dynamical and chemical evolution. In this study, we determine ages using only photometry and parallaxes, in combination with interstellar extinction maps, and spectroscopic metallicities and alpha abundances from the latest data release (DR8) of the LAMOST survey. In contrast with previous age estimates, we do not use spectroscopic effective temperatures or surface gravities, and thus we rely on the excellent precision and accuracy of the Gaia photometry. We use a new version of the publicly available SPInS code with improved features, including the on-the-fly computation of the autocorrelation time and the automatic convergence evaluation. We determine reliable age estimates for 35,096 and 243,768 sub-giant and main-sequence turn-off stars in the LAMOST DR8 low- and medium-resolution surveys with typical uncertainties smaller than 10<!PCT!>. In addition, we successfully test our method on more than 4,000 stars of 14 well-studied open and globular star clusters covering a wide range of ages, confirming the reliability of our age and uncertainty estimates.

A novel deep learning‐based technique for efficient characterization of engineered cementitious composites cracks for durability assessment

AbstractEngineered Cementitious Composites also known as Strain‐hardening cementitious composites (SHCCs) has unique cracking patterns like cracks that have tiny widths and showcase high density. All of this makes it difficult and laborious to compute crack parameters from crack patterns. Unfortunately, this is an essential part of assessing durability and micromechanical modeling. SHSnet is developed to perform end‐to‐end semantic segmentation of SHCC cracks. SHSnet is efficient, attention based deep encoder‐decoder network with large receptive field. Loss function based on Tversky function were used for training the model. SHSnet with loss function shows promising result with mPrecision, mF1Score and mIoU of 0.87, 0.84 and 0.83 respectively for complex SHCC cracks while requiring at least an order of fewer computational parameters than those in the literature. An image processing unit is then used to estimate the width, number, and length of the cracks from the segmentation mask. Test results show that the computed crack parameters with SHSnet are exactly the same as that computed with an optical microscope but require ~100× less time. Results demonstrate that SHSnet works equally well in SHCCs with different surface textures, crack density, and widths; the final result was far superior to a conventional technique. This technique also shows promising results in an automatic evaluation of crack parameters relevant to durability and visualizing crack patterns even in the presence of artifacts during progressive testing. The results also demonstrate the necessity to accurately and densely calculate crack length and maximum crack width; else the durability results are expected to be significantly more conservative than the actual value.

Train & Constrain: Phonologically Informed Tongue Twister Generation from Topics and Paraphrases

Abstract Previous work in phonologically and phonetically grounded language generation has mainly focused on domains such as puns and poetry. In this article, we present new work on the generation of English tongue twisters - a form of language that is required to be conditioned on a phoneme level to maximize sound overlap, while maintaining semantic consistency with an input topic or phrase and still being grammatically correct. We present TwisterLister, a pipeline for generating phonologically informed tongue twisters from large language models (LLMs) that we use to generate TwistList 2.0, the largest annotated dataset of tongue twisters to date, consisting of 17K+ examples from a combination of human and LLM authors. Our generation pipeline involves the use of a phonologically constrained vocabulary alongside LLM prompting to generate novel, non-derivative tongue twister examples.We additionally present the results of automatic and human evaluation of smaller models trained on our generated dataset to demonstrate the extent to which phonologically motivated language types can be generated without explicit injection of phonological knowledge. Additionally, we introduce a phoneme-aware constrained decoding module (PACD) that can be integrated into an autoregressive language model and demonstrate that this method generates good quality tongue twisters both with and without fine-tuning the underlying language model. We also design and implement a range of automatic metrics for the task of tongue twister generation that is phonologically motivated and captures the unique essence of tongue twisters, primarily based on phonemic edit distance (PED).

Implementation of a Whisper Architecture-Based Turkish Automatic Speech Recognition (ASR) System and Evaluation of the Effect of Fine-Tuning with a Low-Rank Adaptation (LoRA) Adapter on Its Performance

This paper focuses on the implementation of the Whisper architecture to create an automatic speech recognition (ASR) system optimized for the Turkish language, which is considered a low-resource language in terms of speech recognition technologies. Whisper is a transformer-based model known for its high performance across numerous languages. However, its performance in Turkish, a language with unique linguistic features and limited labeled data, has yet to be fully explored. To address this, we conducted a series of experiments using five different Turkish speech datasets to assess the model’s baseline performance. Initial evaluations revealed a range of word error rates (WERs) between 4.3% and 14.2%, reflecting the challenges posed by Turkish. To improve these results, we applied the low-rank adaptation (LoRA) technique, which is designed to fine-tune large-scale models efficiently by introducing a reduced set of trainable parameters. After fine-tuning, significant performance improvements were observed, with WER reductions of up to 52.38%. This study demonstrates that fine-tuned Whisper models can be successfully adapted for Turkish, resulting in a robust and accurate end-to-end ASR system. This research highlights the applicability of Whisper in low-resource languages and provides insights into the challenges of and strategies for improving speech recognition performance in Turkish.

Quantum-Inspired Fusion for Open-Domain Question Answering

Open-domain question-answering systems need models capable of referencing multiple passages simultaneously to generate accurate answers. The Rational Fusion-in-Decoder (RFiD) model focuses on differentiating between causal relationships and spurious features by utilizing the encoders of the Fusion-in-Decoder model. However, RFiD reliance on partial token information limits its ability to determine whether the corresponding passage is a rationale for the question, potentially leading to inappropriate answers. To address this issue, we propose a Quantum-Inspired Fusion-in-Decoder (QFiD) model. Our approach introduces a Quantum Fusion Module (QFM) that maps single-dimensional into multi-dimensional hidden states, enabling the model to capture more comprehensive token information. Then, the classical mixture method from quantum information theory is used to fuse all information. Based on the fused information, the model can accurately predict the relationship between the question and passage. Experimental results on two prominent ODQA datasets, Natural Questions and TriviaQA, demonstrate that QFiD outperforms the strong baselines in automatic evaluations.

AI upscaling Supporting Image Alignment in Photogrammetric Reconstruction

Abstract. This research aims to investigate and analyze the contribution provided by preventive image processing using AI in the SfM data acquisition process. Specifically, the objective is to observe qualities and defects of "AI upscaling" integrated into the normal workflow of digital restitution, with the hypothesis that greater sharpness and resolution can lead to better alignments and better model generations. Other similar experiments have been carried out previously on the AI intervention in the photos to improve the alignments, but a generic procedure and with untrained public AI has not yet been experimented. In the first phase of the research, the most suitable AI upscaling method for the purpose is selected, processing example images and comparing them with the original and various settings of the AI parameters to maintain likeness. In the second phase, the procedure efficiency was evaluated in several ways: through the automatic evaluation of Agisoft Metashape's Image Quality, by directly comparing the resulting model with the standard digital reconstruction of the same objects, comparing with a model produced with the native AI upscaling of the return program, and finally comparing with a high-level survey without defects. This test was repeated for four case studies specifically diversified by several factors, including subject scale, level of detail, lighting, color, blur, and type of camera used. Clear improvements emerge in all cases of AI upscaling, even reaching successful reconstructions that would not have started with the canonical method. It can therefore be stated that this preventive upscaling procedure is decisive in cases of low quality or damaged datasets, while in cases of already qualitatively sufficient photos it can be a useful support for increasing the level of detail.

Design of a Web-Based Monitoring and Evaluation Application for Mentoring Using the Rapid Application Development Method at The Faculty of Computer Science UNIKA Santo Thomas Medan

The academic and thesis supervision processes at the Faculty of Computer Science, UNIKA Santo Thomas Medan, currently face various challenges, including a lack of clear structure in scheduling and difficulties in monitoring student progress. To address these issues, this research aims to develop a web-based application that supports monitoring and evaluation of the supervision process. The application is designed using the Rapid Application Development (RAD) method, which facilitates flexible and responsive development tailored to user needs. With features such as supervision scheduling, student progress tracking, and automatic evaluation, this application is expected to enhance the effectiveness of supervision, facilitate communication between lecturers and students, and support more structured academic management within the faculty. The implementation of this application has successfully improved the efficiency of the supervision process and ensured that students receive more structured guidance.

C anton MT: Investigating Back-Translation and Model-Switch Mechanisms for Cantonese-English Neural Machine Translation

This paper investigates the development and evaluation of machine translation models from Cantonese to English (and backward), where we propose a novel approach to tackle low-resource language translations. Despite recent improvements in Neural Machine Translation (NMT) models with Transformer-based architectures, Cantonese, a language with over 80 million native speakers, has below-par State-of-the-art commercial translation models due to a lack of resources. The main objectives of the study are to develop a model that can effectively translate Cantonese to English and evaluate it against state-of-the-art commercial models. To achieve this, a new parallel corpus has been created by combining different available corpora online with preprocessing and cleaning. In addition, a monolingual Cantonese dataset has been created through web scraping to aid the synthetic parallel corpus generation. Following the data collection process, several approaches, including fine-tuning models, back-translation , and model switch , have been used. The translation quality of models has been evaluated with multiple quality metrics, including lexicon-based metrics (SacreBLEU and hLEPOR) and embedding-space metrics (COMET and BERTscore). Based on the automatic metrics, the best model is selected and compared against the 2 best commercial translators using a new human evaluation framework HOPES . The best model proposed in this investigation (NLLB-mBART) with model switch mechanisms has reached comparable and even better automatic evaluation scores against State-of-the-art commercial models (Bing and Baidu Translators), with a SacreBLEU score of 16.8 on our test set. Furthermore, an open-source web application has been developed to allow users to translate between Cantonese and English, with the different trained models available for effective comparisons between models from this investigation and users. CantonMT is available at https://github.com/kenrickkung/CantoneseTranslation

Automating Comment Generation for Smart Contract from Bytecode

Recently, smart contracts have played a vital role in automatic financial and business transactions. To help end users without programming background to better understand the logic of smart contracts, previous studies have proposed models for automatically translating smart contract source code into their corresponding code summaries. However, in practice, only 13% of smart contracts deployed on the Ethereum blockchain are associated with source code. The practical usage of these existing tools is significantly restricted. Considering that bytecode is always necessary when deploying smart contracts, in this paper, we first introduce the task of automatically generating smart contract code summaries from bytecode. We propose a novel approach, named S mart BT ( Smart contract B ytecode T ranslator) for automatically translating smart contract bytecode into fine-grained natural language description directly. Two key challenges are posed for this task: structural code logic hidden in bytecode and the huge semantic gap between bytecode and natural language descriptions. To address the first challenge, we transform bytecode into CFG (Control-Flow Graph) to learn code structural and logic details. Regarding the second challenge, we introduce an information retrieval component to fetch similar comments for filling the semantic gap. Then the structural input and semantic input are used to build an attentional sequence-to-sequence neural network model. The copy mechanism is employed to copy rare words directly from similar comments and the coverage mechanism is employed to eliminate repetitive outputs. The automatic evaluation results show that SmartBT outperforms a set of baselines by a large margin, and the human evaluation results show the effectiveness and potential of SmartBT in producing meaningful and accurate comments for smart contract code from bytecode directly.

Instrumentation and automatic evaluation of event indicators alongside railway lines: a first attempt to comply with article 90 of the French mobility legislation on transport

According to the Article 90 of the French Mobility Orientation Law (LOM) No. 2019-1428 (December 24, 2019), the Testing and Measurement Laboratory (LEM) of the Régie Autonome Des Transports Parisiens (RATP) 1 has been at the forefront in determining and evaluating indicators of annoyance due to event-related noise from railway transport infrastructure. Six measurement points located near tramway, metro, and suburban trains transport infrastructures were equipped with specific instrumentation allowing for measuring and automatically detecting railway and non-railway sound events (road traffic, alert sirens, and other sound activities), this multiplicity of noise sources necessitated a comprehensive approach to data collection and analysis, ensuring accurate assessment of the impact of railway transport infrastructure on surrounding area. Measurements were carried out continuously for 15 days. A post-processing system was also developed to analyze over a dozen noise indicators for each category of sound events. This article presents the technical solutions developed and deployed by the laboratory in a few months to address the regulatory challenges posed by the experimental phase of Article 90 of the LOM law.

Piano performance evaluation dataset with multilevel perceptual features

This study aims to build a comprehensive dataset that enables the automatic evaluation of piano performances. In real-world piano performance, especially within the realm of classical piano music, we encounter a vast spectrum of performance variations. The challenge lies in how to effectively evaluate these performances. We must consider three critical aspects: (1) It is essential to gauge how performers express the music and how listeners perceive the performance, rather than focusing on the compositional characteristics of the musical piece. (2) Beyond fundamental elements like pitch and duration, we must also embrace higher-level features such as interpretation. (3) Such evaluation should be done by experts to discern the nuanced performances. Regrettably, there exists no dataset that addresses these challenging evaluation tasks. Therefore, we introduce a pioneering dataset PercePiano, annotated by music experts, with more extensive features capable of representing these nuanced aspects effectively. It encapsulates piano performance with a wide range of perceptual features that are recognized by musicians. Our evaluation benchmark includes a novel metric designed to accommodate the inherent subjectivity of perception. Furthermore, we propose an enhanced baseline framework that grounds performance on score data, aligning model predictions with human perception. Harnessing the aligned features enhances the baseline performance and proves to be adaptable to various model structures. In conclusion, our research opens new possibilities for comprehensive piano performance evaluation.

Towards automatic farrowing monitoring-A Noisy Student approach for improving detection performance of newborn piglets.

Nowadays, video monitoring of farrowing and automatic video evaluation using Deep Learning have become increasingly important in farm animal science research and open up new possibilities for addressing specific research questions like the determination of husbandry relevant indicators. A robust detection performance of newborn piglets is essential for reliably monitoring the farrowing process and to access important information about the welfare status of the sow and piglets. Although object detection algorithms are increasingly being used in various scenarios in the field of livestock farming, their usability for detecting newborn piglets has so far been limited. Challenges such as frequent animal occlusions, high overlapping rates or strong heterogeneous animal postures increase the complexity and place new demands on the detection model. Typically, new data is manually annotated to improve model performance, but the annotation effort is expensive and time-consuming. To address this problem, we propose a Noisy Student approach to automatically generate annotation information and train an improved piglet detection model. By using a teacher-student model relationship we transform the image structure and generate pseudo-labels for the object classes piglet and tail. As a result, we improve the initial detection performance of the teacher model from 0.561, 0.838, 0.672 to 0.901, 0.944, 0.922 for the performance metrics Recall, Precision and F1-score, respectively. The results of this study can be used in two ways. Firstly, the results contribute directly to the improvement of piglet detection in the context of birth monitoring systems and the evaluation of the farrowing progress. Secondly, the approach presented can be transferred to other research questions and species, thereby reducing the problem of cost-intensive annotation processes and increase training efficiency. In addition, we provide a unique dataset for the detection and evaluation of newborn piglets and sow body parts to support researchers in the task of monitoring the farrowing process.

Automobile exterior emotional design method based on deep learning and multiple views imagery integrating calculation

In the context of the increasingly dynamic experience economy, users’ emotional experience with automobile has become a leading factor in consumption, making automobile exterior crucial carriers of emotional needs and design expression. However, users’ emotion expression is from three-dimensional automobile entities, with a single perspective potentially causing deviations. Thus, this paper presents a novel automobile exterior emotion design method based on deep learning and multiple views imagery integrating calculation. This approach enables a seamless transition from users’ emotional needs to automobile exterior design. First, we explore users’ emotional evaluation of automobile exterior using the semantic differential method. The improved model of ResNet101-Attention mechanism was used to predict users’ emotion. Additionally, based on above emotional evaluation values, this study proposed a calculation method of multiple views automobile imagery integrating using the Analytic Hierarchy Process and Entropy Weight, calculate the automobile imagery weights of different views, and obtain the final automobile exterior emotional imagery. Finally, the Captum toolkit and the LoRA fine-tuning training technique are used to construct an automobile exterior design optimization generation model, producing a design that meets users’ emotion needs. By comparing the results obtained using both manual and automatic evaluations, and comparing to classical deep learning models. we verify the model’s validity and the advantages of the proposed method. This intelligent method assists designers in quickly identifying users’ responses for automobile design and provides automobile design that meet their emotional needs

ARDs-YOLO: Intelligent detection of asphalt road damages and evaluation of pavement condition in complex scenarios

Pavement distress detection is the key to the development of road maintenance programs. However, traditional methods with long detection period and strong subjectivity are difficult to meet the rapid and large-scale automatic detection and evaluation of pavement technical condition. In this paper, we propose a method based on YOLOv5 combined with the initialized anchor frame selection strategy of intersection and merger ratio metrics, the multidimensional information interaction attention mechanism module ARDs, and the lightweight up-sampling operator CARAFE. An asphalt pavement distress dataset ARDs-5 containing 21,797 pavement images is constructed using a vehicle-mounted CCD camera. The proposed method outperforms the comparison frontier baseline on the self-constructed dataset with a mAP of 75.8%. Finally, ARD-YOLO was applied to evaluate the pavement condition of asphalt highways in Liaoning Province, China, and achieved excellent performance. The experiments show that ARD-YOLO has high practical value for pavement distress detection and evaluation in complex environments.

An automatic workflow for the quantitative evaluation of bit wear based on computer vision

As global oil exploration ventures into deeper and more complex territories, drilling bit wear and damage have emerged as significant constraints on drilling efficiency and safety. Despite the publication of official bit wear evaluation standards by the International Association of Drill Contractors (IADC), the current lack of quantitative and scientific evaluation techniques means that bit wear assessments rely heavily on engineers' experience. Consequently, forming a standardized database of drilling bit information to underpin the mechanisms of bit wear and facilitate optimal design remains challenging. Therefore, an efficient and quantitative evaluation of bit wear is crucial for optimizing bit performance and improving penetration efficiency.This paper introduces an automatic standard workflow for the quantitative evaluation of bit wear and the design of a comprehensive bit information database. Initially, a method for acquiring images of worn bits at the drilling site was developed. Subsequently, the wear classification and grading models based on computer vision were established to determine bit status. The wear classification model focuses on the positioning and classification of bit cutters, while the wear grading model quantifies the extent of bit wear. After that, the automatic evaluation method of the bit wear is realized. Additionally, bit wear evaluation software was designed, integrating all necessary functions to assess bit wear in accordance with IADC standards. Finally, a drilling bit database was created by integrating bit wear data, logging data, mud-logging data, and basic drilling bit data.This workflow represents a novel approach to collecting and analyzing drilling bit information at drilling sites. It holds potential to facilitate the creation of a large-scale information database for the entire lifecycle of drilling bits, marking the inception of intelligent analysis, design, and manufacture of drilling bits, thereby enhancing performance in challenging drilling conditions.

Struck-out handwritten word detection and restoration for automatic descriptive answer evaluation

Unlike objective type evaluation, descriptive answer evaluation is challenging due to unpredictable answers and free writing style of answers. Because of these, descriptive answer evaluation has received special attention from many researchers. Automatic answer evaluation is useful for the following situations. It can avoid human intervention for marking, eliminates bias marking and most important is that it can save huge manpower. To develop an efficient and accurate system, there are several open challenges. One such open challenge is cleaning the document, which includes struck-out words removal and restoring the struck-out words. In this paper, we have proposed a system for struck-out handwritten word detection and restoration for automatic descriptive answer evaluation. The work has two stages. In the first stage, we explore the combination of ResNet50 and the diagonal line (principal and secondary diagonal lines) segmentation module for detecting words and then classifying struck-out words using a classification network. In the second stage, we explore the combination of U-Net as a backbone and Bi-LSTM for predicting pixels that represent actual text information of the struck-out words based on the relationship between sequences of pixels for restoration. Experimental results on our dataset and standard datasets show that the proposed model is impressive for struck-out word detection and restoration. A comparative study with the state-of-the-art methods shows that the proposed approach outperforms the existing models in terms of struck-out word detection and restoration.

Visual interpretation of deep learning model in ECG classification: A comprehensive evaluation of feature attribution methods

Feature attribution methods can visually highlight specific input regions containing influential aspects affecting a deep learning model's prediction. Recently, the use of feature attribution methods in electrocardiogram (ECG) classification has been sharply increasing, as they assist clinicians in understanding the model's decision-making process and assessing the model's reliability. However, a careful study to identify suitable methods for ECG datasets has been lacking, leading researchers to select methods without a thorough understanding of their appropriateness. In this work, we conduct a large-scale assessment by considering eleven popular feature attribution methods across five large ECG datasets using a model based on the ResNet-18 architecture. Our experiments include both automatic evaluations and human evaluations. Annotated datasets were utilized for automatic evaluations and three cardiac experts were involved for human evaluations. We found that Guided Grad-CAM, particularly when its absolute values are utilized, achieves the best performance. When Guided Grad-CAM was utilized as the feature attribution method, cardiac experts confirmed that it can identify diagnostically relevant electrophysiological characteristics, although its effectiveness varied across the 17 different diagnoses that we have investigated.

NLP and OCR based Automatic Answer Script Evaluation System

NLP and OCR based Automatic Answer Script Evaluation System

An open-source fine-tuned large language model for radiological impression generation: a multi-reader performance study

BackgroundThe impression section integrates key findings of a radiology report but can be subjective and variable. We sought to fine-tune and evaluate an open-source Large Language Model (LLM) in automatically generating impressions from the remainder of a radiology report across different imaging modalities and hospitals.MethodsIn this institutional review board-approved retrospective study, we collated a dataset of CT, US, and MRI radiology reports from the University of California San Francisco Medical Center (UCSFMC) (n = 372,716) and the Zuckerberg San Francisco General (ZSFG) Hospital and Trauma Center (n = 60,049), both under a single institution. The Recall-Oriented Understudy for Gisting Evaluation (ROUGE) score, an automatic natural language evaluation metric that measures word overlap, was used for automatic natural language evaluation. A reader study with five cardiothoracic radiologists was performed to more strictly evaluate the model’s performance on a specific modality (CT chest exams) with a radiologist subspecialist baseline. We stratified the results of the reader performance study based on the diagnosis category and the original impression length to gauge case complexity.ResultsThe LLM achieved ROUGE-L scores of 46.51, 44.2, and 50.96 on UCSFMC and upon external validation, ROUGE-L scores of 40.74, 37.89, and 24.61 on ZSFG across the CT, US, and MRI modalities respectively, implying a substantial degree of overlap between the model-generated impressions and impressions written by the subspecialist attending radiologists, but with a degree of degradation upon external validation. In our reader study, the model-generated impressions achieved overall mean scores of 3.56/4, 3.92/4, 3.37/4, 18.29 s,12.32 words, and 84 while the original impression written by a subspecialist radiologist achieved overall mean scores of 3.75/4, 3.87/4, 3.54/4, 12.2 s, 5.74 words, and 89 for clinical accuracy, grammatical accuracy, stylistic quality, edit time, edit distance, and ROUGE-L score respectively. The LLM achieved the highest clinical accuracy ratings for acute/emergent findings and on shorter impressions.ConclusionsAn open-source fine-tuned LLM can generate impressions to a satisfactory level of clinical accuracy, grammatical accuracy, and stylistic quality. Our reader performance study demonstrates the potential of large language models in drafting radiology report impressions that can aid in streamlining radiologists’ workflows.

DiBiMT: A Gold Evaluation Benchmark for Studying Lexical Ambiguity in Machine Translation

Abstract Despite the remarkable progress made in the field of Machine Translation (MT), current systems still struggle when translating ambiguous words, especially when these express infrequent meanings. In order to investigate and analyze the impact of lexical ambiguity on automatic translations, several tasks and evaluation benchmarks have been proposed over the course of the last few years. However, works in this research direction suffer from critical shortcomings. Indeed, existing evaluation datasets are not entirely manually curated, which significantly compromises their reliability. Furthermore, current literature fails to provide detailed insights into the nature of the errors produced by models translating ambiguous words, lacking a thorough manual analysis across languages. With a view to overcoming these limitations, we propose Disambiguation Biases in MT (DiBiMT), an entirely manually-curated evaluation benchmark for investigating disambiguation biases in eight language combinations and assessing the ability of both commercial and non-commercial systems to handle ambiguous words. We also examine and detail the errors produced by models in this scenario by carrying out a manual error analysis in all language pairs. Additionally, we perform an extensive array of experiments aimed at studying the behavior of models when dealing with ambiguous words. Finally, we show the ineffectiveness of standard MT evaluation settings for assessing the disambiguation capabilities of systems and highlight the need for additional efforts in this research direction and ad-hoc testbeds such as DiBiMT.Our benchmark is available at: https://nlp.uniroma1.it/dibimt/.