Automatic Evaluation Research Articles

AI-assisted automatic MRI-based tongue volume evaluation in motor neuron disease (MND)

PurposeMotor neuron disease (MND) causes damage to the upper and lower motor neurons including the motor cranial nerves, the latter resulting in bulbar involvement with atrophy of the tongue muscle. To measure tongue atrophy, an operator independent automatic segmentation of the tongue is crucial. The aim of this study was to apply convolutional neural network (CNN) to MRI data in order to determine the volume of the tongue.MethodsA single triplanar CNN of U-Net architecture trained on axial, coronal, and sagittal planes was used for the segmentation of the tongue in MRI scans of the head. The 3D volumes were processed slice-wise across the three orientations and the predictions were merged using different voting strategies. This approach was developed using MRI datasets from 20 patients with ‘classical’ spinal amyotrophic lateral sclerosis (ALS) and 20 healthy controls and, in a pilot study, applied to the tongue volume quantification to 19 controls and 19 ALS patients with the variant progressive bulbar palsy (PBP).ResultsConsensus models with softmax averaging and majority voting achieved highest segmentation accuracy and outperformed predictions on single orientations and consensus models with union and unanimous voting. At the group level, reduction in tongue volume was not observed in classical spinal ALS, but was significant in the PBP group, as compared to controls.ConclusionUtilizing single U-Net trained on three orthogonal orientations with consequent merging of respective orientations in an optimized consensus model reduces the number of erroneous detections and improves the segmentation of the tongue. The CNN-based automatic segmentation allows for accurate quantification of the tongue volumes in all subjects. The application to the ALS variant PBP showed significant reduction of the tongue volume in these patients and opens the way for unbiased future longitudinal studies in diseases affecting tongue volume.

Study of Subjective Answer Evaluation using Natural Language Processing and Machine Learning

Now a days, as we are moving towards automation there is a need for an automatic subjective answer evaluation system. Manual evaluation is a time and energy-consuming task. Currently, the online answer evaluation is available for mcq based question, hence evaluation of the theory answer is hectic for the checker. Evaluator manually checks the answer and allot the marks. Sometimes quality of evaluation may change according to mood of evaluator.The current system takes more manpower and time to evaluate the answer. It will be very helpful for educational institutions if the process of evaluation of descriptive answers is automated to capably assess student’s exam answer sheets. The system will evaluate the answer based upon the number of words and number of letters from extracted text from the pre-processed data

A Comprehensive Analysis of the Effectiveness of Large Language Models as Automatic Dialogue Evaluators

Automatic evaluation is an integral aspect of dialogue system research. The traditional reference-based NLG metrics are generally found to be unsuitable for dialogue assessment. Consequently, recent studies have suggested various unique, reference-free neural metrics that better align with human evaluations. Notably among them, large language models (LLMs), particularly the instruction-tuned variants like ChatGPT, are shown to be promising substitutes for human judges. Yet, existing works on utilizing LLMs for automatic dialogue evaluation are limited in their scope in terms of the number of meta-evaluation datasets, mode of evaluation, coverage of LLMs, etc. Hence, it remains inconclusive how effective these LLMs are. To this end, we conduct a comprehensive study on the application of LLMs for automatic dialogue evaluation. Specifically, we analyze the multi-dimensional evaluation capability of 30 recently emerged LLMs at both turn and dialogue levels, using a comprehensive set of 12 meta-evaluation datasets. Additionally, we probe the robustness of the LLMs in handling various adversarial perturbations at both turn and dialogue levels. Finally, we explore how model-level and dimension-level ensembles impact the evaluation performance. All resources are available at https://github.com/e0397123/comp-analysis.

ParaGuide: Guided Diffusion Paraphrasers for Plug-and-Play Textual Style Transfer

Textual style transfer is the task of transforming stylistic properties of text while preserving meaning. Target "styles" can be defined in numerous ways, ranging from single attributes (e.g. formality) to authorship (e.g. Shakespeare). Previous unsupervised style-transfer approaches generally rely on significant amounts of labeled data for only a fixed set of styles or require large language models. In contrast, we introduce a novel diffusion-based framework for general-purpose style transfer that can be flexibly adapted to arbitrary target styles at inference time. Our parameter-efficient approach, ParaGuide, leverages paraphrase-conditioned diffusion models alongside gradient-based guidance from both off-the-shelf classifiers and strong existing style embedders to transform the style of text while preserving semantic information. We validate the method on the Enron Email Corpus, with both human and automatic evaluations, and find that it outperforms strong baselines on formality, sentiment, and even authorship style transfer.

Interpretable Long-Form Legal Question Answering with Retrieval-Augmented Large Language Models

Many individuals are likely to face a legal dispute at some point in their lives, but their lack of understanding of how to navigate these complex issues often renders them vulnerable. The advancement of natural language processing opens new avenues for bridging this legal literacy gap through the development of automated legal aid systems. However, existing legal question answering (LQA) approaches often suffer from a narrow scope, being either confined to specific legal domains or limited to brief, uninformative responses. In this work, we propose an end-to-end methodology designed to generate long-form answers to any statutory law questions, utilizing a "retrieve-then-read" pipeline. To support this approach, we introduce and release the Long-form Legal Question Answering (LLeQA) dataset, comprising 1,868 expert-annotated legal questions in the French language, complete with detailed answers rooted in pertinent legal provisions. Our experimental results demonstrate promising performance on automatic evaluation metrics, but a qualitative analysis uncovers areas for refinement. As one of the only comprehensive, expert-annotated long-form LQA dataset, LLeQA has the potential to not only accelerate research towards resolving a significant real-world issue, but also act as a rigorous benchmark for evaluating NLP models in specialized domains. We publicly release our code, data, and models.

LLMEval: A Preliminary Study on How to Evaluate Large Language Models

Recently, the evaluation of Large Language Models has emerged as a popular area of research. The three crucial questions for LLM evaluation are ``what, where, and how to evaluate''. However, the existing research mainly focuses on the first two questions, which are basically what tasks to give the LLM during testing and what kind of knowledge it should deal with. As for the third question, which is about what standards to use, the types of evaluators, how to score, and how to rank, there hasn't been much discussion. In this paper, we analyze evaluation methods by comparing various criteria with both manual and automatic evaluation, utilizing onsite, crowd-sourcing, public annotators and GPT-4, with different scoring methods and ranking systems. We propose a new dataset, LLMEval and conduct evaluations on 20 LLMs. A total of 2,186 individuals participated, leading to the generation of 243,337 manual annotations and 57,511 automatic evaluation results. We perform comparisons and analyses of different settings and conduct 10 conclusions that can provide some insights for evaluating LLM in the future. The dataset and the results are publicly available at https://github.com/llmeval. The version with the appendix are publicly available at https://arxiv.org/abs/2312.07398.

Watermarking Conditional Text Generation for AI Detection: Unveiling Challenges and a Semantic-Aware Watermark Remedy

To mitigate potential risks associated with language models (LMs), recent AI detection research proposes incorporating watermarks into machine-generated text through random vocabulary restrictions and utilizing this information for detection. In this paper, we show that watermarking algorithms designed for LMs cannot be seamlessly applied to conditional text generation (CTG) tasks without a notable decline in downstream task performance. To address this issue, we introduce a simple yet effective semantic-aware watermarking algorithm that considers the characteristics of conditional text generation with the input context. Compared to the baseline watermarks, our proposed watermark yields significant improvements in both automatic and human evaluations across various text generation models, including BART and Flan-T5, for CTG tasks such as summarization and data-to-text generation. Meanwhile, it maintains detection ability with higher z-scores but lower AUC scores, suggesting the presence of a detection paradox that poses additional challenges for watermarking CTG.

CAMEL: Capturing Metaphorical Alignment with Context Disentangling for Multimodal Emotion Recognition

Understanding the emotional polarity of multimodal content with metaphorical characteristics, such as memes, poses a significant challenge in Multimodal Emotion Recognition (MER). Previous MER researches have overlooked the phenomenon of metaphorical alignment in multimedia content, which involves non-literal associations between concepts to convey implicit emotional tones. Metaphor-agnostic MER methods may be misinformed by the isolated unimodal emotions, which are distinct from the real emotions blended in multimodal metaphors. Moreover, contextual semantics can further affect the emotions associated with similar metaphors, leading to the challenge of maintaining contextual compatibility. To address the issue of metaphorical alignment in MER, we propose to leverage a conditional generative approach for capturing metaphorical analogies. Our approach formulates schematic prompts and corresponding references based on theoretical foundations, which allows the model to better grasp metaphorical nuances. In order to maintain contextual sensitivity, we incorporate a disentangled contrastive matching mechanism, which undergoes curricular adjustment to regulate its intensity during the learning process. The automatic and human evaluation experiments on two benchmarks prove that, our model provides considerable and stable improvements in recognizing multimodal emotion with metaphor attributes.

Amplifying Diversity and Quality in Commonsense Knowledge Graph Completion (Student Abstract)

Conventional commonsense knowledge graph completion (CKGC) methods provide inadequate sequence when fine-tuning or generating stages and incorporate full fine-tuning, which fail to align with the autoregressive model's pre-training patterns and have insufficient parameter efficiency. Moreover, decoding through beam or greedy search produces low diversity and high similarity in generated tail entities. Hence, we resort to prefix-tuning and propose a lightweight, effective pipeline to enhance the quality and diversity of extracted commonsense knowledge. Precisely, we measure head entity similarity to yield and then concatenate top-k tuples before each target tuple for prefix-tuning the source LM, thereby improving the efficiency and speed for pretrained models; then, we design a penalty-tailored diverse beam search (p-DBS) for decoding tail entities, producing a greater quantity and diversity of generated commonsense tuples; besides, a filter strategy is utilized to filter out invalid commonsense knowledge. Through extensive automatic evaluations, including ChatGPT scoring, our method can extract diverse, novel, and accurate commonsense knowledge (CK).

Improving Automatic VQA Evaluation Using Large Language Models

8 years after the visual question answering (VQA) task was proposed, accuracy remains the primary metric for automatic evaluation. VQA Accuracy has been effective so far in the IID evaluation setting. However, our community is undergoing a shift towards open-ended generative models and OOD evaluation. In this new paradigm, the existing VQA Accuracy metric is overly stringent and underestimates the performance of VQA systems. Thus, there is a need to develop more robust automatic VQA metrics that serve as a proxy for human judgment. In this work, we propose to leverage the in-context learning capabilities of instruction-tuned large language models (LLMs) to build a better VQA metric. We formulate VQA evaluation as an answer-rating task where the LLM is instructed to score the accuracy of a candidate answer given a set of reference answers. We demonstrate the proposed metric better correlates with human judgment compared to existing metrics across several VQA models and benchmarks. We hope wide adoption of our metric will contribute to better estimating the research progress on the VQA task. We plan to release the evaluation code and collected human judgments.

Improving the Robustness of Knowledge-Grounded Dialogue via Contrastive Learning

Knowledge-grounded dialogue (KGD) learns to generate an informative response based on a given dialogue context and external knowledge (e.g., knowledge graphs; KGs). Recently, the emergence of large language models (LLMs) and pre-training techniques has brought great success to knowledge-grounded dialogue. However, when building KGD systems in real applications, there are various real-world noises that are inevitable to face. For example, the dialogue context might involve perturbations such as misspellings and abbreviations. In addition, KGs typically suffer from incompletion and also might contain erroneous and outdated facts. Such real-world noises pose a challenge to the robustness of KGD systems and hinder their applications in the real world. In this paper, we propose an entity-based contrastive learning framework for improving the robustness of KGD. Specifically, we make use of the entity information in a KGD sample to create both its positive and negative samples which involve semantic-irrelevant and semantic-relevant perturbations, respectively. The contrastive learning framework ensures the KGD model is aware of these two types of perturbations, thus could generate informative responses with the potentially noisy inputs in real applications. Experimental results on three widely-used benchmark datasets show that our method achieves new state-of-the-art performance in terms of automatic evaluation scores, verifying its effectiveness and potentiality. Furthermore, we show that our method is able to generate better responses than comparison models in both the noisy and the few-shot settings.

Beyond Entities: A Large-Scale Multi-Modal Knowledge Graph with Triplet Fact Grounding

Much effort has been devoted to building multi-modal knowledge graphs by visualizing entities on images, but ignoring the multi-modal information of the relation between entities. Hence, in this paper, we aim to construct a new large-scale multi-modal knowledge graph with triplet facts grounded on images that reflect not only entities but also their relations. To achieve this purpose, we propose a novel pipeline method, including triplet fact filtering, image retrieving, entity-based image filtering, relation-based image filtering, and image clustering. In this way, a multi-modal knowledge graph named ImgFact is constructed, which contains 247,732 triplet facts and 3,730,805 images. In experiments, the manual and automatic evaluations prove the reliable quality of our ImgFact. We further use the obtained images to enhance model performance on two tasks. In particular, the model optimized by our ImgFact achieves an impressive 8.38% and 9.87% improvement over the solutions enhanced by an existing multi-modal knowledge graph and VisualChatGPT on F1 of relation classification. We release ImgFact and its instructions at https://github.com/kleinercubs/ImgFact.

PoetryDiffusion: Towards Joint Semantic and Metrical Manipulation in Poetry Generation

Controllable text generation is a challenging and meaningful field in natural language generation (NLG). Especially, poetry generation is a typical one with well-defined and strict conditions for text generation which is an ideal playground for the assessment of current methodologies. While prior works succeeded in controlling either semantic or metrical aspects of poetry generation, simultaneously addressing both remains a challenge. In this paper, we pioneer the use of the Diffusion model for generating sonnets and Chinese SongCi poetry to tackle such challenges. In terms of semantics, our PoetryDiffusion model, built upon the Diffusion model, generates entire sentences or poetry by comprehensively considering the entirety of sentence information. This approach enhances semantic expression, distinguishing it from autoregressive and large language models (LLMs). For metrical control, its constraint control module which can be trained individually enables us to flexibly incorporate a novel metrical controller to manipulate and evaluate metrics (format and rhythm). The denoising process in PoetryDiffusion allows for the gradual enhancement of semantics and flexible integration of the metrical controller which can calculate and impose penalties on states that stray significantly from the target control distribution. Experimental results on two datasets demonstrate that our model outperforms existing models in terms of automatic evaluation of semantic, metrical, and overall performance as well as human evaluation. Codes are released to https://github.com/ChorlingLau/PoetryDiffusion.

Evaluation of Various Free Software Options for Catphan 504 Phantom Analysis

In computed tomography, image quality tests are important to guarantee a correct medical diagnosis and a better cost and benefit for the patient. Purpose: the purpose of this study is to analyse the images reconstructed with different thorax and bone convolution filters using popular free-use software in the field of medical physics, for the Catphan 504 phantom. Methods: a total of 14 scans were performed using the chest protocol, with convolution filters (FC) FC30, FC35, FC50, FC51, FC52, FC53, FC55, FC56, FC80, FC81, FC83, FC84, FC85, FC86, on the 16-channel Canon Aquilion Lightning CT scanner using the Catphan 504 phantom. Image quality parameters evaluated were: noise, uniformity, linearity of CT numbers, and high spatial resolution with MTF 50% and MTF 10%. The images were evaluated using software such as ImageJ, Script Python, and free software for automatic evaluation of the Catphan 504 phantom, CTQA-cp, SPICE-CT, and Pylinac. Results: the tests carried out with the Catphan 504 phantom were analysed by the software and agreed with each other (with p>0.05), except for Pylinac. The results obtained with Pylinac had a significant difference for the uniformity, slice thickness, and MTF10% tests, this being the code that was the furthest away from the results obtained by the other codes. Conclusions: the ImageJ, Spice-CT, and CTQA-cp software showed consistent results for the tests performed, while Pylinac had limitations in calculating the standard deviation for the noise test and showed significant differences in some tests when compared to the other software.

Mobilenet V3-transformer, a lightweight model for image caption

Image captioning is a task of generating natural language descriptions for images, which is important and challenging in computer vision. This task involves both visual and linguistic understanding, which makes it complex and difficult to solve. In this paper, we propose a novel and efficient image captioning model, named MobileNet V3-Transformer (Mob-Tran), which combines the advantages of both convolutional and transformer architectures. Our model uses the improved MobileNet V3 and the transformer’s encoder as the encoder to extract and enhance visual features from images, and uses the transformer’s decoder as the decoder to generate captions based on the encoded features. The MobileNet V3 model used in this experiment has had its classifier removed. This experiment combined automatic and human evaluation to evaluate multiple models, including MobileNet V3-Transformer, using ten automatic evaluation metrics (BLEU_1, BLEU_2, BLEU_3, BLEU_4, CIDEr, ROUGE, METEOR, Model Storage Size, Model Training Time and Model Inference Time) and five human evaluation criteria (Grammaticality, Adequacy, Logic, Readability, and Humanness). The experimental results demonstrate that our model is capable of producing high-quality image captions with low complexity and high efficiency.

Determinants of automatic age and race bias: ingroup-outgroup distinction salience moderates automatic evaluations of social groups.

The present research investigates whether ingroup-outgroup distinction salience moderates automatic intergroup bias (i.e., more positive evaluations of ingroup targets relative to outgroup targets) toward multiply categorizable social targets. In two experiments, we manipulated the salience of participants' social identity based on age vs. race, respectively. Afterwards, we measured automatic evaluations of social targets varying in age and race. Young White participants exhibited higher automatic race bias when their racial identity (i.e., White in contrast to Black) was salient. Conversely, they exhibited higher automatic age bias when their age identity (i.e., young in contrast to old) was salient. Going beyond previous research, we show that it is sufficient to direct participants' attention to their ingroup-identity in contrast to the respective outgroup to cause changes in automatic intergroup bias. This is important because it provides a strong test of the hypothesis that ingroup-outgroup distinction salience moderates automatic intergroup bias.

Improvements in Similarity Measurement Method for Emotion Classification of Japanese Sentences

This study proposes the following three improvements to address the shortfalls of existing methods for emotion classification based on similarity measures between Japanese sentences. (1) The proposed method utilizes, as a similarity measure, an automatic evaluation metric for machine translation based on correlation considering global word order. Similarity measures used in existing methods are too strict or lax in their consideration of the order of words during the processing of Japanese sentences. They fail to incorporate the fact that Japanese exhibits a higher degree of freedom in word order compared to English. (2) Rather than word-level alignment, the proposed method aligns two sentences at the level of a base phrase constituted by one or more content words followed by zero or more function words. Word-level alignment, used by existing methods, may cause inappropriate matching, because it independently processes content words and function words which constitute a base phrase. (3) The proposed method integrates, using linear interpolation, similarity scores by the alignment of word-level and those of base phrase-level. Integration can avoid data sparsity, namely a problem that a similarity score between two sentences becomes zero, occurring when base phrase-level alignment is made. The experimental results found that (1) the proposed method demonstrated significantly higher classification accuracy than existing alternatives; (2) the base phrase-level alignment was effective because the alignment exhibited higher classification accuracy than the word-level alignment; and (3) integration of the word-level and base phrase-level alignments was valid because the integrated method outperformed the base phrase-level alignment. Key Words: Emotion classification, Similarity measure, Japanese, Word order, Base phrase, Linear interpolation

BNoteHelper: A Note-based Outline Generation Tool for Structured Learning on Video-sharing Platforms

Usually generated by ordinary users and often not particularly designed for learning, the videos on video-sharing platforms are mostly not structured enough to support learning purposes, although they are increasingly leveraged for that. Most existing studies attempt to structure the video using video summarization techniques. However, these methods focus on extracting information from within the video and aiming to consume the video itself. In this article, we design and implement BNoteHelper, a note-based video outline prototype that generates outline titles by extracting user-generated notes on Bilibili, using the BART model fine-tuned on a built dataset. As a browser plugin, BNoteHelper provides users with video overview and navigation as well as note-taking template, via two main features: outline table and navigation marker. The model and prototype are evaluated through automatic and human evaluations. The automatic evaluation reveals that, both before and after fine-tuning, the BART model outperforms T5-Pegasus in BLEU and Perplexity metrics. Also, the results from user feedback reveal that the generation outline sourced from notes is preferred by users over that sourced from video captions due to its more concise, clear, and accurate characteristics but also too general with less details and diversities sometimes. Two features of the video outline are also found to have respective advantages, especially in holistic and fine-grained aspects. Based on these results, we propose insights into designing a video summary from the user-generated creation perspective, customizing it based on video types, and strengthening the advantages of its different visual styles on video-sharing platforms.

Affective Prompt-Tuning-Based Language Model for Semantic-Based Emotional Text Generation

The large language models based on transformers have shown strong text generation ability. However, due to the need for significant computing resources, little work has been done to generate emotional text using language models such as GPT-2. To address this issue, the authors proposed an affective prompt-tuning-based language model (APT-LM) equipped with an affective decoding (AD) method, aiming to enhance emotional text generation with limited computing resources. In detail, the proposed model incorporates the emotional attributes into the soft prompt by using the NRC emotion intensity lexicon and updates the additional parameters while freezing the language model. Then, it steers the generation toward a given emotion by calculating the cosine distance between the affective soft prompt and the candidate tokens generated by the language model. Experimental results show that the proposed APT-LM model significantly improves emotional text generation and achieves competitive performance on sentence fluency compared to baseline models across automatic evaluation and human evaluation.

Intelligence Sparse Sensor Network for Automatic Early Evaluation of General Movements in Infants.

General movements (GMs) have been widely used for the early clinical evaluation of infant brain development, allowing immediate evaluation of potential development disorders and timely rehabilitation. The infants' general movements can be captured digitally, but the lack of quantitative assessment and well-trained clinical pediatricians presents an obstacle for many years to achieve wider deployment, especially in low-resource settings. There is a high potential to explore wearable sensors for movement analysis due to outstanding privacy, low cost, and easy-to-use features. This work presents a sparse sensor network with soft wireless IMU devices (SWDs) for automatic early evaluation of general movements in infants. The sparse network consisting of only five sensor nodes (SWDs) with robust mechanical properties and excellent biocompatibility continuously and stably captures full-body motion data. The proof-of-the-concept clinical testing with 23 infants showcases outstanding performance in recognizing neonatal activities, confirming the reliability of the system. Taken together with a tiny machine learning algorithm, the system can automatically identify risky infants based on the GMs, with an accuracy of up to 100% (99.9%). The wearable sparse sensor network with an artificial intelligence-based algorithm facilitates intelligent evaluation of infant brain development and early diagnosis of development disorders.