Model For Task Research Articles

A Review of the Limitations of Language Models in NLP

Abstract. In the field of Natural Language Processing (NLP), large neural language models have successfully been applied to a variety of tasks, including machine translation and reading comprehension. These models often learn the structure of language (such as grammar and semantics) by predicting the next word or character. However, there's an overly optimistic assessment of the current state of natural language understanding, which assumes that models can handle text at a semantic level. This article primarily explores the successes of large neural language models (such as BERT and GPT-2) in numerous tasks that require the understanding of linguistic meaning in the domain of NLP. Language models trained purely on form cannot learn meaning, as understanding meaning involves the relationship between linguistic form and non-linguistic communicative intent. The article aims to guide scientific research in the field of Natural Language Understanding (NLU) by clearly distinguishing the concepts of form and meaning. The learning, generalization, and other capabilities of different models under various attributes show significant differences, suggesting that a combination of different models and properties can achieve unexpected results. The simple learning combinations of some models may offer new insights into the understanding of linguistic meaning by models.

Layer-Wise Learning Rate Optimization for Task-Dependent Fine-Tuning of Pre-Trained Models: An Evolutionary Approach

The superior performance of large-scale pre-trained models, such as Bidirectional Encoder Representations from Transformers (BERT) and Generative Pre-trained Transformer (GPT), has received increasing attention in both academic and industrial research and has become one of the current research hotspots. A pre-trained model refers to a model trained on large-scale unlabeled data, whose purpose is to learn general language representation or features for fine-tuning or transfer learning in subsequent tasks. After pre-training is complete, a small amount of labeled data can be used to fine-tune the model for a specific task or domain. This two-stage method of “pre-training+fine-tuning” has achieved advanced results in natural language processing (NLP) tasks. Despite widespread adoption, existing fixed fine-tuning schemes that adapt well to one NLP task may perform inconsistently on other NLP tasks given that different tasks have different latent semantic structures. In this article, we explore the effectiveness of automatic fine-tuning pattern search for layer-wise learning rates from an evolutionary optimization perspective. Our goal is to use evolutionary algorithms to search for better task-dependent fine-tuning patterns for specific NLP tasks than typical fixed fine-tuning patterns. Experimental results on two real-world language benchmarks and three advanced pre-training language models show the effectiveness and generality of the proposed framework.

MedSegBench: A comprehensive benchmark for medical image segmentation in diverse data modalities

MedSegBench is a comprehensive benchmark designed to evaluate deep learning models for medical image segmentation across a wide range of modalities. It covers a wide range of modalities, including 35 datasets with over 60,000 images from ultrasound, MRI, and X-ray. The benchmark addresses challenges in medical imaging by providing standardized datasets with train/validation/test splits, considering variability in image quality and dataset imbalances. The benchmark supports binary and multi-class segmentation tasks with up to 19 classes and uses the U-Net architecture with various encoder/decoder networks such as ResNets, EfficientNet, and DenseNet for evaluations. MedSegBench is a valuable resource for developing robust and flexible segmentation algorithms and allows for fair comparisons across different models, promoting the development of universal models for medical tasks. It is the most comprehensive study among medical segmentation datasets. The datasets and source code are publicly available, encouraging further research and development in medical image analysis.

Self-supervised learning-based multi-source spectral fusion for fruit quality evaluation: A case study in mango fruit ripeness prediction

Rapid and non-destructive techniques for fruit quality evaluation are widely concerned in modern agro-industry. Spectroscopy is one of the most commonly used techniques in this field. With the growing popularity of various spectroscopic instruments, it is indeed worthwhile to explore modeling with multi-source spectral data to achieve more accurate predictions. Nonetheless, a major challenge is acquiring enough labeled samples, as measuring fruit chemical values is laborious, expensive, and time-consuming, which hinders the development of a reliable prediction model. Therefore, this study aims to develop a model for predicting the internal chemical composition of fruits by integrating multi-source spectral fusion combined with self-supervised learning (SSL). A visible (Vis) and near-infrared (NIR) spectral dataset related to dry matter content (DMC) prediction in mango fruit is used as an example to validate the effectiveness of the proposed method. To obtain multi-source spectral data, the Vis and NIR portions are processed as two separate spectral ranges. An SSL pre-training is performed utilizing a large amount of raw unlabeled spectral data to extract general knowledge, which is subsequently migrated to a downstream task for fine-tuning. The experimental results indicate that the multi-source spectral fusion model performs better than the single-source spectral model. Moreover, SSL solves the data scarcity problem and outperforms non-pre-trained models in downstream DMC prediction tasks with less computational overhead. Remarkably, utilizing only <10 % of the total samples is sufficient to achieve a performance close to 99 % of the best results. The presented method has great potential in spectral analysis of food and agro-products.

AutoXAI4Omics: an automated explainable AI tool for omics and tabular data.

Machine learning (ML) methods offer opportunities for gaining insights into the intricate workings of complex biological systems, and their applications are increasingly prominent in the analysis of omics data to facilitate tasks, such as the identification of novel biomarkers and predictive modeling of phenotypes. For scientists and domain experts, leveraging user-friendly ML pipelines can be incredibly valuable, enabling them to run sophisticated, robust, and interpretable models without requiring in-depth expertise in coding or algorithmic optimization. By streamlining the process of model development and training, researchers can devote their time and energies to the critical tasks of biological interpretation and validation, thereby maximizing the scientific impact of ML-driven insights. Here, we present an entirely automated open-source explainable AI tool, AutoXAI4Omics, that performs classification and regression tasks from omics and tabular numerical data. AutoXAI4Omics accelerates scientific discovery by automating processes and decisions made by AI experts, e.g. selection of the best feature set, hyper-tuning of different ML algorithms and selection of the best ML model for a specific task and dataset. Prior to ML analysis AutoXAI4Omics incorporates feature filtering options that are tailored to specific omic data types. Moreover, the insights into the predictions that are provided by the tool through explainability analysis highlight associations between omic feature values and the targets under investigation, e.g. predicted phenotypes, facilitating the identification of novel actionable insights. AutoXAI4Omics is available at: https://github.com/IBM/AutoXAI4Omics.

Sudden death of quantum advantage in correlation generations.

Quantum noise is one of the most profound obstacles to implementing large-scale quantum algorithms and schemes. In particular, the dynamical process by which quantum noise, varying in strength from 0 to critical levels, affects and destroys quantum advantage has not been well understood. Meanwhile, correlation generation serves as a precious theoretical model for information processing tasks, where quantum advantage can be precisely quantified. In this study, we show that this model provides valuable insights into the understanding of this dynamical process. We prove that, as the strength of quantum noise continuously increases from 0, the quantum advantage diminishes gradually and eventually vanishes. Unexpectedly, in some cases, we observe the phenomenon of a sudden death of quantum advantage: When the noise strength exceeds a certain threshold, the quantum advantage abruptly disappears from a substantial level. This phenomenon, once again, reveals the tremendous impact of noise on quantum information processing tasks.

EDDINet: Enhancing drug–drug interaction prediction via information flow and consensus constrained multi-graph contrastive learning

Predicting drug–drug interactions (DDIs) is crucial for understanding and preventing adverse drug reactions (ADRs). However, most existing methods inadequately explore the interactive information between drugs in a self-supervised manner, limiting our comprehension of drug–drug associations. This paper introduces EDDINet: Enhancing Drug-Drug Interaction Prediction via Information Flow and Consensus-Constrained Multi-Graph Contrastive Learning for precise DDI prediction. We first present a cross-modal information-flow mechanism to integrate diverse drug features, enriching the structural insights conveyed by the drug feature vector. Next, we employ contrastive learning to filter various biological networks, enhancing the model’s robustness. Additionally, we propose a consensus regularization framework that collaboratively trains multi-view models, producing high-quality drug representations. To unify drug representations derived from different biological information, we utilize an attention mechanism for DDI prediction. Extensive experiments demonstrate that EDDINet surpasses state-of-the-art unsupervised models and outperforms some supervised baseline models in DDI prediction tasks. Our approach shows significant advantages and holds promising potential for advancing DDI research and improving drug safety assessments. Our codes are available at: https://github.com/95LY/EDDINet_code.

UTILIZING RETRIEVAL-AUGMENTED GENERATION IN LARGE LANGUAGE MODELS TO ENHANCE INDONESIAN LANGUAGE NLP

The improvement of Large Language Models (LLM) such as ChatGPT through Retrieval-Augmented Generation (RAG) techniques has urgency in the development of natural language translation technology and dialogue systems. LLMs often experience obstacles in addressing special requests that require information outside the training data. This study aims to discuss the use of Retrieval-Augmented Generation (RAG) on large-scale language models to improve the performance of Natural Language Processing (NLP) in Indonesian, which has so far been poorly supported by high-quality data and to overcome the limitations of traditional language models in understanding the context of Indonesian better. The method used is a combination of retrieval capabilities (external information search) with generation (text generation), where the model utilizes broader and more structured basic data through the retrieval process to produce more accurate and relevant text. The data used includes the Indonesian corpus of the 30 Juz Quran translation into Indonesian. The results of the trial show that the RAG approach significantly improves the performance of the model in various NLP tasks, including token usage optimization, text classification, and context understanding, by increasing the accuracy and relevance of the results

Machine Learning-Potato Leaf Disease Detection App (MR-PoLoD)

Potato production in Indonesia has grown very rapidly, making Indonesia the largest potato producer in Southeast Asia. However, there are challenges for farmers in growing potatoes. Such as treating potatoes for various diseases. 2 diseases will occur in potato plants if not treated quickly, namely early blight disease caused by the fungus Alternaria solani and late blight disease caused by the microorganism Phytophthora infestans. The project "Potato Plant Leaf Disease Detector (MR-PoLod)" aims to design an android application that can classify leaves on potato plants into 3 classifications, namely healthy, early, and late blight disease. This application uses the CNN (Convolutional Neural Network) Machine Learning Algorithm because currently, CNN is recognized as the most efficient and effective model in pattern and image recognition tasks. This application uses the Python programming language which is rich in library and framework availability so that it can meet the needs of machine learning and image classification tasks. The total data used for training data, data validation and data testing is 3165 images. With each division of the data process on the training data of 70%, validation of 15% testing of 15% to test the effectiveness of the model that has been created. The performance of MR-PoLod for each class, obtained a precision value, recall, and f1-score of 0.99. Likewise, the accuracy value achieved by the model is 0.99 or 99%. Thus, the expected application can facilitate farmers in classifying diseases on potato plant leaves.

Network architecture search methods for constructing efficient fault diagnosis models in rotating machinery

Abstract The development of high-performance fault diagnosis models for specific tasks requires substantial expertise. Neural Architecture Search (NAS) offers a promising solution, but most NAS methodologies are hampered by lengthy search durations and low efficiency, and few researchers have applied these methods within the fault diagnosis domain. This paper introduces a novel differentiable architecture search (DARTS) method tailored for constructing efficient fault diagnosis models for rotating machinery, designed to rapidly and effectively search for network models suitable for specific datasets. Specifically, this study constructs a completely new and advanced search space, incorporating various efficient, lightweight convolutional operations to reduce computational complexity. To enhance the stability of the differentiable network architecture search process and reduce fluctuations in model accuracy, this study proposes a novel Multi-scale Pyramid Squeeze Attention (MPSA) module. This module aids in the learning of richer multi-scale feature representations and adaptively recalibrates the weights of multi-dimensional channel attention. The proposed method was validated on two rotating machinery fault datasets, demonstrating superior performance compared to manually designed networks and general network search methods, with notably improved diagnostic effectiveness.

A Metric-Based Detection System for Large Language Model Texts

More efforts are being put into improving Large Language Models’ (LLM) capabilities than into dealing with their implications. Current LLMs are able to generate high quality texts seemingly indistinguishable from those written by human experts. While offering great potentials, such breakthroughs also pose new challenges for safe and ethical uses of LLMs in education, science, and a multitude of other areas. To add up, the majority of current approaches in LLM text detection are either computationally expensive or need access to the LLMs’ internal computations, both of which hinder their public accessibility. With such motivation, this paper presents a novel metric learning paradigm for detection of LLM-generated texts that is able to balance among computational costs, accessibility, and performances. Specifically, the detection is based on learning a similarity function between a given text and an equivalent example generated by LLMs that outputs high values for LLM-LLM text pairs and low values for LLM-human text pairs. In terms of architecture, the detection framework includes a pretrained language model for the text embedding task and a newly designed deep metric model. The metric component can be trained on triplets or pairs of same-context instances to signify the distances between human texts and LLM ones while reducing that among LLM texts. Next, we develop five datasets totalling over 95,000 contexts and triplets of responses in which one from human and two from GPT-3.5 TURBO or GPT-4 TURBO for benchmarking. Experiment studies show that our best architectures maintain F1 scores in between 0.87 to 0.95 across the tested corpora in multiple experiment settings. The metric framework also demands significantly less time in training and inference compared to the RoBERTa, LLaMA 3, Mistral v0.3, and Ghostbuster, while keeping 90% to 150% performances of the best benchmark.

Debiasing vision-language models for vision tasks: a survey

Debiasing vision-language models for vision tasks: a survey

Policy analysis combining artificial intelligence and text mining technology in the perspective of educational informatization

In order to explore the application potential of artificial intelligence (AI) and text mining technology in educational policy analysis and evaluate their impact on the psychological perception of policy audiences, this study firstly introduces the application of AI and text mining technology in education. Secondly, it explores the application of psychological theories in educational policy analysis. Finally, this study constructs an educational policy text analysis model and verifies the feasibility of the optimized model through performance comparison experiments and case analysis. The experimental results show that the optimized model exhibits higher accuracy, recall rate, and F1 score compared to traditional models when handling educational policy text analysis tasks with different data volumes. This finding highlights the importance of optimizing models for specific tasks and the potential of improving the understanding and analysis capabilities of models for specific text types through careful adjustments. In addition, the application of psychological theories to the analysis of educational policy texts provides a new perspective and method for understanding the impact of policies on audience psychological states, which helps in formulating more effective and humanized policies. Therefore, the study has certain reference significance for the use of AI and text mining technology to support educational policy analysis and formulation, providing valuable insights and guidance for future related research and practice.

Performance Analysis of Different Types of Nn Models for Target Recognition

The objective of this research is to conduct a comprehensive performance analysis of various types of neural network (NN) models for target recognition. Specifically, this study focuses on evaluating the effectiveness and efficiency of yolov8n, yolov8s, yolov8m models in target recognition tasks. Leveraging cutting-edge technologies such as OpenCV, the research is aimed at developing a robust methodology for assessing the performance of these NN models. Through meticulous analysis, this study aims to provide insights into the strengths and weaknesses of each model, facilitating informed decision-making for practical applications1. This paper presents the process of designing and conducting the performance analysis. The study discusses the implications of the findings for future developments in target recognition systems.

Smart Caption: Intelligent Image Description Using Transformer Decoder

The need for automated image description systems has grown significantly with the rise of multimedia content across diverse domains such as social media, digital libraries, and accessibility technologies. Smart Caption presents a novel approach for generating intelligent and context-aware image descriptions by utilizing a combination of Vision Transformers (ViTs) and Natural Language Processing (NLP) Transformer decoders. Unlike traditional convolution-based methods, Vision Transformers treat images as sequences of patches, enabling them to capture global image features more effectively. These extracted visual features are then processed by a Transformer-based decoder to produce coherent and contextually appropriate captions, bridging the gap between image recognition and natural language generation. Our approach focuses on leveraging the attention mechanisms inherent in both Vision and NLP Transformers to enhance the quality of image descriptions. The ViT architecture is designed to focus on relevant regions within an image, while the NLP Transformer decoder is adept at generating fluent and detailed descriptions by attending to the most significant visual features. This two-stage process improves the precision of object detection, relationship understanding, and scene context in generated captions. Experimental results demonstrate that Smart Caption out performs traditional methods in terms of accuracy, contextual relevance, and fluency, marking a significant step forward in the field of automated image captioning. Through this research, we aim to provide a scalable, efficient, and intelligent solution for image description, with potential applications in areas such as accessibility tools for visually impaired users, digital content indexing, and automated content generation. Our findings highlight the strengths of transformer-based models in bridging vision and language tasks, offering promising directions for future research in multimodal AI..

Deep Learning for Text Sentiment Analysis: A Survey

Abstract. With the continuous improvement of many factors such as corpus, computing power, data scale, and laboratory conditions, how to quickly and accurately understand the emotional tendency in the text has become an important issue in many fields. The current state of affairs and the trajectory that deep learning is pursuing in the realm of text emotion analysis hold significant importance. This article first introduces the definition, importance and classification of emotional analysis, including the analysis of emotional polarity, emotional detection and fine-grained emotion. Subsequently, the key technologies of deep learning were discussed and the application of deep learning in natural language processing, especially LSTM, Bert, and their specific applications in emotional analysis within deep learning models. Then reveal the advantages and limitations of each model by comparing the performance of different deep learning models in emotional analysis tasks. Finally, in combination with current deep learning research, this study summarizes the challenges and research trends in the field of text emotional analysis.

A Novel Defect Segmentation Model via Integrating Enhanced U-Net with Convolutional Block Attention Mechanisms

Abstract. Defect segmentation in industrial quality control is pivotal for maintaining high product surface standards, but it remains a challenge due to complex defect patterns and varying backgrounds. Traditional segmentation methods often struggle with sensitivity and specificity, leading to either missed defects or false positives. In order to address the above issues this paper presents an advanced defect segmentation model that integrates a Convolutional Block Attention Module (CBAM) with an enhanced U-Net architecture. Our method leverages the spatial and channel-wise attention mechanisms to focus more accurately on subtle defects and remove background interference, enhancing the discriminative power of the feature maps generated by the U-Net. Experimental results on SD-saliency-900 industrial datasets demonstrate significant improvements in segmentation accuracy and robustness compared to existing state-of-the-art methods, confirming the effectiveness of integrating attention mechanisms in deep learning models for complex defect segmentation tasks. Specifically, our model achieved a mean Intersection over Union (mIoU) of 0.87, which is an increase of 5% over the traditional U-Net model and a 4% improvement compared to the DeepLabv3+ model. These results indicate a substantial enhancement in defect detection performance, validating the superiority of our approach.

Addressing imperfect symmetry: A novel symmetry-learning actor-critic extension

Symmetry, a fundamental concept to understand our environment, often oversimplifies reality from a mathematical perspective. Humans are a prime example, deviating from perfect symmetry in terms of appearance and cognitive biases (e.g. having a dominant hand). Nevertheless, our brain can easily overcome these imperfections and efficiently adapt to symmetrical tasks. The driving motivation behind this work lies in capturing this ability through reinforcement learning. To this end, we introduce Adaptive Symmetry Learning (ASL) — a model-minimization actor-critic extension that addresses incomplete or inexact symmetry descriptions by adapting itself during the learning process. ASL consists of a symmetry fitting component and a modular loss function that enforces a common symmetric relation across all states while adapting to the learned policy. The performance of ASL is compared to existing symmetry-enhanced methods in a case study involving a four-legged ant model for multidirectional locomotion tasks. The results show that ASL can recover from large perturbations and generalize knowledge to hidden symmetric states. It achieves comparable or better performance than alternative methods in most scenarios, making it a valuable approach for leveraging model symmetry while compensating for inherent perturbations.

Comparative Analysis of Chatbots Using Large Language Models for Web Development Tasks

In this study, we compare the performance of five chatbots using large language models (LLMs) in handling web development tasks. Three human testers asked each chatbot nine predefined questions related to creating a simple website with a dynamic form and database integration. The questions covered tasks such as generating a web document structure, designing a layout, creating a form, and implementing database queries. The chatbots’ outputs were ranked based on accuracy, completeness, creativity, and security. The experiment reveals that conversational chatbots are adept at managing complex tasks, while programming assistants require more precisely formulated tasks or the ability to generate new responses to address irrelevant outputs. The findings suggest that conversational chatbots are more capable of handling a broader range of web development tasks with minimal supervision, whereas programming assistants need more precise task definitions to achieve comparable results. This study contributes to understanding the strengths and limitations of various LLM-based chatbots in practical coding scenarios, offering insights for their application in web development.

Data-driven prediction of flow fields in a needle-ring-net electrohydrodynamic pump system

In this paper, a data-mechanism hybrid modeling method for efficiently obtaining an electrohydrodynamic flow field is proposed. First, a backpropagation (BP) model with high accuracy is trained to get the value of essential parameter q0 for the mechanism simulation of flow fields. Subsequently, the mechanism model is used to generate a database for flow field reconstruction. Three machine learning algorithms, namely, BP neural network, random forest regression (RFR), and convolutional neural network (CNN), are employed to predict and reconstruct the flow behaviors of a needle-ring-net electrohydrodynamic pump. The RFR model demonstrates higher accuracy and precision in predicting velocity and pressure in the flow field compared to the BP and CNN models. The use of machine learning models for flow field prediction can significantly reduce the computational time while maintaining the computational accuracy. Additionally, an analysis assessing the impact of varying dataset sizes on the prediction accuracy of the model is conducted. The results indicate that the size of the dataset significantly influences the model predictive performance. Specifically, larger datasets are suggested to enhance both the accuracy and the generalization capabilities of the model. This observation highlights the critical role of dataset size in optimizing the performance of machine learning models for predictive tasks in engineering applications. These results offer important references for improving the design and optimization of electrohydrodynamic pumps.