Generalization Capability Of Model Research Articles

ExpeL: LLM Agents Are Experiential Learners

The recent surge in research interest in applying large language models (LLMs) to decision-making tasks has flourished by leveraging the extensive world knowledge embedded in LLMs. While there is a growing demand to tailor LLMs for custom decision-making tasks, finetuning them for specific tasks is resource-intensive and may diminish the model's generalization capabilities. Moreover, state-of-the-art language models like GPT-4 and Claude are primarily accessible through API calls, with their parametric weights remaining proprietary and unavailable to the public. This scenario emphasizes the growing need for new methodologies that allow learning from agent experiences without requiring parametric updates. To address these problems, we introduce the Experiential Learning (ExpeL) agent. Our agent autonomously gathers experiences and extracts knowledge using natural language from a collection of training tasks. At inference, the agent recalls its extracted insights and past experiences to make informed decisions. Our empirical results highlight the robust learning efficacy of the ExpeL agent, indicating a consistent enhancement in its performance as it accumulates experiences. We further explore the emerging capabilities and transfer learning potential of the ExpeL agent through qualitative observations and additional experiments.

Compositional Generalization for Multi-Label Text Classification: A Data-Augmentation Approach

Despite significant advancements in multi-label text classification, the ability of existing models to generalize to novel and seldom-encountered complex concepts, which are compositions of elementary ones, remains underexplored. This research addresses this gap. By creating unique data splits across three benchmarks, we assess the compositional generalization ability of existing multi-label text classification models. Our results show that these models often fail to generalize to compositional concepts encountered infrequently during training, leading to inferior performance on tests with these new combinations. To address this, we introduce a data augmentation method that leverages two innovative text generation models designed to enhance the classification models' capacity for compositional generalization. Our experiments show that this data augmentation approach significantly improves the compositional generalization capabilities of classification models on our benchmarks, with both generation models surpassing other text generation baselines. Our codes available at https://github.com/yychai74/LD-VAE.

CLIP-Gaze: Towards General Gaze Estimation via Visual-Linguistic Model

Gaze estimation methods often experience significant performance degradation when evaluated across different domains, due to the domain gap between the testing and training data. Existing methods try to address this issue using various domain generalization approaches, but with little success because of the limited diversity of gaze datasets, such as appearance, wearable, and image quality. To overcome these limitations, we propose a novel framework called CLIP-Gaze that utilizes a pre-trained vision-language model to leverage its transferable knowledge. Our framework is the first to leverage the vision-and-language cross-modality approach for gaze estimation task. Specifically, we extract gaze-relevant feature by pushing it away from gaze-irrelevant features which can be flexibly constructed via language descriptions. To learn more suitable prompts, we propose a personalized context optimization method for text prompt tuning. Furthermore, we utilize the relationship among gaze samples to refine the distribution of gaze-relevant features, thereby improving the generalization capability of the gaze estimation model. Extensive experiments demonstrate the excellent performance of CLIP-Gaze over existing methods on four cross-domain evaluations.

Symmetric Self-Paced Learning for Domain Generalization

Deep learning methods often suffer performance degradation due to domain shift, where discrepancies exist between training and testing data distributions. Domain generalization mitigates this problem by leveraging information from multiple source domains to enhance model generalization capabilities for unseen domains. However, existing domain generalization methods typically present examples to the model in a random manner, overlooking the potential benefits of structured data presentation. To bridge this gap, we propose a novel learning strategy, Symmetric Self-Paced Learning (SSPL), for domain generalization. SSPL consists of a Symmetric Self-Paced training scheduler and a Gradient-based Difficulty Measure (GDM). Specifically, the proposed training scheduler initially focuses on easy examples, gradually shifting emphasis to harder examples as training progresses. GDM dynamically evaluates example difficulty through the gradient magnitude with respect to the example itself. Experiments across five popular benchmark datasets demonstrate the effectiveness of the proposed learning strategy.

Memory-Efficient Prompt Tuning for Incremental Histopathology Classification

Recent studies have made remarkable progress in histopathology classification. Based on current successes, contemporary works proposed to further upgrade the model towards a more generalizable and robust direction through incrementally learning from the sequentially delivered domains. Unlike previous parameter isolation based approaches that usually demand massive computation resources during model updating, we present a memory-efficient prompt tuning framework to cultivate model generalization potential in economical memory cost. For each incoming domain, we reuse the existing parameters of the initial classification model and attach lightweight trainable prompts into it for customized tuning. Considering the domain heterogeneity, we perform decoupled prompt tuning, where we adopt a domain-specific prompt for each domain to independently investigate its distinctive characteristics, and one domain-invariant prompt shared across all domains to continually explore the common content embedding throughout time. All domain-specific prompts will be appended to the prompt bank and isolated from further changes to prevent forgetting the distinctive features of early-seen domains. While the domain-invariant prompt will be passed on and iteratively evolve by style-augmented prompt refining to improve model generalization capability over time. In specific, we construct a graph with existing prompts and build a style-augmented graph attention network to guide the domain-invariant prompt exploring the overlapped latent embedding among all delivered domains for more domain-generic representations. We have extensively evaluated our framework with two histopathology tasks, i.e., breast cancer metastasis classification and epithelium-stroma tissue classification, where our approach yielded superior performance and memory efficiency over the competing methods.

Optimizing Matrix Factorization for Personalized Recommendations Using Ridge Regularization

Personalized recommendation systems have become indispensable tools for enhancing user engagement and satisfaction in various online platforms. Matrix Factorization (MF) algorithms serve as fundamental techniques in these systems, allowing for the efficient modeling of user-item interactions and the generation of tailored recommendations. However, ensuring the robustness and generalization capability of MF models remains a challenge, particularly in the presence of sparse and noisy datasets. In this study, we focus on optimizing MF for personalized recommendations through the incorporation of L2 regularization techniques. By introducing a penalty term based on the squared Frobenius norm of the user and item matrices, L2 regularization promotes the learning of more stable and generalized latent representations, thereby mitigating overfitting. We aim to investigate the impact of L2 regularization on recommendation performance and to demonstrate its effectiveness in improving the accuracy and robustness of MF-based recommendation systems. We conduct comprehensive experiments on real-world datasets, evaluating the performance of L2-regularized MF models against baseline approaches. Our results indicate that L2 regularization significantly enhances recommendation accuracy and generalization performance, highlighting its potential to optimize MF for personalized recommendations in diverse application domains

Investigation of Deepfake Voice Detection Using Speech Pause Patterns: Algorithm Development and Validation

Background The digital era has witnessed an escalating dependence on digital platforms for news and information, coupled with the advent of “deepfake” technology. Deepfakes, leveraging deep learning models on extensive data sets of voice recordings and images, pose substantial threats to media authenticity, potentially leading to unethical misuse such as impersonation and the dissemination of false information. Objective To counteract this challenge, this study aims to introduce the concept of innate biological processes to discern between authentic human voices and cloned voices. We propose that the presence or absence of certain perceptual features, such as pauses in speech, can effectively distinguish between cloned and authentic audio. Methods A total of 49 adult participants representing diverse ethnic backgrounds and accents were recruited. Each participant contributed voice samples for the training of up to 3 distinct voice cloning text-to-speech models and 3 control paragraphs. Subsequently, the cloning models generated synthetic versions of the control paragraphs, resulting in a data set consisting of up to 9 cloned audio samples and 3 control samples per participant. We analyzed the speech pauses caused by biological actions such as respiration, swallowing, and cognitive processes. Five audio features corresponding to speech pause profiles were calculated. Differences between authentic and cloned audio for these features were assessed, and 5 classical machine learning algorithms were implemented using these features to create a prediction model. The generalization capability of the optimal model was evaluated through testing on unseen data, incorporating a model-naive generator, a model-naive paragraph, and model-naive participants. Results Cloned audio exhibited significantly increased time between pauses (P<.001), decreased variation in speech segment length (P=.003), increased overall proportion of time speaking (P=.04), and decreased rates of micro- and macropauses in speech (both P=.01). Five machine learning models were implemented using these features, with the AdaBoost model demonstrating the highest performance, achieving a 5-fold cross-validation balanced accuracy of 0.81 (SD 0.05). Other models included support vector machine (balanced accuracy 0.79, SD 0.03), random forest (balanced accuracy 0.78, SD 0.04), logistic regression, and decision tree (balanced accuracies 0.76, SD 0.10 and 0.72, SD 0.06). When evaluating the optimal AdaBoost model, it achieved an overall test accuracy of 0.79 when predicting unseen data. Conclusions The incorporation of perceptual, biological features into machine learning models demonstrates promising results in distinguishing between authentic human voices and cloned audio.

EC-Net: General image tampering localization network based on edge distribution guidance and contrastive learning

General Image Tampering Localization (GITL) is a challenging task that aims to locate tampered regions in manipulated images accurately. Despite notable achievements in recent years, existing methods still struggle with generalization and pixel-level segmentation, particularly in the face of diverse tampering forms and invisible traces. To address these issues, we propose EC-Net, a GITL network based on edge distribution guidance and contrastive learning. EC-Net decomposes the task into two phases: the coarse locating phase and the fine locating phase. In the coarse locating phase, a contrastive learning strategy is incorporated to improve the model's generalization capabilities. Additionally, an edge distribution prediction method is introduced to anticipate the global contour information of tampered regions. In the fine locating phase, we introduced a cross-scale fusion mechanism to synthesize feature information across different scales, enhancing the model's ability to discern subtle changes in tampered regions. Additionally, an edge reconstruction mechanism is introduced to restore details, thereby improving the accuracy of the final localization results. Evaluation on seven datasets shows that EC-Net significantly outperforms seven state-of-the-art methods, improving localization performance by 0.095 (F1), 0.088 (IoU), and 0.043 (AUC) compared to the second-best method on average. Additionally, we present a tampered image dataset (RLS26K) for model auxiliary training, featuring diverse real-life scenes with various tampering types, diverse styles, and rich content.

Enhancing Image Classification Accuracy Based on AlexNet

This study explores the potential of deep learning for small-scale image classification tasks through the utilization of the classical AlexNet model on the CIFAR-10 dataset. The methodology involves meticulous architectural adjustments, comprehensive data preprocessing, and strategic optimization of the learning rate decay, resulting in substantial improvements in model performance. In addition, innovative data augmentation techniques are introduced to enhance the model's robustness and generalization capabilities. The experimental outcomes unequivocally underscore the efficacy of deep learning in addressing small-scale image classification challenges, offering versatile applications across diverse domains such as image recognition, autonomous driving, and medical diagnostics. Acknowledging the dynamic nature of deep learning, ongoing research endeavors are warranted. Future directions may encompass the exploration of more intricate neural network architectures, advanced data augmentation methodologies, and the implementation of interpretability tools to augment both performance and comprehensibility. This study provides compelling evidence of deep learning's aptitude for small-scale image classification tasks, offering valuable insights and guidance for future research and practical implementations. As the field of deep learning continues to evolve, this work aims to serve as a valuable reference and catalyst for further advancements in the domain.

Optimization and Improvement of Recurrent Neural Networks in Stock Price Prediction

Recurrent Neural Networks (RNNs) are a significant branch of deep learning, particularly suitable for handling data with temporal dependencies. In the financial domain, stock price prediction is a highly focused problem. Traditional stock price prediction methods, often based on linear models, fail to fully capture the nonlinear dynamics of stock prices. RNNs, with their memory capabilities, can capture the long-term dependencies of stock prices, hence offering great application potential in stock price prediction. However, RNNs also face challenges when processing stock price data, as stock prices are influenced by numerous factors and are highly complex and uncertain. Additionally, stock price data often contains noise and outliers, impacting the model’s predictive performance. To address these issues, incremental improvements in prediction accuracy and model generalization capabilities are achieved through optimizing the model structure and improving data preprocessing methods. As technology advances and algorithms continue to innovate, the application of RNNs in stock price prediction will become more widespread and in-depth.

Prediction of chemical reaction yields with large-scale multi-view pre-training

Developing machine learning models with high generalization capability for predicting chemical reaction yields is of significant interest and importance. The efficacy of such models depends heavily on the representation of chemical reactions, which has commonly been learned from SMILES or graphs of molecules using deep neural networks. However, the progression of chemical reactions is inherently determined by the molecular 3D geometric properties, which have been recently highlighted as crucial features in accurately predicting molecular properties and chemical reactions. Additionally, large-scale pre-training has been shown to be essential in enhancing the generalization capability of complex deep learning models. Based on these considerations, we propose the Reaction Multi-View Pre-training (ReaMVP) framework, which leverages self-supervised learning techniques and a two-stage pre-training strategy to predict chemical reaction yields. By incorporating multi-view learning with 3D geometric information, ReaMVP achieves state-of-the-art performance on two benchmark datasets. Notably, the experimental results indicate that ReaMVP has a significant advantage in predicting out-of-sample data, suggesting an enhanced generalization ability to predict new reactions. Scientific Contribution: This study presents the ReaMVP framework, which improves the generalization capability of machine learning models for predicting chemical reaction yields. By integrating sequential and geometric views and leveraging self-supervised learning techniques with a two-stage pre-training strategy, ReaMVP achieves state-of-the-art performance on benchmark datasets. The framework demonstrates superior predictive ability for out-of-sample data and enhances the prediction of new reactions.

Optimizing Time-Series forecasting using stacked deep learning framework with enhanced adaptive moment estimation and error correction

Time-series forecasting is essential for decision-making activities, and deep learning models have shown promising results in this field. However, existing models face challenges in capturing the complex non-linear patterns and randomness of real-world time-series data. To address this issue, we propose a novel stacked deep learning framework that combines a bi-directional long-short-term memory network, convolutional 1D, and max-pooling-1D layers in each stack. We also introduce an enhanced version of adaptive moment estimation and an error correction method to improve the model's accuracy and convergence speed. We evaluate our model's performance using four real-world datasets (air quality, PM2.5, electricity price, and securities lending) and compare it with existing models based on statistical measures such as mean absolute error, mean squared error, and standard deviation. The results demonstrate that our proposed model outperforms existing models significantly, achieving a ∼ 52 % improvement in mean absolute error for air quality data, a ∼ 36 % improvement in mean squared error for the PM2.5 dataset, and a ∼ 27.5 % improvement in mean squared error for electricity price data. The enhanced adaptive moment estimation and error correction method further improve the model's accuracy and generalization capabilities, making it suitable for multi-domain data sets. Our proposed model has potential applications in various fields, including manufacturing, traffic prediction, weather forecasting, environmental monitoring, finance, and energy management.

Raman spectroscopy and fusion machine learning algorithm: A novel approach to identify dairy fraud

In addressing food safety verification, ensuring the authenticity of food labels remains of utmost importance. With the objective of enhancing label validation and fraud detection, our research introduces an innovative fusion model utilizing Raman spectroscopy technology. This model is uniquely designed to differentiate various dairy products and provide a quantitative assessment of their nutritional components. To achieve this, we integrated four foundational machine learning algorithms: Light Gradient Boosting Machine (LightGBM), Support Vector Machine (SVM), Random Forest (RF), and extreme Gradient Boosting (XGBoost). These algorithms individually showcased a classification accuracy exceeding 90% for differentiating dairy brands and achieved a fat prediction Normalized Root Mean Square Error (NRMSE) under 13%. However, when these algorithms were synergistically combined in our fusion model, we observed a remarkable improvement in classification accuracy, reaching 99%. Additionally, for the pivotal task of predicting fat content in dairy samples, the fusion model manifested an unparalleled performance, with metrics indicating R2 = 0.98, Root Mean Square Error (RMSE) = 0.2 g/100 mL, and NRMSE = 6.2%. Notably, while the efficiency of the individual models fluctuated depending on the training objectives, the fusion model consistently outperformed in all scenarios. In essence, our fusion model's adaptability, precision, and generalization capabilities underscore its potential as an invaluable tool in the domain of dairy product quality assessment.

A Data-Driven Method for Ship Motion Forecast

Accurate forecasting of ship motion is of great significance for ensuring maritime operational safety and working efficiency. A data-driven ship motion forecast method is proposed in this paper, aiming at the problems of low generalization of a single forecast model and insufficient forecast accuracy under unknown conditions. First, the fluid dynamics simulations of the ship are carried out under multiple node conditions based on overset mesh technology, and the obtained motion data is used for training the Bidirectional Long Short-term Memory network models. One or more pre-trained forecast models would be selected based on the correlation of condition nodes when forecasting ship motion under non-node conditions. The Golden Jackal Optimization Algorithm is used to compute the regression coefficient of each node model in real time, and finally, the dynamic model average is calculated. The results show that the method proposed in this study can accurately forecast the pitch and heave of the KCS ship in 5 s, 10 s, and 15 s of forecast duration. The accuracy of the multi-order forecast model improves more in longer forecast duration tasks compared with the first-order model. When forecasting ship motion under non-node conditions, the method shows stronger model generalization capabilities.

FedRDR: Federated Reinforcement Distillation-Based Routing Algorithm in UAV-Assisted Networks for Communication Infrastructure Failures

Traditional Internet of Things (IoT) networks have limited coverage and may experience failures due to natural disasters affecting critical IoT devices, making it difficult for them to provide communication services. Therefore, how to establish network communication service more efficiently in the presence of fault points is the problem we solve in this paper. To address this issue, this study constructs a hierarchical multi-domain data transmission architecture for an emergency network with unmanned aerial vehicles (UAVs) employed as core communication devices. This architecture expands the functionality of UAVs as key network devices and provides a theoretical basis for their feasibility as intelligent network controllers and switches. Firstly, the UAV controllers perceive the network status and learn the spatio-temporal characteristics of air-to-ground network links. Secondly, a routing algorithm within the domain based on federated reinforcement distillation (FedRDR) is developed, which enhances the generalization capability of the routing decision model by increasing the training data samples. Simulation experiments are conducted, and the results show that the average communication data size between each domain controller and the server is approximately 45.3 KB when using the FedRDR algorithm. Compared to the transmission of parameters through federated reinforcement learning algorithms, FedRDR reduces the transmitted parameter size by approximately 29%. Therefore, the FedRDR routing algorithm helps to facilitate knowledge transfer, accelerate the training process of intelligent agents within the domain, and reduce communication costs in resource-constrained scenarios for UAV networks and has practical value.

Brain tumor malignancy classification using improved VGG16 based on MRI images

This study introduces a method based on Magnetic Resonance Imaging (MRI) images and Convolutional Neural Networks (CNN) to categorize the malignancy of brain tumors. The major goal of this research is to develop an accurate and reliable model that aids in tumor early identification, leading to better patient outcomes. Specifically, first, the MRI images undergo preprocessing, which includes intensity normalization and random brightness and contrast adjustments for data augmentation. Next, the Visual Geometry Group (VGG16) model serves as a baseline, enhanced with flatten, dense, and dropout layers. Transfer learning is applied to leverage pre-trained features from VGG16, improving the generalization capability of model. A sizable collection of brain MRI pictures is used to train the model. The experimental outcomes on the MRI dataset illustrate the efficacy of the suggested model. The CNN-based approach achieves high accuracy in classifying brain tumor malignancy based on MRI images, indicating its potential in medical image analysis and its significance in early tumor detection and diagnosis. The proposed model serves as a valuable tool for healthcare professionals, aiding in well-informed decisions and providing an ideal medical approach for early detection of brain tumors. This research opens avenues for future work in medical imaging and deep learning, paving the way for improved treatment and diagnosis of brain tumors.

An accelerated lattice Boltzmann method for natural convection coupled with convolutional neural network

A novel approach is proposed in the present study, which combines the lattice Boltzmann method (LBM) with convolutional neural networks and is suitable for square cavity natural convection and conjugate natural convection problems, exhibiting remarkable acceleration capabilities and potentials. The density distribution function obtained from LBM at time t and the temperature distribution function at time t+Δt under different Rayleigh numbers (Ra) are, respectively, utilized as input and output datasets for training and comparison in three convolutional neural networks, aiming to select the optimal coupling model, namely, half-Res-Unet. The coupling model can accurately simulate the natural convection in a square cavity within six times the upper limit of the Ra under the training condition, which can save the central processing unit (CPU) calculation time and the iteration steps by up to 29.2% and 30.3%, respectively. The coupling model is further extended in the current study to incorporate conjugate natural convection, enabling the accurate simulation of temperature distribution under training conditions with a thermal conductivity ratio (Ka) of 25 and an upper limit of Ra increased by 20 times. The corresponding maximum relative errors for the average Nusselt numbers (Nu) are found to be 1.8% and 0.7%, respectively, providing strong evidence for the generalization capability of the coupling model. Furthermore, the coupling model demonstrates a remarkable acceleration performance, as evidenced by its ability to reduce the CPU calculation time by up to 39.6% and iteration steps in the simulation process by 36.5%. It offers valuable insights into the integration of LBM with machine learning techniques, thereby enhancing the computational efficiency of LBM.

Improving deep PROPELLER MRI via synthetic blade augmentation and enhanced generalization

In PROPELLER MRI, obtaining sufficient high-quality blade data remains a challenge, so the efficiency and generalization of deep learning-based reconstruction models are deteriorated. Due to narrow rotated and translated blades acquired in PROPELLER, the technique of data augmentation that is used for deep learning-based Cartesian MRI reconstruction cannot be directly applied. To address the issue, this paper introduces a novel approach for the generation of synthetic PROPELLER blades, and it is subsequently employed in data augmentation for undersampled blades reconstruction. The principal aim of this study is to address the challenges of reconstructing undersampled blades to enhance both image quality and computational efficiency. Evaluation metrics including PSNR, NMSE, and SSIM indicate superior performance of the model trained with augmented data compared to non-augmented counterparts. The synthetic blade augmentation significantly enhances the model's generalization capability and enables robust performance across varying imaging conditions. Furthermore, the study demonstrates the feasibility of utilizing synthetic blades exclusively in the training phase, suggesting a reduced dependency on real PROPELLER blades. This innovation in synthetic blade generation and data augmentation technique contributes to enhanced image quality and improved generalization capability of the associated deep learning model for PROPELLER MRI reconstruction.

DCSPose: A Dual-Channel Siamese Framework for Unseen Textureless Object Pose Estimation

The demand for object pose estimation is steadily increasing, and deep learning has propelled the advancement of this field. However, the majority of research endeavors face challenges in their applicability to industrial production. This is primarily due to the high cost of annotating 3D data, which places higher demands on the generalization capabilities of neural network models. Additionally, existing methods struggle to handle the abundance of textureless objects commonly found in industrial settings. Finally, there is a strong demand for real-time processing capabilities in industrial production processes. Therefore, in this study, we introduced a dual-channel Siamese framework to address these challenges in industrial applications. The architecture employs a Siamese structure for template matching, enabling it to learn the matching capability between the templates constructed from high-fidelity simulated data and real-world scenes. This capacity satisfies the requirements for generalization to unseen objects. Building upon this, we utilized two feature extraction channels to separately process RGB and depth information, addressing the limited feature issue associated with textureless objects. Through our experiments, we demonstrated that this architecture effectively estimates the three-dimensional pose of objects, achieving a 6.0% to 10.9% improvement compared to the state-of-the-art methods, while exhibiting robust generalization and real-time processing capabilities.

A-MASA: Arabic Multi-Domain Aspect-Based Sentiment Analysis Datasets

The rapid growth of Natural Language Processing (NLP) applications has gained interest in various languages, including Arabic. One critical NLP task is Aspect-Based Sentiment Analysis (ABSA). ABSA involves identifying the sentiment expressed toward specific aspects or attributes of entities mentioned in a piece of text, rather than determining the overall sentiment. Despite the increased use of Semitic languages like Arabic in NLP over the past decade, there remains a lack of resources and datasets in Arabic, particularly in dialectical Arabic. Additionally, existing ABSA datasets are often limited to a single domain, which does not support all types of ABSA, such as multi-domain ABSA, nor do they allow for testing a model's generalization capabilities. This study addresses these limitations by constructing multi-domain ABSA datasets in Arabic for the tasks of aspect extraction and aspect polarity detection. The datasets include both real and synthetic data, covering dialectical and modern standard Arabic, and comprising 6,500 records across five domains. These datasets can be utilized in single-domain, cross-domain, and multi-domain experiments for ABSA. Furthermore, we evaluated the constructed datasets using various transformer-based models and assessed the impact of generating synthetic data on the diversity and balance of the data, as well as on model performance. The results reported in this research demonstrate that the data is valid for use in ABSA tasks. Moreover, the models used for evaluation achieved competitive or superior results compared to existing models on various datasets. Additionally, the annotation guidelines we created are presented as recommended practices applicable to different tasks. All datasets used in this study are publicly available for research purposes.