Enhance Generalization Ability Research Articles

A novel non-pretrained deep supervision network for polyp segmentation

In this paper, we propose a non-pretrained deep supervision network (NPD-Net) for polyp segmentation. Unlike previous deep supervision networks that rely on ground truth (GT) or pre-training with GT to supervise deep features(the prediction maps from decoder), we propose a novel deep supervision strategy that directly utilizes the GT encoder (that encodes GT to get its maps) after initialization to mitigate overfitting and enhance generalization ability without pre-training, in other words, a non-pretrained. This strategy makes up the gap of directly using GT for deep supervision while mitigates the risk of overfitting due to leverage the well-train pre-trained weights on a small polyp datasets. In addition, we introduce a simple and efficient parallel dual attention module (PDA) to enhance the global modeling ability. PDA executes spatial and channel attention in parallel, and adopts implicit positional encoding and transpose operation to reduce computational complexity. Finally, NPD-Net is able to effectively supervise deep features, expand the range of context information acquisition and improve segmentation performance, particularly in terms of generalization ability. Our experimental results on five benchmark datasets demonstrate that NPD-Net outperforms other state-of-the-art methods. The code will be available at https://github.com/guobaoxiao/NPD-Net.

Prediction of chemical reaction yields with large-scale multi-view pre-training

Developing machine learning models with high generalization capability for predicting chemical reaction yields is of significant interest and importance. The efficacy of such models depends heavily on the representation of chemical reactions, which has commonly been learned from SMILES or graphs of molecules using deep neural networks. However, the progression of chemical reactions is inherently determined by the molecular 3D geometric properties, which have been recently highlighted as crucial features in accurately predicting molecular properties and chemical reactions. Additionally, large-scale pre-training has been shown to be essential in enhancing the generalization capability of complex deep learning models. Based on these considerations, we propose the Reaction Multi-View Pre-training (ReaMVP) framework, which leverages self-supervised learning techniques and a two-stage pre-training strategy to predict chemical reaction yields. By incorporating multi-view learning with 3D geometric information, ReaMVP achieves state-of-the-art performance on two benchmark datasets. Notably, the experimental results indicate that ReaMVP has a significant advantage in predicting out-of-sample data, suggesting an enhanced generalization ability to predict new reactions. Scientific Contribution: This study presents the ReaMVP framework, which improves the generalization capability of machine learning models for predicting chemical reaction yields. By integrating sequential and geometric views and leveraging self-supervised learning techniques with a two-stage pre-training strategy, ReaMVP achieves state-of-the-art performance on benchmark datasets. The framework demonstrates superior predictive ability for out-of-sample data and enhances the prediction of new reactions.

Enhancing Predictions of Drug Solubility through Multidimensional Structural Characterization Exploitation.

Solubility is not only a significant physical property of molecules but also a vital factor in smallmolecule drug development. Determining drug solubility demands stringent equipment, controlled environments, and substantial human and material resources. The accurate prediction of drug solubility using computational methods has long been a goal for researchers. In this study, we introduce MSCSol, a solubility prediction model that integrates multidimensional molecular structure information. We incorporate a graph neural network with geometric vector perceptrons (GVP-GNN) to encode 3D molecular structures, representing spatial arrangement and orientation of atoms, as well as atomic sequences and interactions. We also employ Selective Kernel Convolution combined with Global and Local attention mechanisms to capture molecular features context at different scales. Additionally, various descriptors are calculated to enrich the molecular representation. For the 2D and 3D structural data of molecules, we design different data augmentation strategies to enhance generalization ability and prevent the model from learning irrelevant information. Extensive experiments on benchmark and independent datasets demonstrate MSCSol's superior performance. Ablation studies further confirm the effectiveness of different modules. Interpretability analysis highlights the importance of various atomic groups and substructures for solubility and verifies that our model effectively captures functional molecular structures and higher-order knowledge. The source code and datasets are freely available at https://github.com/ZiyuFanCSU/MSCSol.

Adaptive Memory Networks With Self-Supervised Learning for Unsupervised Anomaly Detection

Unsupervised anomaly detection aims to build models to effectively detect unseen anomalies by only training on the normal data. Although previous reconstruction-based methods have made fruitful progress, their generalization ability is limited due to two critical challenges. First, the training dataset only contains normal patterns, which limits the model generalization ability. Second, the feature representations learned by existing models often lack representativeness which hampers the ability to preserve the diversity of normal patterns. In this paper, we propose a novel approach called Adaptive Memory Network with Self-supervised Learning (AMSL) to address these challenges and enhance the generalization ability in unsupervised anomaly detection. Based on the convolutional autoencoder structure, AMSL incorporates a self-supervised learning module to learn general normal patterns and an adaptive memory fusion module to learn rich feature representations. Experiments on four public multivariate time series datasets demonstrate that AMSL significantly improves the performance compared to other state-of-the-art methods. Specifically, on the largest CAP sleep stage detection dataset with 900 million samples, AMSL outperforms the second-best baseline by 4%+ in both accuracy and F1 score. Apart from the enhanced generalization ability, AMSL is also more robust against input noise.

Small-sample stacking model for qualitative analysis of aluminum alloys based on femtosecond laser-induced breakdown spectroscopy.

Material characterization using laser-induced breakdown spectroscopy (LIBS) often relies on extensive data for effective analysis. However, data acquisition can be challenging, and the high dimensionality of raw spectral data combined with a large-scale sample dataset can strain computational resources. In this study, we propose a small sample size stacking model based on femtosecond LIBS to achieve accurate qualitative analysis of aluminum alloys. The proposed three-layer stacking algorithm performs data reconstruction and feature extraction to enhance the analysis. In the first layer, random forest spectral feature selection and specific spectral line spreading are employed to reconstruct the data. The second layer utilizes three heterogeneous classifiers to extract features from the reconstructed spectra in different feature spaces, generating second-level reconstructed data. Finally, the third layer utilizes the reconstructed dataset for qualitative prediction. Results indicate that the Stacking algorithm outperforms traditional methods such as k-nearest neighbors (KNN), support vector machine (SVM), and random forest (RF), including those combined with principal component analysis (PCA). The Stacking algorithm achieves an impressive 100% recognition rate in classification, with Accuracy, precision, recall, and F1 scores reaching 1.0. Moreover, as the number of samples decreases, the gap between the recognition accuracy of the Stacking algorithm and traditional approaches widens. For instance, using only 15 spectra for training, the Stacking algorithm achieves a recognition accuracy of 96.47%, significantly surpassing the improved RF's accuracy of 71.76%. Notably, the model demonstrates strong robustness compared to traditional modeling approaches, and the qualitative prediction error remains consistently below 5%. These findings underscore the model's enhanced generalization ability and higher prediction accuracy in small sample machine learning. This research contributes significantly to improving the applicability of the LIBS technique for fast detection and analysis of small samples. It provides valuable insights into the development of effective methodologies for material characterization, paving the way for advancements in the field.

Intelligent Extraction of Terracing Using the ASPP ArrU-Net Deep Learning Model for Soil and Water Conservation on the Loess Plateau

The prevention and control of soil erosion through soil and water conservation measures is crucial. It is imperative to accurately and quickly extract information on these measures in order to understand how their configuration affects the runoff and sediment yield process. In this investigation, intelligent interpretation algorithms and deep learning semantic segmentation models pertinent to remote sensing imagery were examined and scrutinized. Our objective was to enhance interpretation accuracy and automation by employing an advanced deep learning-based semantic segmentation model for the astute interpretation of high-resolution remote sensing images. Subsequently, an intelligent interpretation algorithm model tailored was developed for terracing measures in high-resolution remote sensing imagery. Focusing on Fenxi County in Shanxi Province as the experimental target, in this research we conducted a comparative analysis between our proposed model and alternative models. The outcomes demonstrated that our refined algorithm model exhibited superior precision. Additionally, in this research we assessed the model’s generalization capability by utilizing Wafangdian City in Liaoning Province as another experimental target and performed a comparative analysis with human interpretation. The findings revealed that our model possesses enhanced generalization ability and can substantially augment interpretation efficiency.

A novel framework for extracting moment-based fingerprint features in specific emitter identification

Extensive experiments illustrate that moments and their derivations can act as effective fingerprint features for specific emitter identification. Nevertheless, the lack of mechanistic explanation restricts the moment-based fingerprint features to a trial-based and data-driven technique. To make up for theoretical weakness and enhance generalization ability, we analytically investigate how intentional modulation and unintentional modulation affect moments. A framework for extracting moment-based fingerprint features is proposed through fine-segmenting slices. Fingerprint features are extracted, followed by segmenting signals into a combination of sinewaves and calculating their moments. The proposed framework shows advantages in mechanism interpretability and generalizing ability. Simulations and experiments verified the correctness and effectiveness of the proposed framework.

EchoWrite 2.0: A Lightweight Zero-Shot Text-Entry System Based on Acoustics

Limited by size, shape, and other factors, it is rather inconvenient to interact with new smart devices by traditional methods. Acoustic-based methods following a machine learning approach have been put forward to resolve this problem in previous works. But they possess limitations of heavy training overhead, low performance for unseen users, and intensive computation cost. Following our previous work in this area, we further overcome shortcomings of existing work and propose a lightweight and zero-shot text-entry system for unseen users based on acoustic sensing. The key novelty of this work is proposing a new model training strategy including dataset construction and augmentation methods to effectively enhance generalization ability of a simple learning model with as few training data as possible, based on our insight into the problem. We design and implement a real-time Android application system called <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">EchoWrite 2.0</monospace> to validate our idea with extensive experiments. Results show that <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">EchoWrite 2.0</monospace> can recognize digits, English letters, and words with an accuracy of 85.3%, 73.2%, and 96.9%, respectively, for unseen users without providing any data to the learning model. The comparison with related work in different aspects shows overall superiority of <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">EchoWrite 2.0</monospace> .

Rock slope displacement prediction based on multi-source information fusion and SSA-DELM model

In order to solve the inefficient use of multi-source heterogeneous data information cross fusion and the low accuracy of prediction of landslide displacement, the current research proposed a new prediction model combining variable selection, sparrow search algorithm, and deep extreme learning machine. A cement mine in Fengxiang, Shaanxi Province, was studied as a case. The study first identified the variables related to landslide displacement of rock slope, and removed redundant variables by using Pearson correlation and gray correlation analysis. To avoid the impacts of random input weights and random thresholds in the DELM model, the SSA algorithm is used to optimize the model’s parameters, which can generate the optimal parameter combinations. The results showed an enhanced generalization ability of the model by removal of redundant variables by Pearson correlation and gray correlation analysis, and higher accuracy in the prediction of landside displacement of rock slope by SSA-DELM compared to other traditional machine learning algorithms. The current study is significant in the literature on rock slope disaster analysis.

COVID-19 detection in CT and CXR images using deep learning models.

Infectious diseases pose a threat to human life and could affect the whole world in a very short time. Corona-2019 virus disease (COVID-19) is an example of such harmful diseases. COVID-19 is a pandemic of an emerging infectious disease, called coronavirus disease 2019 or COVID-19, caused by the coronavirus SARS-CoV-2, which first appeared in December 2019 in Wuhan, China, before spreading around the world on a very large scale. The continued rise in the number of positive COVID-19 cases has disrupted the health care system in many countries, creating a lot of stress for governing bodies around the world, hence the need for a rapid way to identify cases of this disease. Medical imaging is a widely accepted technique for early detection and diagnosis of the disease which includes different techniques such as Chest X-ray (CXR), Computed Tomography (CT) scan, etc. In this paper, we propose a methodology to investigate the potential of deep transfer learning in building a classifier to detect COVID-19 positive patients using CT scan and CXR images. Data augmentation technique is used to increase the size of the training dataset in order to solve overfitting and enhance generalization ability of the model. Our contribution consists of a comprehensive evaluation of a series of pre-trained deep neural networks: ResNet50, InceptionV3, VGGNet-19, and Xception, using data augmentation technique. The findings proved that deep learning is effective at detecting COVID-19 cases. From the results of the experiments it was found that by considering each modality separately, the VGGNet-19 model outperforms the other three models proposed by using the CT image dataset where it achieved 88.5% precision, 86% recall, 86.5% F1-score, and 87% accuracy while the refined Xception version gave the highest precision, recall, F1-score, and accuracy values which equal 98% using CXR images dataset. On the other hand, and by applying the average of the two modalities X-ray and CT, VGG-19 presents the best score which is 90.5% for the accuracy and the F1-score, 90.3% for the recall while the precision is 91.5%. These results enables to automatize the process of analyzing chest CT scans and X-ray images with high accuracy and can be used in cases where RT-PCR testing and materials are limited.

Enriched global horizontal irradiance prediction using novel ensemble improved backpropagation neural network

Penetration of solar energy into the power grid and smart grid is becoming an urge because of the continuous progress in industrialization and advancement. Requires a high accurate Global Horizontal Irradiance (GHI) prediction to achieve effective penetration of solar energy. This paper proposes a novel Ensemble Improved Backpropagation Neural Network (EIBPNN) with enhanced generalization ability because it is developed based on the various inputs’ individual improved backpropagation neural networks. Hence, the variance of individual IBPNN and input parameters based uncertainty are overcome and has the generic performance capability. The comparative analysis imparts the proposed prediction model results improved GHI prediction than the existing models. The proposed model has enriched GHI prediction with better generalization.

Multiscale similarity ensemble framework for remaining useful life prediction

Accurate prediction of remaining useful life (RUL) is crucially important to perform prognostics and health management. A new similarity-based autoencoder multiscale ensemble (Similarity-based AE MSEN) methodology is proposed in this paper to improve RUL prediction accuracy and characterize RUL prediction uncertainty by considering the differences in equipment degradation rates, monitoring data lengths and fault modes. Firstly, multiscale sliding window sets are designed to divide health index (HI) curves generated by autoencoder into multiscale HI segments, which are used to measure the similarities between training and testing units. Then, multiscale prediction results obtained from similar training units are fused by kernel density estimation to fit a RUL distribution and then provide uncertainty for RUL prediction. The proposed multiscale ensemble strategy can overcome accuracy limitation caused by a fixed time scale and enhance generalization ability. Analysis of experimental datasets shows that the proposed multiscale method achieves state-of -the-art prediction performance.

Improving Nonlinear Interpolation of K-Space Data Using Semi-Supervised Learning and Autoregressive Model.

Parallel magnetic resonance imaging (pMRI) accelerates data acquisition by undersampling k-space through an array of receiver coils. Finding accurate relationships between acquired and missing k-space data determines the interpolation performance and reconstruction quality. Autocalibration signals (ACS) are generally used to learn the interpolation coefficients for reconstructing the missing k-space data. Based on the estimation-approximation error analysis in machine learning, increasing training data size can reduce estimation error and therefore enhance generalization ability of the interpolator, but scanning time will be longer if more ACS data are acquired. We propose to augment training data using unacquired and acquired data outside of ACS region through semi-supervised learning idea and autoregressive model. Local neighbor unacquired k-space data can be used for training tasks and reducing the generalization error. Experimental results show that the proposed method outperforms the conventional methods by suppressing noise and aliasing artifacts.

MiniExpNet: A small and effective facial expression recognition network based on facial local regions

Deep networks based Facial Expression Recognition (FER) have shown excellent performance. Due to high computational cost and memory resources, it is hard to develop these FER models in practical scenarios. In this paper, we design a small and effective deep network for FER based on facial local regions. Firstly, Regions of Interest (ROIs) related to facial Action Units (AUs) are cropped by facial landmarks and represented by Histogram Local Binary Pattern (HLBP) feature. Secondly, an effective and efficient light-weight network named MiniExpNet, is proposed by human–machine collaborative strategy, and only has 60 K parameters. To further improve the performance, we extend the MiniExpNet by integrating the Attention Block and self-distillation mechanism. The Attention Block exploits the importance of different ROIs for FER. The self-distillation mechanism is used for enhancing generalization ability during training. Extensive comparisons are made on three widely-used FER datasets, CK+, RaFD and Oulu-CASIA. Compared to exiting light-weight networks, the proposed model provides a better balance between accuracy, model size and speed.

Support vector machines with the known feature-evolution priors

In traditional SVMs (support vector machines), each feature involved in an object is assumed to contribute equally to the classification accuracy. However, the equal importance assumption for each feature is inappropriate for classifying the objects. The existing weighting strategies usually utilize the fixed scaling factors to measure the importance of each feature and hence they are not good fits for feature evolution. In this paper, with the assumption that a Known Feature-Evolution Prior (KFEP) can be employed to capture importance distribution from the perspective of both feature granularity and object granularity, SVM-based machines called KFEP-SVMs are developed to capture the importance of features dynamically. Also, it is revealed that SVMs with the known feature-evolution priors are equivalent to SVMs on a higher feature space. Besides, we theoretically demonstrate that the proposed machines KFEP-SVMs with the known feature-evolution prior indeed have enhanced generalization ability, compared with the classic SVMs. Experimental results show the effectiveness of the proposed machines KFEP-SVMs.

Online Sequential Complex-Valued ELM for Noncircular Signals: Augmented Structures and Learning Algorithms

Online sequential extreme learning machine (OS-ELM) has become a popular online learning strategy for single-hidden layer feedforward neural networks, and complex-valued signals are ubiquitous in real applications. In order to cater for complex-valued signals, especially for noncircular signals, in this paper we extend OS-ELM to complex domain, and propose two augmented online sequential complex ELM models by incorporating the conjugates of the network input and the hidden layer respectively. In this way, the proposed models are equipped with the capability to capture the complete second-order statistics of noncircular signals, which results in the enhanced generalization ability. The corresponding regularized models are derived to avoid the possible overfitting problem. By exploiting the algebra structure resulted from the augmented architecture, several approaches to reducing the computational complexity are also proposed. Simulation results validate the efficiency of the proposed algorithms.

A Kernel Design Approach to Improve Kernel Subspace Identification

Subspace identification methods, such as canonical variate analysis (CVA), are noniterative tools suitable for the state-space modeling of multi-input, multi-output processes, e.g., industrial processes, using input-output data. To learn nonlinear system behavior, kernel subspace techniques are commonly used. However, the issue of kernel design must be given more attention because the type of kernel can influence the kind of nonlinearities that the model can capture. In this article, a new kernel design is proposed for CVA-based identification, which is a mixture of a global and local kernel to enhance generalization ability and includes a mechanism to vary the influence of each process variable into the model response. During validation, model hyper-parameters were tuned using random search. The overall method is called feature-relevant mixed kernel CVA (FR-MKCVA). Using an evaporator case study, the trained FR-MKCVA models show a better fit to observed data than those of single-kernel CVA, linear CVA, and neural net models under both interpolation and extrapolation scenarios. This work provides a basis for future exploration of deep and diverse kernel designs for system identification.

Rolling-Element Bearing Fault Diagnosis Using Improved LeNet-5 Network.

To address the problems of low recognition accuracy, slow convergence speed and weak generalization ability of traditional LeNet-5 network used in rolling-element bearing fault diagnosis, a rolling-element bearing fault diagnosis method using improved 2D LeNet-5 network is put forward. The following improvements to the traditional LeNet-5 network are made: the convolution and pooling layers are reasonably designed and the size and number of convolution kernels are carefully adjusted to improve fault classification capability; the batch normalization (BN) is adopted after each convolution layer to improve convergence speed; the dropout operation is performed after each full-connection layer except the last layer to enhance generalization ability. To further improve the efficiency and effectiveness of fault diagnosis, on the basis of improved 2D LeNet-5 network, an end-to-end rolling-element bearing fault diagnosis method based on the improved 1D LeNet-5 network is proposed, which can directly perform 1D convolution and pooling operations on raw vibration signals without any preprocessing. The results show that the improved 2D LeNet-5 network and improved 1D LeNet-5 network achieve a significant performance improvement than traditional LeNet-5 network, the improved 1D LeNet-5 network provides a higher fault diagnosis accuracy with a less training time in most cases, and the improved 2D LeNet-5 network performs better than improved 1D LeNet-5 network under small training samples and strong noise environment.

Transfer learning for neural network model in chlorophyll-a dynamics prediction.

Neural network models have been used to predict chlorophyll-a concentration dynamics. However, as model generalization ability decreases, (i) the performance of the models gradually decreases over time; (ii) the accuracy and performance of the models need to be improved. In this study, Transfer learning (TL) is employed to optimize neural network models (including feedforward neural networks (FNN), recurrent neural networks (RNN) and long short-term memory (LTSM)) and overcome these problems. Models using TL are able to reduce the influence of mutable data distribution and enhance generalization ability. Thus, it can improve the accuracy of prediction and maintain high performance in long-term applications. Also, TL is compared with parameter norm penalties (PNP) and dropout-two other methods used to improve model generalization ability. In general, TL has a better prediction effect than PNP and dropout. All the models, including FNN with different architectures, RNN and LSTM, as well as models optimized by PNP, dropout, and TL, are applied to an estuary reservoir in eastern China to predict chlorophyll-a dynamics at 5-min intervals. According to the results of this study, (i) models with TL produce the best prediction results; (ii) the original models and the models with PNP and dropout lose their ability to predict within 3 months, while TL models retain a high prediction accuracy.

Head Pose Estimation in the Wild Assisted by Facial Landmarks Based on Convolutional Neural Networks

Convolutional neural networks (CNNs) exhibit excellent performance on the head pose estimation problem under controllable conditions, but their generalization ability in the wild needs to be improved. To address this issue, we propose an approach involving the introduction of facial landmark information into the task simplifier and landmark heatmap generator constructed before the feed-forward neural network, which can use this information to normalize the face shape into a canonical shape and generate a landmark heatmap based on the transformed facial landmarks to assist in feature extraction, for enhancing generalization ability in the wild. Our method was trained on 300W-LP and tested on AFLW2000-3D. The result shows that for the same feed-forward neural network when our method is used to introduce facial landmark information into a CNN, accuracy improves from 88.5% to 99.0% and mean average error decreases from 5.94° to 1.46° on AFLW2000-3D. Furthermore, we evaluate our method on several datasets used for pose estimation and compare the result with AFLW2000-3D, finding that the features extracted by a CNN could not reflect the head pose efficiently, which limits the performance of the CNN on the head pose estimation problem in wild. By introducing facial landmarks, the CNN could extract features that reflect head pose more efficiently, thereby significantly improving the accuracy of head pose estimation in the wild.