Improve Generalization Ability Research Articles

Factors affected housing prices: taking Boston as an example

Abstract. Real estate price prediction plays a vital role in urban planning, investment decision-making, and risk management. However, existing prediction models often show problems such as insufficient generalization ability and susceptibility to outliers when faced with complex nonlinear relationships, multidimensional features, and noisy data. Therefore, choosing a model that can accurately capture complex patterns and has strong robustness has become the focus of research. This paper introduces the random forest model and compares it with multivariate linear regression, XGBoost, and support vector machine (SVM). Compared with the traditional regression model, the random forest model combines the flexibility of decision trees and the multi-level feature extraction ability of deep learning, and can better handle the complex nonlinear relationships in the Boston housing price dataset. The experimental results show that the random forest model has achieved excellent performance in all evaluation indicators, and the model accuracy indicators are distributed as MSE=8.2502, RMSE=2.8723, MAE=2.0668, and R^2=0.8875. These results show that the random forest model not only outperforms other models in prediction accuracy but also shows significant advantages in dealing with data complexity and improving generalization ability. Therefore, the random forest model provides an efficient and reliable tool for future real estate price prediction research and applications.

Generalization Enhancement of Visual Reinforcement Learning through Internal States.

Visual reinforcement learning is important in various practical applications, such as video games, robotic manipulation, and autonomous navigation. However, a major challenge in visual reinforcement learning is the generalization to unseen environments, that is, how agents manage environments with previously unseen backgrounds. This issue is triggered mainly by the high unpredictability inherent in high-dimensional observation space. To deal with this problem, techniques including domain randomization and data augmentation have been explored; nevertheless, these methods still cannot attain a satisfactory result. This paper proposes a new method named Internal States Simulation Auxiliary (ISSA), which uses internal states to improve generalization in visual reinforcement learning tasks. Our method contains two agents, a teacher agent and a student agent: the teacher agent has the ability to directly access the environment's internal states and is used to facilitate the student agent's training; the student agent receives initial guidance from the teacher agent and subsequently continues to learn independently. From another perspective, our method can be divided into two phases, the transfer learning phase and traditional visual reinforcement learning phase. In the first phase, the teacher agent interacts with environments and imparts knowledge to the vision-based student agent. With the guidance of the teacher agent, the student agent is able to discover more effective visual representations that address the high unpredictability of high-dimensional observation space. In the next phase, the student agent autonomously learns from the visual information in the environment, and ultimately, it becomes a vision-based reinforcement learning agent with enhanced generalization. The effectiveness of our method is evaluated using the DMControl Generalization Benchmark and the DrawerWorld with texture distortions. Preliminary results indicate that our method significantly improves generalization ability and performance in complex continuous control tasks.

Wasserstein local slow feature analysis and its application to process monitoring

Abstract Complex industrial processes are commonly characterized by dynamics, which results from the fact that there exists compensation of the closed-loop control and complex reflux during process operations. In this work, we propose a new method termed Wasserstein local slow feature analysis approach (WLSFA) used for monitoring the dynamic process, which can learn slowly varying information to capture the trend of process variations and characterize dynamics. Specifically, Wasserstein graph embedding based on optimal transport is proposed to preserve the local geometrical structure of raw data so that the reduced-dimensional representations can more faithfully reveal the underlying information of data, enhancing monitoring performance. Moreover, the l_2 -norm orthogonal constraint is incorporated into the objective function to improve generalization ability and alleviate the false alarm rate. On the basis of WLSFA model, a contribution plot in the sense of reconstruction is developed to locate the fault variables, which is beneficial to regulate abnormal conditions towards normal levels. Finally, the efficiency of the proposed approach is illustrated by a benchmark process and the application to a real-world fractionation process.

Multi‐Task Mixture Density Graph Neural Networks for Predicting Catalyst Performance

AbstractGraph neural networks (GNNs) have drawn more and more attention from material scientists and demonstrated a strong capacity to establish connections between structures and properties. However, with only unrelaxed structures provided as input, few GNN models can predict the thermodynamic properties of relaxed configurations with an acceptable level of error. In this work, a multi‐task (MT) architecture based on DimeNet++ and mixture density networks is developed to improve the performance of such task. Taking CO adsorption on Cu‐based single‐atom alloy catalysts as an example, the method can reliably predict CO adsorption energy with a mean absolute error of 0.087 eV from the initial CO adsorption structures without costly first‐principles calculations. Compared to other state‐of‐the‐art GNN methods, the model exhibits improved generalization ability when predicting the catalytic performance of out‐of‐distribution configurations, built with either unseen substrate surfaces or doping species. Further, the enhancement of expressivity has also been demonstrated on the IS2RE predicting task in the Open Catalyst 2020 project. The proposed MT GNN strategy can facilitate the catalyst discovery and optimization process.

Contrastive BiLSTM-enabled Health Representation Learning for Remaining Useful Life Prediction

Remaining useful life (RUL) prediction is of vital significance in prognostics health management tasks. Due to powerful learning capabilities, deep learning methods, particularly long short-term memory (LSTM) have been widely applied in RUL prediction. However, many existing deep learning approaches overlook the inherent ordered relationship between samples in the direct mapping from sliced data to RUL pattern. To capture the faithful and ordered health representation of a given system, a Contrastive Bidirectional LSTM-enabled Health Representation Learning (CBHRL) framework is proposed. Firstly, the supervised contrastive regression loss (SupCR) is implemented to extract continuous health representation. The SupCR is designed to rank the similarity among health representations from different samples, prompting them highly correlated with linear RUL label. Among the process of contrastive learning, the series odd-even decomposition (SOED) method is devised to construct multi-view degradation data, which improves generalization ability. Finally, since the health representation is constructed on basis of similarity, a new similarity prediction method is proposed as the complement of regression prediction method. Experimental results show the health representations extracted by CBHRL achieve improved ratio ranging from a minimum of 17.19% to a maximum of 291.30% in monotonicity, smoothness and trendability.

Graph augmentation for node-level few-shot learning

In graph few-shot learning, few-shot node classification (FSNC) at the node-level is a popular downstream task. Previous FSNC methods primarily rely on meta-learning or metric learning techniques, aiming to mine prior knowledge from the base classes. However, these methods still have some limitations that need to be addressed, namely: (1) conducting multiple tasks for parameter initialization leads to expensive time costs. (2) ignoring the rich information present in novel classes leads to model over-fitting. To address these issues, this paper proposes a novel graph augmentation method for FSNC on graph data, which includes both parameter initialization and parameter fine-tuning. Specifically, the parameter initialization conducts only one multi-classification task on the base classes, improving generalization ability and reducing time costs. The parameter fine-tuning is designed to include two data augmentation modules (i.e., support augmentation and shot augmentation) on the novel classes to mine the rich information, thus alleviating model over-fitting. As a result, this paper introduces the first graph augmentation method for FSNC. Experimental results showed that our method achieves supreme performance, compared with state-of-the-art FSNC methods.

Joint Variational Inference Network for domain generalization

Domain generalization is a machine learning task that involves training a model on a set of domains with the goal of achieving high performance on unseen domains. While most domain generalization methods focus on extracting domain-invariant features, they may ignore domain-specific information beyond object styles which is label-relevant for classification. In this paper, we first propose a two-step data generation process in which the domain label and observed data are sampled from two distributions sequentially, and accordingly develop an end-to-end Joint Variational Inference Network (JVINet) for domain generalization. JVINet is a framework that admits a variational-based discriminative structure, which the domain-specific latent variable is learned and integrated to enhance the feature for classification. When testing on unseen target domains, the potential beneficial domain information is hence utilized to improve generalization ability. The overall objective is to optimize the variational lower bound of the conditional joint likelihood functions for both class and domain labels. We provide theoretical proof that JVINet can achieve an optimal lower bound using a variational-based discriminative approach. To evaluate the effectiveness of our method, we compare it with state-of-the-art methods on both simulated and real-world datasets.

Advancing Ligand Docking through Deep Learning: Challenges and Prospects in Virtual Screening.

Molecular docking, also termed ligand docking (LD), is a pivotal element of structure-based virtual screening (SBVS) used to predict the binding conformations and affinities of protein-ligand complexes. Traditional LD methodologies rely on a search and scoring framework, utilizing heuristic algorithms to explore binding conformations and scoring functions to evaluate binding strengths. However, to meet the efficiency demands of SBVS, these algorithms and functions are often simplified, prioritizing speed over accuracy.The emergence of deep learning (DL) has exerted a profound impact on diverse fields, ranging from natural language processing to computer vision and drug discovery. DeepMind's AlphaFold2 has impressively exhibited its ability to accurately predict protein structures solely from amino acid sequences, highlighting the remarkable potential of DL in conformation prediction. This groundbreaking advancement circumvents the traditional search-scoring frameworks in LD, enhancing both accuracy and processing speed and thereby catalyzing a broader adoption of DL algorithms in binding pose prediction. Nevertheless, a consensus on certain aspects remains elusive.In this Account, we delineate the current status of employing DL to augment LD within the VS paradigm, highlighting our contributions to this domain. Furthermore, we discuss the challenges and future prospects, drawing insights from our scholarly investigations. Initially, we present an overview of VS and LD, followed by an introduction to DL paradigms, which deviate significantly from traditional search-scoring frameworks. Subsequently, we delve into the challenges associated with the development of DL-based LD (DLLD), encompassing evaluation metrics, application scenarios, and physical plausibility of the predicted conformations. In the evaluation of LD algorithms, it is essential to recognize the multifaceted nature of the metrics. While the accuracy of binding pose prediction, often measured by the success rate, is a pivotal aspect, the scoring/screening power and computational speed of these algorithms are equally important given the pivotal role of LD tools in VS. Regarding application scenarios, early methods focused on blind docking, where the binding site is unknown. However, recent studies suggest a shift toward identifying binding sites rather than solely predicting binding poses within these models. In contrast, LD with a known pocket in VS has been shown to be more practical. Physical plausibility poses another significant challenge. Although DLLD models often achieve higher success rates compared to traditional methods, they may generate poses with implausible local structures, such as incorrect bond angles or lengths, which are disadvantageous for postprocessing tasks like visualization. Finally, we discuss the future perspectives for DLLD, emphasizing the need to improve generalization ability, strike a balance between speed and accuracy, account for protein conformation flexibility, and enhance physical plausibility. Additionally, we delve into the comparison between generative and regression algorithms in this context, exploring their respective strengths and potential.

High-Dimensional Analysis for Generalized Nonlinear Regression: From Asymptotics to Algorithm

Overparameterization often leads to benign overfitting, where deep neural networks can be trained to overfit the training data but still generalize well on unseen data. However, it lacks a generalized asymptotic framework for nonlinear regressions and connections to conventional complexity notions. In this paper, we propose a generalized high-dimensional analysis for nonlinear regression models, including various nonlinear feature mapping methods and subsampling. Specifically, we first provide an implicit regularization parameter and asymptotic equivalents related to a classical complexity notion, i.e., effective dimension. We then present a high-dimensional analysis for nonlinear ridge regression and extend it to ridgeless regression in the under-parameterized and over-parameterized regimes, respectively. We find that the limiting risks decrease with the effective dimension. Motivated by these theoretical findings, we propose an algorithm, namely RFRed, to improve generalization ability. Finally, we validate our theoretical findings and the proposed algorithm through several experiments.

Small object detection algorithm incorporating swin transformer for tea buds.

Accurate identification of small tea buds is a key technology for tea harvesting robots, which directly affects tea quality and yield. However, due to the complexity of the tea plantation environment and the diversity of tea buds, accurate identification remains an enormous challenge. Current methods based on traditional image processing and machine learning fail to effectively extract subtle features and morphology of small tea buds, resulting in low accuracy and robustness. To achieve accurate identification, this paper proposes a small object detection algorithm called STF-YOLO (Small Target Detection with Swin Transformer and Focused YOLO), which integrates the Swin Transformer module and the YOLOv8 network to improve the detection ability of small objects. The Swin Transformer module extracts visual features based on a self-attention mechanism, which captures global and local context information of small objects to enhance feature representation. The YOLOv8 network is an object detector based on deep convolutional neural networks, offering high speed and precision. Based on the YOLOv8 network, modules including Focus and Depthwise Convolution are introduced to reduce computation and parameters, increase receptive field and feature channels, and improve feature fusion and transmission. Additionally, the Wise Intersection over Union loss is utilized to optimize the network. Experiments conducted on a self-created dataset of tea buds demonstrate that the STF-YOLO model achieves outstanding results, with an accuracy of 91.5% and a mean Average Precision of 89.4%. These results are significantly better than other detectors. Results show that, compared to mainstream algorithms (YOLOv8, YOLOv7, YOLOv5, and YOLOx), the model improves accuracy and F1 score by 5-20.22 percentage points and 0.03-0.13, respectively, proving its effectiveness in enhancing small object detection performance. This research provides technical means for the accurate identification of small tea buds in complex environments and offers insights into small object detection. Future research can further optimize model structures and parameters for more scenarios and tasks, as well as explore data augmentation and model fusion methods to improve generalization ability and robustness.

Macro-microscopic study on the crack resistance of polyester fiber asphalt mixture under dry-wet cycling and neural network prediction

Asphalt mixture is a composite material with a complex multiphase dispersed system, and its macroscopic mechanical behavior is inherently related to the microstructure characteristics. To investigate the low-temperature crack resistance degradation law of polyester fiber asphalt mixture under different dry-wet cycling conditions, six polyester fiber contents (0%, 0.3%, 0.35%, 0.4%, 0.45%, and 0.5%) were dry-blended into SMA-13 asphalt mixture to prepare fiber-reinforced SCB specimens with pre-notches. Semi-circular bending tests were conducted to test the crack resistance performance of the fiber asphalt mixture after 0, 2, 4, 6, and 8 dry-wet cycles. The experimental results show that the crack resistance performance of specimens with different fiber contents decreases with the increase of dry-wet cycling times, and the influence of salt-dry-wet coupling is greater than that of water-dry-wet coupling. Under the same conditions, the crack resistance index (CRI) increases first and then decreases with the increase of polyester fiber content, reaching its maximum at a polyester fiber content of 0.4%. When the fiber is added at 0.5%, the agglomeration of polyester fibers restricts the low-temperature performance of the specimen. In addition, the DIC technology is used to analyze the trend of horizontal strain variation of specimens after different dry-wet cycles. The results indicate that with the increase of the cycle period, the strain concentration area becomes more apparent, and in the destruction stage, the full-field horizontal strain Exx in the crack concentration zone gradually increases, while the crack resistance decreases. Finally, addressing the limitations of traditional BP neural networks in solving nonlinear problems, a particle swarm optimization algorithm is proposed to optimize the BP neural network, and a PSO-BP neural network model is constructed. Through the analysis of evaluation indicators, it is found that the PSO-BP neural network has improved generalization ability compared to the traditional BP neural network model, and overfitting is reduced, making it an effective tool for predicting the low-temperature performance of polyester fiber asphalt mixtures.

Estimation of road friction coefficient based on WOA-BP neural network

The road friction coefficient is often estimated based on vehicle dynamics models, but the ideal mathematical model is difficult to describe the actual vehicle operation, which in turn leads to deviations between the estimated friction coefficient and the actual situation. In order to take full account of the actual vehicle driving conditions, back propagation (BP) neural network model is often applied to assess the friction coefficient. To further improve the generalization ability of the network model, this paper proposes a method to predict the friction coefficient using the whale optimized algorithm-BP (WOA-BP) neural network model. The whale algorithm improves the weights and thresholds of the BP neural network model to improve generalization ability and convergence speed. First, the vehicle dynamics analysis is utilized to determine the vehicle dynamics parameters as a function of the friction coefficient. Then, the WOA-BP is compared against the BP under various realistic operating scenarios. The results demonstrate that the WOA-BP estimator is accurate and efficient in estimating the friction coefficient, and the proposed method reduces the mean absolute error by 51.9% and the root mean squared error by 49.1% compared to the BP.

An improved model combining machine learning and Kalman filtering architecture for state of charge estimation of lithium-ion batteries

Accurate state of charge (SOC) estimation of lithium-ion batteries is a fundamental prerequisite for ensuring the normal and safe operation of electric vehicles, and it is also a key technology component in battery management systems. In recent years, lithium-ion battery SOC estimation methods based on data-driven approaches have gained significant popularity. However, these methods commonly face the issue of poor model generalization and limited robustness. To address such issues, this study proposes a closed-loop SOC estimation method based on simulated annealing-optimized support vector regression (SA-SVR) combined with minimum error entropy based extended Kalman filter (MEE-EKF) algorithm. Firstly, a probability-based SA algorithm is employed to optimize the internal parameters of the SVR, thereby enhancing the precision of original SOC estimation. Secondly, utilizing the framework of the Kalman filter, the optimized SVR results are incorporated as the measurement equation and further processed through the MEE-EKF, while the ampere-hour integral physical model serves as the state equation, effectively attenuating the measurement noise, enhancing the estimation accuracy, and improving generalization ability. The proposed method is validated through battery testing experiments conducted under three typical operating conditions and one complex and random operating condition with wide temperature variations under only one condition training. The results demonstrate that the proposed method achieves a mean absolute error below 0.60% and a root mean square error below 0.73% across all operating conditions, showcasing a significant improvement in estimation accuracy compared to the benchmark algorithms. The high precision and generalization capability of the proposed method are evident, ensuring accurate SOC estimation for electric vehicles.

Multiview Large Margin Distribution Machine.

Margin distribution has been proven to play a crucial role in improving generalization ability. In recent studies, many methods are designed using large margin distribution machine (LDM), which combines margin distribution with support vector machine (SVM), such that a better performance can be achieved. However, these methods are usually proposed based on single-view data and ignore the connection between different views. In this article, we propose a new multiview margin distribution model, called MVLDM, which constructs both multiview margin mean and variance. Besides, a framework is proposed to achieve multiview learning (MVL). MVLDM provides a new way to explore the utilization of complementary information in MVL from the perspective of margin distribution and satisfies both the consistency principle and the complementarity principle. In the theoretical analysis, we used Rademacher complexity theory to analyze the consistency error bound and generalization error bound of the MVLDM. In the experiments, we constructed a new performance metric, the view consistency rate (VCR), for the characteristics of multiview data. The effectiveness of MVLDM was evaluated using both VCR and other traditional performance metrics. The experimental results show that MVLDM is superior to other benchmark methods.

Self-constructed strategy-based reinforcement LSTM approach for fiber-reinforced polymer non-linear degradation performance analysis

The performance degradation of fiber-reinforced polymer (FRP) is a typical sequential data with a highly non-linear evolution pattern. In this study, a Self-constructed strategy-based reinforcement LSTM approach (short for SCRLA) is proposed to self-accommodate the non-linear sequential data, reduce modeling burden, and improve generalization ability. SCRLA incorporates Bayesian algorithms introducing the uncertainty of hyperparameter optimization by a probabilistic distribution of implicit objectives and self-constructed a high-dimensional, reinforced machine model that can learn and predict non-linear representations. In the case study, the datasets with different properties (one consisting of finite element analysis (FEA) results and one of real experimental (EXP) data) are selected to verify the validity of the degradation performance predictions. It is shown that the reinforced LSTM based on SCRLA is more suitable for the non-linear degradation performance analysis of FRP, especially with higher prediction accuracy for EXP data. The establishment of the approach and model provides a feasible idea and framework for the prediction of composites' sequential performance.

Intelligent rolling bearing compound fault diagnosis based on frequency-domain Gramian angular field and convolutional neural networks with imbalanced data

The effective fault feature extraction is the core of rolling bearing fault diagnosis. However, rolling bearings usually operate in normal state and fault duration is very short, which will cause imbalance in fault diagnosis data, thus leading to difficulty in fault feature extraction and low diagnosis accuracy. Meanwhile, mutual interference between multiple fault responses will also lead to poor diagnosis performance. To solve these issues, a novel compound fault diagnosis method with imbalanced data based on frequency-domain Gramian angular field (FGAF) and convolutional neural networks optimized by instance normalization and efficient channel attention (IECNN) is proposed. Firstly, FGAF is adopted to map frequency-domain features of fault signals to the polar coordinate to obtain 2D FGAF feature spectrum. Secondly, an instance normalization module is established to reduce internal covariant shift caused by data distribution discrepancy and improve generalization ability. An efficient channel attention module is constructed to further excavate fault features and improve anti-interference ability. Finally, experiments are conducted under imbalanced dataset and imbalance intensified dataset, and the average accuracy of 99.91% and 99.92% were obtained, respectively, which shows the proposed method has better resistance to data imbalance.

Distributed deep reinforcement learning based on bi-objective framework for multi-robot formation

Improving generalization ability in multi-robot formation can reduce repetitive training and calculation. In this paper, we study the multi-robot formation problem with the ability to generalize the target position. Since the generalization ability of neural network is directly proportional to spatial dimension, we adopt the strategy of using different networks to solve different objectives, so that the network learning can focus on the learning of one objective to obtain better performance. In addition, this paper presents a distributed deep reinforcement learning method based on soft actor–critic algorithm for solving multi-robot formation problem. At the same time, the formation evaluation assignment function is designed to adapt to distributed training. Compared with the original algorithm, the improved algorithm can get higher reward cumulative values. The experimental results show that the proposed algorithm can better maintain the desired formation in the moving process, and the rotation design in the reward function makes the multi-robot system have better flexibility in formation. The comparison of control signal curve shows that the proposed algorithm is more stable. At the end of the experiments, the universality of the proposed algorithm in formation maintenance and formation variations is demonstrated.

Machine learning-aided hydrothermal carbonization of biomass for coal-like hydrochar production: Parameters optimization and experimental verification

Biomass to coal-like hydrochar via hydrothermal carbonization (HTC) is a promising route for sustainability development. Yet conventional experimental method is time-consuming and costly to optimize HTC conditions and characterize hydrochar. Herein, machine learning was employed to predict the fuel properties of hydrochar. Random forest (RF), support vector machine (SVM), and extreme gradient boosting (XGB) models were developed, presenting acceptable prediction performance with R2 at 0.825–––0.985 and root mean square error (RMSE) at 1.119–––5.426, and XGB outperformed RF and SVM. The model interpretation indicated feedstock ash content, reaction temperature, and solid to liquid ratio were the three decisive factors. The optimized XGB multi-task model via feature re-examination illustrated improved generalization ability with R2 at 0.927 and RMSE at 3.279. Besides, the parameters optimization and experimental verification with wheat straw as feedstock further demonstrated the huge application potential of machine learning in hydrochar engineering.

PhyNRnet: Physics-informed Newton-Raphson Network for Forward Kinematics Solution of Parallel Manipulators

Abstract Despite significant performance advantages, the intractable forward kinematics have always restricted the application of parallel manipulators to small posture spaces. Traditional analytical method and Newton-Raphson method usually cannot solve this problem well due to lack of generality or latent divergence. To address this issue, this study employs recent advances in deep learning to propose a novel physics-informed Newton-Raphson network (PhyNRnet) to rapidly and accurately solve this forward kinematics problem for general parallel manipulators. The main strategy of PhyNRnet is to combine the Newton-Raphson method with the neural network, which helps to significantly improve the accuracy and convergence speed of the model. In addition, to facilitate the network optimization, semi-autoregression, hard imposition of initial/boundary conditions (I/BCs), batch normalization, etc. are developed and applied in PhyNRnet. Unlike previous data-driven paradigms, PhyNRnet adopts the physics-informed loss functions to guide the network optimization, which gives the model clear physical meaning and helps improve generalization ability. Finally, the performance of PhyNRnet is verified by three parallel manipulator paradigms with large postures, where the Newton-Raphson method has generally diverged. Besides, the efficiency analysis shows that PhyNRnet consumes only a small amount of time at each time step, which meets the real-time requirements.

Symmetry-aware Neural Architecture for Embodied Visual Navigation

AbstractThe existing methods for addressing visual navigation employ deep reinforcement learning as the standard tool for the task. However, they tend to be vulnerable to statistical shifts between the training and test data, resulting in poor generalization over novel environments that are out-of-distribution from the training data. In this study, we attempt to improve the generalization ability by utilizing the inductive biases available for the task. Employing the active neural SLAM that learns policies with the advantage actor-critic method as the base framework, we first point out that the mappings represented by the actor and the critic should satisfy specific symmetries. We then propose a network design for the actor and the critic to inherently attain these symmetries. Specifically, we use G-convolution instead of the standard convolution and insert the semi-global polar pooling layer, which we newly design in this study, in the last section of the critic network. Our method can be integrated into existing methods that utilize intermediate goals and 2D occupancy maps. Experimental results show that our method improves generalization ability by a good margin over visual exploration and object goal navigation, which are two main embodied visual navigation tasks.