Generalization Ability Of Neural Networks Research Articles

Study on Gesture Recognition Method with Two-Stream Residual Network Fusing sEMG Signals and Acceleration Signals.

Currently, surface EMG signals have a wide range of applications in human-computer interaction systems. However, selecting features for gesture recognition models based on traditional machine learning can be challenging and may not yield satisfactory results. Considering the strong nonlinear generalization ability of neural networks, this paper proposes a two-stream residual network model with an attention mechanism for gesture recognition. One branch processes surface EMG signals, while the other processes hand acceleration signals. Segmented networks are utilized to fully extract the physiological and kinematic features of the hand. To enhance the model's capacity to learn crucial information, we introduce an attention mechanism after global average pooling. This mechanism strengthens relevant features and weakens irrelevant ones. Finally, the deep features obtained from the two branches of learning are fused to further improve the accuracy of multi-gesture recognition. The experiments conducted on the NinaPro DB2 public dataset resulted in a recognition accuracy of 88.25% for 49 gestures. This demonstrates that our network model can effectively capture gesture features, enhancing accuracy and robustness across various gestures. This approach to multi-source information fusion is expected to provide more accurate and real-time commands for exoskeleton robots and myoelectric prosthetic control systems, thereby enhancing the user experience and the naturalness of robot operation.

Distributed deep reinforcement learning based on bi-objective framework for multi-robot formation

Improving generalization ability in multi-robot formation can reduce repetitive training and calculation. In this paper, we study the multi-robot formation problem with the ability to generalize the target position. Since the generalization ability of neural network is directly proportional to spatial dimension, we adopt the strategy of using different networks to solve different objectives, so that the network learning can focus on the learning of one objective to obtain better performance. In addition, this paper presents a distributed deep reinforcement learning method based on soft actor–critic algorithm for solving multi-robot formation problem. At the same time, the formation evaluation assignment function is designed to adapt to distributed training. Compared with the original algorithm, the improved algorithm can get higher reward cumulative values. The experimental results show that the proposed algorithm can better maintain the desired formation in the moving process, and the rotation design in the reward function makes the multi-robot system have better flexibility in formation. The comparison of control signal curve shows that the proposed algorithm is more stable. At the end of the experiments, the universality of the proposed algorithm in formation maintenance and formation variations is demonstrated.

Predicting indoor particle concentration in mechanically ventilated classrooms using neural networks: Model development and generalization ability analysis

This study presents a comprehensive analysis of the model development (structure, parameter settings, and prediction accuracy) and generalization ability of neural networks in the classroom setting. A normalized database for model prediction and verification including indoor parameters from 27 high school classrooms over the course of four semesters was set up (a total of 14,112 sets of data for each variable). Three different models were compared: a back propagation artificial neural network (BP-ANN), a nonlinear autoregressive exogenous neural network (NARX), and a long-term and short-term memory neural network (LSTM). For the prediction of PM1 and PM2.5, which are impacted by outdoor PM concentrations, the best performance was achieved by the NARX model: the mean absolute percentage error (MAPE), root mean squared error (RMSE), and coefficient of determination (R2) were 41–55%, 0.45–1.27 μg/m3, and 0.81–0.87, respectively. The R2 of this model was 22% higher than that of the BP-ANN model and much higher than that of the LSTM model. For the prediction of indoor PM10, which are mostly emitted by indoor sources, relatively good performance was achieved by the LSTM model. The trained NARX model was used successfully to predict indoor time-series PM2.5 concentration and source strength in classrooms in five Chinese cities and five American cities. The calculated indoor source strength was 95.5–119.1 μg/h/P, within the range of the measured indoor source strength. The indoor PM2.5 concentrations calculated by a theoretical equation without the consideration of indoor source were 20.8–60.0% smaller than those calculated by the trained model.

Unsupervised Anomaly Detection Approach Based on Adversarial Memory Autoencoders for Multivariate Time Series

The widespread usage of Cyber Physical Systems (CPSs) generates a vast volume of time series data, and precisely determining anomalies in the data is critical for practical production. Autoencoder is the mainstream method for time series anomaly detection, and the anomaly is judged by reconstruction error. However, due to the strong generalization ability of neural networks, some abnormal samples close to normal samples may be judged as normal, which fails to detect the abnormality. In addition, the dataset rarely provides sufficient anomaly labels. This research proposes an unsupervised anomaly detection approach based on adversarial memory autoencoders for multivariate time series to solve the above problem. Firstly, an encoder encodes the input data into low-dimensional space to acquire a feature vector. Then, a memory module is used to learn the feature vector’s prototype patterns and update the feature vectors. The updating process allows partial forgetting of information to prevent model overgeneralization. After that, two decoders reconstruct the input data. Finally, this research uses the Peak Over Threshold (POT) method to calculate the threshold to determine anomalous samples from normal samples. This research uses a two-stage adversarial training strategy during model training to enlarge the gap between the reconstruction error of normal and abnormal samples. The proposed method achieves significant anomaly detection results on synthetic and real datasets from power systems, water treatment plants, and computer clusters. The F1 score reached an average of 0.9196 on the five datasets, which is 0.0769 higher than the best baseline method.

ASAT: Adaptively scaled adversarial training in time series

Adversarial training is a method for enhancing neural networks to improve the robustness against adversarial examples. Besides the security concerns of potential adversarial examples, adversarial training can also improve the generalization ability of neural networks, train robust neural networks, and provide interpretability for neural networks. In this work, we introduce adversarial training in time series analysis to enhance the neural networks for better generalization ability by taking the finance field as an example. Rethinking existing research on adversarial training, we propose the adaptively scaled adversarial training (ASAT) in time series analysis, by rescaling data at different time slots with adaptive scales. Experimental results show that the proposed ASAT can improve both the generalization ability and the adversarial robustness of neural networks compared to the baselines. Compared to the traditional adversarial training algorithm, ASAT can achieve better generalization ability and similar adversarial robustness.

Application of Machine Learning Combined with Wireless Network in Design of Online Translation System

The importance of translation services has become increasingly prominent with the acceleration of economic globalization. Compared with human translation, machine translation is cheaper and faster, and therefore more suitable for the current era. The current mainstream machine translation method is neural machine translation, which employs machine methods to train on parallel corpora and create translation models. Research into neural machine translation has yielded a wealth of information. Learning and generalization abilities of neural networks have substantially enhanced the effectiveness of neural machine translation. This work applies machine learning and wireless network technology to build an online translation system for real-time translation. First, this work proposes a multigranularity feature fusion method based on a directed acyclic graph, which uses a directed acyclic graph to fuse different granularities as input and obtain a position representation. Secondly, this paper improves the Transformer model and proposes multigranularity position encoding and multigranularity self-attention. Then, on the basis of multigranularity features as input, this work introduces dynamic word vectors to improve the word embedding module, and uses the ELMo model to obtain dynamic word vector embeddings. Finally, this work builds a multigranularity feature-dynamic word vector machine translation model with above strategy, deploys it on server. Users can upload the content to be translated and download the translated content through the wireless network and realize an online translation system based on machine learning and wireless network.

Generative Adversarial Training for Supervised and Semi-supervised Learning.

Neural networks have played critical roles in many research fields. The recently proposed adversarial training (AT) can improve the generalization ability of neural networks by adding intentional perturbations in the training process, but sometimes still fail to generate worst-case perturbations, thus resulting in limited improvement. Instead of designing a specific smoothness function and seeking an approximate solution used in existing AT methods, we propose a new training methodology, named Generative AT (GAT) in this article, for supervised and semi-supervised learning. The key idea of GAT is to formulate the learning task as a minimax game, in which the perturbation generator aims to yield the worst-case perturbations that maximize the deviation of output distribution, while the target classifier is to minimize the impact of this perturbation and prediction error. To solve this minimax optimization problem, a new adversarial loss function is constructed based on the cross-entropy measure. As a result, the smoothness and confidence of the model are both greatly improved. Moreover, we develop a trajectory-preserving-based alternating update strategy to enable the stable training of GAT. Numerous experiments conducted on benchmark datasets clearly demonstrate that the proposed GAT significantly outperforms the state-of-the-art AT methods in terms of supervised and semi-supervised learning tasks, especially when the number of labeled examples is rather small in semi-supervised learning.

A new method to improve the generalization ability of neural networks: A case study of nuclear mass training

As one of the fundamental input quantities in nuclear physics, an accurate calculation of the nucleus mass is necessary. Currently, neural network methods improve the accuracy of mass models. However, the reliability for extrapolation cannot be predicted. In this paper, a training method of neural network method is proposed. A fitting method similar to the macroscopic microscopic mass model is used. After fitting the liquid drop model the neural network method is used for training, and then afterwards the parameters are refitted and the neural network model is retrained, and so on several times. Taking the Weizsäcker-Skyrme (WS) model as an example, for known nuclei, it can improve the calculation results by about 20% over the original results. The extrapolation results for unknown neutron-deficient nuclei are also significantly improved. We believe that this method can calculate the masses of partially unknown nuclei more accurately. Further, it may be possible to improve the extrapolation capability of the neural network method using this approach. It is hoped that the present work can be useful for nuclear physics experiments and applications of neural network methods.

Absorption in the wavemaker by Artificial neural networks (ANNs)

Wave-making theories are becoming available, but their applicability is limited to specific ranges of waves and wavemaker types. Machine learning can also be used to discover nonlinear functional relationships. As a result, based on machine learning, this paper proposes a simple and universal framework for generating and absorbing waves.This framework trains neural networks to determine the transfer function between the wavemaker's free-surface elevation and velocity. To increase the generalization ability of neural networks, penalty term and data augmentation techniques based on wave-making mechanisms are introduced, rather than pure data-driven.As a result, once the wavemaker has the target wave profiles in front of it, it can generate waves while also absorbing reflected waves. Analytical solutions are used to validate the simulated wave profiles and wave orbital velocities, demonstrating that the proposed framework is effective at eliminating the re-reflection wave.The validation for generating the solitary wave and the New Year's wave is then performed, indicating that the generated waves agree very well with the desired wave elevation. The proposed framework can help with wavemaker design in the future, and it does not require any complex theoretical derivation.

Gas path parameter prediction of aero-engine based on an autoregressive discrete convolution sum process neural network

In order to improve the approximation ability of neural network to functional and improve the prediction accuracy of time series data, this paper proposes an autoregressive discrete convolution sum process neural network prediction model and applies it to the prediction of aero-engine gas path performance parameters. Coupling threshold wavelet de-noising method is applied to the preprocessing of engine gas path parameters, which can effectively remove the noise in the time series data. Process neurons are applied to artificial neural networks to expand the computational functions of artificial neurons. The process neuron can not only realize the spatial aggregation operation of discrete input data and the connection weight, but also realize the time aggregation operation of the product integral of the continuous input function and the connection weight function. The output of discrete time series is affected by the current input/output, and also by the historical input/output value. Therefore, an autoregressive feedback link is added to the network topology, and the value of the network's output node is used as the input to adjust the network connection weight. The convolution sum operation of discrete time series data can realize the time accumulation effect, so the discrete convolution sum operator is used to replace the integral operator to realize the time aggregation function of the process neuron. Since the integral operation of the continuous function is avoided, the weight training process of the process neural network is effectively simplified, and the accuracy loss in the continuous-time function fitting process of discrete input samples is effectively avoided. Bayesian regularization network weight learning algorithm is used to solve the problems of slow convergence speed and easy to fall into local optimum of back propagation learning algorithm, and improve the generalization ability of neural network. It can be seen from the prediction simulation results of the two engines that the average relative error and root mean square error of the engine gas path parameter prediction based on the autoregressive discrete convolution sum process neural network model can be reduced to 0.67% and 0.35 respectively. The stable and high-precision prediction results show that the network model and weight learning algorithm proposed in this paper have better robustness in functional approximation, and have higher accuracy in time series data prediction.

Research On Pre-Training Method and Generalization Ability of Big Data Recognition Model of the Internet of Things

The Internet of Things and big data are currently hot concepts and research fields. The mining, classification, and recognition of big data in the Internet of Things system are the key links that are widely of concern at present. The artificial neural network is beneficial for multi-dimensional data classification and recognition because of its strong feature extraction and self-learning ability. Pre-training is an effective method to address the gradient diffusion problem in deep neural networks and could result in better generalization. This article focuses on the performance of supervised pre-training that uses labelled data. In particular, this pre-training procedure is a simulation that shows the changes in judgment patterns as they progress from primary to mature within the human brain. In this article, the state-of-the-art of neural network pre-training is reviewed. Then, the principles of the auto-encoder and supervised pre-training are introduced in detail. Furthermore, an extended structure of supervised pre-training is proposed. A set of experiments are carried out to compare the performances of different pre-training methods. These experiments include a comparison between the original and pre-trained networks as well as a comparison between the networks with two types of sub-network structures. In addition, a homemade database is established to analyze the influence of pre-training on the generalization ability of neural networks. Finally, an ordinary convolutional neural network is used to verify the applicability of supervised pre-training.

SAR Image Classification Using Contrastive Learning and Pseudo-Labels With Limited Data

Synthetic aperture radar (SAR) image classification based on deep learning methods has attracted huge enthusiasm among researchers. Sufficient training data guarantee the generalization ability of neural networks. However, it is quite difficult to collect large amount of high-quality and labeled SAR data in real situations. In this letter, a method based on contrastive learning and pseudo-labels is proposed to solve the few-shot SAR image classification problem. First, a siamese structure is introduced to learn semantic representations of SAR target images. The feature vectors obtained in this stage can reveal the similarity between bi-temporal SAR images. Then, the backbone network of the siamese network is connected to a classification head for instance-level classification. The pseudo-label strategy is adopted to gradually improve the performance. A loss function that changes with iteration is designed to optimize the training process with true labels and pseudo-labels. Besides, considering the imprecision of single-network architecture, we adopt dual-networks and cross-training strategies. The final result is obtained by integrating the results of the dual-networks. Ablation studies show that the designed components of the proposed method are effective. Comparative experiments on the moving and stationary target acquisition and recognition (MSTAR) data set demonstrate the superior performance of the proposed method when dealing with limited training data in the SAR classification.

Bidirectional Retrospective Generation Adversarial Network for Anomaly Detection in Videos

Anomaly detection in videos is the task of identifying frames from a video sequence that depict events that do not conform to expected behavior, which is an extremely challenging task due to the ambiguous and unbounded properties of anomalies. With the development of deep learning, video anomaly detection methods based on deep neural networks have made great progress. The existing methods mainly follow two routes, namely, frame reconstruction and frame prediction. Due to the powerful generalization ability of neural networks, the application of reconstruction-based methods is limited. Recently, anomaly detection methods based on prediction have achieved advanced performance. However, their performance suffers when they cannot guarantee lower prediction errors for normal events. In this paper, we propose a novel future frame prediction model based on a bidirectional retrospective generation adversarial network (BR-GAN) for anomaly detection. To predict a future frame with higher quality for normal events, first, we propose a bidirectional prediction combined with a retrospective prediction method to fully mine the bidirectional temporal information between the predicted frame and the input frame sequence. Then, the intensity and gradient loss between the predicted frame and the actual frame together with an adversarial loss are used for appearance (spatial) constraints. In addition, we propose a sequence discriminator composed of a 3-dimensional (3D) convolutional neural network to capture the long-term temporal relationships between frame sequences composed of predicted frames and input frames; this network plays a crucial role in maintaining the motion (temporal) consistency of the predicted frames for normal events. Such appearance and motion constraints further facilitate future frame prediction for normal events, and thus, the prediction network can be highly capable of distinguishing normal and abnormal patterns. Extensive experiments on benchmark datasets demonstrate that our method outperforms most existing state-of-the-art methods, validating the effectiveness of our method for anomaly detection.

Optimized neural network for temperature extraction from Brillouin scattering spectra

Abstract The nonlinear mapping characteristic of artificial neural network (ANN) is suitable for temperature extraction from Brillouin scattering spectra in optical fiber sensing system. To further improve the generalization ability of neural network, an optimized method for ANN training is proposed in this paper. Firstly, a set of noisy training data with different linewidth under various temperatures and frequency scanning intervals is constructed by using Pseudo-Voigt function and is entered into networks for training. Then, the ANNs with optimized parameters are tested by the measured Brillouin scattering spectra, which are from an established Brillouin optical time domain reflectometry (BOTDR) sensing system. Finally, the temperature distribution information of the sensing fiber is extracted directly. The experimental results show that the ANNs trained by the proposed method obtain better temperature extraction accuracy than that obtained by other ANNs, which indicates that the generalization ability and adaptability of ANN are enhanced for temperature extraction in Brillouin optical fiber sensing system.

Encouraging an appropriate representation simplifies training of neural networks

Abstract A common assumption about neural networks is that they can learn an appropriate internal representation on their own, see e.g. end-to-end learning. In this work we challenge this assumption. We consider two simple tasks and show that the state-of-the-art training algorithm fails, although the model itself is able to represent an appropriate solution. We will demonstrate that encouraging an appropriate internal representation allows the same model to solve these tasks. While we do not claim that it is impossible to solve these tasks by other means (such as neural networks with more layers), our results illustrate that integration of domain knowledge in form of a desired internal representation may improve the generalization ability of neural networks.

Feature Variance Regularization: A Simple Way to Improve the Generalizability of Neural Networks

To improve the generalization ability of neural networks, we propose a novel regularization method that regularizes the empirical risk using a penalty on the empirical variance of the features. Intuitively, our approach introduces confusion into feature extraction and prevents the models from learning features that may relate to specific training samples. According to our theoretical analysis, our method encourages models to generate closer feature distributions for the training set and unobservable true data and minimize the expected risk as well, which allows the model to adapt to new samples better. We provide a thorough empirical justification of our approach, and achieves a greater improvement than other regularization methods. The experimental results show the effectiveness of our method on multiple visual tasks, including classification (CIFAR100, ImageNet, fine-grained datasets) and semantic segmentation (Cityscapes).

Research of Mapping Element Space Method for Uncertain Lurie System Stability Diagnosis

A new diagnosis method of Lurie system stability by using Chaotic time series data was proposed in order to solve stability analysis of flight control system with fault and uncertain. Firstly, the reasons for the instability of the nonlinear system of Ruri leaf was analyzed by using the small gain theory and linear matrix inequalities in the robust control theory. And the stability conditions of Rui leaf system under the condition of uncertainty and failure were proved theoretically. In order to quantify the stability of nonlinear systems in complex Ruri leaves, based on the theory of phase space reconstruction, the nonlinear Ruri system with continuous discrete characteristics was converted into an approximate time discrete equation, mapping it to a low dimensional primitive space, by introducing Q Gauss function into Kernel function, the generalization ability of neural networks are enhanced, realizing the stability analysis method based on the characteristic of primitive migration, which is suitable for various fault and uncertainty conditions, and the relative quantized stability norm can be given. The simulation shows that the present method can effectively solve the stability analysis and determination of flight control system under various factors.

Neural Network-Based Distributed Cooperative Learning Control for Multiagent Systems via Event-Triggered Communication.

In this paper, an event-based distributed cooperative learning (DCL) law is proposed for a group of adaptive neural control systems. The plants to be controlled have identical structures, but reference signals for each plant are different. During control process, each agent intermittently broadcasts its neural network (NN) weight estimation to its neighboring agents under an event-triggered condition that is only based on its own estimated NN weights. If communication topology is connected and undirected, the NN weights of all neural control systems can converge to a small neighborhood of their optimal values. The generalization ability of NNs is guaranteed in the event-triggered context, that is, the approximation domain of each NN is the union of all system trajectories. Furthermore, a strictly positive lower bound on the interevent intervals is also guaranteed to avoid the Zeno behavior. Finally, a numerical example is given to illustrate the effectiveness of the proposed learning law.

Sample Efficient Learning of Path Following and Obstacle Avoidance Behavior for Quadrotors

In this paper we propose an algorithm for the training of neural network control policies for quadrotors. The learned control policy computes control commands directly from sensor inputs and is hence computationally efficient. An imitation learning algorithm produces a policy that reproduces the behavior of a path following control algorithm with collision avoidance. Due to the generalization ability of neural networks, the resulting policy performs local collision avoidance of unseen obstacles while following a global reference path. The algorithm uses a time-free model predictive path-following controller as a supervisor. The controller generates demonstrations by following few example paths. This enables an easy to implement learning algorithm that is robust to errors of the model used in the model predictive controller. The policy is trained on the real quadrotor, which requires collision-free exploration around the example path. An adapted version of the supervisor is used to enable exploration. Thus, the policy can be trained from a relatively small number of examples on the real quadrotor, making the training sample efficient.

Event-triggered cooperative learning from output feedback control for multi-agent systems

Abstract In this paper, the event-triggered distributed cooperative learning from output feedback control is presented for a group of uncertain nonlinear systems whose structures are identical but their reference signals are different. An event-triggered communication scheme is used in the control process to overcome the disadvantages of continuous communication. Meanwhile, the weight estimates of all neural networks (NNs) also converge to a small neighborhood of their optimal values, and the generalization ability of NNs is well guaranteed. Specifically, the trigger condition of each agent is only dependent on its own NN weight estimate. It is proved that the inter-event times are lower bounded by a positive constant to avoid the accumulation of events. Finally, a numerical example is provided to substantiate the proposed scheme.