Generalization Ability Of Network Research Articles

Parallel ensemble learning of convolutional neural networks and local binary patterns for face recognition

Background and ObjectiveFace recognition success rate is influenced by illumination, expression, posture change, and other factors, which is due to the low generalization ability of a single convolutional neural network. A new face recognition method based on parallel ensemble learning of convolutional neural networks (CNN) and local binary patterns (LBP) is proposed to solve this problem. It also helps to improve the low pedestrian detection rate caused by occlusion. MethodsFirst, the LBP operator is employed to extract features of the face texture. After that, 10 convolutional neural networks with 5 different network structures are adopted to further extract features for training, to improve the network parameters and get classification result by using the Softmax function after the layer is fully connected. Finally, the method of parallel ensemble learning is used to generate the final result of face recognition using majority voting. ResultsBy this method, the recognition rates in the ORL and Yale-B face datasets increase to 100% and 97.51%, respectively. In the experiments, the proposed approach is illustrated not only enhances its tolerance to illumination, expression, and posture but also improves the accuracy of face recognition and the poor generalization performance of the model, which is normally caused by the learning algorithm being trapped in a local minimum. Moreover, the proposed method is combined with a pedestrian detection model as a hybrid model for improving the detection rate, which shows in the result that the detection rate is improved by 11.2%. ConclusionIn summary, the proposed approach greatly outperforms other competitive methods.

Robust active flow control over a range of Reynolds numbers using an artificial neural network trained through deep reinforcement learning

This paper focuses on the active flow control of a computational fluid dynamics simulation over a range of Reynolds numbers using deep reinforcement learning (DRL). More precisely, the proximal policy optimization (PPO) method is used to control the mass flow rate of four synthetic jets symmetrically located on the upper and lower sides of a cylinder immersed in a two-dimensional flow domain. The learning environment supports four flow configurations with Reynolds numbers 100, 200, 300, and 400, respectively. A new smoothing interpolation function is proposed to help the PPO algorithm learn to set continuous actions, which is of great importance to effectively suppress problematic jumps in lift and allow a better convergence for the training process. It is shown that the DRL controller is able to significantly reduce the lift and drag fluctuations and actively reduce the drag by ∼5.7%, 21.6%, 32.7%, and 38.7%, at Re = 100, 200, 300, and 400, respectively. More importantly, it can also effectively reduce drag for any previously unseen value of the Reynolds number between 60 and 400. This highlights the generalization ability of deep neural networks and is an important milestone toward the development of practical applications of DRL to active flow control.

On the use of pairwise distance learning for brain signal classification with limited observations.

The increasing access to brain signal data using electroencephalography creates new opportunities to study electrophysiological brain activity and perform ambulatory diagnoses of neurological disorders. This work proposes a pairwise distance learning approach for schizophrenia classification relying on the spectral properties of the signal. To be able to handle clinical trials with a limited number of observations (i.e. case and/or control individuals), we propose a Siamese neural network architecture to learn a discriminative feature space from pairwise combinations of observations per channel. In this way, the multivariate order of the signal is used as a form of data augmentation, further supporting the network generalization ability. Convolutional layers with parameters learned under a cosine contrastive loss are proposed to adequately explore spectral images derived from the brain signal. The proposed approach for schizophrenia diagnostic was tested on reference clinical trial data under resting-state protocol, achieving 0.95 ± 0.05 accuracy, 0.98 ± 0.02 sensitivity and 0.92 ± 0.07 specificity. Results show that the features extracted using the proposed neural network are remarkably superior than baselines to diagnose schizophrenia (+20pp in accuracy and sensitivity), suggesting the existence of non-trivial electrophysiological brain patterns able to capture discriminative neuroplasticity profiles among individuals. The code is available on Github: https://github.com/DCalhas/siamese_schizophrenia_eeg.

Feature transfer based adversarial domain adaptation method for cross-domain road extraction

In order to improve the cross-domain applicability of road segmentation, a feature transfer based adversarial domain adaptation method is presented for cross-domain road extraction. The presented method consists of two main parts, a feature transfer network and an adversarial domain adaption network. The feature transfer network transforms the images of the source domain into the feature space of the target domain. The adversarial domain adaption network learns to distinguish whether the results come from source domain or target domain, to promote the backbone network of road extraction to learn the common features of the road. The experimental results showed that the proposed method improved the generalization ability of the road extraction network and could extract the road target from cross-domain images accurately and effectively. The proposed method can realize the road extraction across domain without any annotation of target domain, so it has good application value.

Feature Variance Regularization: A Simple Way to Improve the Generalizability of Neural Networks

To improve the generalization ability of neural networks, we propose a novel regularization method that regularizes the empirical risk using a penalty on the empirical variance of the features. Intuitively, our approach introduces confusion into feature extraction and prevents the models from learning features that may relate to specific training samples. According to our theoretical analysis, our method encourages models to generate closer feature distributions for the training set and unobservable true data and minimize the expected risk as well, which allows the model to adapt to new samples better. We provide a thorough empirical justification of our approach, and achieves a greater improvement than other regularization methods. The experimental results show the effectiveness of our method on multiple visual tasks, including classification (CIFAR100, ImageNet, fine-grained datasets) and semantic segmentation (Cityscapes).

Reinforcing Neural Network Stability with Attractor Dynamics

Recent approaches interpret deep neural works (DNNs) as dynamical systems, drawing the connection between stability in forward propagation and generalization of DNNs. In this paper, we take a step further to be the first to reinforce this stability of DNNs without changing their original structure and verify the impact of the reinforced stability on the network representation from various aspects. More specifically, we reinforce stability by modeling attractor dynamics of a DNN and propose relu-max attractor network (RMAN), a light-weight module readily to be deployed on state-of-the-art ResNet-like networks. RMAN is only needed during training so as to modify a ResNet's attractor dynamics by minimizing an energy function together with the loss of the original learning task. Through intensive experiments, we show that RMAN-modified attractor dynamics bring a more structured representation space to ResNet and its variants, and more importantly improve the generalization ability of ResNet-like networks in supervised tasks due to reinforced stability.

Dynamic Graph Representation for Occlusion Handling in Biometrics

The generalization ability of Convolutional neural networks (CNNs) for biometrics drops greatly due to the adverse effects of various occlusions. To this end, we propose a novel unified framework integrated the merits of both CNNs and graphical models to learn dynamic graph representations for occlusion problems in biometrics, called Dynamic Graph Representation (DGR). Convolutional features onto certain regions are re-crafted by a graph generator to establish the connections among the spatial parts of biometrics and build Feature Graphs based on these node representations. Each node of Feature Graphs corresponds to a specific part of the input image and the edges express the spatial relationships between parts. By analyzing the similarities between the nodes, the framework is able to adaptively remove the nodes representing the occluded parts. During dynamic graph matching, we propose a novel strategy to measure the distances of both nodes and adjacent matrixes. In this way, the proposed method is more convincing than CNNs-based methods because the dynamic graph method implies a more illustrative and reasonable inference of the biometrics decision. Experiments conducted on iris and face demonstrate the superiority of the proposed framework, which boosts the accuracy of occluded biometrics recognition by a large margin comparing with baseline methods.

Rolling-Element Bearing Fault Diagnosis Using Improved LeNet-5 Network.

To address the problems of low recognition accuracy, slow convergence speed and weak generalization ability of traditional LeNet-5 network used in rolling-element bearing fault diagnosis, a rolling-element bearing fault diagnosis method using improved 2D LeNet-5 network is put forward. The following improvements to the traditional LeNet-5 network are made: the convolution and pooling layers are reasonably designed and the size and number of convolution kernels are carefully adjusted to improve fault classification capability; the batch normalization (BN) is adopted after each convolution layer to improve convergence speed; the dropout operation is performed after each full-connection layer except the last layer to enhance generalization ability. To further improve the efficiency and effectiveness of fault diagnosis, on the basis of improved 2D LeNet-5 network, an end-to-end rolling-element bearing fault diagnosis method based on the improved 1D LeNet-5 network is proposed, which can directly perform 1D convolution and pooling operations on raw vibration signals without any preprocessing. The results show that the improved 2D LeNet-5 network and improved 1D LeNet-5 network achieve a significant performance improvement than traditional LeNet-5 network, the improved 1D LeNet-5 network provides a higher fault diagnosis accuracy with a less training time in most cases, and the improved 2D LeNet-5 network performs better than improved 1D LeNet-5 network under small training samples and strong noise environment.

GP-CNN-DTEL: Global-Part CNN Model With Data-Transformed Ensemble Learning for Skin Lesion Classification.

Precise skin lesion classification is still challenging due to two problems, i.e., (1) inter-class similarity and intra-class variation of skin lesion images, and (2) the weak generalization ability of single Deep Convolutional Neural Network trained with limited data. Therefore, we propose a Global-Part Convolutional Neural Network (GP-CNN) model, which treats the fine-grained local information and global context information with equal importance. The Global-Part model consists of a Global Convolutional Neural Network (G-CNN) and a Part Convolutional Neural Network (P-CNN). Specifically, the G-CNN is trained with downscaled dermoscopy images, and is used to extract the global-scale information of dermoscopy images and produce the Classification Activation Map (CAM). While the P-CNN is trained with the CAM guided cropped image patches and is used to capture local-scale information of skin lesion regions. Additionally, we present a data-transformed ensemble learning strategy, which can further boost the classification performance by integrating the different discriminant information from GP-CNNs that are trained with original images, color constancy transformed images, and feature saliency transformed images, respectively. The proposed method is evaluated on the ISIC 2016 and ISIC 2017 Skin Lesion Challenge (SLC) classification datasets. Experimental results indicate that the proposed method can achieve the state-of-the-art skin lesion classification performance (i.e., an AP value of 0.718 on the ISIC 2016 SLC dataset and an Average Auc value of 0.926 on the ISIC 2017 SLC dataset) without any external data, compared with other current methods which need to use external data.

Why ResNet Works? Residuals Generalize.

Residual connections significantly boost the performance of deep neural networks. However, few theoretical results address the influence of residuals on the hypothesis complexity and the generalization ability of deep neural networks. This article studies the influence of residual connections on the hypothesis complexity of the neural network in terms of the covering number of its hypothesis space. We first present an upper bound of the covering number of networks with residual connections. This bound shares a similar structure with that of neural networks without residual connections. This result suggests that moving a weight matrix or nonlinear activation from the bone to a vine would not increase the hypothesis space. Afterward, an O(1 / √N) margin-based multiclass generalization bound is obtained for ResNet, as an exemplary case of any deep neural network with residual connections. Generalization guarantees for similar state-of-the-art neural network architectures, such as DenseNet and ResNeXt, are straightforward. According to the obtained generalization bound, we should introduce regularization terms to control the magnitude of the norms of weight matrices not to increase too much, in practice, to ensure a good generalization ability, which justifies the technique of weight decay.

Prediction of contact fatigue life of AT40 ceramic coating based on neural network

Purpose With the development of deep learning-based analytical techniques, increased research has focused on fatigue data analysis methods based on deep learning, which are gaining in popularity. However, the application of deep neural networks in the material science domain is mainly inhibited by data availability. In this paper, to overcome the difficulty of multifactor fatigue life prediction with small data sets, Design/methodology/approach A multiple neural network ensemble (MNNE) is used, and an MNNE with a general and flexible explicit function is developed to accurately quantify the complicated relationships hidden in multivariable data sets. Moreover, a variational autoencoder-based data generator is trained with small sample sets to expand the size of the training data set. A comparative study involving the proposed method and traditional models is performed. In addition, a filtering rule based on the R2 score is proposed and applied in the training process of the MNNE, and this approach has a beneficial effect on the prediction accuracy and generalization ability. Findings A comparative study involving the proposed method and traditional models is performed. The comparative experiment confirms that the use of hybrid data can improve the accuracy and generalization ability of the deep neural network and that the MNNE outperforms support vector machines, multilayer perceptron and deep neural network models based on the goodness of fit and robustness in the small sample case. Practical implications The experimental results imply that the proposed algorithm is a sophisticated and promising multivariate method for predicting the contact fatigue life of a coating when data availability is limited. Originality/value A data generated model based on variational autoencoder was used to make up lack of data. An MNNE method was proposed to apply in the small data case of fatigue life prediction.

Learn Generalization Feature via Convolutional Neural Network: A Fault Diagnosis Scheme Toward Unseen Operating Conditions

In recent years, Convolutional neural networks (CNNs) have achieved start-of-art performance in the fault diagnosis field. If there is no available information on the unseen operating conditions, the model trained on the seen operating condition cannot perform well. One of the feasible strategies is to enhance the generalization ability of the network on various seen operating conditions. We introduce the center loss to the traditional CNN and build an end-to-end fault diagnosis framework (called CNN-C). By minimizing the intra-class variations, center loss cluster the learned features across various seen operating conditions. With the joint supervision of the center loss and the softmax loss, the learned features of the same class could minimize the domain difference across various seen operating conditions while the features of different classes are separable. The generalization ability of network is improved on unseen operating conditions. Compared with the shallow methods and traditional CNN, the proposed method is promising to deal with the fault diagnosis tasks of the bearing and gearbox.

A Comparison on Data Augmentation Methods Based on Deep Learning for Audio Classification

Deep learning focuses on the representation of the input data and generalization of the model. It is well known that data augmentation can combat overfitting and improve the generalization ability of deep neural network. In this paper, we summarize and compare multiple data augmentation methods for audio classification. These strategies include traditional methods on raw audio signal, as well as the current popular augmentation of linear interpolation and nonlinear mixing on the spectrum. We explore the generation of new samples, the transformation of labels, and the combination patterns of samples and labels of each data augmentation method. Finally, inspired by SpecAugment and Mixup, we propose an effective and easy to implement data augmentation method, which we call Mixed frequency Masking data augmentation. This method adopts nonlinear combination method to construct new samples and linear method to construct labels. All methods are verified on the Freesound Dataset Kaggle2018 dataset, and ResNet is adopted as the classifier. The baseline system uses the log-mel spectrogram feature as the input. We use mean Average Precision @3 (mAP@3) as the evaluation metric to evaluate the performance of all data augmentation methods.

Evaluating the generalization ability of convolutional neural networks for built-up area extraction in different cities of China

The difficulty of build-up area extraction is due to complexity of remote sensing data in terms of heterogeneous appearance with large intra-class variations and lower inter-class variations. In order to extract the built-up area from Landsat 8-OLI images provided by Google earth engine (GEE), we propose a convolutional neural networks (CNN) utilizing spatial and spectral information synchronously, which is built in Google drive using Colaboratory-Keras. To train a CNN model with good generalization ability, we choose Beijing, Lanzhou, Chongqing, Suzhou and Guangzhou of China as the training sites, which are very different in term of natural environments. The ArcGIS-Model Builder is employed to automatically select 99 332 samples from the 38-m global built-up production of the European Space Agency (ESA) in 2014. The validate accuracy of the five experimental sites is higher than 90%. We compare the results with other existing building data products. The classification results of CNN can be very good for the details of the built-up areas, and greatly reduce the classification error and leakage error. We applied the well-trained CNN model to extract built-up areas of Chengdu, Xi’an, Zhengzhou, Harbin, Hefei, Wuhan, Kunming and Fuzhou, for the sake of evaluating the generalization ability of the CNN. The fine classification results of the eight sites indicate that the generalization ability of the well-trained CNN is pretty good. However, the extraction results of Xi’an, Zhengzhou and Hefei are poor. As for the training data, only Lanzhou is located in the northwest region, so the trained CNN has poor image classification ability in the northwest region of China.

Convolutional Neural Network Assisted Optical Orbital Angular Momentum Identification of Vortex Beams

The rapid and accurate identification of large-scale orbital angular momentum (OAM) modes is crucial for expanding the application of vortex beams (VBs). In this paper, an OAM mode recognition method based on convolutional neural networks (CNNs) is proposed and investigated. We construct an 8-layer CNN possesses complex feature extraction capability and train it to own powerful anti-turbulence competence by feeding the intensity patterns of VBs interfered by Gaussian beam. After supervised training of a large sample set, the CNN model takes on excellent network generalization ability and can well detect VBs with the mode range of [−50,50]. The simulation results indicate that under the influence of weak and medium turbulences, the average recognition accuracy exceeds 99%. Even under strong turbulence, the accuracy also reaches 98.54%. Meanwhile, the identification time is only 1.55ms per OAM mode with Intel(R) Xeon(R) Gold 6148 CPU. Moreover, the influence of different Gaussian beam waists, VB orders, input training sets, and CNN structures on OAM mode recognition performance, is fully studied. These results demonstrate that our proposed method can achieve higher accuracy and higher order OAM mode detection at a fast speed, which contributes a more effective method for the recognition of VBs.

Thermal Error Modeling of Feed Axis in Machine Tools Using Particle Swarm Optimization-Based Generalized Regression Neural Network

Abstract This paper demonstrates the development of a thermal error model that is applied on the feed axis of machine tools and based on the neural network. This model can accurately predict the value of the axial thermal error that appears on machine feed axis. In principle, there is the generalized regression neural network (GRNN), which has the good nonlinear mapping ability and serves to construct the error model. About variables, the data of temperature and axial thermal error of machine feed axis are the inputs and outputs, respectively. The particle swarm optimization (PSO) is a component of this model, which serves to optimize the smoothing factor in GRNN, and the particle swarm optimization-based generalized regression neural network (PSO-GRNN) model is built. From experiment, the datum is acquired from a machining centre in four different feed rates. Thereafter, the back propagation (BP) neural network model, the traditional GRNN model, and the PSO-GRNN model were established, and the data collected from the experimentation are input in three models for prediction. Compared with the other two models used in this paper, the PSO-GRNN model can maintain higher prediction accuracy at different feed speed, and the prediction accuracy of it changes less in different feed rates. The proposed model solved the problem of generalization ability of the neural network at different feed rate, which shows good performance and lays a good foundation for further research like thermal error compensation.

Research of Mapping Element Space Method for Uncertain Lurie System Stability Diagnosis

A new diagnosis method of Lurie system stability by using Chaotic time series data was proposed in order to solve stability analysis of flight control system with fault and uncertain. Firstly, the reasons for the instability of the nonlinear system of Ruri leaf was analyzed by using the small gain theory and linear matrix inequalities in the robust control theory. And the stability conditions of Rui leaf system under the condition of uncertainty and failure were proved theoretically. In order to quantify the stability of nonlinear systems in complex Ruri leaves, based on the theory of phase space reconstruction, the nonlinear Ruri system with continuous discrete characteristics was converted into an approximate time discrete equation, mapping it to a low dimensional primitive space, by introducing Q Gauss function into Kernel function, the generalization ability of neural networks are enhanced, realizing the stability analysis method based on the characteristic of primitive migration, which is suitable for various fault and uncertainty conditions, and the relative quantized stability norm can be given. The simulation shows that the present method can effectively solve the stability analysis and determination of flight control system under various factors.

Domain Adaptation With Neural Embedding Matching

Domain adaptation aims to exploit the supervision knowledge in a source domain for learning prediction models in a target domain. In this article, we propose a novel representation learning-based domain adaptation method, i.e., neural embedding matching (NEM) method, to transfer information from the source domain to the target domain where labeled data is scarce. The proposed approach induces an intermediate common representation space for both domains with a neural network model while matching the embedding of data from the two domains in this common representation space. The embedding matching is based on the fundamental assumptions that a cross-domain pair of instances will be close to each other in the embedding space if they belong to the same class category, and the local geometry property of the data can be maintained in the embedding space. The assumptions are encoded via objectives of metric learning and graph embedding techniques to regularize and learn the semisupervised neural embedding model. We also provide a generalization bound analysis for the proposed domain adaptation method. Meanwhile, a progressive learning strategy is proposed and it improves the generalization ability of the neural network gradually. Experiments are conducted on a number of benchmark data sets and the results demonstrate that the proposed method outperforms several state-of-the-art domain adaptation methods and the progressive learning strategy is promising.

Experimental Investigation on the Spatial-temporal Evolution of the hole formation during Laser Helical Drilling

In order to study the influence of process parameters of femtosecond laser helical machining on the formation of micro-holes, we selected main parameters of the femtosecond laser helical drilling process, used 304 stainless steel as the target material, designed a 5-factor and 5-level orthogonal experiment based on the L25 (56) orthogonal table, and analyzed the significance levels of the impacts of single pulse energy, repetition rate, rotation rate, rate of focus downward movement, and air-blowing pressure on the ablation depth. Based on back propagation (BP) neural network, we established the relationship model between five factors and the material ablation depth, and trained the network with orthogonal experiment data. We used the data of the additional drilling experiments to test the generalization ability of the established network, and the results show that the prediction error of the model is within 3%. We designed a single-factor experiment for the characteristic parameter of the helical drilling--the rate of focus downward movement--and obtained the effect of the rate of focus downward movement on the formation of the micro-holes during the process of femtosecond laser helical drilling.

Dynamic Deformation Reconstruction of Variable Section WING with Fiber Bragg Grating Sensors.

In order to monitor the variable-section wing deformation in real-time, this paper proposes a dynamic reconstruction algorithm based on the inverse finite element method and fuzzy network to sense the deformation of the variable-section beam structure. Firstly, based on Timoshenko beam theory and inverse finite element framework, a deformation reconstruction model of variable-section beam element was established. Then, considering the installation error of the fiber Bragg grating (FBG) sensor and the dynamic un-modeled error caused by the difference between the static model and dynamic model, the real-time measured strain was corrected using a solidified fuzzy network. The parameters of the fuzzy network were learned using support vector machines to enhance the generalization ability of the fuzzy network. The loading deformation experiment shows that the deformation of the variable section wing can be reconstructed with the proposed algorithm in high precision.