Training Process Of Neural Network Research Articles

Machine‐learning‐based methods for output‐only structural modal identification

In this study, we propose a machine-learning-based approach to identify the modal parameters of the output-only data for structural health monitoring (SHM) that makes full use of the characteristic of independence of modal responses and the principle of machine learning. By taking advantage of the independence feature of each mode, we use the principle of unsupervised learning, making the training process of the deep neural network becomes the process of modal separation. A self-coding deep neural network is designed to identify the structural modal parameters from the vibration data of structures. The mixture signals, that is, the structural response data, are used as the input of the neural network. Then we use a complex loss function to restrict the training process of the neural network, making the output of the third layer the modal responses we want, and the weights of the last two layers are mode shapes. The deep neural network is essentially a nonlinear objective function optimization problem. A novel loss function is proposed to constrain the independent feature with consideration of uncorrelation and non-Gaussianity to restrict the designed neural network to obtain the structural modal parameters. A numerical example of a simple structure and an example of actual SHM data from a cable-stayed bridge are presented to illustrate the modal parameter identification ability of the proposed approach. The results show the approach's good capability in blindly extracting modal information from system responses.

Optimizing the Weights and Thresholds in Dendritic Neuron Model Using the Whale Optimization Algorithm

In recent years, with the great success of dendritic neuron model (DNM) in various fields, the application of intelligent optimization algorithms in dendritic neuron model has attracted increasing attention of researchers. The training process of neural network is regarded as one of the great challenges of machine learning because of its non-linear nature and unknown optimal parameters. The traditional training algorithm of DNM is prone to fall into local optimum and speed of convergence slowly and so on, resulting in the problem of accuracy and low efficiency. In this paper, for solving the classification problem of dendritic neural model, an innovative intelligent optimization algorithm which named whale optimization algorithm (WOA), is applied to the training of DNM for the first time. Compared with six traditional and classic intelligent optimization algorithms in four classic datasets, the results indicate that WOA-DNM has good performance in various aspects, and its advantage is remarkable.

Residual Life Prediction of Lithium Battery Based on Small Sample Data Sets

In the recycling process of lithium-ion battery, the poor external environment will lead to the degradation of battery performance, so it is necessary to predict the residual life of lithiumion battery. However, the aging cycle of lithium-ion battery is long, so it is difficult to obtain sufficient aging data in a short time. Aiming at the problem of low prediction accuracy of lithium battery residual life under the condition of small samples, a BP neural network modelling algorithm fused with expert knowledge is proposed. Firstly, indirect health factors were extracted from the aging data of lithium battery in NASA PCOE laboratory. Secondly, the initial weight and threshold of BP neural network were optimized by genetic algorithm, and the initial multiplier value of the augmented Lagrange function was set according to Spearman correlation coefficient. Finally, the expert knowledge is integrated into the training process of BP neural network through the augmented Lagrange multiplier method. Simulation results show that the proposed algorithm has higher prediction accuracy than the traditional BP neural network under the condition of small samples.

Diagnosis Model of Volleyball Skills and Tactics Based on Artificial Neural Network

With the development of China’s sports industry, the technical and tactical level of the team is required to be higher and higher. This study mainly discusses the diagnostic model of volleyball technique and tactics based on artificial neural network. With the help of the correlation function in Matlab neural network toolbox, in the training process of volleyball technical and tactical evaluation neural network, the sample data of volleyball technical evaluation index is repeatedly simulated and studied, and finally the network parameters with the minimum error and the highest accuracy are saved as the network model for subsequent verification and evaluation. The middle layer is the hidden layer, which makes the network approach the result of volleyball experts’ evaluation of the technology by adjusting the weights of neurons. The last layer is the output layer, which outputs the actual evaluation results of volleyball experts on the technology. Through repeated training and comparison of input samples, the maximum number of training times of BP network for volleyball technical and tactical evaluation is determined to be 32. Some common experience of estimating hidden node number is provided by trial-and-error method. On this basis, the number of hidden nodes to minimize the network error is finally determined to be 4 through repeated training and comparison. In the process of network diagnosis, the average difference between the evaluation score of network output and the score of actual experts is less than 1%, which reaches a very high precision. It shows that the volleyball skill evaluation model based on BP neural network is feasible in technology and the result is relatively reliable.

Improving Semi-Supervised Learning for Audio Classification with FixMatch

Including unlabeled data in the training process of neural networks using Semi-Supervised Learning (SSL) has shown impressive results in the image domain, where state-of-the-art results were obtained with only a fraction of the labeled data. The commonality between recent SSL methods is that they strongly rely on the augmentation of unannotated data. This is vastly unexplored for audio data. In this work, SSL using the state-of-the-art FixMatch approach is evaluated on three audio classification tasks, including music, industrial sounds, and acoustic scenes. The performance of FixMatch is compared to Convolutional Neural Networks (CNN) trained from scratch, Transfer Learning, and SSL using the Mean Teacher approach. Additionally, a simple yet effective approach for selecting suitable augmentation methods for FixMatch is introduced. FixMatch with the proposed modifications always outperformed Mean Teacher and the CNNs trained from scratch. For the industrial sounds and music datasets, the CNN baseline performance using the full dataset was reached with less than 5% of the initial training data, demonstrating the potential of recent SSL methods for audio data. Transfer Learning outperformed FixMatch only for the most challenging dataset from acoustic scene classification, showing that there is still room for improvement.

Data driven Dirichlet sampling on manifolds

This article presents a new method to sample on manifolds, based on the Dirichlet distribution. The proposed strategy allows to completely respect the underlying manifold around which data are observed, and to do massive sampling with low computational effort. This can be very helpful, for instance, in neural networks training process, as well as in uncertainty analysis and stochastic optimization. Due to its simplicity and efficiency, we believe that the new method has great potential. Three manifolds (two dimensional ring, Mobius strip and spider geometry) are used to test the proposed methodology, and then it is employed to an engineering application, related to the bearing coefficients of a rotating machine. In the application, data are augmented to train a neural network.

Reinforcement Learning for Robust Trajectory Design of Interplanetary Missions

This paper investigates the use of reinforcement learning for the robust design of low-thrust interplanetary trajectories in presence of severe uncertainties and disturbances, alternately modeled as Gaussian additive process noise, observation noise, and random errors in the actuation of the thrust control, including the occurrence of a missed thrust event. The stochastic optimal control problem is recast as a time-discrete Markov decision process to comply with the standard formulation of reinforcement learning. An open-source implementation of the state-of-the-art algorithm Proximal Policy Optimization is adopted to carry out the training process of a deep neural network, used to map the spacecraft (observed) states to the optimal control policy. The resulting guidance and control network provides both a robust nominal trajectory and the associated closed-loop guidance law. Numerical results are presented for a typical Earth–Mars mission. To validate the proposed approach, the solution found in a (deterministic) unperturbed scenario is first compared with the optimal one provided by an indirect technique. The robustness and optimality of the obtained closed-loop guidance laws is then assessed by means of Monte Carlo campaigns performed in the considered uncertain scenarios.

Epileptic EEG Signal Classification Using Convolutional Neural Network Based on Multi-Segment of EEG Signal

High performance in the epileptic electroencephalogram (EEG) signal classification is an important step in diagnosing epilepsy. Furthermore, this classification is carried out to determine whether the EEG signal from a person's examination results is categorized as an epileptic signal or not (healthy). Several automated techniques have been proposed to assist neurologists in classifying these signals. In general, these techniques have yielded a high average accuracy in classification, but the performance still needs to be improved. Therefore, we propose a convolutional neural network based on multi-segment of EEG signals to classify epileptic EEG signals. This method is built to overcome data limitations in the convolutional neural network training process and add the ensemble combination process. The multi-segment of EEG signal is formed by splitting the signal without overlapping each channel and converting it into the spectrogram image based on the short-time Fourier transform value. The spectrogram image is then used as input for the convolutional neural network in in-depth training and testing. The convolutional neural network model of the training results is used to classify each EEG signal segment on each test channel before entering the ensemble combination stage for the final classification. To evaluate the performance of our proposed method, we used the Bonn EEG dataset. The dataset consists of five EEG records labelled as A, B, C, D, and E. The experiments on several datasets (AB-C, AB-D, AB-E, AB-CD, AB-CDE, and AB-CD-E) which were arranged from the dataset showed that our proposed method (with segment) performs better than without segment. Our proposed method yielded the best average of classification accuracy which is 99.33%, 100%, 100%, 99.5%, 99.8%, and 99.4% for the AB-C, AB-D, AB-E, AB-CD, AB-CDE, and AB-CD-E.By these results, the proposed method can outperform several other methods on the same dataset.

Broad Colorization.

The scribble- and example-based colorization methods have fastidious requirements for users, and the training process of deep neural networks for colorization is quite time-consuming. We instead proposed an automatic colorization approach with no dependence on user input and no need to endure long training time, which combines local features and global features of the input gray-scale images. Low-, mid-, and high-level features are united as local features representing cues existed in the gray-scale image. The global feature is regarded as data prior to guiding the colorization process. The local broad learning system is trained for getting the chrominance value of each pixel from the local features, which could be expressed as a chrominance map according to the position of pixels. Then, the global broad learning system is trained to refine the chrominance map. There are no requirements for users in our approach, and the training time of our framework is an order of magnitude faster than the traditional methods based on deep neural networks. To increase the user's subjective initiative, our system allows users to increase training data without retraining the system. Substantial experimental results have shown that our approach outperforms state-of-the-art methods.

Symmetrical filters in convolutional neural networks

Symmetry is present in nature and science. In image processing, kernels for spatial filtering possess some symmetry (e.g. Sobel operators, Gaussian, Laplacian). Convolutional layers in artificial feed-forward neural networks have typically considered the kernel weights without any constraint. We propose to investigate the impact of a symmetry constraint in convolutional layers for image classification tasks, taking our inspiration from the processes involved in the primary visual cortex and common image processing techniques. The goal is to determine if it is necessary to learn each weight of the filters independently, and the extent to which it is possible to enforce symmetrical constraints on the filters throughout the training process of a convolutional neural network by modifying the weight update preformed during the backpropagation algorithm and to evaluate the change in performance. The symmetrical constraint reduces the number of free parameters in the network, and it is able to achieve near identical performance. We address the following cases: x/y-axis symmetry, point reflection, and anti-point reflection. The performance is evaluated on four databases of images representing handwritten digits. The results support the conclusion that while random weights offer more freedom to the model, the symmetry constraint provides a similar level of performance while decreasing substantially the number of free parameters in the model. Such an approach can be valuable in phase-sensitive applications that require a linear phase property throughout the feature extraction process.

Event Prediction Technology Based on Graph Neural Network

With the rapid development of the mobile Internet and the continuous expansion of network scale, the network security situation is becoming increasingly severe, and the endless network security threats have put forward higher requirements for network security performance. Based on the above background, the purpose of this paper is to explore the event prediction technology based on graph neural network. Due to the slow convergence of the network event prediction and evaluation model, the untimely risk assessment and inaccurate safety prediction caused by the incomplete parameter setting of the prediction model have become prominent problems in this field. This paper proposes an event prediction technology based on graph neural network. This method first uses genetic algorithms to optimize the weights in the training process of the graph neural network, which overcomes the blindness of initial weight selection and improves the training efficiency of the graph neural network; the KDDCup99 data set is used to conduct experiments on the above two methods respectively. Verification and analysis. The simulation and comparison experiments respectively verify that the neural network-based network security situation assessment and prediction method proposed in this paper can realize the assessment and prediction of the network situation more efficiently and accurately.

Artificial Neural Networks in Modeling of Dewaterability of Sewage Sludge

Mechanical dewatering is a key process in the management of sewage sludge. However, the drainage efficiency depends on a number of factors, from the type and dose of the conditioning agent to the parameters of the drainage device. The selection of appropriate methods and parameters of conditioning and dewatering of sewage sludge is the task of laboratory work. This work can be accelerated through the use of artificial neural network (ANNs). The paper discusses the possibilities of using ANNs in predicting the dewatering efficiency of physically conditioned sludge. The input variables were only four parameters characterizing the conditioning methods and the dewatering method by centrifugation. These were the dose of the sludge skeleton builders (cement, gypsum, fly ash, and liquid glass), sonication parameters (sonication amplitude and time), and relative centrifugal force. Dewatering efficiency parameters such as sludge hydration and separation factor were output variables. Due to the nature of the research problem, two nonlinear networks were selected: a multilayer perceptron and a radial neural network. Based on the results of the prediction of artificial neural networks, it was found that these networks can be used to forecast the effectiveness of municipal sludge dewatering. The prediction error did not exceed 1.0% of the real value. ANN can therefore be useful in optimizing the dewatering process. In the case of the conducted research, it was the optimization of the sludge dewatering efficiency as a function of the type and parameters of conditioning factors. Therefore, it is possible to predict the dewatering efficiency of sludge that has not been tested in the laboratory, for example, with the use of other doses of physical conditioner. However, the condition for correct prediction and optimization was the use of a large dataset in the neural network training process.

Towards effective learning for face super-resolution with shape and pose perturbations

Recent development of convolutional neural networks (CNNs) has activated a lot of studies and boosted the performance greatly on image super-resolution. This paper addresses the issue of face super-resolution, which attracts a lot of interests in the photographic industry. We propose to make use of the face-specific priors to enhance the performance of face super-resolution with the convolutional neural networks. Classical facial prior models represent the 2D facial shape in a compact low-dimensional space expressed by principal components. Here, we impose perturbations on the low-dimensional space and generate face samples with novel appearance. First, we conduct 2D facial image perturbations through 2D facial landmarks. Then, we carry on the study with perturbations on 3D facial landmarks. Facial pose and shape are perturbated to generate novel appearances of a single 2D facial image. These novel facial samples are then fed into the training process of the convolutional neural networks for face super-resolution. The experimental results demonstrate that the proposed method is adaptable to various networks and achieves superior performance for the face super-resolution task.

Analysis of training of deep neural networks with heterogeneous architecture while detecting malicious network traffic

The given article presents the study of the training process of a composite heterogeneous neural network with deep architecture. A brief description of the architecture of the analyzed neural network system has been given. The key nodes of computational load distribution on each of the layers of the neural network have been tracked. The monitoring results have been analyzed and the following conclusions have been made: the most resource-intensive part of the system during training is the LSTM network. The results of the study of structural features of the neural network hidden layers have been presented. Graphs have been constructed and there has been carried out the study of a statistical distribution of weights on each unique layer of the architecture: on a fully connected layer, on the first hidden layer of the subnet-encoder, on the second hidden layer of the subnet-encoder, on the first and second fully connected output layers of the neural network. Based on the research results, a qualitative assessment of the effectiveness and accuracy of the entire neural network system has been given.

Identification of groundwater contamination sources and hydraulic parameters based on bayesian regularization deep neural network.

Simultaneous identification of various features of groundwater contamination sources and hydraulic parameters, such as hydraulic conductivities, can result in high-nonlinear inverse problem, which significantly hinders identification. A surrogate model was proposed to relieve computational burden caused by massive callings to simulation model in identification. However, shallow learning surrogate model may show limited fitting ability to high nonlinear problem. Thus, in this study, a simulation-optimization method based on Bayesian regularization deep neural network (BRDNN) surrogate model was proposed to efficiently solve high-nonlinear inverse problem. This method identified eight variables including locations and release intensities of two pollution sources and hydraulic conductivities of two partitions. Three hidden layers were employed in the BRDNN surrogate model, which profoundly improved the fitting capacity of nonlinear mapping relationship to the simulation model. Furthermore, Bayesian regularization was applied in the training process of neural network to solve overfitting problem. The results indicated that BRDNN was capable of establishing input-output interplay of high nonlinear inverse problem, which substantially reduced computational cost while ensuring a desirable level of accuracy. The utility of simulation-optimization on the basis of BRDNN surrogate model provided stable and reliable inversion results for groundwater contamination sources and hydraulic parameters.

Negative-ResNet: noisy ambulatory electrocardiogram signal classification scheme

With recently successful applications of deep learning in computer vision and general signal processing, deep learning has shown many unique advantages in medical signal processing. However, data labelling quality has become one of the most significant issues for AI applications, especially when it requires domain knowledge (e.g. medical image labelling). Besides, there might be noisy labels in practical datasets, which impairs the training process of neural networks. In this paper, we propose a semi-supervised algorithm for training data samples with noisy labels by performing selected positive learning and negative learning. To verify the effectiveness of the proposed scheme, we designed a portable ECG patch-RealCare and applied the algorithm on a real-life dataset. Our experimental results show that we can achieve an accuracy of 91.0%, which is 6.2% higher than a normal training process with ResNet. There are 65 patients in our dataset, and we randomly picked 2 patients to perform validation.

Signal Sorting Algorithm of Hybrid Frequency Hopping Network Station Based on Neural Network

In a non-cooperative frequency hopping communication system, the frequency hopping network station sorting of the received hybrid signals plays an important role and becomes an active research area in recent years. In order to solve the problem that the currently widely used clustering algorithm can not achieve satisfactory accuracy. In this paper, we propose a signal sorting method for hybrid frequency hopping network stations by applying the neural network to classify the frequency hopping description words of signals. Additionally, the conjugate gradient algorithm is utilized in the neural network training process to improve the convergence speed. Simulation results demonstrate that when compared with the clustering algorithm, the proposed algorithm converges with fewer iterations and delivers better sorting accuracy, especially in a low signal to noise ratio environment.

Chess as Sequential Data in a Chess Match Outcome Prediction Using Deep Learning with Various Chessboard Representations

Moves made by chess players during the match are certainly some kind of sequence. In this work, we tried to answer the question if such data could be interpreted as sequential. To achieve that, a model build with LSTM layers only was designed. Results performed by the model are justifying that thesis. A novel model architecture was also proposed and trained on multiple data types—chess moves and chess game metadata. Its purpose was to perform as high classification accuracy as possible—we managed to achieve it with a result close to 69%. Moreover, we compared a couple of chessboard representation methods, more precisely a bitmap input and algebraic input, to check which one is more relevant for the neural networks training process. Contrary to what one might suppose, better scores were reached for bitmap input, which from the theoretical point of view carries out less information than algebraic input.

Marketing forecasting based on Big Data information

In the paper discusses the use of big data as a tool to increase data transfer speed while providing access to multidimensional data in the process of forecasting product sales in the market. In this paper discusses modern big data tools that use the MapReduce model. The big data presented in this article is a single, centralized source of information across your entire domain. In the paper also proposes the structure of a marketing analytics system that includes many databases in which transactions are processed in real time. For marketing forecasting of multidimensional data in Matlab, a neural network is considered and built. For training and building a network, it is proposed to construct a matrix of input data for presentation in a neural network and a matrix of target data that determine the output statistical information. Input and output data in the neural network is presented in the form of a 5x10 matrix, which represents static information about 10 products for five days of the week. The application of the Levenberg-Marquardt algorithm for training a neural network is considered. The results of the neural network training process in Matlab are also presented. The obtained forecasting results are given, which allows us to conclude about the advantages of a neural network in multivariate forecasting in real time.

Signal Sorting Algorithm of Hybrid Frequency Hopping Network Station Based on Neural Network

In non-cooperative frequency hopping communication system, the frequency hopping network station sorting of the received hybrid signals plays an important role and becomes an active research area in recent years. In order to solve the problem that the currently widely used clustering algorithm cannot achieve satisfactory accuracy. In this paper, we propose a signal sorting method for hybrid frequency hopping network stations by applying the neural network to classify the frequency hopping description words of signals. Additionally, the conjugate gradient algorithm is utilized in the neural network training process to improve the convergence speed. Once the neural network training is finished, only one frequency hopping description word of the input signal is required to obtain its own network station label in real time. Simulation results demonstrate that when compared with the clustering algorithm, the proposed algorithm converges with less iterations and delivers better sorting accuracy, especially in a low signal to noise ratio environment.