Learning Process Of Neural Network Research Articles

An interpretable convolutional neural network with multi-wavelet kernel fusion for intelligent fault diagnosis

Deep learning (DL) has been widely developed and applied in gearbox fault diagnosis of smart manufacturing systems due to its powerful feature representation ability. However, typical DL-based fault diagnosis approaches do not correlate the domain information or mechanism knowledge into the neural network learning process. This causes issues for the reliability and stability of fault diagnosis engineering applications. To address this problem, we propose a novel interpretable DL model named Multi-Wavelet Kernel Convolution Neural Network (MWKCNN) for fault diagnosis in this article. First, a feature extraction layer named the Multi-Wavelet Kernel Convolution (MWKC) layer is constructed based on the continuous wavelet transform (CWT) to locate and detect the impulse signatures of faults. It can capture sufficient feature representations by involving signal processing knowledge. Then, to balance the influence of different wavelet kernels, we design a kernel weights recalibration (KWR) module to dynamically assign different weights to different wavelet kernels. Finally, the performance of the proposed method is validated using two gearbox datasets. In addition, we illustrate the interpretability of the proposed method by analyzing the peculiarity of different wavelet kernels, the variation of kernel weights, and the visualization of learned features.

Semantic modeling of cell damage prediction: a machine learning approach at human-level performance in dermatology

Machine learning is transforming the field of histopathology. Especially in classification related tasks, there have been many successful applications of deep learning already. Yet, in tasks that rely on regression and many niche applications, the domain lacks cohesive procedures that are adapted to the learning processes of neural networks. In this work, we investigate cell damage in whole slide images of the epidermis. A common way for pathologists to annotate a score, characterizing the degree of damage for these samples, is the ratio between healthy and unhealthy nuclei. The annotation procedure of these scores, however, is expensive and prone to be noisy among pathologists. We propose a new measure of damage, that is the total area of damage, relative to the total area of the epidermis. In this work, we present results of regression and segmentation models, predicting both scores on a curated and public dataset. We have acquired the dataset in collaborative efforts with medical professionals. Our study resulted in a comprehensive evaluation of the proposed damage metrics in the epidermis, with recommendations, emphasizing practical relevance for real world applications.

Pre-Design of Multi-Band Planar Antennas by Artificial Neural Networks

In this communication, artificial neural networks are used to estimate the initial structure of a multiband planar antenna. The neural networks are trained on a set of selected normalized multiband antennas characterized by time-efficient modal analysis with limited accuracy. Using the Deep Learning Toolbox in Matlab, several types of neural networks have been created and trained on the sample planar multiband antennas. In the neural network learning process, suitable network types were selected for the design of these antennas. The trained networks, depending on the desired operating bands, will select the appropriate antenna geometry. This is further optimized using Newton’s method in HFSS. The use of the neural pre-design concept speeds up and simplifies the design of multiband planar antennas. The findings presented in this paper will be used to refine and accelerate the design of planar multiband antennas.

Decentralized Resource Allocation-Based Multiagent Deep Learning in Vehicular Network

Resource allocation (RA) has a significant impact on vehicular network performance. With high mobility, RA is more challenging, as the number of vehicles in close proximity changes dynamically in the nonstationary environment. In this article, we propose a multiagent double deep Q-networks scheme to stabilize the system and maximize the sum-capacity of the vehicle-to-infrastructure (V2I) links, while satisfying the reliability and delay constraints for vehicle-to-vehicle (V2V) links. To avoid interference caused by unstable V2V links, a transmission mode selection is considered in the scheme design. In addition, we introduce a binarized weight algorithm to accelerate the deep neural network learning process and, therefore, improve the computational complexity of our scheme. Through extensive simulations and complexity analysis, we demonstrate that the proposed scheme yields excellent performance in terms of the sum-rate and probability rate of V2I and V2V communication modes. We also compare the proposed scheme with binarized weights with other algorithms in terms of accuracy evaluation.

Adaptive critic learning for event‐triggered safe control of nonlinear safety‐critical systems

AbstractIn this paper, an event‐triggered safe control method based on adaptive critic learning (ACL) is proposed for a class of nonlinear safety‐critical systems. First, a safe cost function is constructed by adding a control barrier function (CBF) to the traditional quadratic cost function; the optimization problem with safety constraints that is difficult to deal with by classical ACL methods is solved. Subsequently, the event‐triggered scheme is introduced to reduce the amount of computation. Further, combining the properties of CBF with the ACL‐based event‐triggering mechanism, the event‐triggered safe Hamilton–Jacobi–Bellman (HJB) equation is derived, and a single critic neural network (NN) framework is constructed to approximate the solution of the event‐triggered safe HJB equation. In addition, the concurrent learning method is applied to the NN learning process, so that the persistence of excitation (PE) condition is not required. The weight approximation error of the NN and the states of the system are proven to be uniformly ultimately bounded (UUB) in the safe set with the Lyapunov theory. Finally, the availability of the presented method can be validated through the simulation.

Improved Cascade Correlation Neural Network Model Based on Group Intelligence Optimization Algorithm

The Cascade Correlation learning algorithm is a special supervised learning algorithm for artificial neural network architecture. The optimization algorithm in the traditional neural network has the disadvantages of a single optimization goal, slow convergence speed, and can easily fall into local area, which cannot fully meet the key elements in the cascade correlation learning algorithm. In comparison, the group intelligence optimization algorithm can take into account these key elements in the optimization process at the same time, and obtain better optimization results. In this paper, we propose the single-objective optimization algorithm jDE-B and the multi-objective optimization algorithm MOEA-T, and improve the network expansion mode in the learning process of Cascade Correlation neural networks. We investigate the effect of applying the group intelligent optimization algorithm in the Cascade Correlation learning algorithm. Experimental results show that our improved algorithm is able to enhance the ability of the Cascade Correlation neural network to fit problems, reduce the number of hidden units and the depth of the network, and optimize the network structure.

Evaluation of the effectiveness of training and the use of neural networks to predict biological age.

Neural network training is widely used in various educational fields: staff training, mastering the curriculum at school and university, forming recommendations for private use by respondents, for teaching pensioners health-saving techniques. It is relevant to analyze the learning process of neural networks and evaluate their effectiveness on various models. For the study, a model was chosen for predicting the index of biological aging of a person based on the characteristics of his personality. To train neural networks, a data matrix of 1,632 people aged 35 to 70 years was compiled. Output variable: biological aging index; input variables: gender, age, family and professional status, place of residence, body type, type of functional asymmetry, style of relationships with people, as well as personal resources. Four neural networks were trained: for men and women, for working professionals and for pensioners. Results: 1) trained neural networks give different recommendations for men and women of pre-retirement and post-retirement age. 2) The effectiveness of predicting the biological aging index using neural networks turned out to be significantly high for all trained programs. 3) Neural networks can be used to model various social situations and identify what changes this will lead to for output variables. Situations were modeled: a) if all single adults get married, b) if all family adults break up, c) if everyone receives the recommendations of psychologists on the selection of personal resources and will use them. The neural network has issued a forecast: it is better for adult women not to change their family status/it is better for adult men to change their status. The use of personal resources selected by psychologists is effective for everyone.

HOW IMPORTANT ARE ACTIVATION FUNCTIONS IN REGRESSION AND CLASSIFICATION? A SURVEY, PERFORMANCE COMPARISON, AND FUTURE DIRECTIONS

Inspired by biological neurons, the activation functions play an essential part in the learning process of any artificial neural network (ANN) commonly used in many real-world problems. Various activation functions have been proposed in the literature for classification as well as regression tasks. In this work, we survey the activation functions that have been employed in the past as well as the current state-of-the-art. In particular, we present various developments in activation functions over the years and the advantages as well as disadvantages or limitations of these activation functions. We also discuss classical (fixed) activation functions, including rectifier units, and adaptive activation functions. In addition to discussing the taxonomy of activation functions based on characterization, a taxonomy of activation functions based on applications is presented. To this end, the systematic comparison of various fixed and adaptive activation functions is performed for classification datasets such as MNIST, CIFAR-10, and CIFAR-100. In recent years, a physics-informed machine learning framework has emerged for solving problems related to scientific computations. For this purpose, we also discuss various requirements for activation functions that have been used in the physics-informed machine learning framework. Furthermore, various comparisons are made among different fixed and adaptive activation functions using various machine learning libraries such as TensorFlow, PyTorch, and JAX. Our findings show that activation functions such as rectified linear unit (ReLU) and its variants, which are currently the state-of-the-art for many classification problems, do not work well in physics-informed machine learning frameworks due to the stringent requirement of the existence of derivatives, whereas other activation functions such as hyperbolic tangent, swish, and sine give better performance, with superior results achieved with adaptive activation functions, especially for multiscale problems.

Training Learning Weights of Elman Neural Network Using Salp Swarm Optimization Algorithm

Elman Neural Network (ENN) is one of the most common type of recurrent neural network (RNN). It is frequently applied to numerous applications such as classification and predication. However, the ENN model, suffers from serious flaws like network stagnancy and delayed convergence during training. To enhance the effectiveness of the ENN and address the mention problems, numerous studies have been conducted to speed up the learning process of ENN. Many optimization techniques are being used and each algorithm has its own characteristics in terms of efficiency. Some of the algorithms inspire from nature, while others come from the swarm's combined behaviour. Different numbers of optimization methods, including Particle Swarm Optimization (PSO), Bat Algorithm, and Cuckoo Search (CS) are applied to improve the ENN's learning process. As a result, this study suggested using Salp Swarm algorithm (SSA) to address these issues and speed up the ENN algorithm's learning process. The SSA, which is based on the typical behaviour of Salp swarms, is proposed in this research as a novel optimization technique. Due to its effectiveness and Salp intelligent behaviour, a novel Salp algorithm improves the learning process of the artificial neural network (ANN), and ENN models. Therefore this study proposed Salp swarm artificial neural network (SSANN) and Salp swarm Elman neural network (SSElmanNN). Performance of the suggested models is evaluated against ANN, back propagation neural network (BPNN), along with ENN in term of accuracy and Mean Square Error (MSE). Two datasets such as IRIS and Credit Card are used for simulation. The simulation results demonstrate that the suggested models outperform the other algorithms used in this study in terms of MSE and accuracy.

A scale conjugate neural network learning process for the nonlinear malaria disease model

<p>The purpose of this work is to provide a stochastic framework based on the scale conjugate gradient neural networks (SCJGNNs) for solving the malaria disease model of pesticides and medication (MDMPM). The host and vector populations are divided in the mathematical form of the malaria through the pesticides and medication. The stochastic SCJGNNs procedure has been presented through the supervised neural networks based on the statics of validation (12%), testing (10%), and training (78%) for solving the MDMPM. The optimization is performed through the SCJGNN along with the log-sigmoid transfer function in the hidden layers along with fifteen numbers of neurons to solve the MDMPM. The accurateness and precision of the proposed SCJGNNs is observed through the comparison of obtained and source (Runge-Kutta) results, while the small calculated absolute error indicate the exactitude of designed framework based on the SCJGNNs. The reliability and consistency of the SCJGNNs is observed by using the process of correlation, histogram curves, regression, and function fitness.</p>

Complexity emerging from simplicity: Bifurcation analysis of the weights time series in a feedforward neural network

Artificial neural networks (ANN), inspired by biological neural networks and the human brain, have attracted neuroscience researchers’ interest to simulate cognitive processes in recent years. However, the learning mechanisms inside the artificial neural networks are still unclear. Time series analysis of the weights of a network during the training stage can illustrate the dynamics behind the weights and provide new insight into the learning process.In this study, we investigate the dynamics of the weights of a three-neuron feed-forward artificial neural network based on weights recursive maps and time series. The dynamics of the weights are analyzed from the perspective of chaos theory fundamentals such as bifurcation analysis, equilibrium point analysis, and coexisting attractor emergence. While the previous works only examine the effect of in-network factors on the weights dynamics, here we investigate both in-network and out-of-network factors, including parameters, network structure, network input and output, bias, and activation function. The results show that weights time series exhibit rich nonlinear dynamics, including intermittency and chaos, even in the simplest structures of the network. Bifurcation analysis shows the emergence of several coexisting attractors, meaning that the learning process in an artificial neural network, and maybe in the human brain, takes place on different attractors. Hence, this is the initial condition of the brain that determines in which attractor the learning should occur.For the first time, using the recursive map of the weights, we introduce a novel discrete neuronal model with learning capability, which maps an input to an output. This ability distinguishes our model from the previous ones that only simulate the neuron electrical voltage without any learning process inside. This model can simulate cognitive phenomena such as neuronal synchronization in near-realistic condition during the training stage.

Learning from undercoded clinical records for automated International Classification of Diseases (ICD) coding.

To develop an unbiased objective for learning automatic coding algorithms from clinical records annotated with only partial relevant International Classification of Diseases codes, as annotation noise in undercoded clinical records used as training data can mislead the learning process of deep neural networks. We use Medical Information Mart for Intensive Care III as our dataset. We employ positive-unlabeled learning to achieve unbiased loss estimation, which is free of misleading training signal. We then utilize reweighting mechanism to compensate for the imbalance between positive and negative samples. To further close the performance gap caused by poor quality annotation, we integrate the supervision provided by the automatic annotation tool Medical Concept Annotation Toolkit which can ease the heavy burden of manual validation. Our benchmarking results show that positive-unlabeled learning with reweighting outperforms competitive baseline methods over a range of missing label ratios. Integrating supervision provided by annotation tool further boosted the performance. Considering the annotation noise and severe imbalance, unbiased loss estimation and reweighting mechanism are both important for learning from undercoded clinical records. Unbiased loss requires the estimation of false negative ratios and estimation through trained models is practical and competitive. The combination of positive-unlabeled learning with reweighting and supervision provided by the annotation tool is a promising solution to learn from undercoded clinical records.

Selective local binary pattern with convolutional neural network for facial expression recognition

<span lang="EN-US">Variation in images in terms of head pose and illumination is a challenge in facial expression recognition. This research presents a hybrid approach that combines the conventional and deep learning, to improve facial expression recognition performance and aims to solve the challenge. We propose a selective local binary pattern (SLBP) method to obtain a more stable image representation fed to the learning process in convolutional neural network (CNN). In the preprocessing stage, we use adaptive gamma transformation to reduce illumination variability. The proposed SLBP selects the discriminant features in facial images with head pose variation using the median-based standard deviation of local binary pattern images. We experimented on the Karolinska directed emotional faces (KDEF) dataset containing thousands of images with variations in head pose and illumination and Japanese female facial expression (JAFFE) dataset containing seven facial expressions of Japanese females’ frontal faces. The experiments show that the proposed method is superior compared to the other related approaches with an accuracy of 92.21% on KDEF dataset and 94.28% on JAFFE dataset.</span>

Fourier Ptychographic Microscopy Reconstruction Method Based on Residual Transfer Networks

Fourier ptychographic microscopy reconstruction mostly adopts the traditional alternating iterative phase recovery method and optimization method, which has high computational complexity, high redundancy of image acquisition data, low reconstruction quality and high time consumption. In this paper, the model of residual transfer networks based on Resnet152 is proposed for Fourier ptychographic microscopy reconstruction, the learning process of deep convolution neural network is introduced, and the image reconstruction method based on deep learning realizes the end-to-end reconstruction of low-resolution images to high-resolution images. Through comparative experiments and analysis, the residual network can overcome the gradient explosion, make the feature information more complete and efficient, and the incremental up-sampling reconstruction network has higher image quality, lower computational complexity and shorter running time.

Application of Meta-Heuristic Algorithms for Training Neural Networks and Deep Learning Architectures: A Comprehensive Review.

The learning process and hyper-parameter optimization of artificial neural networks (ANNs) and deep learning (DL) architectures is considered one of the most challenging machine learning problems. Several past studies have used gradient-based back propagation methods to train DL architectures. However, gradient-based methods have major drawbacks such as stucking at local minimums in multi-objective cost functions, expensive execution time due to calculating gradient information with thousands of iterations and needing the cost functions to be continuous. Since training the ANNs and DLs is an NP-hard optimization problem, their structure and parameters optimization using the meta-heuristic (MH) algorithms has been considerably raised. MH algorithms can accurately formulate the optimal estimation of DL components (such as hyper-parameter, weights, number of layers, number of neurons, learning rate, etc.). This paper provides a comprehensive review of the optimization of ANNs and DLs using MH algorithms. In this paper, we have reviewed the latest developments in the use of MH algorithms in the DL and ANN methods, presented their disadvantages and advantages, and pointed out some research directions to fill the gaps between MHs and DL methods. Moreover, it has been explained that the evolutionary hybrid architecture still has limited applicability in the literature. Also, this paper classifies the latest MH algorithms in the literature to demonstrate their effectiveness in DL and ANN training for various applications. Most researchers tend to extend novel hybrid algorithms by combining MHs to optimize the hyper-parameters of DLs and ANNs. The development of hybrid MHs helps improving algorithms performance and capable of solving complex optimization problems. In general, the optimal performance of the MHs should be able to achieve a suitable trade-off between exploration and exploitation features. Hence, this paper tries to summarize various MH algorithms in terms of the convergence trend, exploration, exploitation, and the ability to avoid local minima. The integration of MH with DLs is expected to accelerate the training process in the coming few years. However, relevant publications in this way are still rare.

Gemini connector

Spectacular advances have been made in the field of machine vision over the past decade. While this discipline is traditionally driven by geometric models, neural networks have proven to be superior in some applications and have significantly expanded the limits of what is possible. At the same time, conventional graphic models describe the relationship between images and the associated scene with textures and light in a physically realistic manner and are an important part of photogrammetry. Differential renderers combine these approaches by enabling gradient-based optimization in fixed structures of a graphics pipeline and thus adapt the learning process of neural networks. This fusion of formalized knowledge and machine learning motivates the idea of a modular differentiable renderer in which physical and statistical models can be recombined depending on the use case. We therefore present Gemini Connector: an initiative for the modular development and combination of differentiable physical models and neural networks. We examine opportunities and problems and motivate the idea with the extension of a differentiable rendering pipeline to include models of underwater optics for the analysis of deep sea images. Finally, we discuss use cases, especially within the Cross-Domain Fusion initiative.

A multivariate binary decision tree classifier based on shallow neural network

In this paper, a novel decision tree classifier based on shallow neural networks with piecewise and nonlinear transformation activation functions are presented. A shallow neural network is recursively employed into linear and non-linear multivariate binary decision tree methods which generates splitting nodes and classifier nodes. Firstly, a linear multivariate binary decision tree with a shallow neural network is proposed which employs a rectified linear unit function. Secondly, there is presented a new activation function with non-linear property which has good generalization ability in learning process of neural networks. The presented method shows high generalization ability for linear and non-linear multivariate binary decision tree models which are called a Neural Network Decision Tree (NNDT). The proposed models with high generalization ability ensure the classification accuracy and performance. A novel split criterion of generating the nodes which focuses more on majority objects of classes on the current node is presented and employed in the new NNDT models. Furthermore, a shallow neural network based NNDT models are converted into a hyperplane based linear and non-linear multivariate decision trees which has high speed in the processing classification decisions. Numerical experiments on publicly available datasets have showed that the presented NNDT methods outperform the existing decision tree algorithms and other classifier methods.

Design and Implementation of a Multidimensional Visualization Reconstruction System for Old Urban Spaces Based on Neural Networks.

This article presents an in-depth study and analysis of the construction of a convolutional neural network model and multidimensional visualization system of old urban space and proposes the design of a multifaceted visualization reconstruction system of old urban space based on a neural network. It also quantitatively analyzes the essential spatial attribute characteristics of urban shadow areas as nodes of the overall urban dynamic network in three dimensions—spatial connection strength, spatial connection distance, and spatial connection direction—summarizes the characteristics of urban old spatial structure from the perspective of a dynamic network, and then proposes the model of urban old spatial design from the perspective of an active network. The shallow depth of the network structure is used to reduce the parameters in the learning process of reconfigurable convolutional neural networks using data sets so that the model learns more general features. For the situation where the number of data sets is small, data augmentation is used to expand the size of the data sets and improve the recognition accuracy of the reconfigurable convolutional neural network. A real-time update method of multifaceted data visualization for big data scenarios is proposed and implemented to reduce the network load and network latency caused by charts of multidimensional data changes, reduce the data error rate, and maintain the system stability in the old urban space concurrency scenario.

Analysis of Mobile Robot Control by Reinforcement Learning Algorithm

This work presents a Deep Reinforcement Learning algorithm to control a differentially driven mobile robot. This study seeks to explain the influence of different definitions of the environment with a mobile robot on the learning process. In our study, we focus on the Reinforcement Learning algorithm called Deep Deterministic Policy Gradient, which is applicable to continuous action problems. We investigate the effectiveness of different noises, inputs, and cost functions in the neural network learning process. To examine the feature of the presented algorithm, a number of simulations were run, and their results are presented. In the simulations, the mobile robot had to reach a target position in a way that minimizes distance error. Our goal was to optimize the learning process. By analyzing the results, we wanted to recommend a more efficient choice of input and cost functions for future research.

Recognition and Detection Methods of Artificial Intelligence in Computer Network Faults under the Background of Big Data

With the widespread use of computers and the rapid development of Internet technology, computer application technology has become more and more important in people’s work and life. The article mainly studies particle swarm optimization (PSO) and radial basis neural network function (RBF). Particle swarm optimization is an evolutionary swarm intelligence algorithm, such as nonderivative node transfer function or gradient information loss. Because its principle is simple and easy to implement, it can deal with some problems that cannot be solved by traditional methods. It is widely used in neural networks, and has achieved good results in many fields such as network training, performance optimization, and system mismanagement. RBF neural network is a feedforward neural network, which overcomes the shortcomings of traditional neural network learning process that the convergence is highly dependent on the initial value and can only be partially converged. This paper organically combines PSO algorithm and RBF neural network to study the detection and detection of computer network faults. The results show that although the prediction error of the improved network model on the experimental test set is still only 89.3% different from that of the SVM model, its convergence time is reduced to 0.90699885 s, which can effectively detect and identify computer network faults.