Output Layer Of Network Research Articles

Activation Functions for Neural Networks: Application and Performance-based Comparison

Past decade has seen explosive growth of Deep Learning (DL) algorithms based on Artificial Neural Networks (ANNs) and its applications in vast emerging domains to solve real world complex problems. The DL architecture uses Activation Functions (AFs), to perform the task of finding relationship between the input feature and the output. Essential building blocks of any ANN are AFs which bring the required non-linearity of the output in the Output layer of network. Layers of ANNs are combinations of linear and nonlinear AFs. Most extensively used AFs are Sigmoid, Hyperbolic Tangent (Tanh), Rectified Linear Unit (ReLU) etc to name a few. Choosing an AF for a particular AF depends on various factors such as Nature of Application, Design of ANN, Optimizers used in the network, Complexity of Data etc. This paper presents a survey on most widely used AFs along with the important consideration while selecting an AF on a specific problem domain. A broad guideline on selecting an AF based on the literature survey has been presented to help researchers in employing suitable AF in their problem domain.

FSKT‐GE: Feature maps similarity knowledge transfer for low‐resolution gaze estimation

AbstractThe limited of texture details information in low‐resolution facial or eye images presents a challenge for gaze estimation. To address this, FSKT‐GE (feature maps similarity knowledge transfer for low‐resolution gaze estimation) is proposed, a gaze estimation framework consisting of both a high resolution (HR) network and low resolution (LR) network with the identical structure. Rather than mere feature imitation, this issue is addressed by assessing the cosine similarity of feature layers, emphasizing the distribution similarity between the HR and LR networks. This enables the LR network to acquire richer knowledge. This framework utilizes a combination loss function, incorporating cosine similarity measurement, soft loss based on probability distribution difference and gaze direction output, along with a hard loss from the LR network output layer. This approach on low‐resolution datasets derived from Gaze360 and RT‐Gene datasets is validated, demonstrating excellent performance in low‐resolution gaze estimation. Evaluations on low‐resolution images obtained through 2×, 4×, and 8× down‐sampling are conducted on two datasets. On the Gaze360 dataset, the lowest mean angular errors of 10.97°, 11.22°, and 13.61° were achieved, while on the RT‐Gene dataset, the lowest mean angular errors of 6.73°, 6.83°, and 7.75° were obtained.

A real-time method for detecting bottom defects of lithium batteries based on an improved YOLOv5 model

Defect detection of lithium batteries is a crucial step in lithium battery production. However, traditional detection methods mainly rely on the human eyes to observe the bottom defects of lithium battery products, which have low detection accuracy and slow detection speed. To solve this practical problem, an improved YOLOv5s model is proposed in this paper. Firstly, a new layer of the network output layer is added to improve the detection effect of small defects. Secondly, to extract important information in the feature maps, the convolutional block attention module attention mechanism is added to the YOLOv5s model. Finally, a new position loss function is used to improve the accuracy of the position prediction of the model. The experimental results indicate that the improved YOLOv5s model can accurately and quickly detect three types of defects on the bottom surface of lithium batteries. Specifically, the loss and mean average precision (mAP) of the improved YOLOv5s model are 0.033 94 and 67.5% respectively. Compared with the traditional YOLOv5s model, the loss of the improved YOLOv5s model is reduced by 31%. As well as, the mAP of the improved YOLOv5s model is increased by 4.3% on the lithium battery defect dataset. Compared with the YOLOv3, YOLOv3-spp, retinanet and YOLOv4, the mAP of the improved YOLOv5s model increased by 5.4%, 0.7%, 11.9% and 3.7% respectively. Compared with other improved YOLOv5 algorithms used in various fields, the mAP of the proposed model on the lithium battery dataset is the highest. The detection speed of the improved YOLOv5s model reaches 111 frames per second, which can meet the real-time detection requirements. The improved YOLOv5s model has board application prospects in the industrial production of lithium batteries.

DPA‐UNet rectal cancer image segmentation based on visual attention

SummaryLow segmentation accuracy and background noise exisit in rectal cancer lesion segmentation. To segment colorectal tumors in CT images accurately, we propose an improved U‐Net network for colorectal tumor image segmentation dilated‐pyramid‐attention U‐Net (DPA‐UNet). In this paper, we use the U‐Net network and combine techniques such as dilated convolution, weighted feature pyramid structure (W‐FPN), and convolutional block attention module (CBAM) mechanism. Firstly, CBAM and W‐FPN are combined to extract dense pixel‐level features for pixel labeling. Secondly, after the third network output layer, three serially dilated depth‐separable dilated convolutional layers with dilation rates of 1, 2, and 4, are added respectively to expand the feature map receptive field. Finally, the DPA‐UNet model is compared and analyzed with other new network structures. The experimental results show that DPA‐UNet achieves automatic segmentation of the colorectal cancer image region of interest (ROI).

Improved YOLOv3 detection method for PCB plug-in solder joint defects based on ordered probability density weighting and attention mechanism

Printed Circuit Board (PCB) is the heart component of electronic products, and its defect detection is the basic requirement of PCB quality control in the production process. Traditional visual detection methods need artificial design features, so their detection accuracy is poor, and the rate of missed and false detection is high. To solve the above problems, this paper proposes an improved YOLOv3 (You Only Look Once) detection method for PCB plug-in solder spot defects based on the combination of the ordered probability density weighting and the attention mechanism. First, in order to obtain a higher priority priori box, the ordered probability density weighting (OWA) method is used to optimize the multiple sets of a priori boxes generated by K-means. Then, to get more effective feature information, the Squeeze-and-Excitation mechanism (SE) is added to the backbone network. In the feature detection network, the Convolutional Block Attention Module (CBAM) attention mechanism is joined, at the same time in the inspection network output layer three layer feature are fusions. Finally, in order to accelerate the convergence speed of model and improve the accuracy of the model, the network loss function was improved by using the generalized joint generalized intersection over union (GIoU), and the COCO data model was applied to PCB solder spot defect training by transfer learning method. After testing, the average detection accuracy of improved network is improved from 84.35% to 96.69%, and the improved network has better convergence than the original network. The study shows that the improved method based on YOLOv3 is more suitable for industrial application of PCB plug-in solder spot defect detection.

A physics‐guided neural network‐based approach to velocity model calibration for microseismic data

ABSTRACTThe physics‐guided neural network framework combines the effectiveness of data‐driven and physics‐based models, and it is, therefore, becoming increasingly popular in geophysical applications. We present a physics‐guided neural network–based approach to calibrate velocity models for microseismic data. In our implementation, the physics‐guided neural network comprises of a user‐selected number of fully connected layers, a scaling and shifting layer and a forward modelling operator layer. We input the observed P‐ and S‐wave arrival times to the neural network. In the forward pass, the network's output layer produces normalized P‐ and S‐wave velocities for the subsurface model. The scaling and shifting layer converts the normalized output to realistic velocity values. The forward modelling operator (i.e. a ray‐shooting algorithm) layer computes traveltimes using the velocities from the preceding scaling and shifting layer and the known source–receiver locations. We then evaluate a loss function that compares the predicted traveltimes with the input observed arrival times, and update network's weights and bias parameters. We also use a weight‐based saliency measure to evaluate whether the selected network architecture (i.e. number of hidden layers and neurons) is optimal for the model calibration problem. Finally, using synthetic data examples, we demonstrate that our unsupervised physics‐guided neural network–based approach can provide robust velocity model and uncertainty estimates.

Mining of Weak Fault Information Adaptively Based on DNN Inversion Estimation for Fault Diagnosis of Rotating Machinery

Fault feature extraction plays a significant role in bearing fault diagnosis, especially in incipient fault period. Recently, deep neural network has been favored by many researchers due to its excellent hierarchical feature extraction capabilities. The existing diagnosis methods based on deep neural network mostly take original condition monitoring data as input and further convert fault diagnosis into pattern recognition issues. Although it improves the level of intelligent diagnosis, it is still confronted with practical problems. Since deep neural network includes many non-linear mapping layers, the extracted fault-related features show a high degree of abstraction, which reduces the practical level of this technology. Aimed at this problem, a mining method of understandably weak fault information is proposed based on deep neural network inversion estimation. From perspective of neuron response degree, this method mines the most sensitive input pattern that maximizes the neuron’s activation value of network output layer in the original input feature space. It can intuitively mine the weak fault information from the original signal. Two bearing experiments verify the effectiveness and reliability of the proposed method.

Statistical Mechanics of Deep Linear Neural Networks: The Backpropagating Kernel Renormalization

The success of deep learning in many real-world tasks has triggered an intense effort to understand the power and limitations of deep learning in the training and generalization of complex tasks, so far with limited progress. In this work, we study the statistical mechanics of learning in Deep Linear Neural Networks (DLNNs) in which the input-output function of an individual unit is linear. Despite the linearity of the units, learning in DLNNs is nonlinear, hence studying its properties reveals some of the features of nonlinear Deep Neural Networks (DNNs). Importantly, we solve exactly the network properties following supervised learning using an equilibrium Gibbs distribution in the weight space. To do this, we introduce the Back-Propagating Kernel Renormalization (BPKR), which allows for the incremental integration of the network weights starting from the network output layer and progressing backward until the first layer's weights are integrated out. This procedure allows us to evaluate important network properties, such as its generalization error, the role of network width and depth, the impact of the size of the training set, and the effects of weight regularization and learning stochasticity. BPKR does not assume specific statistics of the input or the task's output. Furthermore, by performing partial integration of the layers, the BPKR allows us to compute the properties of the neural representations across the different hidden layers. We have proposed an extension of the BPKR to nonlinear DNNs with ReLU. Surprisingly, our numerical simulations reveal that despite the nonlinearity, the predictions of our theory are largely shared by ReLU networks in a wide regime of parameters. Our work is the first exact statistical mechanical study of learning in a family of DNNs, and the first successful theory of learning through successive integration of DoFs in the learned weight space.

Airborne infrared aircraft target detection algorithm based on YOLOv4-tiny

Aiming at the problem that the current aerial infrared aircraft targets are blurred, low contrast, and susceptible to noise interference, which can lead to inaccurate recognition, an improved YOLOv4-tiny infrared aircraft target detection method based on cavity convolution is proposed. First, in order to make full use of the shallow features, add a parallel branch after the feature layer whose output size of YOLOv4-tiny is; then, after the first network output layer, add three parallel holes of 1, 3 respectively. 5. The depth of 5 can separate the hollow convolution layer to expand the feature map receptive field; finally, the feature fusion network is improved, the final output feature layer of the network, the improved anti-residual block extraction feature, and the size is adjusted after convolution. The prediction result is processed and output by yolo head. Experiments on the aerial infrared aircraft data set show that compared with the original YOLOv4-tiny, the detection accuracy is increased by 4.29% and the detection effect is significantly improved under the premise of less loss of detection frame rate and small weight.

Crowd Evacuation Guidance Based on Combined Action Reinforcement Learning

Existing crowd evacuation guidance systems require the manual design of models and input parameters, incurring a significant workload and a potential for errors. This paper proposed an end-to-end intelligent evacuation guidance method based on deep reinforcement learning, and designed an interactive simulation environment based on the social force model. The agent could automatically learn a scene model and path planning strategy with only scene images as input, and directly output dynamic signage information. Aiming to solve the “dimension disaster” phenomenon of the deep Q network (DQN) algorithm in crowd evacuation, this paper proposed a combined action-space DQN (CA-DQN) algorithm that grouped Q network output layer nodes according to action dimensions, which significantly reduced the network complexity and improved system practicality in complex scenes. In this paper, the evacuation guidance system is defined as a reinforcement learning agent and implemented by the CA-DQN method, which provides a novel approach for the evacuation guidance problem. The experiments demonstrate that the proposed method is superior to the static guidance method, and on par with the manually designed model method.

Prediction of the Slope Solute Loss Based on BP Neural Network

The existence of soil macropores is a common phenomenon. Due to the existence of soil macropores, the amount of solute loss carried by water is deeply modified, which affects watershed hydrologic response. In this study, a new improved BP (Back Propagation) neural network method, using Levenberg–Marquand training algorithm, was used to analyze the solute loss on slopes taking into account the soil macropores. The rainfall intensity, duration, the slope, the characteristic scale of macropores and the adsorption coefficient of ions, are used as the variables of network input layer. The network middle layer is used as hidden layer, the number of hidden nodes is five, and a tangent transfer function is used as its neurons transfer function. The cumulative solute loss on the slope is used as the variable of network output layer. A linear transfer function is used as its neurons transfer function. Artificial rainfall simulation experiments are conducted in indoor experimental tanks in order to verify this model. The error analysis and the performance comparison between the proposed method and traditional gradient descent method are done. The results show that the convergence rate and the prediction accuracy of the proposed method are obviously higher than that of traditional gradient descent method. In addition, using the experimental data, the influence of soil macropores on slope solute loss has been further confirmed before the simulation.

Diagnosis Support Model of Cardiomegaly Based on CNN Using ResNet and Explainable Feature Map

Medical expert support systems using medical data significantly contribute to the field of medicine by utilizing intelligent image analysis technology. Such medical systems must be validated and must have a transparent internal structure. Therefore, the technique of analyzing the results of the medical system is an important factor. This study proposes a diagnosis support model of cardiomegaly based on CNN using ResNet and explainable feature map. To configure the model, initially, a cardiomegaly diagnosis model is configured using ResNet, and a chest X-ray data set is used and learned. As the result changes resulting from the input changes of the model configured for model diagnosis support are detected, an explainable feature map for analysis is implemented. The input changes of the model are made as each pixel of the required image is reversed in sequence and the changes of the neural network output layer are saved on the feature map. As a result, the configured analysis method provides information inside the neural network through an accuracy result close to 80% and a visually expressed feature map. In particular, this interpretation method can be extended to a more general form in a medical support system that analyzes patterns.

Phase retrieval wavefront sensing based on image fusion and convolutional neural network

The conventional phase retrieval wavefront sensing approaches mainly refer to a series of iterative algorithms, such as G-S algorithms, Y-G algorithms and error reduction algorithms. These methods use intensity information to calculate the wavefront phase. However, most of the traditional phase retrieval algorithms are difficult to meet the real-time requirements and depend on the iteration initial value used in iterative transformation or iterative optimization to some extent, so their practicalities are limited. To solve these problems, in this paper, a phase-diversity phase retrieval wavefront sensing method based on wavelet transform image fusion and convolutional neural network is proposed. Specifically, the image fusion method based on wavelet transform is used to fuse the point spread functions at the in-focus and defocus image planes, thereby simplifying the network inputs without losing the image information. The convolutional neural network (CNN) can directly extract image features and fit the required nonlinear mapping. In this paper, the CNN is utilized to establish the nonlinear mapping between the fusion images and wavefront distortions (represented by Zernike polynomials), that is, the fusion images are taken as the input data, and the corresponding Zernike coefficients as the output data. The network structure of the training in this paper has 22 layers, they are 1 input layer, 13 convolution layers, 6 pooling layers, 1 flatten layer and 1 full connection layer, that is, the output layer. The size of the convolution kernel is 3 × 3 and the step size is 1. The pooling method selects the maximum pooling and the size of the pooling kernel is 2 × 2. The activation function is ReLU, the optimization function is Adam, the loss function is the MSE, and the learning rate is 0.0001. The number of training data is 10000, which is divided into three parts: training set, validation set, and test set, accounting for 80%, 15% and 5% respectively. Trained CNN can directly output the Zernike coefficients of order 4–9 to a high precision, with these fusion images serving as the input, which is more in line with the real-time requirements. Abundant simulation experiments prove that the wavefront sensing precision is root-mean-square(RMS) 0.015<i>λ</i>, when the dynamic range of the wavefront is the aberration of low spatial frequency within 1.1<i>λ</i> of RMS value (i.e. the dynamic range of Zernike coefficients of order 4–9 is <inline-formula><tex-math id="M600">\begin{document}$[- 0.5\lambda \,, \, 0.5\lambda]$\end{document}</tex-math><alternatives><graphic xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="5-20201362_M600.jpg"/><graphic xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="5-20201362_M600.png"/></alternatives></inline-formula>). In practical application, according to the system aberration characteristics, the number of network output layer units can be changed and the network structure can be adjusted based on the method presented in this paper, thereby training the new network suitable for higher order aberration to realize high-precision wavefront sensing. It is also proved that the proposed method has certain robustness against noise, and when the relative defocus error is within 7.5%, the wavefront sensor accuracy is acceptable. With the improvement of image resolution, the wavefront sensing accuracy is improved, but the number of input data of the network also increases with the sampling rate increasing, and the time cost of network training increases accordingly.

Naturalistic Driver Intention and Path Prediction Using Recurrent Neural Networks

Understanding the intentions of drivers at intersections is a critical component for autonomous vehicles. Urban intersections that do not have traffic signals are a common epicenter of highly variable vehicle movement and interactions. We present a method for predicting driver intent at urban intersections through multi-modal trajectory prediction with uncertainty. Our method is based on recurrent neural networks combined with a mixture density network output layer. To consolidate the multi-modal nature of the output probability distribution, we introduce a clustering algorithm that extracts the set of possible paths that exist in the prediction output and ranks them according to probability. To verify the method’s performance and generalizability, we present a real-world dataset that consists of over 23 000 vehicles traversing five different intersections, collected using a vehicle-mounted lidar-based tracking system. An array of metrics is used to demonstrate the performance of the model against several baselines.

Infrared Target Extraction Based on Immune Extension Neural Network

This paper addresses the difficulties of target extraction from low contrast low resolution infrared images. The extension neural network is integrated with the negative selection algorithm (NSA) in artificial immunity. Based on the specific recognition principle of long non-coding RNA (lncRNA) in biological immune system, the mature self-extension detectors are calculated to extract infrared target and background. Upon the convergence of clustering process, the output layer of network merges the categories corresponding to the target to be detected. The proposed method, it needs less detectors than the traditional NSA algorithm. Compared with several target detection algorithms, the proposed algorithm has higher detection accuracy and higher time efficiency.

Neural network–based speed control method and experimental verification for electromagnetic direct drive vehicle robot driver

The article presents an application of neural network to an electromagnetic direct drive vehicle robot driver. To achieve the accuracy and adaptability of speed control for driving vehicle with manual transmission and automatic transmission in various test conditions, a new neural network–based speed control method of the direct drive vehicle robot driver is presented. The mechanical actuators of the direct drive vehicle robot driver are directly driven through electromagnetic linear actuator. The coordinated control system structure of the direct drive vehicle robot driver is described. The mechanical actuators include the driving mechanical legs and the shift manipulator of the direct drive vehicle robot driver. The driving mechanical legs are consisted of the throttle mechanical leg, the brake mechanical leg, and the clutch mechanical leg. The displacement of the mechanical actuators represent the displacement of the driving mechanical legs, and the displacement of the shift manipulator. The number of the network hidden node is five, and a tangent transfer function is used as the neurons transfer function. The actual vehicle speed is as the variable of the network output layer, and a linear transfer function is as the neurons transfer function. The neural network training algorithm adopts the Levenberg–Marquardt training algorithm. The vehicle tests manipulated by the robot driver in different driving conditions are conducted. The proposed control method of the robot driver is verified and compared with the performances of other control method and human driver.

Prediction on Steel Corrosion Amount After the Concrete Cracking Due to Corrosion Expansion Based on Generalized Neural Network

Generalized neural network is applied to establish the network model of steel corrosion amount after the concrete cracking due to corrosion expansion, crack width along the ribs, bar diameter, concrete compressive strength and concrete cover thickness are used as the neurons of the input layer of network, and the loss rate of steel corrosion section is used as the output layer of network. The research results show that the predicted results of generalized neural network are in good agreement with the measured data, and the precision can meet the requirement of engineering practice, which provides an effective solution for the prediction of steel corrosion amount after the concrete cracking due to corrosion expansion.

Learning Fuzzy Network Using Sequence Bound Global Particle Swarm Optimizer

This paper proposes an algorithm for classification by learning fuzzy network with a sequence bound global particle swarm optimizer. The aim of this work can be achieved in two folded. Fold one provides an explicit mapping of an input features from original domain to fuzzy domain with a multiple fuzzy sets and the second fold discusses the novel sequence bound global particle swarm optimizer for evolution of optimal set of connection weights between hidden layer and output layer of the fuzzy network. The novel sequence bound global particle swarm optimizer can solve the problem of premature convergence when learning the fuzzy network plagued with many local optimal solutions. Unlike multi-layer perceptron with many hidden layers it has only single hidden layer. The output layer of this network contains one neuron. This network advocates a simple and understandable architecture for classification. The experimental studies show that the classification accuracy of the proposed algorithm is promising and superior to other alternatives such as multi-layer perceptron and radial basis function network.

Evolving neural network using a genetic algorithm for predicting the deformation modulus of rock masses

The determination of deformation modulus of rock masses is one of the most difficult tasks in the field of rock mechanics. Due to the high cost and measurement difficulties of in situ tests in modulus determination, the predictive models using regression based statistical methods, back propagation neural networks (BPNN) and fuzzy systems are recently employed for the indirect estimation of the modulus. Among these methods, the BPNN has been reported to be very useful in modeling the rock material behavior, such as deformation modulus, by many researchers. Despite its extensive applications, design and structural optimization of BPNN are still done via a time-consuming reiterative trial-and-error approach. This research focuses on the efficiency of the genetic algorithm (GA) in design and optimizing the BPNN structure and its application to predict the deformation modulus of rock masses. GA is utilized to find the optimal number of neurons in hidden layer, learning rates and momentum coefficients of hidden and output layers of network. Then the result is compared with that of trial-and-error procedure. For the purpose, a database including 120 data sets was employed from four dam sites and power house locations in Iran. Taking advantages of performance criteria such as MSE, MAE, r, proved that the GA-ANN model gives superior predictions over the trial-and-error model.

Determining the structure of a radial basis function network for prediction of nonlinear hydrological time series

A network using radial basis functions (RBFs) as the mapping function in the evolutionary equation for prediction of time series is presented. An RBF network requires the determination of the number of centres of RBFs, their receptive field widths and the linear weights of the network output layer. Traditionally, the number of centres of RBFs is fixed. In this paper, methods to estimate the widths of the receptive fields and the number of centres for the RBFs are introduced. The latter is based on the concept of the generalized degrees of freedom. The linear weights are determined by the least-squares method. The proposed method is then applied to make predictions of six sets of data: two theoretical functions that are known to become chaotic under certain parameter conditions (Henon map and Lorenz map), and four real-time series (discharge data from the Mekong River in Thailand and Laos, and from the Chao Phraya River in Thailand, and sea-surface temperature anomaly data). The results are at least one order of magnitude better than those obtained by a similar model with fixed number of centres as well as by a linear model and a stochastic model for most of the data sets.