Weights Of Neural Network Research Articles

Do Neural Network Weights Account for Classes Centers?

The exploitation of deep neural networks (DNNs) as descriptors in feature learning challenges enjoys apparent popularity over the past few years. The above tendency focuses on the development of effective loss functions that ensure both high feature discrimination among different classes, as well as low geodesic distance between the feature vectors of a given class. The vast majority of the contemporary works rely their formulation on an empirical assumption about the feature space of a network's last hidden layer, claiming that the weight vector of a class accounts for its geometrical center in the studied space. This article at hand follows a theoretical approach and indicates that the aforementioned hypothesis is not exclusively met. This fact raises stability issues regarding the training procedure of a DNN, as shown in our experimental study. Consequently, a specific symmetry is proposed and studied both analytically and empirically that satisfies the above assumption, addressing the established convergence issues. More specifically, the aforementioned symmetry suggests that all weight vectors are unit, coplanar, and their vector summation equals zero. Such a layout is proven to ensure a more stable learning curve compared against the corresponding ones succeeded by popular models in the field of feature learning.

Optimal tracking control for unknown nonlinear systems with uncertain input saturation: A dynamic event-triggered ADP algorithm

This paper is devoted to proposing a novel dynamic event-triggered adaptive dynamic programming (ADP) algorithm to tackle the tracking control problem of unknown nonlinear systems with uncertain input saturation. Initially, an augmented system is established to address the investigated optimal tracking control issue with a discounted cost function. An online identifier based on neural networks (NNs) is designed to recover the unknown system dynamics. Furthermore, the controller design consists of two parts, including the optimal control policy for the nominal system and the NN-based feedforward compensator. In particular, the latter leverages adaptive NN techniques to cope with uncertain input constraints which can be deemed as uncertain saturation nonlinearities. A critic NN is constructed to obtain the approximate optimal control policy for the nominal system. Moreover, rather than traditional time-triggered and static event-triggered control schemes, an adaptive dynamic event-triggering condition is designed to determine the system sampling as well as the controller execution to further reduce transmission and computation burden. The introduction of an internal dynamic variable in possession of a positive constant lower bound can realize the dynamic adjustment of triggering threshold and effectively exclude Zeno behavior. Notice that the weights of identifier NN and critic NN are updated simultaneously, only at the event-triggering instants. Subsequently, the Lyapunov-based theoretical analysis is conducted to confirm the uniform ultimate boundedness of the tracking error and all the NN weight estimation errors. Eventually, the feasibility and effectiveness of the developed algorithm are validated by means of two simulation examples.

Coherent Blending of Biophysics-Based Knowledge with Bayesian Neural Networks for Robust Protein Property Prediction.

Predicting properties of proteins is of interest for basic biological understanding and protein engineering alike. Increasingly, machine learning (ML) approaches are being used for this task. However, the accuracy of such ML models typically degrades as test proteins stray further from the training data distribution. On the other hand, models that are more data-free, such as biophysics-based models, are typically uniformly accurate over all of the protein space, even if inferior for test points close to the training distribution. Consequently, being able to cohesively blend these two types of information within one model, as appropriate in different parts of the protein space, will improve overall importance. Herein, we tackle just this problem to yield a simple, practical, and scalable approach that can be easily implemented. In particular, we use a Bayesian formulation to integrate biophysical knowledge into neural networks. However, in doing so, a technical challenge arises: Bayesian neural networks (BNNs) enable the user to specify prior information only on the neural network weight parameters, rather than on the function values given to us from a typical biophysics-based model. Consequently, we devise a principled probabilistic method to overcome this challenge. Our approach yields intuitively pleasing results: predictions rely more heavily on the biophysical prior information when the BNN epistemic uncertainty─uncertainty arising from a lack of training data rather than sensor noise─is large and more heavily on the neural network when the epistemic uncertainty is small. We demonstrate this approach on an illustrative synthetic example, on two examples of protein property prediction (fluorescence and binding), and for generality on one small molecule property prediction problem.

Tikhonov-Tuned Sliding Neural Network Decoupling Control for an Inverted Pendulum

This paper introduces the concept of intelligent control using Tikhonov regularization for nonlinear coupled systems. This research is driven by the increasing demand for advanced control techniques and aims to explore the impact of Tikhonov regularization on these systems. The primary objective is to determine the optimal regularization term and its integration with other control methods to enhance intelligent control for nonlinear coupled systems. Tikhonov regularization is a technique employed to adjust neural network weights and prevent overfitting. Additionally, the incorporation of ReLU activation function in the neural network simplifies thearchitecture, avoiding issues like gradient explosion, and optimizes controller performance. Furthermore, sliding surfaces are designed to improve control system stability and robustness. The proposed Tikhonov-tuned sliding neural network (TSN) controller ensures both stability and superior system performance. The methodology emphasizes the importance of determining optimal neural network weights and regularization terms to prevent overfitting, facilitating accurate predictions in inverted pendulum control system applications. To assess the functionality and stability of TSN, this paper employs simulations and experimental implementations to control both the rotary inverted pendulum and the arm-driven inverted pendulum. The results indicate that the proposed TSN methodologies are effective and feasible.

Machine learning analysis of heat transfer and electroosmotic effects on multiphase wavy flow: a numerical approach

PurposeThis study aims to present a unique hybrid metaheuristic approach to solving the nonlinear analysis of hall currents and electric double layer (EDL) effects in multiphase wavy flow by merging the firefly algorithm (FA) and the water cycle algorithm (WCA).Design/methodology/approachNonlinear Hall currents and EDL effects in multiphase wavy flow are originally described by partial differential equations, which are then translated into an ordinary differential equation model. The hybrid FA-WCA technique is used to take on the optimization challenge and find the best possible design weights for artificial neural networks. The fitness function is efficiently optimized by this hybrid approach, allowing the optimal design weights to be determined.FindingsThe proposed strategy is shown to be effective by taking into account multiple variables to arrive at a single answer. The numerical results obtained from the proposed method exhibit good agreement with the reference solution within finite intervals, showcasing the accuracy of the approach used in this study. Furthermore, a comparison is made between the presented results and the reference numerical solutions of the Hall Currents and electroosmotic effects in multiphase wavy flow problem.Originality/valueThis comparative analysis includes various performance indices, providing a statistical assessment of the precision, efficiency and reliability of the proposed approach. Moreover, to the best of the authors’ knowledge, this is a new work which has not been explored in existing literature and will add new directions to the field of fluid flows to predict most accurate results.

On Neuroevolution of Multi-Input Compositional Pattern Producing Networks: A Case of Entertainment Computing, Edge Devices, and Smart Cities

This work presents a novel approach by utilizing Heterogeneous Activation Neural Networks (HA-NNs) to evolve the weights of Artificial Neural Networks (ANNs) for reinforcement learning in console and arcade computer games like Atari's Breakout and Sonic the Hedgehog. It is the first study to explore the potential of HA-NNs as potent ANNs in solving gaming-related reinforcement learning problems. Additionally, the proposed solution optimizes data transmission over networks for edge devices, marking a novel application of HA-NNs. The study achieved outstanding results, outperforming recent works in benchmark environments like CartPole-v1, Lunar Lander Continuous, and MountainCar-Continuous, with HA-NNs and ANNs evolved using the Neuroevolution of Augmenting Topologies (NEAT) algorithm. Notably, the key advancements include exceptional scores of 500 in CartPole-v1 and 98.2 in Mountain Car Continuous, demonstrating the efficacy of HA-NNs in reinforcement learning tasks. Beyond gaming, the research addresses the challenge of efficient data communication between edge devices, which has the potential to enhance performance in smart cities while reducing the load on edge devices and supporting seamless entertainment experiences with minimal commuting. This work pioneers the application of HA-NNs in reinforcement learning for computer games and introduces a novel approach for optimizing edge device communication, promising significant advancements in the fields of AI, neural networks, and smart city technologies.

Evaluation of oral English teaching quality based on BP neural network optimized by improved crow search algorithm

Oral English teaching quality evaluation is a complex nonlinear relationship, which is affected by many factors and has low accuracy. Aiming at the problem, a teaching quality evaluation method based on a BP neural network optimized by the improved crow search algorithm (ICSA) is proposed. First, ICSA is put forward and five algorithms are used to compare with the proposed algorithm on 10 benchmarks functions. The results show that ICSA outperforms the other five algorithms on 10 functions. Second, a feature selection method based on the improved binary crow search algorithm (BICSA) is used to select teaching quality evaluation indexes, and 10 standard datasets from the UCI repository are used for testing experiments. Finally, an oral English teaching evaluation model based on BP neural network is designed, in which BICSA is used for feature selection and ICSA is used to optimize the initial weights of the BP neural network. In the experiment, we designed 5 first-grade indexes and 15 second-grade indexes, and then we collects 23 groups of oral English teaching quality data. BICSA selected 10 features from a set of 15 features. Experimental results show that this method can effectively evaluate the quality of oral English teaching with high accuracy and real-time performance.

Model-based neuroadaptive event-triggered tracking consensus control for nonlinear multiagent systems with input delay

In this article, a neuroadaptive event-triggered control methodology is proposed to tackle the leader-following tracking consensus issue of uncertain nonlinear multiagent systems (MASs) with input delay and external disturbances. The proposed approach incorporates radial bias function neural networks (RBF NNs) and the integral compensation technique, effectively addressing the issues of nonlinear uncertainty and input delay, respectively. For the sake of reducing the transmission of information, we adopt the strategy that the neural network (NN) weight update law is only changed at the event-triggered instant, and a novel adaptive model and related controller are constructed. Simultaneously, this paper establishes an impulsive dynamic system to analyze the stability of the closed-loop system by extending the Lyapunov theorem to continuous and reset dynamics. Sufficient conditions for achieving leader-following tracking consensus are derived without the Zeno behavior. Finally, a numerical example is presented to illustrate the validity of the proposed method.

Uncertainty-aware trustworthy weather-driven failure risk predictor for overhead contact lines

Overhead contact lines (OCLs) are electric transmission lines that power railways, which are constantly threatened by external weather and environmental factors due to their outdoor location. Hence, for the long-term functioning of railway lines, a weather-driven risk predictor is an essential tool. Current prediction methods mainly adopt a single-point estimation system with fixed weights of neural networks and therefore cannot propagate the uncertainties within the data and model, resulting in unreliable predictions. To enhance safety-risk prevention capabilities, this paper proposes an uncertainty-aware trustworthy weather-driven failure-risk approach for OCLs, in a probabilistic deep multitask learning framework. Firstly, a deep Gaussian process is employed as the backbone model to deal with imbalanced weather-related failure datasets with limited fault samples. Moreover, a multi-task learning framework is embedded to simultaneously predict the multiple weather-driven failure risks under lightning, wind and haze. Finally, the predictive uncertainty is decomposed into epistemic and aleatory uncertainties, where epistemic and aleatory uncertainties account for the uncertainty within the model and data, respectively. Extensive experiments on actual OCLs are carried out to demonstrate the effectiveness of the proposed approach, which can effectively capture the predictive uncertainty and provide trustworthy predictive decisions of mitigating operational risk for railway operators.

Crow Way: An Optimization Technique for generating the Weight and Bias in Deep CNN

Fine-tuning Convolutional Neural Networks (CNNs) weights and biases are essential for solving difficult machine learning problems. We provide a hybrid optimizations strategy that combines the benefits of Crowd Search (CSA) and Brainstorm Optimization (BSO). CSA-BSO optimizes CNNs. CSA-BSO optimizes CNN weights. Our fitness measures determine each crow's location and speed throughout the CSA phase. The BSO phase mixes optimal replies with random ones to create new solutions. The best new ideas move on. After a certain number of iterations, we alternate between the CSA and BSO phases and pick the best solution as the CNN's final state. We design and test the CSA-BSO approach on many CNN architectures and datasets. Experimentally, the hybrid algorithm outperforms CSA and BSO in convergence time and solution quality. CSA and BSO collaborate to efficiently employ feasible regions and investigate a bigger search field, boosting optimizations. Stable and adaptable, CSA-BSO supports many CNN architectures and data formats. This study's main contribution is the CSA-BSO, a unique hybrid optimizations approach for convolutional neural network weight and bias optimizations. It beats CSA and BSO. The recommended strategy improves CNN picture classification, object recognition, and NLP.

PSiamRML: Target Recognition and Matching Integrated Localization Algorithm Based on Pseudo-Siamese Network

The positioning function of unmanned aerial vehicles (UAVs) is a challenging and fundamental research topic and is the premise for UAVs to realize autonomous navigation. The disappearance of satellite signals makes it challenging to achieve accurate positioning. Thus, visual positioning algorithms based on computer vision have been proposed in recent years and these algorithms have produced good results. However, these algorithms have relatively simple functions and cannot perceive the environment. Their versatility is poor, and mismatching often occurs, which affects the positioning accuracy. Aiming to address the need for integrated target recognition, target matching, and positioning of UAVs, we propose an algorithm that integrates the target recognition, matching, and positioning functions by combining the single-shot multibox detector (SSD) algorithm with the deep feature matching algorithm. This algorithm is based on the idea of pseudo-Siamese networks and the SSD algorithm, introducing a deep feature matching method to directly calculate the correspondence between two images. The main idea is to use the VGG network trained by the SSD target recognition algorithm to extract deep features, without any special training for feature matching. Finally, by sharing neural network weights, the integrated design of target recognition and image-matching localization algorithms is achieved. Mismatches between the real-time and reference images are addressed by introducing the grid-based motion statistics algorithm to optimize the matching result and improve the correct matching efficiency of the target. The University-Release dataset was used to compare and analyze the performance of the proposed algorithm to verify its superiority and feasibility. The results show that the matching accuracy of the PSiamRML algorithm is generally good and that it significantly compensates for changes in the contrast, scale, brightness, blur, deformation, and so on, apart from improving the stability and robustness. Finally, a matching test scenario with aerial images captured by an S1000 six-rotor UAV served to verify the effectiveness and practicability of the PSiamRML algorithm.

Transformer fault diagnosis based on Chaotic Particle Swarm Optimization RBF Neural Network

This article uses logistic chaotic mapping to improve the particle swarm algorithm parameters and construct the chaotic particle swarm optimization (CPSO) algorithm. Then, the CPSO algorithm is used to optimize the width, weight, and center values of the Radial Basis Function Neural Networks (RBFNN) to improve the RBFNN model used to diagnose transformer fault types. Results show that the CPSO-RBFNN model has a small mean square error and high accuracy in diagnosing transformer faults.

Neural image caption generator based on crossbar array design of memristor module

The crossbar array design method based on a single memristor provides an excessive writing error and cannot achieve the network’s requirements for the generation of image captions. In order to solve this problem, this paper highlights the following techniques. (1) A crossbar array of memristor modules is built, and the writing error is theoretically reduced to a maximum of 0.0619‰. (2) A method of linearly mapping the weights of neural networks to memristance values is proposed, and a memristive neural network based on a memristor module is designed. (3) A neural image caption generator based on VGG-16-LSTM is designed in light of the crossbar array of the memristor module. The design can achieve a maximum of 99.38% of the reference network in BLEU scoring performance. The performance of neural networks based on memristor modules under different input conditions is discussed, and the paper’s best hardware design scheme, named it-10, is proposed. The it-8 simulation can obtain 97.37% of the reference network in BLEU scoring performance, and it requires far less demanding input conditions. Moreover, this paper finds a way to design a novel crossbar array based on the memristor module and make a breakthrough in memristive neural network solution for the cross-modal image caption generation task.

Recognition of Alzheimer’s Patients in Emotional States Based on the Optimal Convolutional Neural Network and Electroencephalography

Introduction: Accurate diagnosis of Alzheimer’s disease in the early stages plays an important role in patient care, and preventive measures should be taken before irreversible brain damage occurs. With increasing age, there are changes in memory, which is normal, but the symptoms of Alzheimer’s disease are more than temporary forgetfulness. Early and intelligent diagnosis of Alzheimer’s disease in different situations can greatly help patients and physicians. Method: In the proposed method, a convolutional neural network will be used to improve the recognition of people with Alzheimer’s disease from healthy people in emotional states. First, the required pre -processing is done on the electroencephalography signal, and then, it will be applied as an input to the network. Next, the genetic algorithm is used to optimize the weights of the convolutional neural network. Results: The research shows that the frontal lobe of the brain is related to emotions and the use of F3 and F4 channels reflects more information compared to other channels, so with this information, the process of recognizing Alzheimer’s patients in emotional states is better . Conclusion: The proposed method was evaluated with other categories in valence and arousal states. It was observed that this method has a better efficiency compared to other methods with an accuracy of 92.3% in valence and 94.3% in arousal in recognizing people with Alzheimer’s disease.

State of Charge Estimation for Power Battery Using Improved Extended Kalman Filter Method Based on Neural Network

In order to enhance the accuracy of the traditional extended Kalman filter (EKF) algorithm in the estimation of the state of charge (SoC) of power batteries, we first derived the state space equation and measurement equation of lithium power batteries based on the Thevenin battery model and the modified Ampere-Hour integralalgorithm. Then, the basic principles of EKF, backpropagation neural networks (BPNNs), and a biogeography-based optimization (BBO) algorithm were analyzed, and the arc curve mobility model was used to improve the global search ability of the BBO algorithm. By combining these three algorithms, this paper proposes a BP neural network method based on the BBO algorithm. This method uses the BBO algorithm to optimize the incipient weight and threshold of the BP neural network and uses this improved neural network to modify the estimated value of the extended Kalman filter algorithm (BBOBP-EKF). Finally, the BBOBP-EKF algorithm, the extended Kalman filter algorithm based on the BP neural network (BP-EKF), and the EKF algorithm are used to estimate the error value of the SOC of a power battery, and according to the experimental data, it was confirmed that the proposed BBOBP-EKF algorithm has been improved compared to other algorithms with respect to each error index term, in which the maximum error is 1% less than that of the BP-EKF algorithm and 2.4% less than that of the EKF algorithm, the minimum error is also the smallest, and the estimation accuracy is improved compared to the traditional algorithms.

A Hybrid Weight Quantization Strategy for Memristive Neural Networks

Due to the ability to store data and process information, the memristor-based neuromorphic system has attracted extensive attention. Its efficient parallel computing approach allows it to implement neural networks in hardware. However, due to the limitation of the range of memristor conductance, it is difficult to represent high-precision weights in memristive neural network. During the off-chip learning, it is crucial to find an efficient weight quantization scheme and map it to the memristor array. Therefore, a hybrid weight quantization strategy that combines uniform and non-uniform quantization is proposed to overcome these problems. Specifically, the curve fitting of pulse modulation for tantalum oxide-based memristor is carried out, and the mapping rules of weights are proposed to simplify the process of reading verification. Furthermore, the hybrid quantization strategy is proposed and applied to a multilayer perceptron and a convolutional neural network, respectively. The effectiveness and robustness of the hybrid quantization scheme are verified in the MNIST dataset. Experiments show that the proposed hybrid quantization scheme can achieve 99.26% accuracy at 4 bits and tolerate 20% noise interference. The simulation results in this paper also provide an effective solution for the hardware implementation of memristive neural networks.

Coarse-Grained Modeling Using Neural Networks Trained on Structural Data.

We propose a method of bottom-up coarse-graining, in which interactions within a coarse-grained model are determined by an artificial neural network trained on structural data obtained from multiple atomistic simulations. The method uses ideas of the inverse Monte Carlo approach, relating changes in the neural network weights with changes in average structural properties, such as radial distribution functions. As a proof of concept, we demonstrate the method on a system interacting by a Lennard-Jones potential modeled by a simple linear network and a single-site coarse-grained model of methanol-water solutions. In the latter case, we implement a nonlinear neural network with intermediate layers trained by atomistic simulations carried out at different methanol concentrations. We show that such a network acts as a transferable potential at the coarse-grained resolution for a wide range of methanol concentrations, including those not included in the training set.

Neural network observer based on fuzzy auxiliary sliding-mode-control for nonlinear systems

This research suggests the use of neural network observers based on fuzzy auxiliary sliding mode control for the fault estimation and isolation of quadrotor unmanned aerial vehicle sensors. The fuzzy auxiliary sliding-mode-control-based adaptive approach for neural network observer is used for the approximation, reconstruction, and isolation of unknown faults by utilizing the multi-layer neural network. The neural network weight parameters are updated adaptively by using the fuzzy auxiliary sliding-mode-control approach. In conventional neural networks, the gradient descent approach-based back-propagation procedures are adapted for the training of the neural network. In this research, a new concept of the nonlinear controller such as the fuzzy auxiliary sliding-mode-control method is adapted for neural network online training, in which the fuzzy auxiliary sliding-mode-control is used as the learning approach; the neural network is used as a control process that calculates the stable and dynamic learning rates. By the consideration of unknown faults diagnosis and isolation, the online learning approach used in this research has shown the faults estimation, reconstruction, and isolation abruptly and with high accuracy compared to conventional approaches as well the approach used in literature. Strategies adopted in the literature are not capable of fault detection, estimation, and reconstruction with high accuracy and abruptness compared to the method adopted in this research. The presented approach is validated by using the nonlinear dynamics of quadrotor unmanned aerial vehicles, results show the accuracy, abruptness, and efficiency compared to the algorithms adapted in the literature. It is suggested that the proposed strategy can be integrated into nonlinear systems fault diagnosis, fault isolation, and for increasing the system performance.

Robust tracking control for uncertain Euler–Lagrange systems via dynamic event‐triggered and self‐triggered ADP

AbstractThis article investigates the robust tracking control problem for a class of Euler–Lagrange systems in presence of parameter uncertainties and external disturbances. Through system transformation and theoretical analysis, an adaptive dynamic programming (ADP) algorithm with two adaptive neural networks (NNs) and a suitable triggering mechanism is proposed to attain the robust stability of the closed‐loop system. A single critic NN is leveraged to implement the approximate optimal controller design. Particularly, an NN‐based feedforward compensation is developed to cope with the uncertainties with unknown bounds. Two different triggering mechanisms are respectively constructed to reduce the budget of sampling, communication and computation, namely the dynamic event‐triggering mechanism (DETM) and the self‐triggering mechanism (STM). The DETM is utilized to decide the update of remote controller and critic NN weight, which can yield a larger inter‐event interval than the static event‐triggering mechanism. Also, the Zeno‐free behavior is guaranteed. Moreover, it is a novel attempt to introduce the STM into ADP design, which relaxes the demand of dedicated hardware online monitoring the event‐triggering condition. Then it is demonstrated that all signals in the closed‐loop system are uniformly ultimately bounded (UUB) via Lyapunov‐based stability analysis. Finally, a simulation example of 2‐link robotic system is implemented to verify the feasibility and effectiveness of the proposed algorithm.

System for quantitative diagnosis of COVID-19 associated Pneumonia based on Superpixels with deep learning and chest CT

COVID-19 is a disease caused by the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) that can lead to complications such as acute respiratory distress syndrome, acute heart injury and secondary infections in a relatively high proportion of patients and, consequently, lead to significant mortality. The definitive diagnosis of COVID-19 is performed by real-time Polymerase Chain Reaction (RT-PCR). However, as RT-PCR results usually take longer to be completed than computed tomography (CT), the latter has had an important role in the detection of patients infected with COVID-19. A rough estimate of the extent of lung involvement by the disease is also important and considered an additional criterion for deciding on discharge or hospitalization. Recent research has adopted deep neural networks and other machine learning approaches to detect the presence of lung infection caused by COVID-19. However, the extent of lung involvement (volume) caused by the disease has been only superficially investigated. In this work, we created an end-to-end computer vision system to automatically quantify the Percentage of Infection (POI) in chest CT images of COVID-19 cases confirmed by laboratorial analysis. We evaluated the performance of three well-known deep neural networks, trained using three different strategies: (1) no transfer learning with randomly initialized weights; (2) transfer learning using with ImageNet weights without fine-tuning; and (3) transfer learning with weight fine-tuning. Data augmentation and dropout were used during training to reduce overfitting and increase the generalization capacity of the models. Our approach consists of segmenting a chest CT with the SLIC Superpixels method and classifying each segment (superpixel) into a specific class (COVID or non-COVID). We used the weights of the best deep neural network to classify the superpixels and quantify the COVID-19 infection by calculating the POI on chest CT. The results indicate that deep learning models can be successfully used to support radiologists in the quantitative diagnosis of lung infection caused by COVID-19, reaching an accuracy of up to 98.4% with Inception-Resnet-v2 architecture.