Output Of Neural Network Research Articles

The Generation of Visually Credible Adversarial Examples with Genetic Algorithms

An adversarial example is an input that a neural network misclassifies although the input differs only slightly from an input that the network classifies correctly. Adversarial examples are used to augment neural network training data, measure the vulnerability of neural networks, and provide intuitive interpretations of neural network output that humans can understand. Although adversarial examples are defined in the literature as similar to authentic input from the perspective of humans, the literature measures similarity with mathematical norms that are not scientifically correlated with human perception. Our main contributions are to construct a genetic algorithm (GA) that generates adversarial examples more similar to authentic input than do existing methods and to demonstrate with a survey that humans perceive those adversarial examples to have greater visual similarity than existing methods. The GA incorporates a neural network, and we test many parameter sets to determine which fitness function, selection operator, mutation operator, and neural network generate adversarial examples most visually similar to authentic input. We establish which mathematical norms are most correlated with human perception, which permits future research to incorporate the human perspective without testing many norms or conducting intensive surveys with human subjects. We also document a tradeoff between speed and quality in adversarial examples generated by GAs and existing methods. Although existing adversarial methods are faster, a GA provides higher-quality adversarial examples in terms of visual similarity and feasibility of adversarial examples. We apply the GA to the Modified National Institute of Standards and Technology (MNIST) and Canadian Institute for Advanced Research (CIFAR-10) datasets.

Visual Parking Occupancy Detection Using Extended Contextual Image Information via a Multi-Branch Output ConvNeXt Network.

Along with society's development, transportation has become a key factor in human daily life, increasing the number of vehicles on the streets. Consequently, the task of finding free parking slots in metropolitan areas can be dramatically challenging, increasing the chance of getting involved in an accident and the carbon footprint, and negatively affecting the driver's health. Therefore, technological resources to deal with parking management and real-time monitoring have become key players in this scenario to speed up the parking process in urban areas. This work proposes a new computer-vision-based system that detects vacant parking spaces in challenging situations using color imagery processed by a novel deep-learning algorithm. This is based on a multi-branch output neural network that maximizes the contextual image information to infer the occupancy of every parking space. Every output infers the occupancy of a specific parking slot using all the input image information, unlike existing approaches, which only use a neighborhood around every slot. This allows it to be very robust to changing illumination conditions, different camera perspectives, and mutual occlusions between parked cars. An extensive evaluation has been performed using several public datasets, proving that the proposed system outperforms existing approaches.

Design and hardware in loop testing of an intelligent controller for power quality improvement in a complex micro grid

The wide-ranging use of renewable energy sources (RES) in the modern era leads to the association of distributed non-conventional energy resources in contrary to large-scale and localized conventional non-renewable sources. The prevalent use of distributed sources produces various Power-Quality issues. A thorough review of the literature provides numerous approaches for Power Quality (PQ) improvements; however, most techniques can improve only a few PQ problems. The new technical challenges like power loss, stability, reliability and power quality (PQ) issues are observed in present micro grid scenario. In this regard, this research work is an attempt to address PQ improvement in a three-phase realistic complex micro grid (integration of three simple micro grids to the point of common coupling. Improvement of various PQ factors for example sag/swell, unbalancing, power factor (Pf), total harmonic distortion (THD), communication delay and addition of impedance has been addressed in this research work. In this research article, a Kernel based Deep Auto Regressive Exogenous Output Neural Network (KDNARX) controller has been proposed and applied to improve power quality issues in a complex or realistic three phase complex micro grid (RTPCMG) in grid connected mode of operation. The RTPCMG is a combination of three simple grids with individual ratings of (4 kW, 4 kW and 6 kW). Photo voltaic cell, wind generator (WG), fuel cell (FC) and battery energy storage system (BESS) are the constituent distributed generators (DGs) for the proposed micro grid. The PQ improvement by application of proposed KDNARX controller to the DG inverter switching, has been overly tested in 12 different power quality issues under various operating conditions. The Kernel parameters are optimized by a new optimization technique called Cosine Chaotic PSO (CCPSO) algorithm. The results proved the strength of the new control technique ensuring PQ improvement and stability study of a three phase AC micro grid. Also, the potentiality of the new control scheme is compared with ANN and NARX and its superiority has been proved. Further, some case studies are validated through hardware in loop (HIL) environment thus justifying its real time implementation.

Open-circuit fault diagnosis method for inverters using deep learning and the evidence reasoning rule

Inverters having high voltage levels, high power density, and high integration are widely used. However, many high-frequency switch units also increase the probability of failure. Therefore, developing an accurate and stable fault diagnosis method is necessary. This paper proposes a fault diagnosis algorithm based on deep learning and the evidence reasoning (ER) rule. It not only ensures high diagnostic accuracy, but also enhances the stability of the diagnostic results. The algorithm takes the three-phase voltage source inverter as the research object and extracts the three-phase current signals with different types of faults as features. First, Convolutional and Deep Neural Network methods were utilized independently to determine a preliminary diagnosis. Second, the softmax functions of the Convolutional and Deep Neural Network outputs provided the probability distribution of the fault category, which was used as the evidence body for the ER rule to construct the fusion diagnosis. In addition, a new method of determining the reliability and the importance factors of the evidence was proposed in which the evaluation index of the deep-learning diagnosis result was applied. Finally, the final classification result was obtained using the ER rule. The proposed method can effectively enhance the accuracy and robustness compared with a single classifier.

Parameter optimization of a pure electric sweeper dust port by a backpropagation neural network combined with a whale algorithm

Abstract. Optimizing the structure of the suction port is the key to effectively improving the performance of the sweeping vehicle. The CFD (computational fluid dynamics) method and gas–solid two-phase flow model are used to analyse the influence rule of the structural parameters and the height above ground on the cleaning effect, which is verified by real vehicle tests. The data set was established by an orthogonal test method, and a BP (backpropagation) neural network was used to fit the structural parameters and evaluation indexes. The fitting errors were all within 5 %, indicating that the fitting results of this method were good. According to the fitting relation of the BP neural network output, the whale algorithm should be further used to solve the optimal structural parameters. The results show that the optimal parameter combination is β=63∘, d=168 mm and h=12 mm. The energy consumption of the optimized model is reduced, and the internal airflow loss is reduced. The particle residence time becomes shorter, and the particle can flow out from the outlet faster, thus improving the dust absorption effect. The research can provide a theoretical reference for performance optimization and parameter matching of sweepers.

Prediction of thermo-mechanical performance for effusion cooling by machine learning method

Fast and accurate evaluation of performance of cooling structures is very critical to the design of hot-section components in gas turbines. In the present study, machine learning method was developed to predict thermo-mechanical performance for effusion cooling on flat plate. The geometric and thermodynamic parameters including blowing ratio, density ratio, hole inclination angle, hole pitch and spacing were studied. Latin hypercube sampling was used for the design of numerical experiments, and the data sets for training model were generated by computational fluid dynamics (CFD) solutions. A kind of deep neural networks (DNNs) was used to predict the distributions of solid temperature and stress. Based on the outputs of neural network, the effects of the geometric and thermodynamic parameters on thermo-mechanical performance were investigated in detail. The results show that thermal stress decreases with the increases of density ratio and blowing ratio, while the increases of hole spacing and pitch cause the increase of thermal stress. Compared with the CFD results, the proposed DNN method has mean absolute percent error of 0.5% and 0.08% and root mean square error of 6.95 K and 5.1 MPa as applying for predicting solid temperature and stress. On the whole, machine learning is a promising method for modelling effusion cooling with high accuracy.

A multivariable output neural network approach for simulation of plug-in hybrid electric vehicle fuel consumption

This study is laser focused on the simulation of fuel consumption and fuel economy label parameters of plug-in hybrid electric vehicles. While fuel economy is a key factor in the design of plug-in hybrid electric vehicles, a fuel economy label can educate customers about the economic advantage of purchasing a particular car. The fuel economy label of a PHEV consists of parameters like driving range, electrical energy consumption, fuel economy for city, highway, and combined use, battery recharge time, and fuel consumption rates. The study used an inverse function model of an artificial neural network to simulate and calculate the parameters of the fuel economy labels of PHEVs. Firstly, the selected parameters of the fuel economy label of plug-in hybrid electric vehicles were used to develop a single output model. The output variable of the single output model was then merged with dummy functions to form input variables for the inverse function model. The output variables simulated were engine size in litres; estimated driving range when the battery is fully charged in km, battery recharged time in hours, city fuel consumption (L/100 ​km), highway fuel consumption (L/100 ​km), combined fuel consumption (L/100 ​km), estimated driving range when the tank is full, carbon dioxide (CO2) emission in grams/km, electric motor power in kW, number of cylinders, and electrical charges consumed in kWh/100 ​km. Different cases of input variables were considered for the inverse function model. The accuracy of the model was 29.1 times greater than that of the conventional inverse artificial neural network model.

Stability of Deep Neural Networks via Discrete Rough Paths

Using rough path techniques, we provide a priori estimates for the output of Deep Residual Neural Networks in terms of both the input data and the (trained) network weights. As trained network weights are typically very rough when seen as functions of the layer, we propose to derive stability bounds in terms of the total $p$-variation of trained weights for any $p\in[1,3]$. Unlike the $C^1$-theory underlying the neural ODE literature, our estimates remain bounded even in the limiting case of weights behaving like Brownian motions, as suggested in [arXiv:2105.12245]. Mathematically, we interpret residual neural network as solutions to (rough) difference equations, and analyse them based on recent results of discrete time signatures and rough path theory.

Track geometry prediction for Laser Metal Deposition based on on-line artificial vision and deep neural networks

Laser Metal Deposition (LMD) is an additive manufacturing technology that attracts great interest from the industry, thanks to its potential to realize parts with complex geometries in one piece, and to repair damaged ones, while maintaining good mechanical properties. Nevertheless, the complexity of this process has limited its widespread adoption, since different part geometries, strategies and boundary conditions can yield very different results in terms of external shapes and inner flaws. Moreover, monitoring part quality during the process execution is very challenging, as direct measurements of both structural and geometrical properties are mostly impracticable. This work proposes an on-line monitoring and prediction approach for LMD that exploits coaxial melt pool images, together with process input data, to estimate the size of a track deposited by LMD. In particular, a novel deep learning architecture combines the output of a convolutional neural network (that takes melt pool images as inputs) with scalar variables (process and trajectory data). Various network architectures are evaluated, suggesting to use at least three convolutional layers. Furthermore, results imply a certain degree of invariance to the number and size of dense layers. The effectiveness of the proposed method is demonstrated basing on experiments performed on single tracks deposited by LMD using powders of Inconel 718, a relevant material for the aerospace and automotive sectors. • LMD process requires on-line control for achieving predictable results. • Matrix and scalar variables are mixed in the proposed deep network structure. • Geometry prediction by AI is tested on single-track experiments. • Image information extracted by the CNN improves on-line track size prediction.

Reservoir Computing for Solving Ordinary Differential Equations

There is a wave of interest in using physics-informed neural networks for solving differential equations. Most of the existing methods are based on feed-forward networks, while recurrent neural networks solvers have not been extensively explored. We introduce a reservoir computing (RC) architecture, an echo-state recurrent neural network capable of discovering approximate solutions that satisfy ordinary differential equations (ODEs). We suggest an approach to calculate time derivatives of recurrent neural network outputs without using back-propagation. The internal weights of an RC are fixed, while only a linear output layer is trained, yielding efficient training. However, RC performance strongly depends on finding the optimal hyper-parameters, which is a computationally expensive process. We use Bayesian optimization to discover optimal sets in a high-dimensional hyper-parameter space efficiently and numerically show that one set is robust and can be transferred to solve an ODE for different initial conditions and time ranges. A closed-form formula for the optimal output weights is derived to solve first-order linear equations in a one-shot backpropagation-free learning process. We extend the RC approach by solving nonlinear systems of ODEs using a hybrid optimization method consisting of gradient descent and Bayesian optimization. Evaluation of linear and nonlinear systems of equations demonstrates the efficiency of the RC ODE solver.

Distributed cooperative control with collision avoidance for spacecraft swarm reconfiguration via reinforcement learning

This article investigates an adaptive, distributed, and cooperative control strategy for the problem of spacecraft swarm reconfiguration, which involves assembling the spacecraft at a close distance to one another while avoiding collisions and keeping spacecraft far from an obstacle. The key idea is to transform their opposite indices into equivalent ones by using soft and hard constraints. The proposed control strategy is inspired by the actor–critic framework of reinforcement learning (RL) algorithms: The soft constraint is designed by using a critic neural network (NN) for assembling and avoiding obstacles, while collisions among the spacecraft are prevented based on the hard constraint established in an artificial potential field (APF). By drawing support from this idea of equivalent transformation, the adaptive, distributed, and cooperative controller is devised by using an actor NN of the RL algorithm, an APF, and Backstepping control technology. The action NNs are used to estimate the input signals of the desired control and the undesired effects due to disturbance from the APF, and the expected control performance is then obtained by minimizing the output of the critic NN. The computational burden incurred by the NNs is significantly reduced by reducing the number of parameters that need to be learned by NNs. Lyapunov stability theory is used to guarantee that all signals in this closed-loop system are ultimately uniformly bounded to ensure its stability. The results of simulations of a swarm of spacecraft demonstrated the effectiveness of the proposed control strategy.

Practical phase retrieval using double deep image priors

Phase retrieval (PR) consists of recovering complex-valued objects from their oversampled Fourier magnitudes and takes a central place in scientific imaging. A critical issue around PR is the typical nonconvexity in natural formulations and the associated bad local minimizers. The issue is exacerbated when the support of the object is not precisely known and hence must be overspecified in practice. Practical methods for PR hence involve convolved algorithms, e.g., multiple cycles of hybrid input-output (HIO) + error reduction (ER), to avoid the bad local minimizers and attain reasonable speed, and heuristics to refine the support of the object, e.g., the famous shrinkwrap trick. Overall, the convolved algorithms and the support-refinement heuristics induce multiple algorithm hyperparameters, to which the recovery quality is often sensitive. In this work, we propose a novel PR method by parameterizing the object as the output of a learnable neural network, i.e., deep image prior (DIP). For complex-valued objects in PR, we can flexibly parametrize the magnitude and phase, or the real and imaginary parts separately by two DIPs. We show that this simple idea, free from multi-hyperparameter tuning and support-refinement heuristics, can obtain superior performance than gold-standard PR methods. For the session: Computational Imaging using Fourier Ptychography and Phase Retrieval.

Dynamics evolution prediction from time series data with recurrent neural networks in a complex system

Time series data can be used to predict the dynamical behaviors without knowing equation model of a system. In this study, long-short term memory (LSTM) neural network is implemented to construct a complex dynamical system from data series. The network is trained through minimizing the loss function to obtain the optimal weight matrices of LSTM cells. We find that the LSTM network can well ”learn” the information of the complex system. The data series generated from periodic orbits of a nonlinear system can be exactly predicted by comparing the output of neural networks with the real complex system. For the chaotic data series, the time evolution of trajectories could exactly match the actual system in the short-term data. Moreover, the long-term ergodic behavior of the complex system remains in our prediction, although such chaotic data series are quite sensitive to the initial conditions and the ensuing increase in uncertainty.

Neural Network Approximation of Refinable Functions

In the desire to quantify the success of neural networks in deep learning and other applications, there is a great interest in understanding which functions are efficiently approximated by the outputs of neural networks. By now, there exists a variety of results which show that a wide range of functions can be approximated with sometimes surprising accuracy by these outputs. For example, it is known that the set of functions that can be approximated with exponential accuracy (in terms of the number of parameters used) includes, on one hand, very smooth functions such as polynomials and analytic functions and, on the other hand, very rough functions such as the Weierstrass function, which is nowhere differentiable. In this paper, we add to the latter class of rough functions by showing that it also includes refinable functions. Namely, we show that refinable functions are approximated by the outputs of deep ReLU neural networks with a fixed width and increasing depth with accuracy exponential in terms of their number of parameters. Our results apply to functions used in the standard construction of wavelets as well as to functions constructed via subdivision algorithms in Computer Aided Geometric Design.

Normalization effects on deep neural networks

We study the effect of normalization on the layers of deep neural networks of feed-forward type. A given layer $ i $ with $ N_{i} $ hidden units is allowed to be normalized by $ 1/N_{i}^{\gamma_{i}} $ with $ \gamma_{i}\in[1/2,1] $ and we study the effect of the choice of the $ \gamma_{i} $ on the statistical behavior of the neural network's output (such as variance) as well as on the test accuracy on the MNIST data set. We find that in terms of variance of the neural network's output and test accuracy the best choice is to choose the $ \gamma_{i} $'s to be equal to one, which is the mean-field scaling. We also find that this is particularly true for the outer layer, in that the neural network's behavior is more sensitive in the scaling of the outer layer as opposed to the scaling of the inner layers. The mechanism for the mathematical analysis is an asymptotic expansion for the neural network's output. An important practical consequence of the analysis is that it provides a systematic and mathematically informed way to choose the learning rate hyperparameters. Such a choice guarantees that the neural network behaves in a statistically robust way as the $ N_i $ grow to infinity.

Neural State-Space Models: Empirical Evaluation of Uncertainty Quantification

Effective quantification of uncertainty is an essential and still missing step towards a greater adoption of deep-learning approaches in different applications, including mission-critical ones. In particular, investigations on the predictive uncertainty of deep-learning models describing non-linear dynamical systems are very limited to date. This paper is aimed at filling this gap and presents preliminary results on uncertainty quantification for system identification with neural state-space models. We frame the learning problem in a Bayesian probabilistic setting and obtain posterior distributions for the neural network's weights and outputs through approximate inference techniques. Based on the posterior, we construct credible intervals on the outputs and define a surprise index which can effectively diagnose usage of the model in a potentially dangerous out-of-distribution regime, where predictions cannot be trusted.

Model-Based Reinforcement Learning and Neural-Network-Based Policy Compression for Spacecraft Rendezvous on Resource-Constrained Embedded Systems

Autonomous spacecraft rendezvous is very challenging in increasingly complex space missions. In this article, we present our approach model-based reinforcement learning for spacecraft rendezvous guidance (MBRL4SRG). We build a Markov decision process model based on the Clohessy-Wiltshire equation of spacecraft dynamics and use dynamic programming to solve it and generate the decision table as the optimal agent policy. Since the onboard computing system of spacecraft is resource constrained in terms of both memory size and processing speed, we train a neural network (NN) as a compact and efficient function approximation to the tabular representation of the decision table. The NN outputs are formally verified using the verification tool ReluVal, and the verification results show that the robustness of the NN is maintained. Experimental results indicate that MBRL4SRG achieves lower computational overhead than the conventional proportional–integral–derivative algorithm and has higher trustworthiness and better computational efficiency during training than the model-free reinforcement learning algorithms.

Learning for Precision Motion of an Interventional X-ray System: Add-on Physics-Guided Neural Network Feedforward Control

Tracking performance of physical-model-based feedforward control for interventional X-ray systems is limited by hard-to-model parasitic nonlinear dynamics, such as cable forces and nonlinear friction. In this paper, these nonlinear dynamics are compensated using a physics-guided neural network (PGNN), consisting of a physical model, embedding prior knowledge of the dynamics, in parallel with a neural network to learn hard-to-model dynamics. To ensure that the neural network learns only unmodelled Effects, the neural network output in the subspace spanned by the physical model is regularized via an orthogonal projection-based approach, resulting in complementary physical model and neural network contributions. The PGNN feedforward controller reduces the tracking error of an interventional X-ray system by a factor of 5 compared to an optimally tuned physical model, successfully compensating the unmodeled parasitic dynamics.

Activation Functions Effect on Fractal Coding Using Neural Networks

Activation functions play an essential role in converting the output of the artificial neural network into nonlinear results, since without this nonlinearity, the results of the network will be less accurate. Nonlinearity is the mission of all nonlinear functions, except for polynomials. The activation function must be differentiable for backpropagation learning. This study’s objective is to determine the best activation functions for the approximation of each fractal image. Different results have been attained using Matlab and Visual Basic programs, which indicate that the bounded function is more helpful than other functions. The non-linearity of the activation function is important when using neural networks for coding fractal images because the coefficients of the Iterated Function System are different according to the different types of fractals. The most commonly chosen activation function is the sigmoidal function, which produces a positive value. Other functions, such as tansh or arctan, whose values can be positive or negative depending on the network input, tend to train neural networks faster. The coding speed of the fractal image is different depending on the appropriate activation function chosen for each fractal shape. In this paper, we have provided the appropriate activation functions for each type of system of iterated functions that help the network to identify the transactions of the system.

Explaining the Outputs of Convolutional Neural Network - Recurrent Neural Network (CNN-RNN) based Apparent Personality Detection Models using the Class Activation Maps

This study aims to use the Class Activation Map (CAM) visualisation technique to understand the outputs of apparent personality detection models based on a combination of Convolutional Neural Networks (CNN) and Recurrent Neural Networks (RNN). The ChaLearn Looking at People First Impression (CVPR'17) dataset is used for experimentation in this study. The dataset consists of short video clips labelled with the Big Five personality traits. Two deep learning models were designed to predict apparent personality with VGG19 and ResNet152 as base models. Then the models were trained using the raw frames extracted from the videos. The highest accurate models from each architecture were chosen for feature visualisation. The test dataset of the CVPR'17 dataset is used for feature visualisation. To identify the feature's contribution to the network's output, the CAM XAI technique was applied to the test dataset and calculated the heatmap. Next, the bitwise intersection between the heatmap and background removed frames was measured to identify how much features from the human body (including facial and non-facial data) affected the network output. The findings revealed that nearly 35%-40% of human data contributed to the output of both models. Additionally, after analysing the heatmap with high-intensity pixels, the ResNet152 model was found to identify more human-related data than the VGG19 model, achieving scores of 46%-51%. The two models have different behaviour in identifying the key features which influence the output of the models based on the input.