Weights Of Neural Network Research Articles

Short-term electric load forecasting using particle swarm optimization-based convolutional neural network

Short-term electric load forecasting is essential for the operation of power systems and the power market, including economic dispatch, unit commitment, peak load shaving, load management and bidding strategy development. This paper presents a novel method that uses a hybrid convolutional neural network (CNN) that is cascaded with a fully-connected network, to conduct 24h-ahead electric load forecasting in power systems. Spearman’s rank-order correlation, which serves as input size information for the CNN, was used to measure the cross-correlation between two sets of observations (chronological loads) of time shifts or lags. Both max and average pooling operations were utilized to extract data features in concatenation while spatial and conventional dropouts were employed to avoid overfitting in the hybrid CNN. The proposed method used two loops to yield the optimal CNN. The outer loop optimizes the structure and hyperparameters (such as kernel size) of the hybrid CNN by a proposed method of enhanced elite-based particle swarm optimization (EEPSO) while the inner loop optimizes the parameters (such as values of a kernel) and weights in neural networks using the Adam optimizer. The EEPSO is based on the mean search and chaotic descending inertia weight. The proposed method was used to study the 2018 and 2019 hourly load data in Taiwan Power Company. Simulation results show that the proposed method outperforms the well-known ARIMA, Radial Basis Function Network, Support Vector Regression, Long Short-term Memory and their hybrid variants.

Approximation of smooth functionals using deep ReLU networks

In recent years, deep neural networks have been employed to approximate nonlinear continuous functionals F defined on Lp([−1,1]s) for 1≤p≤∞. However, the existing theoretical analysis in the literature either is unsatisfactory due to the poor approximation results, or does not apply to the rectified linear unit (ReLU) activation function. This paper aims to investigate the approximation power of functional deep ReLU networks in two settings: F is continuous with restrictions on the modulus of continuity, and F has higher order Fréchet derivatives. A novel functional network structure is proposed to extract features of higher order smoothness harbored by the target functional F. Quantitative rates of approximation in terms of the depth, width and total number of weights of neural networks are derived for both settings. We give logarithmic rates when measuring the approximation error on the unit ball of a Hölder space. In addition, we establish nearly polynomial rates (i.e., rates of the form exp−a(logM)b with a>0,0<b<1) when measuring the approximation error on a space of analytic functions.

Enhanced regularization for on-chip training using analog and temporary memory weights

In-memory computing techniques are used to accelerate artificial neural network (ANN) training and inference tasks. Memory technology and architectural innovations allow efficient matrix–vector multiplications, gradient calculations, and updates to network weights. However, on-chip learning for edge devices is quite challenging due to the frequent updates. Here, we propose using an analog and temporary on-chip memory (ATOM) cell with controllable retention timescales for implementing the weights of an on-chip training task. Measurement results for Read–Write timescales are presented for an ATOM cell fabricated in GlobalFoundries’ 45 nm RFSOI technology. The effect of limited retention and its variability is evaluated for training a fully connected neural network with a variable number of layers for the MNIST hand-written digit recognition task. Our studies show that weight decay due to temporary memory can have benefits equivalent to regularization, achieving a ∼33% reduction in the validation error (from 3.6% to 2.4%). We also show that the controllability of the decay timescale can be advantageous in achieving a further ∼26% reduction in the validation error. This strongly suggests the utility of temporary memory during learning before on-chip non-volatile memories can take over for the storage and inference tasks using the neural network weights. We thus propose an algorithm-circuit codesign in the form of temporary analog memory for high-performing on-chip learning of ANNs.

Implementation of Quantum Optimization Algorithm

Quantum computing is a quickly growing research field. This article introduces the basic concepts of quantum computing, recent developments in quantum searching Quantum computers are computing devices that can theoretically have computed power that is many orders of magnitude greater than that of conventional computers. The quantum bit, or qubit, which is the quantum state of electrons in an atom, is the fundamental data unit in a quantum computer. Qubits have the potential to exist in several superposed states at once, making them capable of carrying much more data than do standard two-state bits. The mathematical basis of the proportionality of qubit states is like that of the input weights of neural networks. The technology of quantum computers has made some progress, but much more work needs to be done in terms of research and development before it can be used in everyday life. There are also views that suggest this outcome is impossible. A recent field of study known as quantum computing (QC) combines components of computing, physics, and mathematics. Interest in quantum computing is rising among academics, technologists, and businesspeople. For the past ten years, it has given researchers in the scientific, technological, and industrial disciplines a place to conduct their work. The fundamentals of QC have been developed using notions from quantum physics. The parallel processing functionality in QC has simplified the currently employed algorithms. Numerous optimization-related challenges and issues were solved with the aid of this function. Intelligent computational techniques that were inspired by quantum mechanics have been applied in a variety of fields. A qubit's ability to be in superposition is one of its characteristics that distinguishes it from a conventional bit. Superposition is one of the fundamental principles. Entanglement is another one of quantum physics' counterintuitive occurrences. When each particle's quantum state cannot be characterized separately from the quantum state of the other particle(s), a pair or group of particles are said to be entangled

Application of BP Neural Network Improved by Fireworks Algorithm on Suspender Damage Prediction of Long-Span Half-Through Arch Bridge

In recent decades, with the large-scale construction and rapid development of half-through arch bridges, as well as the increase of bridge service time, the suspender damage of arch bridge has become increasingly prominent. Therefore, real-time monitoring and regular detection of the health of arch bridge suspenders and timely detection and accurate judgment of the damage location and extent of suspenders are of great engineering significance for evaluating the reliability and residual life of arch bridge structures. By analyzing the main difficulties and existing problems of suspender damage identification, this paper takes the change rate of modal curvature as the damage index, introduces fireworks algorithm into the neural network model, optimizes the optimization process of neural network weight and threshold, and proposes a prediction model based on improved BP neural network by fireworks algorithm. According to the measured data of the damage degree of a long-span arch bridge in daily monitoring and on-site inspection, the proposed prediction method is applied to verify the effectiveness and accuracy in engineering health detection. On this basis, the improved BP neural network by fireworks algorithm is used to predict the suspender damage of a certain long-span half-through arch bridge, which provides an important basis for the actual bridge safety assessment.

BRoCoM: A Bayesian Framework for Robust Computing on Memristor Crossbar

Memristor crossbar arrays are considered to be a promising platform for neuromorphic computing. To deploy a trained neural network model on memristor crossbars, memristors need to be programmed to the corresponding weight values. In fact, due to device based process variation and noise, deviations of the stored weights from the trained weights are inevitable, thereby causing the degradation of the actual inference performance. This paper proposes a unified Bayesian inference based framework, BRoCoM, which connects device nonidealities and algorithmic training together for robust computing on memristor crossbars. BRoCoM is able to incorporate different levels of nonidealities into prior weight distribution, and transform robustness optimization to Bayesian neural network training, the weights of neural networks are optimized to accommodate uncertainties and minimize inference degradation. Experimental results confirm the capability of the proposed BRoCoM to achieve stable inference performance while tolerating the non-ideal effects of process variation and noise.

Neural-Network-Based Secure State Estimation Under Energy-Constrained Denial-of-Service Attacks: An Encoding-Decoding Scheme

This paper is concerned with the secure state estimation issue for a class of networked nonlinear systems under energy-constrained denial-of-service (EC-DoS) cyber-attacks and encoding-decoding scheme (EDS). The information transmissions between sensors and the estimator are executed via a bandwidth-limited communication network, on which the EDS is deployed to convert transmitted signals into finite-length codewords for the purpose of improving transmission efficiency. The EC-DoS attacks, whose intention is to jeopardize the network-based signal transmissions by overloading the communication resource, are assumed to occur in an intermittent way with bounded occurrence frequency/durations owing to the inherent energy constraints on the attackers. Considering the worst case of such EC-DoS attacks, a neural-network (NN)-based state estimator is constructed to generate the desired state estimates for the underlying networked nonlinear system. By employing the Lyapunov stability theory, the estimation error dynamics of the system state and the neural-network weight are jointly analyzed within a unified framework. Subsequently, sufficient conditions are obtained for the existence of the desired NN-based state estimator, and then both the desired estimator gain matrix and the NN tuning parameters are characterized. Finally, the validity of our estimation approach is confirmed by an example.

Deep Learning for Continuous-Time Leader Synchronization in Graphical Games Using Sampling and Deep Neural Networks

Abstract We propose a novel deep learning-based approach for the problem of continuous-time leader synchronization in graphical games on large networks. The problem setup is to deploy a distributed and coordinated swarm to track the trajectory of a leader while minimizing local neighborhood tracking error and control costs for each agent. The goal of our work is to develop optimal control policies for continuous-time leader synchronization in graphical games using deep neural networks. We discretize the agents model using sampling to facilitate the modification of gradient descent methods for learning optimal control policies. The distributed swarm is deployed for a certain amount of time while keeping the control input of each agent constant during each sampling period. After collecting state and input data at each sampling time during one iteration, we update the weights of a deep neural network for each agent using collected data to minimize a loss function that characterizes the agents' local neighborhood tracking error and the control cost. A modified gradient descent method is presented to overcome existing limitations. The performance of the proposed method is compared with two reinforcement learning-based methods in terms of robustness to initial neural network weights and initial local neighborhood tracking errors and the scalability to networks with a large number of agents. Our approach has been shown to achieve superior performance compared with the other two methods.

Development of the solution search method using the population algorithm of global search optimization

The research objects are decision making support systems. subject research is a decision making process in management tasks using bio-inspired algorithms. A method of finding solutions using the population algorithm of global search optimization is proposed. Joint use of invasive weed algorithm is proposed, genetic algorithm and evolving artificial neural networks are improved. The method has the following sequence of actions: ‒ an input of initial data; ‒ processing of initial data taking into account the degree of uncertainty; ‒ formation of the optimization vector; ‒ creation of descendant vectors; ‒ ordering of vectors in descending order; ‒ reducing the dimensionality of the feature space; ‒ teaching knowledge bases. The peculiarity of the proposed method lies in the placement of agents-weeds, taking into account the uncertainty of the initial data, improved procedures for reducing the space of signs about the analysis object state. Training of synaptic weights of an artificial neural network, type and parameters of the membership function and the architecture of individual elements, and the architecture of an artificial neural network as a whole is carried out. The proposed method was simulated in the MathСad 14 software environment. The task to be solved during the simulation was to determine the route of the ships in the operational zones of the Black and Azov seas in the conditions of hybrid actions of the enemy. The use of the method makes it possible to increase the efficiency of data processing at the level of 21–27 % due to the use of additional improved procedures. The proposed method should be used to solve the problems of evaluating complex and dynamic processes in the interests of solving national security problems

Development of a method for the search of solutions in the sphere of national security using bio-inspired algorithms

The object of research are decision support systems. The subject of research is the decision-making process in management problems using bio-inspired algorithms. A method for the search of solutions in the field of national security using bio-inspired algorithms is proposed. The proposed method is based on a combination of an artificial bat algorithm and evolving artificial neural networks. The method has the following sequence of actions: ‒ input of initial data; ‒ processing of initial data taking into account the degree of uncertainty; ‒ numbering of bat agents (BA); ‒ placement of bat agents taking into account the degree of uncertainty about the state of the analysis object in the search space; ‒ setting the initial BA speed and the echolocation frequency of each BA; ‒ starting a local search; ‒ launching a global search; ‒ training knowledge bases of bat agents. The originality of the proposed method consists in the arrangement of bat agents taking into account the uncertainty of initial data, improved global and local search procedures taking into account the noise degree of data about the state of the analysis object. Another feature of the proposed method is the use of an improved procedure for training bat agents. The training procedure consists in learning the synaptic weights of an artificial neural network, the type and parameters of the membership function, the architecture of individual elements and the architecture of the artificial neural network as a whole. The method makes it possible to increase the efficiency of data processing at the level of 13–21 % due to the use of additional improved procedures. The proposed method should be used to solve the problems of evaluating complex and dynamic processes in the interests of solving national security problems

Transfer Learning for the Visual Arts: The Multi-modal Retrieval of Iconclass Codes

Iconclass is an iconographic thesaurus, which is widely used in the digital heritage domain to describe subjects depicted in artworks. Each subject is assigned a unique descriptive code, which has a corresponding textual definition. The assignment of Iconclass codes is a challenging task for computational systems, due to the large number of available labels in comparison to the limited amount of training data available. Transfer learning has become a common strategy to overcome such a data shortage. In deep learning, transfer learning consists in fine-tuning the weights of a deep neural network for a downstream task. In this work, we present a deep retrieval framework, which can be fully fine-tuned for the task under consideration. Our work is based on a recent approach to this task, which already yielded state-of-the-art performance, although it could not be fully fine-tuned yet. This approach exploits the multi-linguality and multi-modality that is inherent to digital heritage data. Our framework jointly processes multiple input modalities, namely, textual and visual features. We extract the textual features from the artwork titles in multiple languages, whereas the visual features are derived from photographic reproductions of the artworks. The definitions of the Iconclass codes, containing useful textual information, are used as target labels instead of the codes themselves. As our main contribution, we demonstrate that our approach outperforms the state-of-the-art by a large margin. In addition, our approach is superior to the M 3 P feature extractor and outperforms the multi-lingual CLIP in most experiments due to the better quality of the visual features. Our out-of-domain and zero-shot experiments show poor results and demonstrate that the Iconclass retrieval remains a challenging task. We make our source code and models publicly available to support heritage institutions in the further enrichment of their digital collections.

Hybrid classification framework for chronic kidney disease prediction model

ABSTRACT ‘Chronic kidney disease (CKD) – or chronic renal failure (CRF) is a term that encompasses all degrees of decreased kidney function, from damaged–at risk through mild, moderate, and severe chronic kidney failure’. As a risky factor, the disease has steadily turned out to be a major cause of death and morbidity. Accordingly, ultrasound (US) is significant in enhancing the rates of early recognition of CKD. Here, a new CKD detection model is introduced that includes ‘(1) Pre-processing (2) segmentation (3) Feature extraction, and (4) Classification’. Improved Gaussian filtering is used for pre-processing, and watershed-based segmentation is carried out. Additionally, features like the ROI, mean intensity, and the projected Local Vector Pattern (LVP) are retrieved. The ‘Optimized Neural Network (NN) and Long Short-Term Memory (LSTM)’ are then provided the output of the features. Additionally, using Self Updated Cat Swarm Optimization, the weights of NN are adjusted in order to increase the classifier's prediction accuracy (SU-CSO). The categorized output is then calculated by averaging the results from the improved NN and LSTM. Lastly, it is demonstrated that the proposed strategy is superior to other options.

The cascade backstepping controller design for supercavitating vehicle based on RBFNN observer

Aiming at the model uncertainty in design process of longitudinal motion control system of supercavitating vehicle, a backstepping controller is designed by combining with Radial Basis Function neural network (RBFNN). Considering that the environment of the supercavitating vehicle is complex during its traversal, it is not possible to measure all states directly. The calculation of planing force with nonlinear characteristics is related to vertical velocity, therefore, the accuracy of vertical velocity has impact on the value of planing force directly. A state observer is designed to estimate vertical velocity. In this paper, based on the backstepping method, the uncertain part of matrix coefficient in cascade control model of supercavitating vehicle is approximated by RBFNN. The output of the observer approximates the state variables, then, a depth tracking control law of the vehicle is obtained according to the observed values and the weight of neural network is calculated by Lyapunov function. Finally, it is proved that the control law can ensure the uniform ultimate boundedness of the closed-loop system.

Cyber-security attack prediction using cognitive spectral clustering technique based on simulated annealing search

Data protection and security is a big challenging portion in a modern technical world against the cyber-attacks like; ransomware, man-in-the-middle, DDoS, etc. In order to overcome this scenario, there are lot of artificial intelligence framework have been introduced to detect and classify the cyber-attacks. In particular, neural networks, with their solid speculation execution ability, are capable to address an extensive variety of cyber-attacks. This article frames the training and testing of a neural network group such a way to deal with detection of cyber-attack using cognitive spectral clustering technique based on simulated annealing search method. The optimization of individual networks can be made by using adaptive memetic algorithm with simulated annealing search. It is used to enhance the neural network weights and hidden neurons respectively. This algorithm is a combination of both local and global search enhancement method and used to get rid of the premature convergence, and used to achieve the adaptive search output. The testing outcome of the proposed framework shows a better result 99.5% of overall accuracy, and effectively adaptive in terms of detecting the cyber-attacks.

Sepsis Prediction by Using a Hybrid Metaheuristic Algorithm: A Novel Approach for Optimizing Deep Neural Networks.

The early diagnosis of sepsis reduces the risk of the patient's death. Gradient-based algorithms are applied to the neural network models used in the estimation of sepsis in the literature. However, these algorithms become stuck at the local minimum in solution space. In recent years, swarm intelligence and an evolutionary approach have shown proper results. In this study, a novel hybrid metaheuristic algorithm was proposed for optimization with regard to the weights of the deep neural network and applied for the early diagnosis of sepsis. The proposed algorithm aims to reach the global minimum with a local search strategy capable of exploring and exploiting particles in Particle Swarm Optimization (PSO) and using the mental search operator of the Human Mental Search algorithm (HMS). The benchmark functions utilized to compare the performance of HMS, PSO, and HMS-PSO revealed that the proposed approach is more reliable, durable, and adjustable than other applied algorithms. HMS-PSO is integrated with a deep neural network (HMS-PSO-DNN). The study focused on predicting sepsis with HMS-PSO-DNN, utilizing a dataset of 640 patients aged 18 to 60. The HMS-PSO-DNN model gave a better mean squared error (MSE) result than other algorithms in terms of accuracy, robustness, and performance. We obtained the MSE value of 0.22 with 30 independent runs.

Neural network models for atmospheric residue desulfurization using real plant data with novel training strategies

Oil refining industry has been using processes to convert heavy oil into high-value materials due to environmental regulations and decreasing demand for heavy oil. Atmospheric Residue Desulfurization (ARDS) process is important as a pre-treatment process for removing impurities from heavy oil. In this paper, neural network (NN) models are developed to predict the impurity removal amounts in the ARDS process using real plant data. To improve the reliability and reflect the chemical reaction characteristics, three training strategies are proposed: (1) restricting NN weights to positive values, (2) penalizing imbalances for two identical reactor trains, and (3) updating NNs with the latest 4-week data in a receding manner. The trained models satisfy the chemical consistency and provide well-balanced predictions. The model with a pre-defined aging factor structure can learn the catalyst aging status. The maximum percent error in the predicted values is smaller than or equal to 9.20%.

Decentralized Adaptive Quantized Dynamic Surface Control for a Class of Flexible Hypersonic Flight Vehicles with Input Quantization

A control strategy for a certain class of hypersonic flight aircraft dynamic models with unknown parameters is proposed in this article. The strategy is adaptive dynamic surface input quantization control. To address the issues in conventional inversion control, a first-order low-pass filter and an adaptive parameter minimum learning law are introduced in the control system design process. This method has the following features: (1) it solves the problem of repeated differentiation of the virtual control law in the conventional back-stepping method, greatly simplifying the control law structure; (2) by using the norm of the neural network weight vector as the adaptive adjustment parameter instead of updating each element online, the number of adaptive adjustment parameters is significantly reduced, improving the execution efficiency of the controller; (3) the introduced hysteresis quantizer overcomes the disadvantage of the quantization accuracy deterioration when the input value is too low in the logarithm quantizer, improving the accuracy of the quantizer. Stability analysis has shown that all signals in the closed-loop system are semi-globally uniformly bounded, and simulation results have verified the effectiveness of the proposed adaptive quantized control scheme.

Optimal Adaptive Control of Uncertain Nonlinear Continuous-Time Systems With Input and State Delays.

In this article, an actor-critic neural network (NN)-based online optimal adaptive regulation of a class of nonlinear continuous-time systems with known state and input delays and uncertain system dynamics is introduced. The temporal difference error (TDE), which is dependent upon state and input delays, is derived using actual and estimated value function and via integral reinforcement learning. The NN weights of the critic are tuned at every sampling instant as a function of the instantaneous integral TDE. A novel identifier, which is introduced to estimate the control coefficient matrices, is utilized to obtain the estimated control policy. The boundedness of the state vector, critic NN weights, identification error, and NN identifier weights are shown through the Lyapunov analysis. Simulation results are provided to illustrate the effectiveness of the proposed approach.

Booth Encoding-Based Energy Efficient Multipliers for Deep Learning Systems

Artificial intelligence on edge is a growing research field. In this paper, we propose a novel re-encoding scheme for reducing the size of the weights of deep neural networks (DNNs). The proposed re-encoding scheme exploits the Booth encoding scheme and the power-of-two (PO2) quantization to allow for very low energy computations during the inference of the neural networks with minimal loss in classification accuracy. We demonstrate the advantages of the proposed re-encoding scheme by computing a convolutional neural network (CNN) and a linear neural network on the proposed Extended Exact Multiplier and the proposed PO2 Multiplier. Our proposed PO2 quantization and re-encoding method reduce the model size for the CNN by 30.77% and the model size of the linear neural network by 49.86%. Furthermore, our multipliers reduce the inference energy for CNN by 50.6% and for the linear neural network by 90.1%. The PO2 Multiplier is proposed for the sensor-end computation of the linear neural network with a 77.32% reduction in the area relative to an exact Booth multiplier and it reduces the inference energy consumption of the linear neural network by 93.2% when compared to the unmodified exact multiplier. Our proposed scheme can be used to improve the energy consumption during inference for most Booth multipliers with minor modifications to the re-encoding signal arrangements. We also demonstrate that the proposed re-encoding scheme paired with the proposed multipliers outperforms all the existing designs in terms of resource utilization with a minimal impact on the inference accuracy of the neural networks.

Optimal prescribed performance tracking control of nonlinear motor driven systems via adaptive dynamic programming

AbstractAlthough optimal regulation problem has been well studied, resolving optimal tracking control via adaptive dynamic programming (ADP) has not been completely resolved, particularly for nonlinear uncertain systems. In this paper, an online adaptive learning method is developed to realize the optimal tracking control design for nonlinear motor driven systems (NMDSs), which adopts the concept of ADP, unknown system dynamic estimator (USDE), and prescribed performance function (PPF). To this end, the USDE in a simple form is first proposed to address the NMDSs with bounded disturbances. Then, based on the estimated unknown dynamics, we define an optimal cost function and derive the optimal tracking control. The derived optimal tracking control is divided into two parts, that is, steady‐state control and optimal feedback control. The steady‐state control can be obtained with the tracking commands directly. The optimal feedback control can be obtained via the concept of ADP based on the PPF; this contributes to improving the convergence of critic neural network (CNN) weights and tracking accuracy of NMDSs. Simulations are provided to display the feasibility of the designed control method.