Error Backpropagation Method Research Articles

Computational approach based on use of multilayer neural networks in classification of rail line states

The article considers the issue of classification of the states of rail lines using multilayer neural networks. It is shown that with significant external disturbing influences, the most promising direction is the retraining of classification models using neural networks. The representativeness of the sample is proposed to be realized by applying the scaling of informative features. It is proposed to minimize classification errors by gradient optimization methods. In the work during classification of rail lines states by neural networks, at the first stage, artificial classes of states are formed. At the second stage, the classification of states by the trained neural network is considered, classification errors are identified. On the basis of these errors the network is retrained until acceptable results are obtained. The article presents neuron learning algorithms: Hebb’s algorithm, error backpropagation method (for training a multilayer neural network), Konohen’s, Hopfield’s, Hamming’s algorithm and network. In the presented revision of the classification model, the correction is carried out according to the Widrow-Hoff algorithm.

Analysis of the application of neural network technologies in predicting equipment failure in production

The relevance. The article considers the solution to the problem of analyzing and predicting changes in the dynamics of measuring instrument readings and identifying patterns describing the course and possible periods of failures of technological process equipment by constructing a predictive time series model. A critically important infrastructure facility of a thermal power plant was selected as the studied one – a water pump for pumping process water into neutralizer tanks (the first stage of chemical water purification) located in the chemical workshop of the Krasnoyarsk CHP-3. The general methodology of model construction, requirements for input data arrays, preprocessing algorithms for the formation of samples used for training and testing models are described. To build a predictive model, the work used elements such as recurrent layers of long-term short- term memory LSTM, MinMaxScaler regularization, Dropout layer to reduce the effect of retraining, the ‘Adam’ optimizer using error back propagation methods and optimization using the steepest descent method using conjugate gradients, the learning error function is the average root of the error (Root Mean Squared Error, RMSE), as well as plot_predictions visualization. The algorithms are implemented in Python, in the Anaconda Jupyter Nootebook core. Based on the results of testing the training, the model showed sufficient performance of this method, since the time spent on creating the model and obtaining a forecast plays an important role when launching the model in real production conditions. In conclusion, recommendations are given to improve the accuracy of the forecast, the developed interface of the production system is presented and the directions of future research in this area are outlined.

Sparse-firing regularization methods for spiking neural networks with time-to-first-spike coding

The training of multilayer spiking neural networks (SNNs) using the error backpropagation algorithm has made significant progress in recent years. Among the various training schemes, the error backpropagation method that directly uses the firing time of neurons has attracted considerable attention because it can realize ideal temporal coding. This method uses time-to-first-spike (TTFS) coding, in which each neuron fires at most once, and this restriction on the number of firings enables information to be processed at a very low firing frequency. This low firing frequency increases the energy efficiency of information processing in SNNs. However, only an upper limit has been provided for TTFS-coded SNNs, and the information-processing capability of SNNs at lower firing frequencies has not been fully investigated. In this paper, we propose two spike-timing-based sparse-firing (SSR) regularization methods to further reduce the firing frequency of TTFS-coded SNNs. Both methods are characterized by the fact that they only require information about the firing timing and associated weights. The effects of these regularization methods were investigated on the MNIST, Fashion-MNIST, and CIFAR-10 datasets using multilayer perceptron networks and convolutional neural network structures.

Online Adaptive Neurochaotic Fuzzy Controller Design to Reduce the Seismic Response of Buildings Equipped with Active Tuned Mass Damper System

This paper presents a novel adaptive neurochaotic fuzzy control system based on type-2 fuzzy systems to reduce seismic responses in multistory structures with active tuned mass dampers under near-field and far-field earthquakes. In this proposed control system, the whole parameters of the plant are assumed to be completely unknown, the structural model is estimated using a multilayer perceptron neural network, and the system’s Jacobian is extracted. The online estimation model is used, and the controller parameters are adaptively trained using the extended Kalman filter and error back-propagation method. Subsequently, the control force is applied to the active tuned mass damper, and the control objectives are met. The adaptive controller does not require initial settings, and a fractional-order proportional-integral-derivative controller is added to maximize stability and robustness against seismic vibration. A simple adaptive controller optimized by a particle swarm is also presented as an innovation. Comparing the performance of the improved simple adaptive controller and adaptive neurochaotic fuzzy controller, the proposed controllers appear more efficient and accurate. However, the superiority of the novel adaptive neurochaotic fuzzy over the improved simple adaptive controller in reducing maximum displacement, acceleration, drift, and base shear while maintaining acceptable performance under parametric uncertainties and seismic conditions is substantial.

Intelligent Forecasting of Electricity Consumption in Managing Energy Enterprises in Order to Carry out Energy-Saving Measures

The concept of "Digital Transformation 2030", which defines the national goals and strategic objectives of the development of the Russian Federation for the period up to 2030, specifies specialized goals and objectives that are an important message for the introduction of intelligent information management technologies in the electric power industry. The main challenges for the transition to digital transformation are the increase in the rate of growth of tariffs for the end consumer, the increasing wear and tear of the network infrastructure, the presence of excessive network construction and the increase in requirements for the quality of energy consumption. The determining factor in the possibility of developing an effective energy policy is the forecasting of electricity consumption using artificial intelligence methods. One of the methods for implementing the above is the development of an artificial neural network (ANN) to obtain an early forecast of the amount of required (consumed) electricity. The obtained predictive values open up the possibility not only to build a competent energy policy by increasing the energy efficiency of an energy company, but also to carry out specialized energy-saving measures in order to optimize the organization’s budget. The solution to this problem is presented in the form of an artificial neural network (ANN) of the second generation. The main advantages of this ANN are its versatility, fast and accurate learning, as well as the absence of the need for a large amount of initial da-ta for a qualitative forecast. The ANN itself is based on the classical neuron and the error back-propagation method with their further modification. The coefficients of learning rate and sensitivity have been added to the error backpropagation method, and the coefficient of response to anomalies in the time series has been introduced into the neuron. This made it possible to significantly improve the learning rate of the artificial neural network and improve the accuracy of predictive results. The results presented by this study can be taken as a guideline for energy companies when making decisions within the framework of energy policy, including when carrying out energy saving measures, which will be especially useful in the current economic realities.

Artifical Neural Network for Forecasting Electricity Consumption in Energy Enterprises

The concept of "Digital Transformation 2030", which defines the national goals and strategic objectives of the development of the Russian Federation for the period up to 2030, specifies specialized goals and objectives. These goals and objectives are an important message for the introduction of intelligent information management systems based on digital technologies in the electric power industry in general and energy consumption in particular. The main challenges for the transition to digital transformation are: increasing the rate of tariff growth for the end consumer; increasing deterioration of the network infrastructure; the presence of excessive network construction; increasing requirements for the quality of energy consumption. Based on the above, this article is devoted to the development of a method for the formation of an intelligent con­trol system at an energy enterprise by obtaining an early forecast of the amount of electricity required. The predicted values will help not only to increase the energy efficiency of the company through the implementation of specialized energy saving measures, but also to reduce the financial costs of electricity. The solution to this problem is presented in the form of an intelligent neural network system. The main advantages of this artificial neural network are versatility, high-speed and accurate learning, as well as a small amount of training data. The artificial neural network itself is based on the freely distributed TensorFlow machine learning software library, and a modified error backpropagation method is used as training, the main difference of which is the addition of an artificial neural network learning rate increase factor. The results of the analysis of the effectiveness of the method showed that the proposed intelligent neural network system is more accurate (including relative to similar solutions): the average error does not exceed 3.08 % over the entire time of the experiment. This will allow companies to carry out energy saving measures, which will be especially useful in the current economic realities.

Cascaded Compressed Sensing Networks

For deep networks with complex nonlinearity, the structure analysis and design remain challenging. In the letter, we propose to understand and build deep networks as a cascade of compressed sensing. Each compressed sensing module consists of two layers, corresponding to the two data transforms involved in compressed sensing, namely the sparse transform over dictionaries and the measurement over random projection. The two transforms can be viewed as a process of feature generation and extraction, thus amenable to network construction. The proposed construction has two advantages. First, it can be analyzed with compressed sensing theory. Second, it enables networks to be learned layer by layer by back-propagating the target via compressed sensing, and empirically the layerwise learning method tends to outperform the conventional error back-propagation method in the presence of noise.

A system based on an artificial neural network of the second generation for decision support in especially significant situations

Currently, specialized system models implemented on the basis of decision support in exceptional (emergency) situations (states) using machine learning, artificial intelligence (including using neural networks) to reproduce, predict and prevent (or minimize the risk of consequences) in exceptional situations are useful and are becoming increasingly popular. Floods also fall under such exceptional situations and states, on the basis of which the task of early forecasting of an exceptional situation arises, using the example of rising water levels at stationary hydrological posts in order to prevent (or minimize the risk) the transition of the territory management system under consideration to an exceptional state (emergency situation). To solve this problem, a decision support system is proposed for early prediction of water rise levels, based on a neural network (intelligent) analysis of retrospective data (code of a stationary hydrological post / automatic station, date, water level at a stationary hydrological post / automatic station, atmospheric pressure, wind speed, snow cover thickness, amount of precipitation, time and air temperature) in order to calculate the values of water levels for 5 days in advance. The artificial neural network itself is based on the freely distributed TensorFlow machine learning software library, and a modified error backpropagation method is used as training, the main difference of which is the addition of an artificial neural network (ANN) learning rate increase factor. A mathematical model of the subject area is constructed in the form of a multidimensional space and an integrated multidimensional data model. A single integrated model describes the subject area quite fully and can be used as a single source of information, which provides system information support, defined as a set of distributed databases. All this makes it possible to improve the accuracy of forecast values of water levels at stationary hydrological posts and automatic stations. The main difficulty in early forecasting is a sharp rise in water levels over a short period of time (usually 1 day) for various reasons. Our neural network model takes this into account and allows us to give more accurate results. The results of the analysis of the effectiveness of the method showed that the proposed decision support system is more accurate: the average error does not exceed 55.68 cm for the entire flood period, the average error between the real and calculated values in the framework of forecasting for 5 days does not exceed 2.10 % compared to existing common methods/systems (8.36 %). This will give the necessary time to special services to carry out anti–flood measures to prepare for the protection of technical facilities of enterprises.

Stacked recurrent neural network based high precision pointing coupled control of the spacecraft and telescopes

In some space missions especially in the field of space gravitational wave detection, the telescope needs to point to a certain target through attitude movement and pointing control. In several mainstream gravitational wave detection missions, the detector usually consists of a cluster of three identical satellites, flying in a quasi-equilateral triangular formation with a big edge length, so every satellite needs two telescopes to point each other and constitute three giant Michelson-Type interferometers. Therefore, a satellite platform system with two telescopes is researched in this paper. This research helps to characterize the attitude motion of a telescope for space astronomical observation or space gravitational wave detection, provides new method on the telescope’s high-precision pointing control. For this purpose, we derive a satellite-telescope coupling attitude model, design the sliding mode controller for satellite and the stacked recurrent neural network adaptive controller for telescope. In the stacked recurrent neural network adaptive controller design, a sliding mode control technology is adopted. In addition, we propose a combinatorial optimization method for network weights in the stacked recurrent neural network training process, that is, the output layer is corrected by the adaptive law, and the correction of other layers adopt the error backpropagation method. Finally, a numerical simulation method verifies the effectiveness of the controller design.

Solving the initial value problem of ordinary differential equations by Lie group based neural network method.

To combine a feedforward neural network (FNN) and Lie group (symmetry) theory of differential equations (DEs), an alternative artificial NN approach is proposed to solve the initial value problems (IVPs) of ordinary DEs (ODEs). Introducing the Lie group expressions of the solution, the trial solution of ODEs is split into two parts. The first part is a solution of other ODEs with initial values of original IVP. This is easily solved using the Lie group and known symbolic or numerical methods without any network parameters (weights and biases). The second part consists of an FNN with adjustable parameters. This is trained using the error back propagation method by minimizing an error (loss) function and updating the parameters. The method significantly reduces the number of the trainable parameters and can more quickly and accurately learn the real solution, compared to the existing similar methods. The numerical method is applied to several cases, including physical oscillation problems. The results have been graphically represented, and some conclusions have been made.

Global fast terminal sliding mode based radial basis function neural network for accurate fault estimation in nonlinear systems

The problem of quick and accurate fault estimation in nonlinear systems is addressed in this article by combining the technique of radial basis function neural network (RBFNN) and global fast terminal sliding mode control (GFTSMC) concept. A new strategy to update the neural network weights, by using the global fast terminal sliding surface instead of conventional error back propagation method, is introduced to achieve real time, quick and accurate fault estimation which is critical for fault tolerant control system design. The combination of online learning ability of RBFNN, to approximate any nonlinear function, and finite time convergence property of GFTSMC ensures quick detection and accurate estimation of faults in real time. The effectiveness of the proposed strategy is demonstrated through simulations using a nonlinear model of a commercial aircraft and considering a wide range of sensors and actuators faults. The simulation results show that the proposed method is capable of quick and accurate online fault estimation in nonlinear systems and shows improved performance as compared to conventional RBFNN and other techniques existing in literature.

Model of a quantum neural network based on a multilevel activation function

This article proposes a model of a quantum neural network based on a multilevel activation function, which has high performance. This performance is achieved by combining such unique components as: the quantum weight vector, quantum activity values and the dot product operator. In addition, algorithms for learning weight parameters and quantum intervals are presented, in particular, the Levenberg-Marquardt algorithm is improved to improve the quality of the error back propagation method in the process of learning weight parameters, which will also reduce the likelihood of quantum errors.

Approaches to the choice of tools for constructing a methodology for learning to work with Big Data

The article actualizes the need for teaching students to work with Big Data technologies. Big Data is a promising and fundamental industry that requires a large number of qualified specialists in various fields. The aim of the work is to describe the concept of determining a set of hardware, software, algorithmic and methodological tools (taking into account the contingent of students and the capabilities of the educational institution) for building a methodology for teaching a discipline related to the study of Big Data processing methods. There are two main sectors of stakeholders who need specialists in the field of Big Data. A detailed comparative analysis of software solutions that support Big Data processing is carried out. The article describes the methodology for constructing a course for teaching students technologies for processing and analyzing Big Data. A plan for organizing a lecture course and laboratory practice with consideration of subtasks is proposed for students to perform during training. The composition and methodology of independent work of students in the discipline related to the study of Big Data, using a learning management system such as Moodle, are discussed. An example of implementing data processing by means of the RapidMiner Studio package using a multi-layer neural network training algorithm using the error back propagation method is presented.

Intelligent handwritten recognition using hybrid CNN architectures based-SVM classifier with dropout

Text recognition in Arabic handwritten scripts is an active research field. These recognition systems face numerous challenges, including enormous open data-bases, infinite variation in people’s handwriting, and freestyle. In this manuscript, Authors model deep learning architecture which can efficiently be utilized to recognizing Arabic handwritten scripts. This work explored a new model for both single font and multi-font type which concentrate on two common classifiers which are: Support Vector Machine (SVM) along with Convolutional Neural Network (CNN). Furthermore, authors protected the proposed model against the issue of over-fitting because of the strong performance of dropout technique. Both classification and feature extraction are done automatically. In the light of the error backpropagation method analysis, authors also have been proposed an innovative depth neural network training rule for maximum interval minimum classification error. In the meantime, max-margin minimum classification error (M3CE) and cross entropy are analyzed and hybridized to obtain better outcomes. Authors tested the proposed model on AHDB, AHCD, HACDB, and IFN/ENIT databases. The proposed model performance is compared with the accuracies of text recognition gained from state-of-the-art Arabic text recognition. The proposed model delivers favorable results.

РАЗРАБОТКА НЕЙРОННОЙ СЕТИ ДЛЯ РЕШЕНИЯ ЗАДАЧИ РАСПОЗНАВАНИЯ РЕЧИ

The article discusses a method for solving the problem of speech recognition on the example of recognizing individual words of a limited dictionary using a forward propagation neural network trained by the error back propagation method. The goal was to create a neural network model for recognizing the solution of individual words, analyze the training characteristics and behavior of the constructed neural network. Based on the input data and output requirements, a feedback neural network selected. To train the selected neural network model, a back propagation algorithm was chosen. The developed neural network demonstrated the expected behavior associated with learning and generalization errors. It found that even if the generalization error decreases as the learning sequence increases, the errors begin to fluctuate regardless of the introduction of a dynamic learning rate. The network sufficiently trained to meet the generalization error requirements, but there is stillroom to improve the generalization error. Practical results of training the constructed neural network at different sizes of the training sample presented.

РАЗРАБОТКА НЕЙРОННОЙ СЕТИ ДЛЯ РЕШЕНИЯ ЗАДАЧИ РАСПОЗНАВАНИЯ РЕЧИ

The article discusses a method for solving the problem of speech recognition on the example of recognizing individual words of a limited dictionary using a forward propagation neural network trained by the error back propagation method. The goal was to create a neural network model for recognizing the solution of individual words, analyze the training characteristics and behavior of the constructed neural network. Based on the input data and output requirements, a feedback neural network selected. To train the selected neural network model, a back propagation algorithm was chosen. The developed neural network demonstrated the expected behavior associated with learning and generalization errors. It found that even if the generalization error decreases as the learning sequence increases, the errors begin to fluctuate regardless of the introduction of a dynamic learning rate. The network sufficiently trained to meet the generalization error requirements, but there is stillroom to improve the generalization error. Practical results of training the constructed neural network at different sizes of the training sample presented.

Застосування нейронної мережі для оцінювання частотної характеристики багатопроменевого каналу в системах зв’язку з технологією OFDM

In the context of the development of modern technologies, the key role is assigned to the exchange of information. Digital communication systems are at the heart of information technology. To meet the needs of a modern person the requirements for communication systems are constantly increasing. Large information flows require high data rates. At the same time an important task is to reduce the number of errors that occur during data transmission. In OFDM systems, this is achieved by increasing the accuracy of estimating the frequency response of the communication channel. In this paper a neural network of direct propagation is used to estimate the values of the frequency response of the communication channel in OFDM systems. The neural network was designed for conditions when the pilot signals in the OFDM symbol structure are arranged in a combined pattern. Under such conditions the neural network receives noisy values of the frequency response on the pilot subcarriers as input information. Its task is to filter these values from noise and interpolate the values of the frequency response to the data subcarriers. The designed neural network has 32 incoming neurons, 128 outgoing neurons and 2 hidden layers of 8 neurons each. The structure of this neural network was designed with such an approach that the vector of the frequency response of the channel is estimated sequentially by 128 samples with their further combining. The neural network was trained on communication channels with given correlation properties by applying the error backpropagation method. The analysis of the efficiency of the network was carried out by means of statistical modeling using a model example in the Matlab computer-aided design system. The results of estimating the values of the frequency response using a neural network are compared with the results that are given by known methods. The analysis of the efficiency of the neural network showed that it is capable of providing a gain of up to 2 dB in comparison with the method of two-stage estimation of the frequency response estimation for a given model example. The neural network is inferior in the estimation accuracy to the minimum mean square error method, however, it has a lower implementation complexity compared to it.

Нейросетевое моделирование изменения концентрации молочной кислоты в ходе непрерывной ферментации бифидобактерий

In this work the changes in the lactic acid concentration during continuous fermentation of bifidobacteria have been investigated to obtain a neural network mathematical description. The fermentation was carried out under the conditions close to those of the descending colon (maintaining pH of 6.8 with 20% sodium hydroxide; anaerobiosis; the medium dilution rate was 0.04 h-1). This colon section is characterized by a large number of microorganisms, as well as their enormous influence on the host organism. The researches were carried out with the probiotic strain of Bifidobacterium adolescentis VKPM Ac-1662 (ATCC 15703T), the concentrations of the prebiotic oligofructose were varied (2, 5, 10, 15 g/l). Until a dynamic equilibrium state and at least 36 h after that, the concentrations of lactic and acetic acids using the of high performance liquid chromatography, optical density and viable bacteria count (CFU/ml) were measured. The neural network was trained on the basis on the obtained experimental data. The multilayer perceptron was chosen as the main architecture of the neural network. The vectors of the training sample include 6 variables: 5 input and 1 output. The training took place synchronously using the error back propagation method. The general error of the neural network was 1.85%. It was proved that the neural network approach helps to well illustrate the influence of various factors on the course of biotechnological processes; it summarizes the multiple experimental data with an acceptable error. The resulting neural network mathematical description proves that the representativeness of the training sample is important for obtaining the most accurate mathematical description. Further researches are needed to obtain a mathematical description of the change in the all environment components concentration in the form of a complex of the trained artificial neural networks.

Development of the algorithm of keyword search in the Kazakh language text corpus

The issue of semantic text analysis occupies a special place in computational linguistics. Researchers in this field have an increased interest in developing an algorithm that will improve the quality of text corpus processing and probabilistic determination of text content. The results of the study on the application of methods, approaches, algorithms for semantic text analysis in computational linguistics in international and Kazakhstan science led to the development of an algorithm of keyword search in a Kazakh text. The first step of the algorithm was to compile a reference dictionary of keywords for the Kazakh language text corpus. The solution to this problem was to apply the Porter (stemmer) algorithm for the Kazakh language text corpus. The implementation of the stemmer allowed highlighting unique word stems and getting a reference dictionary, which was subsequently indexed. The next step is to collect learning data from the text corpus. To calculate the degree of semantic proximity between words, each word is assigned a vector of the corresponding word forms of the reference dictionary, which results in a pair of a keyword and a vector. And the last step of the algorithm is neural network learning. During learning, the error backpropagation method is used, which allows a semantic analysis of the text corpus and obtaining a probabilistic number of words close to the expected number of keywords. This process automates the processing of text material by creating digital learning models of keywords. The algorithm is used to develop a neurocomputer system that will automatically check the text works of online learners. The uniqueness of the keyword search algorithm is the use of neural network learning for texts in the Kazakh language. In Kazakhstan, scientists in the field of computational linguistics conducted a number of studies based on morphological analysis, lemmatization and other approaches and implemented linguistic tools (mainly translation dictionaries). The scope of neural network learning for parsing of the Kazakh language remains an open issue in the Kazakhstan science. The developed algorithm involves solving one of the problems of effective semantic analysis of the text in the Kazakh language

The method of array antenna constructive synthesis on the basis of neural network approach

The method to decision constructive synthesis of array antennas was conducted. The method usefull when antenna elements can be in discreste states (for example: active element, passive element, excluded item, active element with discrete nominal of output power e.t.c). The method is based on neural network approach. The structure of a neural network consist of a classifying neural network and several approximating neural networks is substantiated. Input signals correspond to phase centers of array antenna elements. Number of output signals in classifying part is equal to discrete status of antenna element. Each approximating part of network has one output signal wich correspond to continious meaning. Separate parts of network preliminary learning with error back propagation method. The genetic algorithm of neural network learning with limited number of training coefficients is proposed. Examples of solving problems of constructive synthesis, with different indicators of the quality of neural network training are given.