Learning Process Of Neural Network Research Articles

Accelerating the learning process of a neural network by predicting the weight coefficient

The purpose of this study is to analyze and implement the acceleration of the neural network learning process by predicting the weight coefficients. The relevance of accelerating the learning of neural networks is touched upon, as well as the possibility of using predictionmodels in a wide range of tasks where it is necessary to build fast classifiers.When data is received from the array of sensors of a chemical unit in real time, it is necessary to be able to predict changes and change the operating parameters. After assessment, this should be done as quickly as possible in order to promptly change the current structure and stateof the resulting substances.Work on speeding up classifiers usually focuses on speeding up the applied classifier. The calculation of the predicted values of the weight coefficients arecarried out using the calculation of the value using the known predictionmodels. The possibility of the combined use of predictionmodels and optimization models was tested to accelerate the learning process of a neural network.The scientific novelty of the study lies in the effectiveness analysis of predictionmodels use in training neural networks. For the experimental evaluation of the effectiveness of predictionmodelsuse, the classification problem was chosen. To solve the experimental problem, the type of neural network “multilayer perceptron”was chosen.The experiment is divided into several stages: initial training of the neural network without a model, and then using predictionmodels; initial training of a neural network without an optimization method, and then using optimization methods; initial training of the neural network using combinations of predictionmodels and optimization methods; measuring the relative error of using predictionmodels, optimization methods and combined use.Models such as “Seasonal Linear Regression”, “Simple Moving Average”, and “Jump”were used in the experiment.The “Jump”model was proposed and developed based on the results of observing the dependence of changes in the values of the weighting coefficient on the epoch. Methods such as “Adagrad”, “Adadelta”,“Adam”were chosen for training neural and subsequent verification of the combined use of predictionmodels with optimization methods.As a result of the study, the effectiveness of the use of predictionmodels in predicting the weight coefficients of a neural network has been revealed. Theidea is proposed and models are used that can significantly reduce the training time of a neural network. The idea of using predictionmodels is that the model of the change in the weight coefficient from the epoch is a time series, which in turn tends to a certain value.As a result of the study, it was found that it is possible to combine predictionmodels and optimization models. Also, predictionmodels do not interfere with optimization models, since they do not affect the formula of the training itself, as a result of which it is possible to achieve rapid training of the neural network. In the practical part of the work, two known predictionmodels and the proposed developed model were used. As a result of the experiment, operating conditions were determined using predictionmodels.

Hybrid Learning Driven by Dynamic Descriptors for Video Classification of Reflective Surfaces

Visual inspection has recently gained increasing importance in the manufacturing industry and is often addressed by means of learning methodologies applied to data obtained from specific lighting and camera system setups. The industrial scenario becomes particularly challenging when the inspection regards reflective objects, which may affect both the data acquisition and the classification decision process, thus limiting the overall performance. In this context, we observe that the dynamics of the reflected light is the key aspect to characterize these surfaces and needs to be accurately exploited to improve the performances of the learning algorithms. To this aim, we propose a combined model-based and data-driven approach designed to detect defects on the reflective surfaces of industrial products, captured as video sequences under coaxial structured illumination. Specifically, a tunable spatial-temporal descriptor of the evolution of the reflected light [dynamic evolution of the light (DEL)] is designed and employed within a hybrid learning (HL) framework, where the learning process of a convolutional neural network (CNN) is driven by the model-based descriptor. This approach is also extended by adopting the similar in nature descriptor dynamic image. The proposed HL solutions are validated against a whole spectrum of state-of-the-art learning procedures and different descriptors. Experiments run on a dataset coming from an actual industrial scenario that confirms the ability of DEL to accurately characterize reflective surfaces and the validity of the HL method, which shows remarkably better performance in fault detection even with respect to modern 3-D CNNs with comparable computational effort.

Multilayer neural networks – as determined systems

The study of the influence of learning speed (η) on the learning process of a multilayer neural network is carried out. The program for a multilayer neural network was written in Python. The learning speed is considered as a constant value and its optimal value at which the best learning is achieved is determined. To analyze the impact of learning speed, a logistic function, which describes the learning process, is used. It is shown that the learning error function is characterized by bifurcation processes that lead to a chaotic state at η> 0.8. The optimal value of the learning speed is determined. The value determines the appearance of the process of doubling the number of local minima, and is η = 0.62 for a three-layer neural network with 4 neurons in each layer. Increasing the number of hidden layers (3 ÷ 30) and the number of neurons in each layer (4 ÷ 150) does not lead to a radical change in the diagram of the logistic function (xn, η), and hence, in the optimal value of the learning speed.

Optimizing Connection Weights in Neural Networks Using Hybrid Metaheuristics Algorithms

The learning process of artificial neural networks is an important and complex task in the supervised learning field. The main difficulty of training a neural network is the process of fine-tuning the best set of control parameters in terms of weight and bias. This paper presents a new training method based on hybrid particle swarm optimization with Multi-Verse Optimization (PMVO) to train the feedforward neural networks. The hybrid algorithm is utilized to search better in solution space which proves its efficiency in reducing the problems of trapping in local minima. The performance of the proposed approach was compared with five evolutionary techniques and the standard momentum backpropagation and adaptive learning rate. The comparison was benchmarked and evaluated using six bio-medical datasets. The results of the comparative study show that PMVO outperformed other training methods in most datasets and can be an alternative to other training methods.

Development of software and algorithms of parallel learning of artificial neural networks using CUDA technologies

The object of research is to parallelize the learning process of artificial neural networks to automate the procedure of medical image analysis using the Python programming language, PyTorch framework and Compute Unified Device Architecture (CUDA) technology. The operation of this framework is based on the Define-by-Run model. The analysis of the available cloud technologies for realization of the task and the analysis of algorithms of learning of artificial neural networks is carried out. A modified U-Net architecture from the MedicalTorch library was used. The purpose of its application was the need for a network that can effectively learn with small data sets, as in the field of medicine one of the most problematic places is the availability of large datasets, due to the requirements for data confidentiality of this nature. The resulting information system is able to implement the tasks set before it, contains the most user-friendly interface and all the necessary tools to simplify and automate the process of visualization and analysis of data. The efficiency of neural network learning with the help of the central processor (CPU) and with the help of the graphic processor (GPU) with the use of CUDA technologies is compared. Cloud technology was used in the study. Google Colab and Microsoft Azure were considered among cloud services. Colab was first used to build a prototype. Therefore, the Azure service was used to effectively teach the finished architecture of the artificial neural network. Measurements were performed using cloud technologies in both services. The Adam optimizer was used to learn the model. CPU duration measurements were also measured to assess the acceleration of CUDA technology. An estimate of the acceleration obtained through the use of GPU computing and cloud technologies was implemented. CPU duration measurements were also measured to assess the acceleration of CUDA technology. The model developed during the research showed satisfactory results according to the metrics of Jaccard and Dyce in solving the problem. A key factor in the success of this study was cloud computing services.

Neural Reduction of Image Data in Order to Determine the Quality of Malting Barley.

Image analysis using neural modeling is one of the most dynamically developing methods employing artificial intelligence. The feature that caused such widespread use of this technique is mostly the ability of automatic generalization of scientific knowledge as well as the possibility of parallel analysis of the empirical data. A properly conducted learning process of artificial neural network (ANN) allows the classification of new, unknown data, which helps to increase the efficiency of the generated models in practice. Neural image analysis is a method that allows extracting information carried in the form of digital images. The paper focuses on the determination of imperfections such as contaminations and damages in the malting barley grains on the basis of information encoded in the graphic form represented by the digital photographs of kernels. This choice was dictated by the current state of knowledge regarding the classification of contamination that uses undesirable features of kernels to exclude them from use in the malting industry. Currently, a qualitative assessment of kernels is carried by malthouse-certified employees acting as experts. Contaminants are separated from a sample of malting barley manually, and the percentages of previously defined groups of contaminations are calculated. The analysis of the problem indicates a lack of effective methods of identifying the quality of barley kernels, such as the use of information technology. There are new possibilities of using modern methods of artificial intelligence (such as neural image analysis) for the determination of impurities in malting barley. However, there is the problem of effective compression of graphic data to a form acceptable for ANN simulators. The aim of the work is to develop an effective procedure of graphical data compression supporting the qualitative assessment of malting barley with the use of modern information technologies. Image analysis can be implemented into dedicated software.

Playing to distraction: towards a robust training of CNN classifiers through visual explanation techniques

The field of deep learning is evolving in different directions, with still the need for more efficient training strategies. In this work, we present a novel and robust training scheme that integrates visual explanation techniques in the learning process. Unlike the attention mechanisms that focus on the relevant parts of images, we aim to improve the robustness of the model by making it pay attention to other regions as well. Broadly speaking, the idea is to distract the classifier in the learning process to force it to focus not only on relevant regions but also on those that, a priori, are not so informative for the discrimination of the class. We tested the proposed approach by embedding it into the learning process of a convolutional neural network for the analysis and classification of two well-known datasets, namely Stanford cars and FGVC-Aircraft. Furthermore, we evaluated our model on a real-case scenario for the classification of egocentric images, allowing us to obtain relevant information about peoples' lifestyles. In particular, we work on the challenging EgoFoodPlaces dataset, achieving state-of-the-art results with a lower level of complexity. The obtained results indicate the suitability of our proposed training scheme for image classification, improving the robustness of the final model.

Hardware‐Friendly Stochastic and Adaptive Learning in Memristor Convolutional Neural Networks

Memristors offer great advantages as a new hardware solution for neuromorphic computing due to their fast and energy‐efficient matrix vector multiplication. However, the nonlinear weight updating property of memristors makes it difficult to be trained in a neural network learning process. Several compensation schemes have been proposed to mitigate the updating error caused by nonlinearity; nevertheless, they usually involve complex peripheral circuits design. Herein, stochastic and adaptive learning methods for weight updating are developed, in which the inaccuracy caused by the memristor nonlinearity can be effectively suppressed. In addition, compared with the traditional nonlinear stochastic gradient descent (SGD) updating algorithm or the piecewise linear (PL) method, which are most often used in memristor neural network, the design is more hardware friendly and energy efficient without the consideration of pulse numbers, duration, and directions. Effectiveness of the proposed method is investigated on the training of LeNet‐5 convolutional neural network. High accuracy, about 93.88%, on the Modified National Institute of Standards and Technology handwriting digits datasets is achieved (with typical memristor nonlinearity as ±1), which is close to the network with complex PL method (94.7%) and is higher than the original nonlinear SGD method (90.14%).

HyperNOMAD

The performance of deep neural networks is highly sensitive to the choice of the hyperparameters that define the structure of the network and the learning process. When facing a new application, tuning a deep neural network is a tedious and time-consuming process that is often described as a “dark art.” This explains the necessity of automating the calibration of these hyperparameters. Derivative-free optimization is a field that develops methods designed to optimize time-consuming functions without relying on derivatives. This work introduces the HyperNOMAD package, an extension of the NOMAD software that applies the MADS algorithm [7] to simultaneously tune the hyperparameters responsible for both the architecture and the learning process of a deep neural network (DNN). This generic approach allows for an important flexibility in the exploration of the search space by taking advantage of categorical variables. HyperNOMAD is tested on the MNIST, Fashion-MNIST, and CIFAR-10 datasets and achieves results comparable to the current state of the art.

Perceptual similarity measurement based on generative adversarial neural networks in graphics design

Measuring the similarity between images is of paramount importance in computer vision. However, the commonly used pixelwise similarity metrics do not match well with perceptual similarity. The purpose of this paper is to propose a visual similarity measurement method, which can be effectively used for plagiarism detection in graphic design. Plagiarism detection of designs refers to the identification and determination of major similarities. It is difficult to carry out the similarity learning process in traditional deep neural network due to the insufficient of training samples. To overcome this problem, a novel scheme is proposed for measuring perceptual similarity of graphics by using a constraint Generative Adversarial Network (GAN) model. The generator of GAN is used to create similar graphics following the common plagiarism features of logo design. Unlike the traditional discriminator which judges the authenticity of the generated image and the original image, the modified discriminator is used to calculate the perceptual similarity of the graphics pair. In graphics design, plagiarism mainly focuses on the changes of shape, color and style, which has certain cognitive subjectivity. Therefore, design experts were invited to participate in a group of cognitive analysis experiments. A perceptual constraint model is established to limit the generation of plagiarized graphics according to “design and visual rationality”. Promising results demonstrate that the proposed method can be used for plagiarism detection of logo design. Given its effectiveness and conceptual simplicity, I hope it can serve as a baseline and contribute to the future research on plagiarism detection of artworks.

Overview of Configuring Adaptive Activation Functions for Deep Neural Networks - A Comparative Study

Recently, the deep neural networks (DNN) have demonstrated many performances in the pattern recognition paradigm. The research studies on DNN include depth layer networks, filters, training and testing datasets. Deep neural network is providing many solutions for nonlinear partial differential equations (PDE). This research article comprises of many activation functions for each neuron. Besides, these activation networks are allowing many neurons within the neuron networks. In this network, the multitude of the functions will be selected between node by node to minimize the classification error. This is the reason for selecting the adaptive activation function for deep neural networks. Therefore, the activation functions are adapted with every neuron on the network, which is used to reduce the classification error during the process. This research article discusses the scaling factor for activation function that provides better optimization for the process in the dynamic changes of procedure. The proposed adaptive activation function has better learning capability than fixed activation function in any neural network. The research articles compare the convergence rate, early training function, and accuracy between existing methods. Besides, this research work provides improvements in debt ideas of the learning process of various neural networks. This learning process works and tests the solution available in the domain of various frequency bands. In addition to that, both forward and inverse problems of the parameters in the overriding equation will be identified. The proposed method is very simple architecture and efficiency, robustness, and accuracy will be high when considering the nonlinear function. The overall classification performance will be improved in the resulting networks, which have been trained with common datasets. The proposed work is compared with the recent findings in neuroscience research and proved better performance.

Research on Economic Mathematical Analysis and Construction Model of Prefabricated Building Structure Based on Improved Neural Network Algorithm

In this article, a mathematical analysis model of economics of prefabricated building structure based on improved neural network algorithm is proposed in order to solve the low analysis accuracy in traditional methods. Firstly, by means of analyzing the costs of materials, labor, and equipment, the economic characteristics of the cost of fabricated building structures are determined. Secondly, the single neuron is analyzed and the weight coefficient is adjusted in accordance with the multilayer neural network structure, so as to minimize the construction error of the economic analysis model of the assembled building structure. Meanwhile, the weight vector is obtained, error-weighted square sum is calculated through choosing an adaptive filter and obtained, and the weight vector is updated by the least squares algorithm. Thirdly, the neural network algorithm training and learning process is designed and improved, the dependent variable is selected, the number of input points is determined, and then, the training and learning process of the improved neural network algorithm is completed. Finally, a fitness function is set to measure the authenticity of dataset, which is further defined as a combination of different weights to construct an economic mathematical analysis model. The experimental results indicate that the analysis results of this method can reach an accuracy up to 96%, so it has a broader application prospect in low-rise buildings.

ПРИМЕНЕНИЕ НЕЙРОННОЙ СЕТИ ДЛЯ УПРАВЛЕНИЯ ТРАЛОВЫМ ПРОМЫСЛОМ

The article highlights the problems of managing trawl fishing, increasing the operation efficiency and reducing the influence of the human factor. There has been considered using 
 a neural network in combination with a mathematical model and BigData technologies for predictive modeling in the process of automatic control of trawl fishing in order to increase its efficiency (to reduce energy and labor costs, to increase fishing productivity). Advantages of the proposed approach are the possibility to account for the above factors neglected in the mathematical model due to the complexity of their mathematical description (e.g. time of the day, time of the year, weather conditions, density, congestion, availability and distribution of food resources, other aquatic species), as well as the possibility of collecting and accumulating data obtained in many fishing operations and from different fishers for their subsequent consideration in the fishery management in the future. There has been proposed a solution based on the corrected output data obtained from 
 a mathematical model and on the output data of a neural network. The weight coefficients of the neural network are extracted from a centralized database using BigData technologies before fishing with a selection criterion for the area and object of fishing. In the course of fishing the input data of the neural network and the final (adjusted) output data of the control are recorded. At the end of fishing the saved data is used in the process of training the neural network, followed by updating the weight coefficients in a centralized database. The neural network learning process occurs between the fisheries on a centralized shared neural network. The adjusted weight coefficients are updated in the general database of fishers.

ALGORITHMIC STABILITY OF DEEP LEARNING NEURAL NETWORKS IN RECOGNIZING THE MICROSTRUCTURE OF MATERIALS

The division of data for training a neural network into training and test data in various proportions to each other is investigated. The question is raised about how the quality of data distribution and their correct annotation can affect the final result of constructing a neural network model. The paper investigates the algorithmic stability of training a deep neural network in problems of recognition of the microstructure of materials. The study of the stability of the learning process makes it possible to estimate the performance of a neural network model on incomplete data distorted by up to 10%. Purpose. Research of the stability of the learning process of a neural network in the classification of microstructures of functional materials. Materials and methods. Artificial neural network is the main instrument on the basis of which produced the study. Different subtypes of deep convolutional networks are used such as VGG and ResNet. Neural networks are trained using an improved backpropagation method. The studied model is the frozen state of the neural network after a certain number of learning epochs. The amount of data excluded from the study was randomly distributed for each class in five different distributions. Results. Investigated neural network learning process. Results of experiments conducted computing training with gradual decrease in the number of input data. Distortions of calculation results when changing data with a step of 2 percent are investigated. The percentage of deviation was revealed, equal to 10, at which the trained neural network model loses its stability. Conclusion. The results obtained mean that with an established quantitative or qualitative deviation in the training or test set, the results obtained by training the network can hardly be trusted. Although the results of this study are applicable to a particular case, i.e., microstructure recognition problems using ResNet-152, the authors propose a simpler technique for studying the stability of deep learning neural networks based on the analysis of a test, not a training set.

THE SPEED OF LEARNING CONVOLUTIONAL NEURAL NETWORKS ON THE GPU AND CPU TO DETECT SYNTHESIZED SPEECH USING SPECTROGRAMS

In this work has been investigated the possibility of using convolutional neural networks to detect synthesized speech. The Python programming language, the TensorFlow library in combination with the high-level Keras API and the ASVspoof 2019 audio database in flac format were used to create the software application. The voice signal of synthesized and natural speech was converted into mel-frequency spectrograms. The structure of a convolutional neural network with high indicators of recognition accuracy is proposed. The learning speed of neural networks on GPU and CPU is compared using the CUDA library. The influence of the batch size parameter on the accuracy of the neural network was investigated. The TensorBoard tool was used to monitor and profile the learning process of neural networks. Keywords: audio deepfake, mel-frequency sound spectrograms, convolutional neural networks, learning speed of neural networks.

Activation Maximization with a Prior in Speech Data

Recently, more and more studies regarding neural networks have been done. However, the learning process of neural networks is often elusive to human beings, which leads to the advent of feature visualization techniques. Activation Maximization (AM) is one of the feature visualization techniques, originally designed for image data. In AM, the input data is optimized to find the data that activates the selected neuron. In this paper, the emotion recognizer’s output is selected as the neuron, and the latent code of a generator (of Generative Adversarial Networks) is optimized instead of the input raw data. The aim of this study is to apply AM to different representations of audio data (waveform-based data and mel-spectrogram-based data) and different model structures (CNN, WaveNet, LSTM), and to find out the most suitable condition for AM in audio domain data. Additionally, we have also tried to visualize the essential features of being a certain class for emotion classification in speech data, using 2 datasets: the Toronto emotional speech set (TESS) and the Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS). The mel-spectrogram-based models were found to be superior to the others, showing the distinctive features of selected emotions. More specifically, the CNN-mel-spectrogram-based model was the best in both qualitative and quantitative (FID score) results. Moreover, as demonstrated in this study, AM can also be employed as an output enhancer for generative models.

CHAOTIC STATES OF A MULTILAYER NEURAL NETWORK

The study of the influence of learning speed η on the learning process of a multilayer neural network was studied. The program for the multilayer neural network was written in Python. We determined the speed of learning at which the best learning is observed. To analyze the impact of learning speed on the learning process, a logistic function was used, which describes the doubling of frequency. It is shown that the learning error function is characterized by bifurcation processes that lead to a chaotic state at η> 0.8. The optimal value of the learning speed is determined, which determines the appearance of the process of doubling the number of local minima. Increasing the number of hidden layers and the number of neurons in each layer does not lead to a radical change in the diagram of the logistics function, and hence the optimal value of the learning speed. It is shown that the increase in the number of hidden layers, as well as the number of neurons in them, with increasing speed of learning is accompanied by the appearance of a chaotic state. The chaotic state is characterized by the lack of a learning process. Using the Fourier transform of the error function, the bifurcation process depending on the learning speed was investigated. On the basis of these researches the program for definition of size of optimum speed of training of a neural network is developed. The text of the article adds program codes for a multilayer neural network with the ability to build a branching diagram and a program to determine the presence or absence of the learning process, taking into account the value of optimal learning speed and error. Key words : multilayer neural network, optimal learning speed, bifurcations, chaotic states.

A Hardware/Software Co-Design Methodology for Adaptive Approximate Computing in clustering and ANN Learning

As one of the most promising energy-efficient emerging paradigms for designing digital systems, approximate computing has attracted a significant attention in recent years. Applications utilizing approximate computing (AxC) can tolerate some loss of quality in the computed results for attaining high performance. Approximate arithmetic circuits have been extensively studied; however, their application at system level has not been extensively pursued. Furthermore, when approximate arithmetic circuits are applied at system level, error-accumulation effects and a convergence problem may occur in computation. Multiple approximate components can interact in a typical datapath, hence benefiting from each other. Many applications require more complex datapaths than a single multiplication. In this paper, a hardware/software co-design methodology for adaptive approximate computing is proposed. It makes use of feature constraints to guide the approximate computation at various accuracy levels in each iteration of the learning process in Artificial Neural Networks (ANNs). The proposed adaptive methodology also considers the input operand distribution and the hybrid approximation. Compared with a baseline design, the proposed method significantly reduces the power-delay product while incurring in only a small loss of accuracy. Simulation and a case study of image segmentation validate the effectiveness of the proposed methodology.

Gradient-Based Algorithm for Tracking the Activity of Neural Network Weights Changing

The research conducted in this paper is in the field of machine learning. The main object of the research is the learning process of an artificial neural network in order to increase its efficiency. The algorithm based on the analysis of retrospective learning data. The dynamics of changes in the values of the weights of an artificial neural network during training is an important indicator of training efficiency. The algorithm proposed in this work is based on changing the weight gradients values. Changing of the gradients weights makes it possible to understand how actively the network weights change during training. This knowledge helps to diagnose the training process and makes an adjusting the training parameters. The results of the algorithm can be used to train an artificial neural network. The network will help to determine the set of measures (actions) needed to optimize the learning process by the algorithm results.

EEG channel selection strategy for deep learning in emotion recognition

Emotions play an important role in everyday life and contribute to physical and mental health. Emotional states can be detected by electroencephalography (EEG signals). Efficient information retrieval from the EEG sensors is a complex and challenging task. Therefore, deep learning methods for EEG signal analysis attract more and more attention. Many researchers emphasize automated feature learning as the motivation for using deep learning approaches. We propose using a limited number of EEG channels as an input for a deep neural network. In the research, we confirm that our electrode selection enhances the learning process of the convolutional neural network. The classification accuracy for the reduced subset of electrodes yields results comparable to the full dataset in a significantly shorter time—the average learning time 58% faster using our proposed strategy.