Neural Network Training Methods Research Articles

IDENTIFICATION OF AREAS OF CORONAVIRUS COVID-19 INCIDENCE SPREADING BASED ON CLUSTER ANALYSIS METHOD

Subject: the use of the mathematical apparatus of neural networks for the scientific substantiation of anti-epidemic measures in order to reduce the incidence of diseases when making effective management decisions. Purpose: to apply cluster analysis, based on a neural network, to solve the problem of identifying areas of incidence. Tasks: to analyze methods of data analysis to solve the clustering problem; to develop a neural network method for clustering the territory of Ukraine according to the nature of the epidemic process COVID-19; on the basis of the developed method, to implement a data analysis software product to identify the areas of incidence of the disease using the example of the coronavirus COVID-19. Methods: models and methods of data analysis, models and methods of systems theory (based on the information approach), machine learning methods, in particular the Adaptive Boosting method (based on the gradient descent method), methods for training neural networks. Results: we used the data of the Center for Public Health of the Ministry of Health of Ukraine distributed over the regions of Ukraine on the incidence of COVID-19, the number of laboratory examined persons, the number of laboratory tests performed by PCR and ELISA methods, the number of laboratory tests of IgA, IgM, IgG; the model used data from March 2020 to December 2020, the modeling did not take into account data from the temporarily occupied territories of Ukraine; for cluster analysis, a neural network of 60 input neurons, 100 hidden neurons with an activation Fermi function and 4 output neurons was built; for the software implementation of the model, the programming language Python was used. Conclusions: analysis of methods for constructing neural networks; analysis of training methods for neural networks, including the use of the gradient descent method for the Adaptive Boosting method; all theoretical information described in this work was used to implement a software product for processing test data for COVID-19 in Ukraine; the division of the regions of Ukraine into zones of infection with the COVID-19 virus was carried out and a map of this division was presented.

Hybrid method of conventional neural network training

The paper proposes a hybrid method for training convolutional neural networks. The method consists of combining second and first-order methods for different elements of the architecture of a convolutional neural network. The hybrid convolution neural network training method allows to achieve significantly better convergence compared to Adam; however, it requires fewer computational operations to implement. Using the proposed method, it is possible to train networks on which learning paralysis occurs when using first-order methods. Moreover, the proposed method could adjust its computational complexity to the hardware on which the computation is performed; at the same time, the hybrid method allows using the mini-packet learning approach. The analysis of the ratio of computations between convolutional neural networks and fully connected artificial neural networks is presented. The mathematical apparatus of error optimization of artificial neural networks is considered, including the method of backpropagation of the error, the Levenberg-Marquardt algorithm. The main limitations of these methods that arise when training a convolutional neural network are analyzed. The analysis of the stability of the proposed method when the initialization parameters are changed. The results of the applicability of the method in various problems are presented.

DEVELOPMENT OF AN ALGORITHM FOR OPTIMIZING ENERGY CONSUMPTION IN CHEMICAL-TECHNOLOGICAL SYSTEMS BASED ON STATISTICAL TRAINING

The purpose of the research. The aim of the study is to develop approaches to solving the problems of optimizing the energy resources of chemical-technological systems based on statistical training. As the main research methods, the article uses graphical and tabular tools for descriptive data analysis to study the dynamics of the structure of energy carriers and determine possible reserves for reducing consumption; method of training neural networks to predict optimal values of energy consumption. Results. The article analyzes the current trends in the energy intensity of the cost of chemical production with an assessment of the degree of transformation of the structure of the energy portfolio and possible reserves for reducing the specific weight of electrical and thermal energy. The method of training neural articles using a regression predictive model was used to determine the minimum possible values of the parameter of energy resources consumption at the upper limit of the range, taking into account the limitations of the technological regulations for the production of chemicals and chemical products. The results of the study are applicable in the development of software complexes for intelligent energy systems, in the process of determining the cause-and-effect relationships of deviations in resource consumption from a given trajectory and the optimal vector of sustainable energy consumption.

A METHOD FOR PREDICTION EXECUTION TIME OF GPU PROGRAMS

The use of coprocessors such as GPU and FPGA is a leading trend in HPC. Therefore a lot of applications from a wide variety of domains were modified for GPUs and successfully used. In this paper, we propose an approach for prediction execution time of CUDA kernels, based on a static analysis of a program source code. The approach is based on building a CUDA core model and a graphics accelerator model. The developed method for estimating the execution time of CUDA kernels is applied to the implementation of matrix multiplication, the Fourier transform and the backpropagation method for training neural networks. As a result of verification, the approach showed good prediction accuracy, especially on low GPU loads.

An annotation-free whole-slide training approach to pathological classification of lung cancer types using deep learning

Deep learning for digital pathology is hindered by the extremely high spatial resolution of whole-slide images (WSIs). Most studies have employed patch-based methods, which often require detailed annotation of image patches. This typically involves laborious free-hand contouring on WSIs. To alleviate the burden of such contouring and obtain benefits from scaling up training with numerous WSIs, we develop a method for training neural networks on entire WSIs using only slide-level diagnoses. Our method leverages the unified memory mechanism to overcome the memory constraint of compute accelerators. Experiments conducted on a data set of 9662 lung cancer WSIs reveal that the proposed method achieves areas under the receiver operating characteristic curve of 0.9594 and 0.9414 for adenocarcinoma and squamous cell carcinoma classification on the testing set, respectively. Furthermore, the method demonstrates higher classification performance than multiple-instance learning as well as strong localization results for small lesions through class activation mapping.

Maximizing minority accuracy for imbalanced pattern classification problems using cost-sensitive Localized Generalization Error Model

Traditional machine learning methods may not yield satisfactory generalization capability when samples in different classes are imbalanced. These methods tend to sacrifice the accuracy of the minority class to improve the overall accuracy without regarding the fact that misclassifications of minority samples usually costs more in many real world applications. Therefore, we propose a neural network training method via a minimization of the cost-sensitive localized generalization error-based objective function (c-LGEM) to achieve a better balance of error yielded by the minority and the majority classes. The c-LGEM emphasizes the minimization of the generalization error of the minority class in a cost-sensitive manner. Experimental results obtained on 16 UCI datasets show that neural networks trained by the c-LGEM yield better performance in comparison to the performance yielded by some existing methods.

Optimization of Energy Consumption in Chemical Production Based on Descriptive Analytics and Neural Network Modeling

Improving the energy efficiency of chemical industries and increasing their environmental friendliness requires an assessment of the parameters of consumption and losses of energy resources. The aim of the study is to develop and test a method for solving the problem of optimizing the use of energy resources in chemical production based on the methodology of descriptive statistics and training of neural networks. Research methods: graphic and tabular tools for descriptive data analysis to study the dynamics of the structure of energy carriers and determine possible reserves for reducing their consumption; correlation analysis with the construction of scatter diagrams to identify the dependences of the range of limit values of electricity consumption on the average rate of energy consumption; a method for training neural networks to predict the optimal values of energy consumption; methods of mathematical optimization and standardization. The authors analyzed the trends in the energy intensity of chemical industries with an assessment of the degree of transformation of the structure of the energy portfolio and possible reserves for reducing the specific weight of electrical and thermal energy; determined the dynamics of energy losses at Russian industrial enterprises; established the correlation dependence of the range of limiting values of power consumption on the average rate of power consumption; determined the optimal limiting limits of the norms for the loss of electrical energy by the example of rubbers of solution polymerization. The results of the study can be used in the development of software complexes for intelligent energy systems that allow tracking the dynamics of consumption and losses of energy resources. Using the results allows you to determine the optimal parameters of energy consumption and identify reserves for improving energy efficiency.

Online learning of neural networks using random projections and sliding window: A case study of a real industrial process

Online Learning of non-stationary data streams is a challenging task. This work presents an online training method for a Single hidden Layer Feedforward neural Network (SLFN) that learns sample-by-sample, using an adjustable sliding window to adapt the network when data has changed. The method presents a fast training procedure, estimating hidden and output layer parameters independently. Tests with four synthetic datasets showed a good accuracy and quick recovery after drift occurrences. The proposed method is also applied to a real dataset from an industrial process in order to address the anomaly detection task, with the network acting as a classifier. Results show that the method is able to detect drifts prior to anomalies in the pre-fault periods, in the real situation that appeared in the industrial dataset.

Performance Analysis of Learning Rate Parameter on Prediction of Signal Power Loss for Network Optimization and Better Generalization

This research work explores the neural network learning capabilities by using a multi-layer perceptron artificial neural network to predict signal power loss by means of dataset from long term evolution network. The analysis of the effect of the learning rate parameter and the adoption of early stopping method during network training have been executed by using varied values of learning rate to ascertain the best learning rate during the neural network training. Also, there were neural network training without the application of learning rate and early stopping method and comparison have been made with the output results as shown in different tables. Output results comparisons have been performed using training regression and performance mean squared error. Two back propagation training algorithms, the Levenberg–Marquardt and the Bayesian Regularization algorithms were employed for the network training and comparison of their prediction abilities examined using same values of learning rates and on application of early stopping method as well as without learning rate and without early stopping method. The result shows an optimal performance of the neural network model on application of 0.005 learning rate and using 75%:15%:15% early stopping method with training regression 0.99267 and performance mean squared error 2.47 using Levenberg–Marquardt and training regression 0.99488 and performance mean squared error of 1.910 using Bayesian Regularization algorithms, respectively. Without application of learning rate and early stopping method, training the network using Levenberg–Marquardt algorithm gives training regression of 0.97111 and performance mean squared error of 7.38 using Levenberg–Marquart algorithm and training regression of 0.99248 and performance mean squared error of 4.42 using Bayesian Regularization algorithm. The margin between the two output results demonstrates the impact and importance of learning rate parameter as well as adopting early stopping method for neural network training for network optimization and better network generalization.

A Training Method for Low Rank Convolutional Neural Networks Based on Alternating Tensor Compose-Decompose Method

Over the past decade, deep learning-based computer vision methods have been shown to surpass previous state-of-the-art computer vision techniques in various fields, and have made great progress in various computer vision problems, including object detection, object segmentation, face recognition, etc. Nowadays, major IT companies are adding new deep-learning-based computer technologies to edge devices such as smartphones. However, since the computational cost of deep learning-based models is still high for edge devices, research is being actively carried out to compress deep learning-based models while not sacrificing high performance. Recently, many lightweight architectures have been proposed for deep learning-based models which are based on low-rank approximation. In this paper, we propose an alternating tensor compose-decompose (ATCD) method for the training of low-rank convolutional neural networks. The proposed training method can better train a compressed low-rank deep learning model than the conventional fixed-structure based training method, so that a compressed deep learning model with higher performance can be obtained in the end of the training. As a representative and exemplary model to which the proposed training method can be applied, we propose a rank-1 convolutional neural network (CNN) which has a structure alternatively containing 3-D rank-1 filters and 1-D filters in the training stage and a 1-D structure in the testing stage. After being trained, the 3-D rank-1 filters can be permanently decomposed into 1-D filters to achieve a fast inference in the test time. The reason that the 1-D filters are not being trained directly in 1-D form in the training stage is that the training of the 3-D rank-1 filters is easier due to the better gradient flow, which makes the training possible even in the case when the fixed structured network with fixed consecutive 1-D filters cannot be trained at all. We also show that the same training method can be applied to the well-known MobileNet architecture so that better parameters can be obtained than with the conventional fixed-structure training method. Furthermore, we show that the 1-D filters in a ResNet like structure can also be trained with the proposed method, which shows the fact that the proposed method can be applied to various structures of networks.

Deep Neural Networks: Selected Aspects of Learning and Application

Training methods for deep neural networks (DNNs) are analyzed. It is shown that maximizing the likelihood function of the distribution of the input data P(x) in the space of synaptic connections of a restricted Boltzmann machine (RBM) is equivalent to minimizing the cross-entropy (CE) of the network error function and minimizing the total mean squared error (MSE) of the network in the same space using linear neurons. The application of DNNs for the detection and recognition of productmarking is considered.

Learning Multi-Granularity Neural Network Encoding Image Classification Using DCNNs for Easter Africa Community Countries

Remote sensing scene classification is a fundamental responsibility of earth observation, aiming at identifying information granular for land cover classification. The multi-granular land use for multi-source remotely sensed image categories is now a principal task in remote sensing data augmentation and data selection. Understanding image representations are meaningful for the scene classification task. Training deep learning model-based approaches has to do with scene classification and brings about fantastic achievement. At the same time, these high-level approaches are computationally expensive and time–consuming. This paper introduces multi-granularity Neural Network Encoding architecture base on InceptionV3, InceptionReseNetV2, VGG16, and DenseNet201 architecture into remote sensing scene classification. To improve performance and to solve intra-class variation for multi-class scene issues remote sensing dataset. By using pre-trained CNN, activation function and ensemble learning have been adopting. InceptionV3 and VGG16 are used to extract features. InceptionResNetV2 use for fine-tuning, which consists of unfreezing the entre model and retraining the new data with a lower learning rate. The proposed fine-tune whole pre-trained model produces better results of test set up to 97.84 % than features extracted by InceptionResNetV2. Also, we use DCNNs ensemble average and weighted average to achieve better outcomes for 97.36% and 99.10%. In our experiments, we attempted to fine-tune the deep convolutional neural networks (DCNNs) training method for remote sensing scene classification on two public datasets UCM, SIRI-WHU, and one dataset I collected through the google earth engine from East Africa Community Countries (EACC) within nine classes within total 2112 labeled images. The results indicate that our proposed fine-tuning of the pre-trained model with few epochs and less computational time increases accuracy.

Механизм обучения нейронной сети для прогноза метеорологической обстановки при использовании ГИС

This article is devoted to the mechanisms of neural network training for forecasting the meteorological situation when using GIS. The structural scheme of the GIS under consideration is proposed as a project solution and the main elements allowing to implement neural networks and their training are defined. The stochastic method is chosen as a tool for neural network training as it suggests the most probable outcome of the event based on the previous sample. The article gives an example of testing neural network training as an application program «Data Processor». The results described in the article allow us to judge about the applicability of the selected neural network training method for forecasting meteorological conditions and using data in geoinformation decision-making systems. Keywords: geoinformation system, synoptic forecast method, hydrodynamic forecast method, aggregator, data processor, knowledge base, deterministic method, expert estimation method, stochastic method, neural network, sampling, probability dispersion.

Deep Belief Network Integrating Improved Kernel-Based Extreme Learning Machine for Network Intrusion Detection

Deep learning has become a research hotspot in the field of network intrusion detection. In order to further improve the detection accuracy and performance, we proposed an intrusion detection model based on improved deep belief network (DBN). Traditional neural network training methods, like Back Propagation (BP), start to train a model with preset parameters such as the randomly initialized weights and thresholds, which may bring some issues, e.g., attracting the model to the local optimal solutions, or requiring a long training period. We use the Kernel-based Extreme Learning Machine (KELM) with the supervised learning ability to replace the BP algorithm in DBN in a bid to ameliorate the situation. Considering the problem of poor classification performance usually caused by randomly initializing kernel parameters with KELM, an enhanced grey wolf optimizer (EGWO) is designed to optimize the parameters of KELM. In order to improve the search ability and optimization ability of the traditional grey wolf optimizer algorithm, a novel optimization strategy combining the inner and outer hunting is introduced. Experiments on KDDCup99, NSL-KDD, UNSW-NB15 and CICIDS2017 datasets show that the proposed DBN-EGWO-KELM algorithm has greater advantages in terms of its accuracy, precision, true positive rate, false positive rate and other evaluation indices compared with BP, RBF, SVM, KELM, LIBSVM, CNN, DBN-KELM and other intrusion detection models, and can effectively meet the requirements of intrusion detection of complex networks.

Artificial applicability labels for improving policies in retrosynthesis prediction

Automated retrosynthetic planning algorithms are a research area of increasing importance. Automated reaction-template extraction from large datasets, in conjunction with neural-network-enhanced tree-search algorithms, can find plausible routes to target compounds in seconds. However, the current method for training neural networks to predict suitable templates for a given target product leads to many predictions that are not applicable in silico. Most templates in the top 50 suggested templates cannot be applied to the target molecule to perform the virtual reaction. Here, we describe how to generate data and train a neural network policy that predicts whether templates are applicable or not. First, we generate a massive training dataset by applying each retrosynthetic template to each product from our reaction database. Second, we train a neural network to perform near-perfect prediction of the applicability labels on a held-out test set. The trained network is then joined with a policy model trained to predict and prioritize templates using the labels from the original dataset. The combined model was found to outperform the policy model in a route-finding task using 1700 compounds from our internal drug-discovery projects.

Hardware-Based Hopfield Neuromorphic Computing for Fall Detection.

With the popularity of smart wearable systems, sensor signal processing poses more challenges to machine learning in embedded scenarios. For example, traditional machine-learning methods for data classification, especially in real time, are computationally intensive. The deployment of Artificial Intelligence algorithms on embedded hardware for fast data classification and accurate fall detection poses a huge challenge in achieving power-efficient embedded systems. Therefore, by exploiting the associative memory feature of Hopfield Neural Network, a hardware module has been designed to simulate the Neural Network algorithm which uses sensor data integration and data classification for recognizing the fall. By adopting the Hebbian learning method for training neural networks, weights of human activity features are obtained and implemented/embedded into the hardware design. Here, the neural network weight of fall activity is achieved through data preprocessing, and then the weight is mapped to the amplification factor setting in the hardware. The designs are checked with validation scenarios, and the experiment is completed with a Hopfield neural network in the analog module. Through simulations, the classification accuracy of the fall data reached 88.9% which compares well with some other results achieved by the software-based machine-learning algorithms, which verify the feasibility of our hardware design. The designed system performs the complex signal calculations of the hardware’s feedback signal, replacing the software-based method. A straightforward circuit design is used to meet the weight setting from the Hopfield neural network, which is maximizing the reusability and flexibility of the circuit design.

Development and application of random forest technique for element level structural damage quantification

This paper presents element level structural damage quantification using an ensemble-based machine learning technique, namely, random forest technique, with acceleration responses from structures. The ensemble-based approach provides a better prediction than an individual model. Random forest is a machine learning algorithm which has several decision trees to perform a task. The proposed approach develops a random forest as a regressor to predict multiple output variables, which is the vector of elemental level damage quantification results of the structure. Damage severity is identified as the reduction in the elemental stiffness parameters. The acceleration responses for single-element and multiple element damage cases are generated and further processed to feed as input to the random forest. The acceleration responses from the sensor nodes are concatenated, and principal component analysis (PCA) is applied to reduce the uncorrelated input dimension. The proposed approach can provide good damage identification results efficiently with much less computational demand and time, compared to the neural network training methods and deep learning models. Moreover, fewer sensors are needed to measure acceleration responses to localise damage and quantify the damage level. To demonstrate the proposed method, a simply supported beam is used as an example in numerical studies. Different levels of noise in the acceleration responses and uncertainty in the finite element modelling are considered. Experimental studies on a steel frame structure with 70 elements are also conducted to investigate the performance of the proposed approach for structural damage quantification. Good identification results are obtained with an efficient training process.

ZigBee wireless smart plug network with RSSI multi‐lateration‐based proximity estimation and parallelised machine learning capabilities for demand response

This study explores how wireless ZigBee technology may be applied to automation of electric loads in residential and commercial spaces, allowing to participate in demand response initiatives. The authors discuss development of a custom smart plug with sensing, wireless communication, and electric load actuation capabilities along with several innovative upgrades. There are many commercially available smart plugs that contain multiple sensors and relays. However, very few provide the ability to effectively estimate the proximity between modules or the ability to perform robust system-wide optimisation. The authors propose two innovative smart plug eco-system improvements. One is the use of a received signal strength indicator (RSSI) multi-lateration-based method to estimate the relative proximities of modules. The RSSI values for almost all transmission paths within the ZigBee network are acquired via the authors' forced network reconfiguration algorithm, addressing the limitations of RSSI observation within a star structure. A second innovation is the development of a parallelised neural network training method for application to load automation. The authors use a k-means clustering algorithm to divide training data into subsets such that training may be parallelised.

An improved model training method for residual convolutional neural networks in deep learning

Residual convolutional neural network (R-CNN) has become a promising method for image recognition in deep learning applications. The application accuracy, as a key indicator, has a close relationship with filter weights in trained R-CNN models. In order to make filters work at full capacity, we find out that lower relevancy between filters in the same layer promotes higher accuracy for R-CNN applications. Furthermore, we propose an improved R-CNN model training method to acquire a higher accuracy and a better generalization ability. In this paper, the main focus is to control the update of filter weights during model training. The key mechanism is achieved through computing the relevancy between filters in the same layer. The relevancy is quantified by a correlation coefficient, e.g., Pearson Correlation Coefficient (PCC). The mechanism takes a larger probability to utilize the updated filter weights with a lower correlation coefficient, and vice versa. In order to validate our proposal, we construct an experiment through PCC on residual networks. The experiment demonstrates that the improved model training method is a promising mean with better generalization ability and higher recognition accuracy (0.52%-1.83%) for residual networks.

Analysis of artificial neural networks training models for airfare price prediction

Air transport is playing an increasing role in the world economy every year. This is facilitated by technological development and the latest developments in the aviation industry, globalization. This paper provides an overview of artificial neural network training methods for airfare predicting. The articles for 2017-2019 were analyzed in order to determine the model with the most accurate prediction. The researchers conducted research on open data collected by themselves and set themselves the goal of creating a model that would advise a user the best time to buy a ticket when the price would be the lowest. The review of the papers by similar themes revealed that the Bagging Regression Tree model has the highest results with an accuracy of 88% and the random forest method has an accuracy of 87%. Civil aviation plays an important role in the economy of each country. Aviation is the best way to cover long distances in comfort in the shortest time. Airlines offer customers a variety of opportunities to travel both within the country and abroad. The main problem of interaction between airlines and customers is the airfare: the former want to sell more at the higher price, and the latter want to buy cheaper. Therefore, companies use their own private algorithms for dynamic pricing and constantly monitor the market situation, responsive to changes in demand and the actions of competitors. This behavior allows them to achieve a balance between the desires of airlines and customers. Scientists are trying to invent a way to predict airfare so that air travelers can buy them at the lowest price. The results of the work in this area provide general rules for the best purchase. For example, according to the article (Udachny, 2016) thebest day to buy a ticket by expedia.com for a domestic flight on the United States is Sunday, and the best period is 57 days before departure. This article provides an overview of the works, the authors of which compared the models of machine learning. Achievements in this area are limited to direct flights of a certain domestic market (USA, India) and 88% accuracy of the forecast (Tziridis et al., 2017). The Bagging Regression Tree model described in the article (Tziridis et al., 2017) can be considered the best result. This trained model can make predictions based only on two parameters: the number of free cargo and the number of days left before departure and has an accuracy of 88%.