Learning Process Of Neural Network Research Articles

Programmable Retention Characteristics in MoS2-Based Atomristors for Neuromorphic and Reservoir Computing Systems.

In this study, we investigate the coexistence of short- and long-term memory effects owing to the programmable retention characteristics of a two-dimensional Au/MoS2/Au atomristor device and determine the impact of these effects on synaptic properties. This device is constructed using bilayer MoS2 in a crossbar structure. The presence of both short- and long-term memory characteristics is proposed by using a filament model within the bilayer transition-metal dichalcogenide. Short- and long-term properties are validated based on programmable multilevel retention tests. Moreover, we confirm various synaptic characteristics of the device, demonstrating its potential use as a synaptic device in a neuromorphic system. Excitatory postsynaptic current, paired-pulse facilitation, spike-rate-dependent plasticity, and spike-number-dependent plasticity synaptic applications are implemented by operating the device at a low-conductance level. Furthermore, long-term potentiation and depression exhibit symmetrical properties at high-conductance levels. Synaptic learning and forgetting characteristics are emulated using programmable retention properties and composite synaptic plasticity. The learning process of artificial neural networks is used to achieve high pattern recognition accuracy, thereby demonstrating the suitability of the use of the device in a neuromorphic system. Finally, the device is used as a physical reservoir with time-dependent inputs to realize reservoir computing by using short-term memory properties. Our study reveals that the proposed device can be applied in artificial intelligence-based computing applications by utilizing its programmable retention properties.

Analysis of The Use of Nguyen Widrow Algorithm in Backpropagation in Kidney Disease

Fast and accurate diagnosis is very important for kidney disease. This research conducts and analyzes by using Nguyen Widrow Algorithm in Back Propagation method in artificial neural network for kidney disease diagnosis with the aim to improve the accuracy in predicting and time efficiency in diagnosing. The Nguyen Widrow algorithm is very capable of accelerating convergence and stabilizing the learning process in artificial neural networks, which is also expected to present a meaningful contribution to the handling of health data. This study uses MATLAB as a platform for algorithm implementation and a dataset of medical records of kidney disease patients collected from a hospital that specializes in treating kidney disease patients. The data pre-processing and artificial neural network modeling stages use the Nguyen Widrow algorithm, while the model training process uses the Back Propagation method. The results showed that the Nguyen Widrow algorithm was able to improve the accuracy of predicting someone suffering from kidney disease compared to using only the Back Propagation method. Analysis of the performance of the model shows a significant improvement in stability and convergence speed during the learning process. This indicates that data processing and medical decision making becomes more efficient. On the other hand, this research also studied the challenges and limitations that will be faced in terms of implementation of the Nguyen Widrow algorithm. Also the sensitivity of the initialization parameters, the need for the quality of the dataset to be used in training the model.This research reveals the ability of the Nguyen Widrow algorithm to improve the performance of artificial neural networks in diagnosing kidney disease. By implementing this algorithm in MATLAB, the results show that the use of the latest data processing technology and analysis tools can provide significant improvements in accuracy and efficiency in the medical field. In addition, this research is expected to provide a new direction in the development of machine learning algorithms for applications in the healthcare field, especially for diagnosing kidney disease. By further utilizing this technology, it contributes significantly to improving the quality of healthcare and treatment outcomes for patients suffering from kidney disease.

Phase transitions in the mini-batch size for sparse and dense two-layer neural networks

The use of mini-batches of data in training artificial neural networks is nowadays very common. Despite its broad usage, theories explaining quantitatively how large or small the optimal mini-batch size should be are missing. This work presents a systematic attempt at understanding the role of the mini-batch size in training two-layer neural networks. Working in the teacher-student scenario, with a sparse teacher, and focusing on tasks of different complexity, we quantify the effects of changing the mini-batch size m. We find that often the generalization performances of the student strongly depend on m and may undergo sharp phase transitions at a critical value mc , such that for m<mc the training process fails, while for m>mc the student learns perfectly or generalizes very well the teacher. Phase transitions are induced by collective phenomena firstly discovered in statistical mechanics and later observed in many fields of science. Observing a phase transition by varying the mini-batch size across different architectures raises several questions about the role of this hyperparameter in the neural network learning process.

A Comparative Study of Javanese Script Classification with GoogleNet, DenseNet, ResNet, VGG16 and VGG19

Purpose: Javanese script is a legacy of heritage or heritage in Indonesia originating from the island of Java needs to be preserved. Therefore, in this study, the classification and identification process of Javanese script letters will be carried out using the CNN method. The purpose of this research is to be able to build a model which can properly classify Javanese script, it can help in the process of recognizing letters in Javanese script easily.Methods: In this study, the Javanese script classification process has been used the transfer learning process of Convolutional Neural Network, namely GoogleNet, DenseNet, ResNet, VGG16 and VGG19. The purpose of using transfer learning is to improve the sequential CNN model, processing can be better and optimal because it utilizes a previously trained model.Result: The results obtained after testing in this study are using the transfer learning method, the GoogleNet model gets an accuracy of 88.75%, the DenseNet model gets an accuracy of 92%, the ResNet model gets an accuracy of 82.75%, the VGG16 model gets an accuracy of 99.25% and the VGG19 model gets an accuracy of 99.50%.Novelty: In previous studies, it is still very rare to discuss the Javanese script classification process using the CNN transfer learning method and which method is the most optimal for performing the Javanese script classification process. In this study, it had been resulted find an effective method to be able to carry out the Javanese script classification process properly and optimally.

COMPARISON OF NEURAL NETWORK ARCHITECTURES TO PREDICTING ELECTRICITY CONSUMPTION BY ENTERPRISE

Forecasting of electricity consumption is a key tool for enterprises, energy supply and power grid organizations. Accurate forecasting enables to plan the distribution of limited resources of the power grid facilities, as well as to manage the demand for electricity. In the context of modern demand management technologies, improving the accuracy of forecasting of electricity consumption becomes especially important.&#x0D; &#x0D; The purpose of the study is to improve the accuracy of predicting power consumption by the power supply object using neural networks.&#x0D; &#x0D; Materials and methods. The work used a data set containing a profile of the enterprise’s capacity for a three-month period, as well as additional data, such as time of day, day of the week, weekends and holidays, month. The data set is divided into training and control parts. Preliminary data processing, neural network architecture design, training and testing were carried out. The criteria for the quality of training are the mean absolute error, the mean square error and the coefficient of determination.&#x0D; &#x0D; Research results. In the work, a comparative analysis of three neural network architectures was performed: a one-dimensional convolutional network, a recurrent neural network with long-term and short-term memory, and WaveNet, on the basis of which their indicators of the quality of power consumption forecasting were evaluated. It was shown that all considered architectures of neural networks are suitable for the use in the issue of predicting power consumption. Long-term and short-term memory networks have shown good results in power prediction due to their ability to handle long-term time dependencies. The WaveNet architecture outperformed both long-term and short-term memory model-based recurrent neural networks and one-dimensional convolutional networks by selected criteria.&#x0D; &#x0D; Conclusions. The study led to the conclusion that the use of neural networks, especially architectures with long-term and short-term memory and WaveNet, is an effective approach for predicting power consumption. The quality of forecasting significantly depends on the choice of hyperparameters and preliminary processing of the initial data. Prospect for further research in this area is studying the influence of various factors on the accuracy of forecasting and optimization of the learning process of neural networks.

Simulation of the of the DeepLabv3 neural network learning process for the agricultural fields segmentation

Objective. Monitoring and determining the state of crops in agricultural production requires the use and improvement of neural network methods of artificial intelligence.The aim of the study is to create a mathematical model of the learning process of the DeepLabV3 neural network for intelligent analysis and segmentation of agricultural fields.Method. Based on the newly formed RGB database of images of agricultural fields, marked up into four classes, a neural network of the DeepLabV3 architecture was developed and trained. Approximations of the learning curve by the modified Johnson function are obtained by the methods of least squares and least modules.Result. A statistical assessment of the quality of training and approximation of neural networks to the DeepLabV3 architecture in combination with ResNet 50 was carried out. The constructed DNN family based on DeepLabV3 with ResNet50 showed the efficiency of recognition and sufficient speed in determining the state of crops.Conclusions. Approximation of the neural network learning diagram to the DeepLabV3 architecture, using a modified Johnson function, allows us to estimate the value of the “saturation” of the simulated dependence and predict the maximum value of the neural network metric without taking into account its possible retraining.

Проектирование прикладного программного продукта распознавания и оценки эмоций с использованием нейронной сети

During the analysis of the most common architectures for building neural networks for image analysis – a direct convolutional layer, a recurrent neural network, a convolutional neural network – the authors determined that the last architecture option for the task is the most suitable. The solutions available on the market, although they allow us to recognize emotions, do not evaluate them, and this is an important result for a wide variety of tasks, because such an assessment allows us to more accurately predict the possible future behavior of both individuals and groups of people. The purpose of this article is to design an application software product (application) and develop a prototype that could perform the functions of analyzing emotions and evaluating them. The objectives of this study include the choice of the architecture of the designed application; the development of an algorithm for the operation of the application; the design of the user interface; a description of the neural network learning process and its results, the model of which was defined in the previous article; demonstration of the prototype (control example). The scientific novelty of the projected application lies in the formation of an assessment of the psycho-emotional state of those people whose images of their faces were evaluated. Such an assessment with recommendations can be widely demanded in various branches of human activity, since it is important to be able to express emotions not only to ordinary people, but also to those whose job is to transfer their emotional state to others: business coaches, journalists, actors, animators, dancers, etc. The presence of emotional intelligence is very important at the moment time.

A unified approach of detecting phase transition in time-varying complex networks

Deciphering the non-trivial interactions and mechanisms driving the evolution of time-varying complex networks (TVCNs) plays a crucial role in designing optimal control strategies for such networks or enhancing their causal predictive capabilities. In this paper, we advance the science of TVCNs by providing a mathematical framework through which we can gauge how local changes within a complex weighted network affect its global properties. More precisely, we focus on unraveling unknown geometric properties of a network and determine its implications on detecting phase transitions within the dynamics of a TVCN. In this vein, we aim at elaborating a novel and unified approach that can be used to depict the relationship between local interactions in a complex network and its global kinetics. We propose a geometric-inspired framework to characterize the network’s state and detect a phase transition between different states, to infer the TVCN’s dynamics. A phase of a TVCN is determined by its Forman–Ricci curvature property. Numerical experiments show the usefulness of the proposed curvature formalism to detect the transition between phases within artificially generated networks. Furthermore, we demonstrate the effectiveness of the proposed framework in identifying the phase transition phenomena governing the training and learning processes of artificial neural networks. Moreover, we exploit this approach to investigate the phase transition phenomena in cellular re-programming by interpreting the dynamics of Hi-C matrices as TVCNs and observing singularity trends in the curvature network entropy. Finally, we demonstrate that this curvature formalism can detect a political change. Specifically, our framework can be applied to the US Senate data to detect a political change in the United States of America after the 1994 election, as discussed by political scientists.

An Enhanced Single-Particle Model Using a Physics-Informed Neural Network Considering Electrolyte Dynamics for Lithium-Ion Batteries

As power sources for electric vehicles, lithium-ion batteries (LIBs) have many advantages, such as high energy density and wide temperature range. In the algorithm design process for LIBs, various battery models with different model structures are needed, among which the electrochemical model is widely used due to its high accuracy. However, the electrochemical model is composed of multiple nonlinear partial differential equations (PDEs) that make the simulating process time-consuming. In this paper, a physics-informed neural network single-particle model (PINN SPM) is proposed to improve the accuracy of the single-particle model (SPM) under high C-rates, while ensuring high solving speed. In PINN SPM, an SPM-Net is designed to solve the distribution of lithium-ion concentration in the electrolyte. In the neural network learning process, a loss function is designed based on the physical constraints brought by the PDEs, which reduces the error of the neural network under dynamic working conditions. Finally, the PINN SPM proposed in this paper can achieve a maximum relative error of up to 1.2% compared with the high-fidelity data generated from the P2D model under various conditions. Additionally, the PINN SPM is 20.8% faster than traditional numerical solution methods with the same computational resources.

Heritage development of traditional culture in folk art education based on the decentralized Internet

Abstract Folk art education is an important way to inherit and develop traditional culture. In this paper, the cascade propagation of typical decentralized Internet-social networks is modeled as a propagation dynamic graph model, and an enhanced graph-aware neural network is proposed through the analysis of the learning process of neural graph networks. A further recurrent graph-aware neural network is proposed for the characteristics of information dissemination in the decentralized Internet, and the transmission and development of traditional culture in folk art education are analyzed based on this network model. In folk art education, the most common type of traditional culture dissemination is ink painting, accounting for 20.32%, which is 6.91%, 12.35%, and 14.86% higher than other types, respectively. From 2014 to 2021, the percentage of Internet-based communication media increased from 12.47% to 24.78%, an increase of 12.31 percentage points. The analysis based on the decentralized Internet can accurately extract the characteristics of traditional culture integrated into folk art education, which helps to inherit further and promote the excellent traditional culture.

An interpretable convolutional neural network with multi-wavelet kernel fusion for intelligent fault diagnosis

Deep learning (DL) has been widely developed and applied in gearbox fault diagnosis of smart manufacturing systems due to its powerful feature representation ability. However, typical DL-based fault diagnosis approaches do not correlate the domain information or mechanism knowledge into the neural network learning process. This causes issues for the reliability and stability of fault diagnosis engineering applications. To address this problem, we propose a novel interpretable DL model named Multi-Wavelet Kernel Convolution Neural Network (MWKCNN) for fault diagnosis in this article. First, a feature extraction layer named the Multi-Wavelet Kernel Convolution (MWKC) layer is constructed based on the continuous wavelet transform (CWT) to locate and detect the impulse signatures of faults. It can capture sufficient feature representations by involving signal processing knowledge. Then, to balance the influence of different wavelet kernels, we design a kernel weights recalibration (KWR) module to dynamically assign different weights to different wavelet kernels. Finally, the performance of the proposed method is validated using two gearbox datasets. In addition, we illustrate the interpretability of the proposed method by analyzing the peculiarity of different wavelet kernels, the variation of kernel weights, and the visualization of learned features.

Semantic modeling of cell damage prediction: a machine learning approach at human-level performance in dermatology

Machine learning is transforming the field of histopathology. Especially in classification related tasks, there have been many successful applications of deep learning already. Yet, in tasks that rely on regression and many niche applications, the domain lacks cohesive procedures that are adapted to the learning processes of neural networks. In this work, we investigate cell damage in whole slide images of the epidermis. A common way for pathologists to annotate a score, characterizing the degree of damage for these samples, is the ratio between healthy and unhealthy nuclei. The annotation procedure of these scores, however, is expensive and prone to be noisy among pathologists. We propose a new measure of damage, that is the total area of damage, relative to the total area of the epidermis. In this work, we present results of regression and segmentation models, predicting both scores on a curated and public dataset. We have acquired the dataset in collaborative efforts with medical professionals. Our study resulted in a comprehensive evaluation of the proposed damage metrics in the epidermis, with recommendations, emphasizing practical relevance for real world applications.

Pre-Design of Multi-Band Planar Antennas by Artificial Neural Networks

In this communication, artificial neural networks are used to estimate the initial structure of a multiband planar antenna. The neural networks are trained on a set of selected normalized multiband antennas characterized by time-efficient modal analysis with limited accuracy. Using the Deep Learning Toolbox in Matlab, several types of neural networks have been created and trained on the sample planar multiband antennas. In the neural network learning process, suitable network types were selected for the design of these antennas. The trained networks, depending on the desired operating bands, will select the appropriate antenna geometry. This is further optimized using Newton’s method in HFSS. The use of the neural pre-design concept speeds up and simplifies the design of multiband planar antennas. The findings presented in this paper will be used to refine and accelerate the design of planar multiband antennas.

Decentralized Resource Allocation-Based Multiagent Deep Learning in Vehicular Network

Resource allocation (RA) has a significant impact on vehicular network performance. With high mobility, RA is more challenging, as the number of vehicles in close proximity changes dynamically in the nonstationary environment. In this article, we propose a multiagent double deep Q-networks scheme to stabilize the system and maximize the sum-capacity of the vehicle-to-infrastructure (V2I) links, while satisfying the reliability and delay constraints for vehicle-to-vehicle (V2V) links. To avoid interference caused by unstable V2V links, a transmission mode selection is considered in the scheme design. In addition, we introduce a binarized weight algorithm to accelerate the deep neural network learning process and, therefore, improve the computational complexity of our scheme. Through extensive simulations and complexity analysis, we demonstrate that the proposed scheme yields excellent performance in terms of the sum-rate and probability rate of V2I and V2V communication modes. We also compare the proposed scheme with binarized weights with other algorithms in terms of accuracy evaluation.

Adaptive critic learning for event‐triggered safe control of nonlinear safety‐critical systems

AbstractIn this paper, an event‐triggered safe control method based on adaptive critic learning (ACL) is proposed for a class of nonlinear safety‐critical systems. First, a safe cost function is constructed by adding a control barrier function (CBF) to the traditional quadratic cost function; the optimization problem with safety constraints that is difficult to deal with by classical ACL methods is solved. Subsequently, the event‐triggered scheme is introduced to reduce the amount of computation. Further, combining the properties of CBF with the ACL‐based event‐triggering mechanism, the event‐triggered safe Hamilton–Jacobi–Bellman (HJB) equation is derived, and a single critic neural network (NN) framework is constructed to approximate the solution of the event‐triggered safe HJB equation. In addition, the concurrent learning method is applied to the NN learning process, so that the persistence of excitation (PE) condition is not required. The weight approximation error of the NN and the states of the system are proven to be uniformly ultimately bounded (UUB) in the safe set with the Lyapunov theory. Finally, the availability of the presented method can be validated through the simulation.

Improved Cascade Correlation Neural Network Model Based on Group Intelligence Optimization Algorithm

The Cascade Correlation learning algorithm is a special supervised learning algorithm for artificial neural network architecture. The optimization algorithm in the traditional neural network has the disadvantages of a single optimization goal, slow convergence speed, and can easily fall into local area, which cannot fully meet the key elements in the cascade correlation learning algorithm. In comparison, the group intelligence optimization algorithm can take into account these key elements in the optimization process at the same time, and obtain better optimization results. In this paper, we propose the single-objective optimization algorithm jDE-B and the multi-objective optimization algorithm MOEA-T, and improve the network expansion mode in the learning process of Cascade Correlation neural networks. We investigate the effect of applying the group intelligent optimization algorithm in the Cascade Correlation learning algorithm. Experimental results show that our improved algorithm is able to enhance the ability of the Cascade Correlation neural network to fit problems, reduce the number of hidden units and the depth of the network, and optimize the network structure.

Evaluation of the effectiveness of training and the use of neural networks to predict biological age.

Neural network training is widely used in various educational fields: staff training, mastering the curriculum at school and university, forming recommendations for private use by respondents, for teaching pensioners health-saving techniques. It is relevant to analyze the learning process of neural networks and evaluate their effectiveness on various models. For the study, a model was chosen for predicting the index of biological aging of a person based on the characteristics of his personality. To train neural networks, a data matrix of 1,632 people aged 35 to 70 years was compiled. Output variable: biological aging index; input variables: gender, age, family and professional status, place of residence, body type, type of functional asymmetry, style of relationships with people, as well as personal resources. Four neural networks were trained: for men and women, for working professionals and for pensioners. Results: 1) trained neural networks give different recommendations for men and women of pre-retirement and post-retirement age. 2) The effectiveness of predicting the biological aging index using neural networks turned out to be significantly high for all trained programs. 3) Neural networks can be used to model various social situations and identify what changes this will lead to for output variables. Situations were modeled: a) if all single adults get married, b) if all family adults break up, c) if everyone receives the recommendations of psychologists on the selection of personal resources and will use them. The neural network has issued a forecast: it is better for adult women not to change their family status/it is better for adult men to change their status. The use of personal resources selected by psychologists is effective for everyone.

HOW IMPORTANT ARE ACTIVATION FUNCTIONS IN REGRESSION AND CLASSIFICATION? A SURVEY, PERFORMANCE COMPARISON, AND FUTURE DIRECTIONS

Inspired by biological neurons, the activation functions play an essential part in the learning process of any artificial neural network (ANN) commonly used in many real-world problems. Various activation functions have been proposed in the literature for classification as well as regression tasks. In this work, we survey the activation functions that have been employed in the past as well as the current state-of-the-art. In particular, we present various developments in activation functions over the years and the advantages as well as disadvantages or limitations of these activation functions. We also discuss classical (fixed) activation functions, including rectifier units, and adaptive activation functions. In addition to discussing the taxonomy of activation functions based on characterization, a taxonomy of activation functions based on applications is presented. To this end, the systematic comparison of various fixed and adaptive activation functions is performed for classification datasets such as MNIST, CIFAR-10, and CIFAR-100. In recent years, a physics-informed machine learning framework has emerged for solving problems related to scientific computations. For this purpose, we also discuss various requirements for activation functions that have been used in the physics-informed machine learning framework. Furthermore, various comparisons are made among different fixed and adaptive activation functions using various machine learning libraries such as TensorFlow, PyTorch, and JAX. Our findings show that activation functions such as rectified linear unit (ReLU) and its variants, which are currently the state-of-the-art for many classification problems, do not work well in physics-informed machine learning frameworks due to the stringent requirement of the existence of derivatives, whereas other activation functions such as hyperbolic tangent, swish, and sine give better performance, with superior results achieved with adaptive activation functions, especially for multiscale problems.

Training Learning Weights of Elman Neural Network Using Salp Swarm Optimization Algorithm

Elman Neural Network (ENN) is one of the most common type of recurrent neural network (RNN). It is frequently applied to numerous applications such as classification and predication. However, the ENN model, suffers from serious flaws like network stagnancy and delayed convergence during training. To enhance the effectiveness of the ENN and address the mention problems, numerous studies have been conducted to speed up the learning process of ENN. Many optimization techniques are being used and each algorithm has its own characteristics in terms of efficiency. Some of the algorithms inspire from nature, while others come from the swarm's combined behaviour. Different numbers of optimization methods, including Particle Swarm Optimization (PSO), Bat Algorithm, and Cuckoo Search (CS) are applied to improve the ENN's learning process. As a result, this study suggested using Salp Swarm algorithm (SSA) to address these issues and speed up the ENN algorithm's learning process. The SSA, which is based on the typical behaviour of Salp swarms, is proposed in this research as a novel optimization technique. Due to its effectiveness and Salp intelligent behaviour, a novel Salp algorithm improves the learning process of the artificial neural network (ANN), and ENN models. Therefore this study proposed Salp swarm artificial neural network (SSANN) and Salp swarm Elman neural network (SSElmanNN). Performance of the suggested models is evaluated against ANN, back propagation neural network (BPNN), along with ENN in term of accuracy and Mean Square Error (MSE). Two datasets such as IRIS and Credit Card are used for simulation. The simulation results demonstrate that the suggested models outperform the other algorithms used in this study in terms of MSE and accuracy.

A scale conjugate neural network learning process for the nonlinear malaria disease model

&lt;abstract&gt; &lt;p&gt;The purpose of this work is to provide a stochastic framework based on the scale conjugate gradient neural networks (SCJGNNs) for solving the malaria disease model of pesticides and medication (MDMPM). The host and vector populations are divided in the mathematical form of the malaria through the pesticides and medication. The stochastic SCJGNNs procedure has been presented through the supervised neural networks based on the statics of validation (12%), testing (10%), and training (78%) for solving the MDMPM. The optimization is performed through the SCJGNN along with the log-sigmoid transfer function in the hidden layers along with fifteen numbers of neurons to solve the MDMPM. The accurateness and precision of the proposed SCJGNNs is observed through the comparison of obtained and source (Runge-Kutta) results, while the small calculated absolute error indicate the exactitude of designed framework based on the SCJGNNs. The reliability and consistency of the SCJGNNs is observed by using the process of correlation, histogram curves, regression, and function fitness.&lt;/p&gt; &lt;/abstract&gt;