Neural Network Training Research Articles

Hardware implementation of backpropagation using progressive gradient descent for in situ training of multilayer neural networks.

Neural network training can be slow and energy-expensive due to the frequent transfer of weight data between digital memory and processing units. Neuromorphic systems can accelerate neural networks by performing multiply-accumulate operations in parallel using nonvolatile analog memory. However, executing the widely used backpropagation training algorithm in multilayer neural networks requires information-and therefore storage-of the partial derivatives of the weight values preventing suitable and scalable implementation in hardware. Here, we propose a hardware implementation of the backpropagation algorithm that progressively updates each layer using in situ stochastic gradient descent, avoiding this storage requirement. We experimentally demonstrate the in situ error calculation and the proposed progressive backpropagation method in a multilayer hardware-implemented neural network. We confirm identical learning characteristics and classification performance compared to conventional backpropagation in software. We show that our approach can be scaled to large and deep neural networks, enabling highly efficient training of advanced artificial intelligence computing systems.

Learning Feynman integrals from differential equations with neural networks

We perform an exploratory study of a new approach for evaluating Feynman integrals numerically. We apply the recently-proposed framework of physics-informed deep learning to train neural networks to approximate the solution to the differential equations satisfied by the Feynman integrals. This approach relies neither on a canonical form of the differential equations, which is often a bottleneck for the analytical techniques, nor on the availability of a large dataset, and after training yields essentially instantaneous evaluation times. We provide a proof-of-concept implementation within the PyTorch framework, and apply it to a number of one- and two-loop examples, achieving a mean magnitude of relative difference of around 1% at two loops in the physical phase space with network training times on the order of an hour on a laptop GPU.

Gradient-free training of recurrent neural networks using random perturbations

Recurrent neural networks (RNNs) hold immense potential for computations due to their Turing completeness and sequential processing capabilities, yet existing methods for their training encounter efficiency challenges. Backpropagation through time (BPTT), the prevailing method, extends the backpropagation (BP) algorithm by unrolling the RNN over time. However, this approach suffers from significant drawbacks, including the need to interleave forward and backward phases and store exact gradient information. Furthermore, BPTT has been shown to struggle to propagate gradient information for long sequences, leading to vanishing gradients. An alternative strategy to using gradient-based methods like BPTT involves stochastically approximating gradients through perturbation-based methods. This learning approach is exceptionally simple, necessitating only forward passes in the network and a global reinforcement signal as feedback. Despite its simplicity, the random nature of its updates typically leads to inefficient optimization, limiting its effectiveness in training neural networks. In this study, we present a new approach to perturbation-based learning in RNNs whose performance is competitive with BPTT, while maintaining the inherent advantages over gradient-based learning. To this end, we extend the recently introduced activity-based node perturbation (ANP) method to operate in the time domain, leading to more efficient learning and generalization. We subsequently conduct a range of experiments to validate our approach. Our results show similar performance, convergence time and scalability when compared to BPTT, strongly outperforming standard node perturbation and weight perturbation methods. These findings suggest that perturbation-based learning methods offer a versatile alternative to gradient-based methods for training RNNs which can be ideally suited for neuromorphic computing applications.

Gradient-free training of recurrent neural networks using random perturbations

Gradient-free training of recurrent neural networks using random perturbations

Experimental and artificial intelligence optimization of paint wastewater (PWW) coagulation using novel Phaseolus vulgaris seed extract (PVSE)

This study investigated the application of artificial intelligence algorithms (AIA) in the coagulation treatment of paint wastewater anchored by novel Phaseolus vulgaris seed extract (PVSE). Untreated wastewater discharge harms the ecosystem, and therefore harmful industrial effluent, such as paint wastewater, must be brought to safe discharge levels before being released into the environment. In addition to AIA, comprehensive characterization tests, coagulation kinetics, and process optimization were also executed. Characterization results revealed that total solid in the PWW was above allowable standard, justifying the need for effective particle decontamination. The XRD and FTIR characterization indicated that PVSE structure is amorphous with abundant amine groups. Results of analysis of variance (ANOVA) obtained from process modeling indicated that the coagulation-flocculation process was a nonlinear quadratic system (F-value = 45.51) which was mostly influenced by PVSE coagulant dosage (F-value = 222.48; standardized effect = 14.85). Artificial intelligence indicated that neural network training effectively captured the nonlinear nature of the system in ANN (RMSE = 0.00040194; R = 0.98497), and ANFIS (RMSE = 0.003961) algorithms. Regression coefficient obtained from process modeling highlighted the suitability of RSM (0.9662), ANN (0.9739), and ANFIS (0.9718) in forecasting the coagulation-flocculation process, while comparative statistical appraisal authenticated the superiority of ANN model over RSM and ANFIS models. The coagulation kinetics experiment, which used a coagulation kinetic model, revealed a constant flocculation constant (Kf-value) for all jar test batches and a strong association between the Menkonu coagulation-flocculation constant (Km) and Kf values. Best removal efficiency of 97.01 % was obtained using ANN coupled genetic algorithm optimization (ANN-GA) at PVSE dosage of 4 g/L, coagulation time of 29 min and temperature of 25.1oC.

Proton spot dose estimation based on positron activity distributions with neural network.

Positron emission tomography (PET) has been investigated for its ability to reconstruct proton-induced positron activity distributions in proton therapy. This technique holds potential for range verification in clinical practice. Recently, deep learning-based dose estimation from positron activity distributions shows promise for in vivo proton dose monitoring and guided proton therapy. This study evaluates the effectiveness of three classical neural network models, recurrent neural network (RNN), U-Net, and Transformer, for proton dose estimating. It also investigates the characteristics of these models, providing valuable insights for selecting the appropriate model in clinical practice. Proton dose calculations for spot beams were simulated using Geant4. Computed tomography (CT) images from four head cases were utilized, with three for training neural networks and the remaining one for testing. The neural networks were trained with one-dimensional (1D) positron activity distributions as inputs and generated 1D dose distributions as outputs. The impact of the number of training samples on the networks was examined, and their dose prediction performance in both homogeneous brain and heterogeneous nasopharynx sites was evaluated. Additionally, the effect of positron activity distribution uncertainty on dose prediction performance was investigated. To quantitatively evaluate the models, mean relative error (MRE) and absolute range error (ARE) were used as evaluation metrics. The U-Net exhibited a notable advantage in range verification with a smaller number of training samples, achieving approximately 75% of AREs below 0.5mm using only 500 training samples. The networks performed better in the homogeneous brain site compared to the heterogeneous nasopharyngeal site. In the homogeneous brain site, all networks exhibited small AREs, with approximately 90% of the AREs below 0.5mm. The Transformer exhibited the best overall dose distribution prediction, with approximately 92% of MREs below 3%. In the heterogeneous nasopharyngeal site, all networks demonstrated acceptable AREs, with approximately 88% of AREs below 3mm. The Transformer maintained the best overall dose distribution prediction, with approximately 85% of MREs below 5%. The performance of all three networks in dose prediction declined as the uncertainty of positron activity distribution increased, and the Transformer consistently outperformed the other networks in all cases. Both the U-Net and the Transformer have certain advantages in the proton dose estimation task. The U-Net proves well suited for range verification with a small training sample size, while the Transformer outperforms others at dose-guided proton therapy.

Proton spot dose estimation based on positron activity distributions with neural network.

Positron emission tomography (PET) has been investigated for its ability to reconstruct proton-induced positron activity distributions in proton therapy. This technique holds potential for range verification in clinical practice. Recently, deep learning-based dose estimation from positron activity distributions shows promise for in vivo proton dose monitoring and guided proton therapy. This study evaluates the effectiveness of three classical neural network models, recurrent neural network (RNN), U-Net, and Transformer, for proton dose estimating. It also investigates the characteristics of these models, providing valuable insights for selecting the appropriate model in clinical practice. Proton dose calculations for spot beams were simulated using Geant4. Computed tomography (CT) images from four head cases were utilized, with three for training neural networks and the remaining one for testing. The neural networks were trained with one-dimensional (1D) positron activity distributions as inputs and generated 1D dose distributions as outputs. The impact of the number of training samples on the networks was examined, and their dose prediction performance in both homogeneous brain and heterogeneous nasopharynx sites was evaluated. Additionally, the effect of positron activity distribution uncertainty on dose prediction performance was investigated. To quantitatively evaluate the models, mean relative error (MRE) and absolute range error (ARE) were used as evaluation metrics. The U-Net exhibited a notable advantage in range verification with a smaller number of training samples, achieving approximately 75% of AREs below 0.5mm using only 500 training samples. The networks performed better in the homogeneous brain site compared to the heterogeneous nasopharyngeal site. In the homogeneous brain site, all networks exhibited small AREs, with approximately 90% of the AREs below 0.5mm. The Transformer exhibited the best overall dose distribution prediction, with approximately 92% of MREs below 3%. In the heterogeneous nasopharyngeal site, all networks demonstrated acceptable AREs, with approximately 88% of AREs below 3mm. The Transformer maintained the best overall dose distribution prediction, with approximately 85% of MREs below 5%. The performance of all three networks in dose prediction declined as the uncertainty of positron activity distribution increased, and the Transformer consistently outperformed the other networks in all cases. Both the U-Net and the Transformer have certain advantages in the proton dose estimation task. The U-Net proves well suited for range verification with a small training sample size, while the Transformer outperforms others at dose-guided proton therapy.

Gasoline Engine Misfire Fault Diagnosis Method Based on Improved YOLOv8

In order to realize the online diagnosis and prediction of gasoline engine fire faults, this paper proposes an improved misfire fault detection algorithm model based on YOLOv8 for sound signals of gasoline engines. The improvement involves substituting a C2f module in the YOLOv8 backbone network by a BiFormer attention module and another C2f module substituted by a CBAM module that combines channel and spatial attention mechanisms which enhance the neural network’s capacity to extract the complex features. The normal and misfire sound signals of a gasoline engine are processed by wavelet transformation and converted to time–frequency images for the training, verification, and testing of convolutional neural network. The experimental results show that the precision of the improved YOLOv8 algorithm model is 99.71% for gasoline engine fire fault tests, which is 2 percentage points higher than for the YOLOv8 network model. The diagnosis time of each sound is less than 100 ms, making it suitable for developing IoT devices for gasoline engine misfire fault diagnosis and driverless vehicles.

Innovative infrastructure to access Brazilian fungal diversity using deep learning.

In the present investigation, we employ a novel and meticulously structured database assembled by experts, encompassing macrofungi field-collected in Brazil, featuring upwards of 13,894 photographs representing 505 distinct species. The purpose of utilizing this database is twofold: firstly, to furnish training and validation for convolutional neural networks (CNNs) with the capacity for autonomous identification of macrofungal species; secondly, to develop a sophisticated mobile application replete with an advanced user interface. This interface is specifically crafted to acquire images, and, utilizing the image recognition capabilities afforded by the trained CNN, proffer potential identifications for the macrofungal species depicted therein. Such technological advancements democratize access to the Brazilian Funga, thereby enhancing public engagement and knowledge dissemination, and also facilitating contributions from the populace to the expanding body of knowledge concerning the conservation of macrofungal species of Brazil.

Quantum Extreme Learning of molecular potential energy surfaces and force fields

Abstract Quantum machine learning algorithms are expected to play a pivotal role in quantum chemistry simulations in the immediate future. One of such key applications is the training of a quantum neural network to learn the potential energy surface and force field of molecular systems. We address this task by using the quantum extreme learning machine paradigm. This particular supervised learning routine allows for a resource efficient training, consisting of a simple linear regression performed on a classical computer. We have tested a setup that can be used to study molecules of any dimension and optimized for immediate use on NISQ devices with a limited number of native gates. We have applied this setup to three case studies: lithium hydride, water, and formamide, carrying out both noiseless simulations and actual implementation on IBM quantum hardware. Compared to other supervised learning routines, the proposed setup requires minimal quantum resources, making it feasible for direct implementation on quantum platforms, while still achieving a high level of predictive accuracy compared to simulations. Our encouraging results pave the way towards the future application to more complex molecules, being the proposed setup scalable.

Quantum Extreme Learning of molecular potential energy surfaces and force fields

Abstract Quantum machine learning algorithms are expected to play a pivotal role in quantum chemistry simulations in the immediate future. One of such key applications is the training of a quantum neural network to learn the potential energy surface and force field of molecular systems. We address this task by using the quantum extreme learning machine paradigm. This particular supervised learning routine allows for a resource efficient training, consisting of a simple linear regression performed on a classical computer. We have tested a setup that can be used to study molecules of any dimension and optimized for immediate use on NISQ devices with a limited number of native gates. We have applied this setup to three case studies: lithium hydride, water, and formamide, carrying out both noiseless simulations and actual implementation on IBM quantum hardware. Compared to other supervised learning routines, the proposed setup requires minimal quantum resources, making it feasible for direct implementation on quantum platforms, while still achieving a high level of predictive accuracy compared to simulations. Our encouraging results pave the way towards the future application to more complex molecules, being the proposed setup scalable.

SURE: A Visualized Failure Indexing Approach using Program Memory Spectrum

Failure indexing is a longstanding crux in software debugging, the goal of which is to automatically divide failures (e.g., failed test cases) into distinct groups according to the culprit root causes, as such multiple faults residing in a faulty program can be handled independently and simultaneously. The community of failure indexing has long been plagued by two challenges: 1) The effectiveness of division is still far from promising. Specifically, existing failure indexing techniques only employ a limited source of software run-time data, for example, code coverage, to be failure proximity and further divide them, which typically delivers unsatisfactory results. 2) The outcome can be hardly comprehensible. Specifically, a developer who receives the division result is just aware of how all failures are divided, without knowing why they should be divided the way they are. This leads to difficulties for developers to be convinced by the division result, which in turn affects the adoption of the results. To tackle these two problems, in this paper, we propose SURE, a vi SU alized failu R e ind E xing approach using the program memory spectrum. We first collect the run-time memory information (i.e., variables’ names and values, as well as the depth of the stack frame) at several preset breakpoints during the execution of a failed test case, and transform the gathered memory information into a human-friendly image (called program memory spectrum, PMS). Then, any pair of PMS images that serve as proxies for two failures is fed to a trained Siamese convolutional neural network, to predict the likelihood of them being triggered by the same fault. Last, a clustering algorithm is adopted to divide all failures based on the mentioned likelihood. In the experiments, we use 30% of the simulated faults to train the neural network, and use 70% of the simulated faults as well as real-world faults to test. Results demonstrate the effectiveness of SURE: It achieves 101.20% and 41.38% improvements in faults number estimation, as well as 105.20% and 35.53% improvements in clustering, compared with the state-of-the-art technique in this field, in simulated and real-world environments, respectively. Moreover, we carry out a human study to quantitatively evaluate the comprehensibility of PMS, revealing that this novel type of representation can help developers better comprehend failure indexing results.

Power-enhanced residual network for function approximation and physics-informed inverse problems

In this study, we investigate how the updating of weights during forward operation and the computation of gradients during backpropagation impact the optimization process, training procedure, and overall performance of the neural network, particularly the multi-layer perceptrons (MLPs). This paper introduces a novel neural network structure called the Power-Enhancing residual network, inspired by highway network and residual network, designed to improve the network's capabilities for both smooth and non-smooth functions approximation in 2D and 3D settings. By incorporating power terms into residual elements, the architecture enhances the stability of weight updating, thereby facilitating better convergence and accuracy. The study explores network depth, width, and optimization methods, showing the architecture's adaptability and performance advantages. Consistently, the results emphasize the exceptional accuracy of the proposed Power-Enhancing residual network, particularly for non-smooth functions. Real-world examples also confirm its superiority over plain neural network in terms of accuracy, convergence, and efficiency. Moreover, the proposed architecture is also applied to solving the inverse Burgers' equation, demonstrating superior performance. In conclusion, the Power-Enhancing residual network offers a versatile solution that significantly enhances neural network capabilities by emphasizing the importance of stable weight updates for effective training in deep neural networks. The codes implemented are available at: https://github.com/CMMAi/ResNet_for_PINN.

Missing genotype imputation in non-model species using self-organizing maps.

Current methodologies of genome-wide single-nucleotide polymorphism (SNP) genotyping produce large amounts of missing data that may affect statistical inference and bias the outcome of experiments. Genotype imputation is routinely used in well-studied species to buffer the impact in downstream analysis, and several algorithms are available to fill in missing genotypes. The lack of reference haplotype panels precludes the use of these methods in genomic studies on non-model organisms. As an alternative, machine learning algorithms are employed to explore the genotype data and to estimate the missing genotypes. Here, we propose an imputation method based on self-organizing maps (SOM), a widely used neural networks formed by spatially distributed neurons that cluster similar inputs into close neurons. The method explores genotype datasets to select SNP loci to build binary vectors from the genotypes, and initializes and trains neural networks for each query missing SNP genotype. The SOM-derived clustering is then used to impute the best genotype. To automate the imputation process, we have implemented gtImputation, an open-source application programmed in Python3 and with a user-friendly GUI to facilitate the whole process. The method performance was validated by comparing its accuracy, precision and sensitivity on several benchmark genotype datasets with other available imputation algorithms. Our approach produced highly accurate and precise genotype imputations even for SNPs with alleles at low frequency and outperformed other algorithms, especially for datasets from mixed populations with unrelated individuals.

Artificial intelligence vision technology application in sustainability evaluation of solar-driven distillation device

The process of transforming brackish water into freshwater utilizing solar thermal energy is referred to as solar-driven distillation device, or solar still. Such device without fossil fuel provides significant environmental and health benefits by reducing air pollutants and remedying brackish water. The yield of purified water from conventional solar stills remains insufficient, prompting the necessity for research to enhance it by understanding the coupling effect between steam flow, heat transfer, and mass transfer. As such, computer-aided brackish water treatment comparison of the hemispherical solar still and other multi-slope solar stills is conducted in this paper, and an automatic steam&heat flow detection algorithm is developed to solve the problem of difficult data acquisition for gas-liquid transport processes. Firstly, double slope, four slope, and hemispherical solar stills are exposed to the sun in outdoor experiments, therefore the solar thermal performance of each still is analyzed through pairwise comparisons. Secondly, a large amount of experimental image data is input into neural network for training, and by continuously adjusting network parameters, the network can accurately recognize different types of images. Finally, the image to be recognized is input into the trained neural network, which outputs the category labels of the image to achieve automatic image recognition. Based on the data above, the best structure of solar-driven device is hemispherical structure, due to that the hemispherical solar still possesses the best performance in terms of distilled water yield, energy efficiency, exergy efficiency and energy payback.

Review of few-shot learning application in CSI human sensing

Wi-Fi sensing has garnered increasing interest for its significant advantages, primarily leveraging Wi-Fi signal fluctuations induced by human activities and advanced neural network algorithms. However, its application faces challenges due to limited generalizability, necessitating frequent data recollection and neural network retraining for adaptation to new environments. To address these limitations, some researchers introduced few-shot learning into Wi-Fi sensing applications because it offers a promising solution with its ability to achieve remarkable performance in novel scenarios using minimal training samples. Despite its potential, a comprehensive review of its applications within this domain remains absent. This study endeavors to fill this gap by exploring prominent Wi-Fi sensing applications that incorporate few-shot learning, aiming to delineate their key features. We categorize few-shot learning approaches into three distinct methodologies: transfer learning, metric learning, and meta-learning, based on their neural network training strategies. Through this classification, we examine representative systems from an application perspective and elucidate the principles of few-shot learning implementation. These systems are evaluated in terms of learning methodology, data modality, and recognition accuracy. Finally, this paper highlights the challenges and future directions for few-shot learning in Channel State Information (CSI) based human sensing, providing a valuable resource for researchers in the field of Wi-Fi human sensing leveraging few-shot learning.

The R-mAtrIx Net

We provide a novel neural network architecture that can: i) output R-matrix for a given quantum integrable spin chain, ii) search for an integrable Hamiltonian and the corresponding R-matrix under assumptions of certain symmetries or other restrictions, iii) explore the space of Hamiltonians around already learned models and reconstruct the family of integrable spin chains which they belong to. The neural network training is done by minimizing loss functions encoding Yang–Baxter equation, regularity and other model-specific restrictions such as hermiticity. Holomorphy is implemented via the choice of activation functions. We demonstrate the work of our neural network on the spin chains of difference form with two-dimensional local space. In particular, we reconstruct the R-matrices for all 14 classes. We also demonstrate its utility as an Explorer, scanning a certain subspace of Hamiltonians and identifying integrable classes after clusterisation. The last strategy can be used in future to carve out the map of integrable spin chains with higher dimensional local space and in more general settings where no analytical methods are available.

Discontinuous permafrost detection from neural network-ensemble learning based electrical resistivity tomography

Electrical resistivity tomography (ERT) is an effective method for detecting the distribution of permafrost. However, the general inversion method of ERT cannot satisfy the engineering designation demand, resulting in the foundation of thaw settlement in discontinuous permafrost regions. In this study, we proposed a neural network-ensemble learning inversion method to improve the detection accuracy of discontinuous permafrost. First, a series of different resistivity distributions was evaluated to establish forward models for the training of a backpropagation neural network (BPNN). The resistivity distributions of the forward models varied with the temperature gradient, similar to the resistivity distribution of real discontinuous permafrost. The bagging algorithm of ensemble learning was then used to optimize the BPNN inversion models. Finally, three discontinuous permafrost resistivity models and two field data examples are considered to demonstrate the feasibility of the proposed inversion model. The inversion results of synthetic and field examples show that the neural network-ensemble learning model achieved a greater inversion effect with better accuracy and less noisy points than a single BPNN model or the Res2Dinv method. The trained ensemble learning inversion method has good application in field permafrost exploration.

Neural network architecture search model for thermal radiation in dense particulate systems

In the CFD-DEM simulation of the many dense particulate systems, particle-scale thermal radiation is an important heat transfer mode under high temperatures. In this work, neural network architecture search model with different layer connection matrix is applied to the view factor regression of the thermal radiation in dense particulate systems. Deep neural networks with 3346 feasible architectures are evaluated by the HyperBand algorithm to find the local optimal solution. Neural architecture search model trained by the big data of the view factor gives a good prediction of the macroscopic radiative properties and it is in general agreement with the empirical correlations and experimental data. The particle–wall radiation decreases strongly with the distance and the maximum interaction depth is about 2.0 times the sphere diameter. The trained deep neural network model provides an efficient data-driven closure to discuss the thermal radiation of the particle–particle and particle–wall interactions in particle bed.

Enhancing deep neural network training efficiency and performance through linear prediction

Deep neural networks have achieved remarkable success in various fields. However, training an effective deep neural network still poses challenges. This paper aims to propose a method to optimize the training effectiveness of deep neural networks, with the goal of improving their performance. Firstly, based on the observation that parameters (weights and bias) of deep neural network change in certain rules during training process, the potential of parameters prediction for improving training efficiency is discovered. Secondly, the potential of parameters prediction to improve the performance of deep neural network by noise injection introduced by prediction errors is revealed. And then, considering the limitations comprehensively, a deep neural network Parameters Linear Prediction method is exploit. Finally, performance and hyperparameter sensitivity validations are carried out on some representative backbones. Experimental results show that by employing proposed Parameters Linear Prediction method, as opposed to SGD, has led to an approximate 1% increase in accuracy for optimal model, along with a reduction of about 0.01 in top-1/top-5 error. Moreover, it also exhibits stable performance under various hyperparameter settings, shown the effectiveness of the proposed method and validated its capacity in enhancing network’s training efficiency and performance.