Neural Network Training Research Articles

Training all-mechanical neural networks for task learning through in situ backpropagation

Recent advances unveiled physical neural networks as promising machine learning platforms, offering faster and more energy-efficient information processing. Compared with extensively-studied optical neural networks, the development of mechanical neural networks remains nascent and faces significant challenges, including heavy computational demands and learning with approximate gradients. Here, we introduce the mechanical analogue of in situ backpropagation to enable highly efficient training of mechanical neural networks. We theoretically prove that the exact gradient can be obtained locally, enabling learning through the immediate vicinity, and we experimentally demonstrate this backpropagation to obtain gradient with high precision. With the gradient information, we showcase the successful training of networks in simulations for behavior learning and machine learning tasks, achieving high accuracy in experiments of regression and classification. Furthermore, we present the retrainability of networks involving task-switching and damage, demonstrating the resilience. Our findings, which integrate the theory for training mechanical neural networks and experimental and numerical validations, pave the way for mechanical machine learning hardware and autonomous self-learning material systems.

Multi-optimization scheme for in-situ training of memristor neural network based on contrastive learning

Multi-optimization scheme for in-situ training of memristor neural network based on contrastive learning

A Machine Learning-Based Approach for Predicting Aerodynamic Coefficients Using Deep Neural Networks and CFD Data

Artificial Intelligence (AI) and Machine Learning (ML) are increasingly being adopted across various fields, including aerodynamics, exhibiting impressive results in complex computational processes and improving prediction accuracy. This study introduces a novel method for airfoil performance assessment through the development and training of a deep Artificial Neural Network (ANN), used for predicting aerodynamic coefficients and pressure distributions, leveraging comprehensive data obtained by using a Computational Fluid Dynamics (CFD) solver. First, an automated CFD solver was developed for obtaining the extensive dataset needed for the effective training of the ANN. The automation process consisted in the generation of a geometry and a mesh, along with the successful integration of the open-source SU2 solver for conducting the aerodynamic simulations, chosen for its versatility and straightforward integration. Once various airfoil analyses were performed and a comprehensive dataset was obtained, data was normalized and the model was trained. Throughout the training process, several model configurations were tested, varying different architectures, hyperparameters and layer settings, until the best-performing layout was chosen. After broad testing and validation, the optimal configuration was identified as being the one to demonstrate the lowest error rates and the most accurate predictions on both training and unseen data, highlighting the model’s generalization capabilities. This Machine Learning-based approach, used as a substitute for traditional methods, provides remarkable accuracy and robustness, capturing complex behaviors and significantly reducing the computational costs associated with CFD simulations.

Minimizing Communication Overhead in Decentralized Deep Neural Networks

Minimizing communication overhead in decentralized deep neural network (DNN) training has become a critical challenge, particularly with the increasing adoption of distributed systems for large-scale machine learning tasks. This paper introduces advanced techniques that leverage gradient compression, adaptive sparsification, and hybrid aggregation to optimize communication efficiency while maintaining model accuracy and convergence rates. Experimental results on benchmark datasets such as CIFAR-10 and ImageNet show that the proposed methods reduce communication costs by up to 70% compared to standard approaches while achieving comparable or superior model accuracy. Additionally, scalability tests on diverse neural network architectures highlight the robustness of the approach, demonstrating efficient performance across varying network sizes and computational setups. These findings underscore the potential of the proposed strategies to enable faster, cost-effective, and sustainable decentralized deep learning systems.

Label flipping attacks in hierarchical federated learning for intrusion detection in IoT

ABSTRACT Federated learning (FL) is a promising approach for distributed training of deep neural networks within Internet of Things (IoT) environments, where the data generated by IoT devices stays local, and only model updates are communicated to a central server. This methodology is particularly relevant for intrusion detection systems in IoT networks, where security is paramount. However, the decentralized nature of FL introduces vulnerabilities, such as the risk of data poisoning by malicious participants. In this paper, we propose a hierarchical federated learning to reduce communication overhead and improve privacy by limiting data spread. We further explore the impact of label-flipping attacks on hierarchical FL systems used in IoT-based intrusion detection. We focus on scenarios where a subset of malicious participants attempts to degrade the global model’s performance by submitting corrupted model updates based on intentionally mislabeled data. Our findings reveal significant decreases in classification accuracy by 10.53% and recall rates, even with a minimal number of compromised participants, primarily affecting the specific classes targeted by the attackers. We further examine how the availability of these malicious nodes influences the attack’s success. To counteract these threats, we introduce a defense mechanism that successfully identifies all malicious clients and mitigates their impact. As a result, the global model’s accuracy is maintained at the original 95% level found during training without the presence of malicious clients, thereby enhancing the resilience of federated learning models in IoT security applications.

Interpreting single-cell and spatial omics data using deep neural network training dynamics.

Single-cell and spatial omics datasets can be organized and interpreted by annotating single cells to distinct types, states, locations or phenotypes. However, cell annotations are inherently ambiguous, as discrete labels with subjective interpretations are assigned to heterogeneous cell populations on the basis of noisy, sparse and high-dimensional data. Here we developed Annotatability, a framework for identifying annotation mismatches and characterizing biological data structure by monitoring the dynamics and difficulty of training a deep neural network over such annotated data. Following this, we developed a signal-aware graph embedding method that enables downstream analysis of biological signals. This embedding captures cellular communities associated with target signals. Using Annotatability, we address key challenges in the interpretation of genomic data, demonstrated over eight single-cell RNA sequencing and spatial omics datasets, including identifying erroneous annotations and intermediate cell states, delineating developmental or disease trajectories, and capturing cellular heterogeneity. These results underscore the broad applicability of annotation-trainability analysis via Annotatability for unraveling cellular diversity and interpreting collective cell behaviors in health and disease.

Neural Network Design and Training for Longitudinal Flight Control of a Tilt-Rotor Hybrid Vertical Takeoff and Landing Unmanned Aerial Vehicle

This paper considers a hybrid vertical take-off and landing (VTOL) unmanned aerial vehicle (UAV). By tilting its propellers, the aircraft can transition from rotary-wing (RW) multirotor mode to fixed-wing (FW) mode and vice versa. A novel architecture of a neural network-based controller (NNC) is presented. An “imitative learning” approach is employed to train the NNC to mimic the response of an expert but computationally expensive model predictive controller (MPC). The resulting NNC approximates the MPC’s solution while significantly decreasing the computational cost. The NNC is trained on the longitudinal axis. Successful simulations and real flight tests prove that the NNC is suitable for the longitudinal axis control of a complex nonlinear system such as the tilt-rotor VTOL UAV through a sequence of transitions between the RW mode to the FW mode, and vice versa, in a forward flight.

Compensated Neural Network Training Algorithm with Minimized Training Dataset for Modeling the Switching Transients of SiC MOSFETs

Accurate modeling of the switching transients of SiC MOSFETs is essential for overvoltage evaluation, EMI prediction, and other critical applications. Due to the fast switching speed, the switching transients of SiC MOSFETs are highly sensitive to parasitic parameters and nonlinear components, making precise modeling challenging. This paper proposes a hybrid model for SiC MOSFET, in which the analytical model is treated as the basis to provide the fundamental waveforms (knowledge-driven), while the neural network (NN) is utilized to fit the high-order and nonlinear features (data-driven). An NN training method with augmented data is proposed to minimize the training datasets. Verification results show that, even though the NN is trained with the data from a single operating condition, the model can accurately predict switching transients of other operating conditions. The proposed methodology has the potential to co-work with the “black-box” or “grey-box” models to enhance the model accuracy.

On the turnpike to design of deep neural networks: Explicit depth bounds

It is well-known that the training of Deep Neural Networks (DNN) can be formalized in the language of optimal control. In this context, this paper leverages classical turnpike properties of optimal control problems to attempt a quantifiable answer to the question of how many layers should be considered in a DNN. The underlying assumption is that the number of neurons per layer—i.e., the width of the DNN—is kept constant. Pursuing a different route than the classical analysis of approximation properties of sigmoidal functions, we prove explicit bounds on the required depths of DNNs based on asymptotic reachability assumptions and a dissipativity-inducing choice of the regularization terms in the training problem. Numerical results obtained for the two spiral task data set for classification indicate that the proposed constructive estimates can provide non-conservative depth bounds.

A merging approach for hole identification with the NMM and WOA-BP cooperative neural network in heat conduction problem

Defect identification is an important issue in structural health monitoring. Herein, originated from inverse techniques, a merging approach is established by the numerical manifold method (NMM) and whale optimization algorithm-back propagation (WOA-BP) cooperative neural network to identify hole defects in heat conduction problems. On the one hand, the NMM can simulate varying hole configurations on a fixed mathematical cover, which eases the generation of “big data” for the training of neural network to a large extent. On the other hand, the WOA, a global optimization algorithm, is adopted to optimize the initial weights and thresholds of the BP neural network to alleviate its frequently encountered local optimum phenomenon. The boundary temperatures of sampling points by the NMM and the associated hole geometries are used for the learning of WOA-BP neural network, which is then applied to predict the hole defects. Numerical examples concerning the detection of circular/ elliptical holes demonstrate that the proposed method possesses higher accuracy and satisfying robustness in holes prediction compared with standard BP network under the same condition. The present work provides a convenient pathway and great potential in application of structural health monitoring.

NeutronSketch: An in-depth exploration of redundancy in large-scale graph neural network training

Graph Neural Networks (GNNs) have achieved notable success in various applications. However, the increasing scale of real-world graphs poses a challenge for efficient GNN training. Recent works propose reducing computations by reducing the scale of the input graph through graph compression or sparsification. However, these methods introduce deep learning models requiring multiple training iterations, which leads to significant additional time overhead. Meanwhile, they fail to consider redundant information that is unhelpful or even harmful for model training in the input graphs. In this work, we propose a universal, one-time redundancy removal method called NeutronSketch to remove the redundant information from the input graph. This method can improve training efficiency while maintaining the model accuracy. In the experiments, we compare NeutronSketch with graph compression, sparsification, and coarsening methods. The results show that NeutronSketch has a faster execution speed and better model accuracy. Additionally, we apply NeutronSketch to the sample-based GNN models. The results show that NeutronSketch reduces input graph scale by an average of 25% compared to the original graph, reducing training time by 10%–90% while maintaining the model accuracy.

Sigmoidal learning rate optimizer for deep neural network training using a two-phase adaptation approach

The optimization of machine learning models is an open research question since an optimal procedure to change the learning rate throughout training is still unknown. Manually defining a learning rate schedule involves troublesome, time-consuming try-and-error procedures to determine hyperparameters, such as learning rate decay epochs and learning rate decay rates, in order to navigate the intricate landscape of loss functions. Although adaptive learning rate optimizers automatize this process, recent studies suggest they may produce overfitting and reduce performance compared to fine-tuned learning rate schedules. Considering that the new machine learning loss function approaches present landscapes with much more saddle points than local minima, we proposed the Training Aware Sigmoidal Optimizer (TASO). This automated two-phase learning rate adaptation mechanism significantly reduces the need for manual hyperparameter tuning. The first phase uses a high learning rate to quickly traverse the numerous saddle points in the error surface, while the second phase uses a low learning rate to gradually approach the center of the local minimum previously found. We compared the proposed approach with commonly used adaptive learning rate schedules such as Adam, RMSProp, and Adagrad. The validation experiments were performed in image and text datasets and showed that TASO outperformed all competing methods in optimal (i.e., performing hyperparameter validation) and suboptimal (i.e., using default hyperparameters) scenarios. In our benchmark tests, TASO demonstrated promising performance, achieving an 8.32% increase in accuracy and a significant 46.62% decrease in training loss on average across various datasets and models. This remarkable performance positions TASO ahead of well-established adaptive optimizers, suggesting higher effectiveness and consistency in performance.

Approaching the Global Nash Equilibrium of Non-Convex Multi-Player Games.

Many machine learning problems can be formulated as non-convex multi-player games. Due to non-convexity, it is challenging to obtain the existence condition of the global Nash equilibrium (NE) and design theoretically guaranteed algorithms. This paper studies a class of non-convex multi-player games, where players' payoff functions consist of canonical functions and quadratic operators. We leverage conjugate properties to transform the complementary problem into a variational inequality (VI) problem using a continuous pseudo-gradient mapping. We prove the existence condition of the global NE as the solution to the VI problem satisfies a duality relation. We then design an ordinary differential equation to approach the global NE with an exponential convergence rate. For practical implementation, we derive a discretized algorithm and apply it to two scenarios: multi-player games with generalized monotonicity and multi-player potential games. In the two settings, step sizes are required to be O(1/k) and O(1/√k) to yield the convergence rates of O(1/ k) and O(1/√k), respectively. Extensive experiments on robust neural network training and sensor network localization validate our theory. Our code is available at https://github.com/GuanpuChen/Global-NE.

Fast prediction of key parameters in FEBA using the COSINE subchannel code and artificial neural network

Numerical techniques have emerged as an essential tool for operators and designers to preemptively acquire key parameters in accidents analysis. However, due to insufficient experience, it is difficult for them to obtain satisfactory numerical results. Moreover, the uncertainty analysis and quantification necessitate the simulation of a substantial number of samples, which requires a significant amount of computational time. Therefore, the development of a fast prediction model becomes imperative. In this work, a prediction model based on the in-house COSINE subchannel code and Multi-Head Perceptron (MHP) is developed. The COSINE subchannel code is employed to provide data sets for training neural networks. Firstly, the numerical results of COSINE subchannel code are compared with experimental data to ensure the accuracy of data sets. Secondly, input features for neural networks are selected by evaluating the impact of input parameters on numerical results, and a series of simulations is carried out to generate data sets. Then, a comparative analysis was conducted between the Multi-Layer Perceptron (MLP) and Support Vector Regression (SVR) models, and the MLP model performs better. Subsequently, the MLP was compared with the MHP, demonstrating the advantage of MHP model. Based on this, the predictions are conducted using the MHP model and the distribution of key parameters is compared with that obtained by COSINE subchannel code. The results illustrate that developed MHP model is an efficient tool for predicting key parameters during the reflooding phase.

Cascade Vertical Federated Learning Towards Straggler Mitigation and Label Privacy Over Distributed Labels

Vertical federated learning (VFL) enables collaborative machine learning on vertically partitioned data with privacy-preservation. Most VFL methods face three daunting challenges in real-world applications. First, most existing VFL methods assume that at least one party holds the complete set of labels of all data samples. However, this assumption often violates the nature of many practical scenarios, where the parties only have partial labels. Second, the heterogeneity and dynamic of computational and communication resources in participated parties may cause the straggler problem and slow down training convergence. Third, the confidential label information could be exposed through malicious parties during VFL. To address these challenges, we propose a novel VFL algorithm named Cascade Vertical Federated Learning (CVFL), in which partitioned labels can be fully utilized to train neural networks with privacy-preservation. To mitigate the straggler problem, we design a novel optimization objective to increase straggler's contribution to the trained models. To mitigate the label privacy risks, we design a novel defense approach to protect the label privacy of CVFL. We conduct comprehensive experiments and the results demonstrate the effectiveness and efficiency of CVFL. Further, the proposed defense approach can achieve a better tradeoff between label privacy and model utility than two widely-used defense approaches.

Improve the energy efficiency of the fruit freeze-drying through the predictive analysis

The considerable energy expenditure involved in the freeze-drying of foods justifies the development of innovative engineering techniques. Artificial intelligence will facilitate mass balance and energy efficiency in future food freeze-drying processes. This study assessed the energy and manufacturing efficiency of the freeze-drying facility through the application of artificial intelligence. Two distinct artificial neural network models were created utilizing real-time data from a factory located in an industrial zone that processed freeze-dried vegetables and kiwi fruit. Analyzing energy efficiency values and production was done using network models constructed from 20 experimental data sets. The Levenberg-Marquardt approach was employed to train neural networks with a multilayer perceptron architecture. The neural network models' prediction values were compared with the experimentally acquired data, and their performance was examined using several performance criteria. The evaluations carried out for 20 different scenarios revealed overall energy efficiency rates ranging from 25.8% to 64.5%. The considerable energy expenditure involved in the freeze-drying of foods justifies the development of innovative engineering techniques. Artificial intelligence will facilitate mass balance and energy efficiency in future food freeze-drying processes.

Learning dynamical systems from noisy data with inverse-explicit integrators

We introduce the mean inverse integrator (MII), a novel approach that improves accuracy when training neural networks to approximate vector fields of dynamical systems using noisy data. This method can be used to average multiple trajectories obtained by numerical integrators such as Runge–Kutta methods. We show that the class of mono-implicit Runge–Kutta methods (MIRK) offers significant advantages when used in connection with MII. When training vector field approximations, explicit expressions for the loss functions are obtained by inserting the training data in the MIRK formulae. This allows the use of symmetric and high-order integrators without requiring the solution of non-linear systems of equations. The combined approach of applying MIRK within MII yields a significantly lower error compared to the plain use of the numerical integrator without averaging the trajectories. This is demonstrated with extensive numerical experiments using data from chaotic and high-dimensional Hamiltonian systems. Additionally, we perform a sensitivity analysis of the loss functions under normally distributed perturbations, providing theoretical support for the favorable performance of MII and symmetric MIRK methods.

SURE: A Visualized Failure Indexing Approach Using Program Memory Spectrum

Failure indexing is a longstanding crux in software debugging, the goal of which is to automatically divide failures (e.g., failed test cases) into distinct groups according to the culprit root causes, as such multiple faults residing in a faulty program can be handled independently and simultaneously. The community of failure indexing has long been plagued by two challenges: (1) The effectiveness of division is still far from promising. Specifically, existing failure indexing techniques only employ a limited source of software runtime data, for example, code coverage, to be failure proximity and further divide them, which typically delivers unsatisfactory results. (2) The outcome can be hardly comprehensible. Specifically, a developer who receives the division result is just aware of how all failures are divided, without knowing why they should be divided the way they are. This leads to difficulties for developers to be convinced by the division result, which in turn affects the adoption of the results. To tackle these two problems, in this article, we propose SURE , a vi SU alized failu R e ind E xing approach using the program memory spectrum (PMS). We first collect the runtime memory information (i.e., variables’ names and values, as well as the depth of the stack frame) at several preset breakpoints during the execution of a failed test case, and transform the gathered memory information into a human-friendly image (called PMS). Then, any pair of PMS images that serve as proxies for two failures is fed to a trained Siamese convolutional neural network, to predict the likelihood of them being triggered by the same fault. Last, a clustering algorithm is adopted to divide all failures based on the mentioned likelihood. In the experiments, we use 30% of the simulated faults to train the neural network, and use 70% of the simulated faults as well as real-world faults to test. Results demonstrate the effectiveness of SURE: It achieves 101.20% and 41.38% improvements in faults number estimation, as well as 105.20% and 35.53% improvements in clustering, compared with the state-of-the-art technique in this field, in simulated and real-world environments, respectively. Moreover, we carry out a human study to quantitatively evaluate the comprehensibility of PMS, revealing that this novel type of representation can help developers better comprehend failure indexing results.

Long Short-Term Memory Networks for the Automated Identification of the Stationary Phase in Tribological Experiments

This study outlines the development and optimization of a Long Short-Term Memory (LSTM) network designed to analyze and classify time-series data from tribological experiments, with a particular emphasis on identifying stationary phases. The process of fine-tuning key hyperparameters was systematically optimized through Bayesian optimization, coupled with K-fold cross-validation to minimize the inherent randomness associated with training neural networks. The refined LSTM network achieved a weighted average accuracy of 84%, demonstrating a high level of agreement between the network’s identified stationary phases and those manually determined by researchers. This result suggests that LSTM networks can reliably mimic manual identification processes in tribological data, providing a promising avenue for automating data analysis. The study underscores the potential of neural networks to transcend their traditional role in predictive modeling within tribology, opening up new possibilities for their application across a broader spectrum of tasks within the field.

AI-based optimization of EM radiation estimates from GSM base stations using traffic data

The fast expansion of mobile networks has sparked worries regarding base station EM radiation's health impacts. Traffic load is commonly ignored when evaluating EM radiation levels using maximum power output. This study proposes utilizing AI and ML on real network traffic data to optimize GSM base station EM radiation estimations. We obtained EM radiation measurements and traffic data from selecting GSM base stations by location and configuration. To predict EM radiation levels, traffic patterns were used to train linear regression, random forests, and neural networks. Base stations were clustered by radiation profile using unsupervised learning. Considering regulatory restrictions and measurement feasibility, an optimization methodology was created to minimize EM radiation estimate inaccuracy. The results show better prediction accuracy than power-based estimations and high generalisability across base station types. Site-specific factors influenced daily EM radiation patterns after clustering. EM radiation levels can be monitored using traffic data and the optimized AI/ML model. This research helps telecom operators and regulators analyze EM radiation more accurately and efficiently. Future projects should include 5G and small cell network extensions and intelligent city platform integration. The suggested method develops data-driven, AI-powered Public Safety and mobile network trust solutions.