Weight Update Research Articles

A 28-nm 8-bit Floating-Point Tensor Core-Based Programmable CNN Training Processor With Dynamic Structured Sparsity

Training deep/convolutional neural networks (DNNs/CNNs) requires a large amount of memory and iterative computation, which necessitates speedup and energy reduction, especially for edge devices with resource/energy constraints. In this work, we present an 8-bit floating-point (FP8) training the processor which implements: 1) highly parallel tensor cores (fused multiply–add trees) that maintain high utilization throughout forward propagation (FP), backward propagation (BP), and weight update (WU) phases of the training process; 2) hardware-efficient channel gating for dynamic output activation sparsity; 3) dynamic weight sparsity (WS) based on group Lasso; and 4) gradient skipping based on the FP prediction error. We develop a custom instruction set architecture (ISA) to flexibly support different CNN topologies and training parameters. The 28-nm prototype chip demonstrates large improvements in floating-point operations (FLOPs) reduction (7.3 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\times$</tex-math> </inline-formula> ), energy efficiency (16.4 TFLOPS/W), and overall training latency speedup (4.7 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\times$</tex-math> </inline-formula> ), for both supervised and self-supervised training tasks.

A Novel Parameter Estimation Method for PMSM by Using Chaotic Particle Swarm Optimization With Dynamic Self-Optimization

In this study, a novel parameter estimation method for permanent magnet synchronous motor (PMSM) of chaotic particle swarm optimization with dynamic self-optimization (DSCPSO) is proposed, where the voltage source inverter (VSI) nonlinearity is estimated simultaneously with the parameters to achieve real-time compensation of VSI nonlinearity. In DSCPSO, the tent chaos theory is introduced into the updating of particle swarm algorithm (PSO) populations, inertia weights and learning factors to enhance its ability to explore potentially better regions. Moreover, a memory tempering annealing (MTA) strategy is employed to guarantee particle pluralistic learning, which combines the superior robustness of the simulated annealing algorithm (SA) while enhancing population diversity. Furthermore, to achieve a reasonable tradeoff between exploration and exploitation, a dynamic lens imaging opposition-based learning (DLIOBL) and domain optimization strategy based on evolutionary information is designed, i.e., DLIOBL in the pre-evolutionary stage guarantees the depth of the exploration learning, while the domain optimization strategy is performed in the post-evolutionary stage accelerates the exploitation operation and avoids the problem of slow convergence in the late stages of PSO. The proposed method is applied to the parameter estimation of PMSM and the experimental results show that, the proposed method can track the VSI nonlinearity and variable parameter better than the conventional method under different working conditions.

A 2-Transistor-2-Capacitor Ferroelectric Edge Compute-in-Memory Scheme With Disturb-Free Inference and High Endurance

This paper proposes C <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> FeRAM, a 2T2C/cell ferroelectric compute-in-memory (CiM) scheme for energy-efficient and high-reliability edge inference and transfer learning. With certain area overhead, C <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> FeRAM achieves the following highlights: (i) compared with FeFET/FeMFET, it achieves disturb-free CiM and much higher write endurance (equal to FeRAM), leading to >100x inference time with <1% accuracy drop for VGG8 in CIFAR-10 dataset, along with the enhanced endurance for weight updates, e.g., CiM-based transfer learning; (ii) compared with 1T1C FeRAM inference cache, the achieved disturb-free feature and CiM capability in C <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> FeRAM lead to improvements of 4x energy, 200x speed, and 3.2e5x life cycles. Such benefits highlight an intriguing solution for future intelligent edge AI.

Efficient Training Method for Memristor-Based Array Using 1T1M Synapse

In this brief, an efficient training method for memristor-based array (crossbar) with one transistor and one memristor (1T1M) synapse is proposed, which enables parallel update of memristor-based arrays trained by stochastic gradient descent within two steps. Voltage ThrEshold Adaptive Memristor (VTEAM) model is utilized to describe memristor characteristics for simulations. On this basis, circuit parameters optimization method compensating the asymmetric and nonlinear weight update is provided for better training results. The effectiveness of proposed training method is evaluated on OR, AND functions and digit recognition task. Simulation results demonstrate the robustness of proposed training method to electrical noise and imperfections of memristors.

ESSENCE: Exploiting Structured Stochastic Gradient Pruning for Endurance-Aware ReRAM-Based In-Memory Training Systems

Processing-in-memory (PIM) enables energyefficient deployment of convolutional neural networks (CNNs) from edge to cloud. Resistive random-access memory (ReRAM) is one of the most commonly used technologies for PIM architectures. One of the primary limitations of ReRAM-based PIM in neural network training arises from the limited write endurance due to the frequent weight updates. To make ReRAM-based architectures viable for CNN training, the write endurance issue needs to be addressed. This work aims to reduce the number of weight reprogrammings without compromising the final model accuracy. We propose the ESSENCE framework with an endurance-aware structured stochastic gradient pruning method, which dynamically adjusts the probability of gradient update based on the current update counts. Experimental results with multiple CNNs and datasets demonstrate that the proposed method can extend ReRAM’s life time for training. For instance, with the ResNet20 network and CIFAR-10 dataset, ESSENCE can save the mean update counts of up to 10.29× compared to the SGD method and effectively reduce the maximum update counts compared with No Endurance method. Furthermore, aggressive tuning method based on ESSENCE can boost the mean update count savings by up to 14.41×.

Can We Find Strong Lottery Tickets in Generative Models?

Yes. In this paper, we investigate strong lottery tickets in generative models, the subnetworks that achieve good generative performance without any weight update. Neural network pruning is considered the main cornerstone of model compression for reducing the costs of computation and memory. Unfortunately, pruning a generative model has not been extensively explored, and all existing pruning algorithms suffer from excessive weight-training costs, performance degradation, limited generalizability, or complicated training. To address these problems, we propose to find a strong lottery ticket via moment-matching scores. Our experimental results show that the discovered subnetwork can perform similarly or better than the trained dense model even when only 10% of the weights remain. To the best of our knowledge, we are the first to show the existence of strong lottery tickets in generative models and provide an algorithm to find it stably. Our code and supplementary materials are publicly available at https://lait-cvlab.github.io/SLT-in-Generative-Models/.

RLEKF: An Optimizer for Deep Potential with Ab Initio Accuracy

It is imperative to accelerate the training of neural network force field such as Deep Potential, which usually requires thousands of images based on first-principles calculation and a couple of days to generate an accurate potential energy surface. To this end, we propose a novel optimizer named reorganized layer extended Kalman filtering (RLEKF), an optimized version of global extended Kalman filtering (GEKF) with a strategy of splitting big and gathering small layers to overcome the O(N^2) computational cost of GEKF. This strategy provides an approximation of the dense weights error covariance matrix with a sparse diagonal block matrix for GEKF. We implement both RLEKF and the baseline Adam in our alphaDynamics package and numerical experiments are performed on 13 unbiased datasets. Overall, RLEKF converges faster with slightly better accuracy. For example, a test on a typical system, bulk copper, shows that RLEKF converges faster by both the number of training epochs (x11.67) and wall-clock time (x1.19). Besides, we theoretically prove that the updates of weights converge and thus are against the gradient exploding problem. Experimental results verify that RLEKF is not sensitive to the initialization of weights. The RLEKF sheds light on other AI-for-science applications where training a large neural network (with tons of thousands parameters) is a bottleneck.

Channel Regeneration: Improving Channel Utilization for Compact DNNs

Overparameterized deep neural networks have redundant neurons that do not contribute to the network's accuracy. In this paper, we introduce a novel channel regeneration technique that reinvigorates these redundant channels by re-initializing its batch normalization scaling factor gamma. This re-initialization of BN gamma promotes regular weight updates during training. Furthermore, we show that channel regeneration encourages the channels to contribute equally to the learned representation and further boosts the generalization accuracy. We apply our technique at regular intervals of the training cycle to improve channel utilization. The solutions proposed in previous works either raise the total computational cost or increase the model complexity. Integrating the proposed channel regeneration technique into the training methodology of efficient architectures requires minimal effort and comes at no additional cost in size or memory. Extensive experiments on several image classification and semantic segmentation benchmarks demonstrate the effectiveness of applying the channel regeneration technique to compact architectures.

Partial-Label Regression

Partial-label learning is a popular weakly supervised learning setting that allows each training example to be annotated with a set of candidate labels. Previous studies on partial-label learning only focused on the classification setting where candidate labels are all discrete, which cannot handle continuous labels with real values. In this paper, we provide the first attempt to investigate partial-label regression, where each training example is annotated with a set of real-valued candidate labels. To solve this problem, we first propose a simple baseline method that takes the average loss incurred by candidate labels as the predictive loss. The drawback of this method lies in that the loss incurred by the true label may be overwhelmed by other false labels. To overcome this drawback, we propose an identification method that takes the least loss incurred by candidate labels as the predictive loss. We further improve it by proposing a progressive identification method to differentiate candidate labels using progressively updated weights for incurred losses. We prove that the latter two methods are model-consistent and provide convergence analysis showing the optimal parametric convergence rate. Our proposed methods are theoretically grounded and can be compatible with any models, optimizers, and losses. Experiments validate the effectiveness of our proposed methods.

Emulating Neuromorphic and In‐Memory Computing Utilizing Defect Engineering in 2D‐Layered WSeOx and WSe2 Thin Films by Plasma‐Assisted Selenization Process

AbstractThe neuromorphic and in‐memory computing using memristors are promising for the building of the next generation computing systems. However, the diffusion dynamics of metal ions/atoms inside the switching medium impose variability in conducting filament (CF) formation, thus limiting their use in von‐Neumann architecture. The precise modulation on the diffusion of metal ions/atoms and their reduction/oxidation probability holds promise to overcome the speed, size, and energy issues of present‐day computers. Here, this study shows that the diffusion of metal ions can be modulated by defects inside the switching medium and confines metal filaments in a precise 1D channel. This filament confinement by the defect engineering leads to an anomalous switching mechanism with two interchangeable modes: unipolar threshold and bipolar modes. The variation between two modes can be modulated by controlling defects in the structures, leading to a uniform switching with low SET/RESET voltage variations of 17.3% and −17.6%, respectively. Moreover, the convolutional neural network is implemented to emulate synaptic plasticity and image recognition to achieve recognition accuracy of 87% due to a highly linear weight update, demonstrating its potential for in‐memory computing.

Thermally Tunable Analog Memristive Behaviors in Sulfurized In2Se3 Nanoflakes for Bio-Plausible Synaptic Devices

Combining the intrinsic superiorities of two-dimensional materials and the emerging demand for neuromorphic computing, two-dimensional memristors have achieved huge advances in materials exploration and synaptic functionality emulation. However, their neuromorphic applications are still in the early stage since digital memristive behaviors for most of them are inconsistent with gradual biological synaptic plasticity. Here, we developed a simple approach to realize analog and thermal tunable memristive behaviors by introducing sulfur vacancies in CVD-grown In2Se3 nanoflakes through secondary sulfurization treatment. The density functional theory and ab initio molecular dynamics simulations confirm that sulfurized and defective In2Se3 can remain thermal stability at elevated temperatures up to 550 K. Therefore, we systematically investigated the temperature-dependent analog and tunable memristive behaviors and realized linear weight update with ultrawide dynamic range in sulfurized In2Se3 at high temperatures. The developed memristive device successfully emulates bio-realistic synaptic functionalities including transformation from short- to long-term plasticity, paired-pulse facilitation, posttetanic potentiation, spike-amplitude-dependent plasticity, spike-rate-dependent plasticity, and spike-time-dependent plasticity effects. It is unveiled that the formation energy of sulfur vacancy is greatly smaller than that of selenium, while their roughly same migration barriers can be modulated by electric field and temperature. Therefore, we put forward that the applied electric field can mediate vacancy migration in the sulfurized In2Se3 to gradually regulate the conductance, thereby realizing the emulation of synaptic plasticity. This work provides a promising approach to designing bio-plausible memristive devices for robust neuromorphic applications at high temperatures.

Bidirectional Synaptic Phototransistor Based on Two-Dimensional Ferroelectric Semiconductor for Mixed Color Pattern Recognition.

Optoelectronic synaptic devices capable of processing multiwavelength inputs are critical for neuromorphic vision hardware, which remains an important challenge. Here, we develop a bidirectional synaptic phototransistor based on a two-dimensional ferroelectric semiconductor of α-In2Se3, which exhibits bidirectional potentiated and depressed synaptic weight update under optical pulse stimulation. Importantly, the bidirectional optoelectronic synaptic behavior can be extended to multiwavelengths (blue, green, and red light), which could be used for color recognition. The mechanism underlying the bidirectional synaptic characteristics is attributed to the gate-configurable barrier heights as revealed by the Kelvin probe force microscopy measurement. The α-In2Se3 device exhibits versatile synaptic plasticity such as paired-pulse facilitation, short- and long-term potentiation, and long-term depression. The bidirectional optoelectronic synaptic weight updates under multiwavelength inputs enable a high accuracy of 97% for mixed color pattern recognition.

A Robust Adaptive Filter for Diffusion Strategy-based Distributed Active Noise Control

Active noise reduction in the existential of impulsive noise poses a serious challenge in a distributed scenario. The conventional centralized multichannel active noise control (MANC) and diffusion filtered-x least mean square (F-xLMS) based distributed active noise control (DANC) systems exhibit deteriorated performance in the existence of impulse noise and outliers. In this regard, a diffusion least logarithmic hyperbolic cosine function (D-Llncosh) based algorithm for DANC system is developed with the goal of mitigating noise in such an environment. It utilizes natural logarithmic of hyperbolic cosine function integrating the mean square error (MSE) and mean absolute error (MAE) as the cost function depending upon the parameter value. The weight update rule of the introduced D-Llncosh-based DANC system is hereby derived. The convergence analysis of the suggested method is also carried out. The proposed system is validated for various types of impulsive noise under different scenarios. The extensive simulation results assert the efficacy of the posited DANC system in comparison with the existing techniques. The suggested method achieves quicker convergence and approximately a 1–5 dB improvement is the noise reduction for different noise scenarios as evident from the obtained results.

Position and Attitude Tracking Finite-Time Adaptive Control for a VTOL Aircraft using Global Fast Terminal Sliding Mode Control

In this work, the position and attitude tracking finite-time adaptive control problem of a type of vertical take-off and landing (VTOL) aircraft system is studied. Here, the dynamic of the VTOL aircraft is subjected to external disturbances and unknown nonlinearities. Firstly, radial basis function neural networks are introduced to approximate these unknown nonlinearities, and adaptive weight update laws are proposed to replace unknown ideal weights. Secondly, for the errors generated in the approximation process and the external disturbances of the aircraft system, adaptive parameter update laws are presented. After that, based on the designed global fast terminal sliding mode control functions and adaptive update laws, we present the position tracking control laws and the roll angle control law. Then, based on this, the adaptive global fast terminal sliding control laws for the given aircraft system are finally obtained. Meanwhile, the stability of the aircraft control system is proven by using Lyapunov stability theory and designed adaptive control laws. It is not only ensured that the outputs of the aircraft system can track the given trajectories, but also ensured that the tracking errors can converge to approximately zero within a finite time. Finally, the validity of the designed adaptive control laws is verified through three numerical examples. It can be obtained that the finite-time tracking problems of the given aircraft system can be achieved at 18.8766 s and 14.6340 s under the given parameters. The results are consistent with the theoretical analysis. In addition, under the control laws proposed in this work, the aircraft system can achieve tracking after 9.443 s and 9.674 s and the tracking errors are basically close to zero, which is significantly superior to other control methods considered in this work.

Neuroadaptive Dynamic Surface Asymptotic Tracking Control of a VTOL Aircraft with Unknown Dynamics and External Disturbances

This work studies the asymptotic tracking control problem of a vertical take-off and landing (VTOL) aircraft with unknown dynamics and external disturbances. The unknown nonlinear dynamics of the VTOL aircraft are approximated via the introduction of radial basis function neural networks. Then, the weight update laws are designed. Furthermore, the parameter update control laws are presented to deal with the errors generated during the approximation process and the external disturbances of the aircraft system. Moreover, first-order filters are introduced to avoid repeated differentiation of the designed virtual control laws, thereby effectively eliminating the “complexity explosion” problem caused by traditional backstepping control. Based on the application of the neural network control method, dynamic surface control technique, weight update laws and parameter update control laws, neuroadaptive dynamic surface control laws for the aircraft system are finally proposed. Theoretical analysis shows that the proposed control law can ensure that the aircraft system asymptotically tracks the reference trajectories and the tracking errors can converge to a small neighborhood of zero by choosing the appropriate designed parameters. Finally, simulation examples are provided to verify the effectiveness of proposed control laws.

Linear and symmetric synaptic weight update characteristics by controlling filament geometry in oxide/suboxide HfOx bilayer memristive device for neuromorphic computing

Memristive devices have been explored as electronic synaptic devices to mimic biological synapses for developing hardware-based neuromorphic computing systems. However, typical oxide memristive devices suffered from abrupt switching between high and low resistance states, which limits access to achieve various conductance states for analog synaptic devices. Here, we proposed an oxide/suboxide hafnium oxide bilayer memristive device by altering oxygen stoichiometry to demonstrate analog filamentary switching behavior. The bilayer device with Ti/HfO2/HfO2−x(oxygen-deficient)/Pt structure exhibited analog conductance states under a low voltage operation through controlling filament geometry as well as superior retention and endurance characteristics thanks to the robust nature of filament. A narrow cycle-to-cycle and device-to-device distribution were also demonstrated by the filament confinement in a limited region. The different concentrations of oxygen vacancies at each layer played a significant role in switching phenomena, as confirmed through X-ray photoelectron spectroscopy analysis. The analog weight update characteristics were found to strongly depend on the various conditions of voltage pulse parameters including its amplitude, width, and interval time. In particular, linear and symmetric weight updates for accurate learning and pattern recognition could be achieved by adopting incremental step pulse programming (ISPP) operation scheme which rendered a high-resolution dynamic range with linear and symmetry weight updates as a consequence of precisely controlled filament geometry. A two-layer perceptron neural network simulation with HfO2/HfO2−x synapses provided an 80% recognition accuracy for handwritten digits. The development of oxide/suboxide hafnium oxide memristive devices has the capacity to drive forward the development of efficient neuromorphic computing systems.

Wi-Fi frame detection via spiking neural networks with memristive synapses

With increasing performance of deep learning, researchers have employed Deep Neural Networks (DNNs) for wireless communications. In particular, mechanisms for detecting Wi-Fi frames that use DNNs demonstrate their excellent performances in terms of detection accuracy. However, DNNs require significant amount of computation resources. Thus, if the DNN-based mechanisms are used in mobile devices or low-end devices, their battery would be quickly depleted. Spiking Neural Networks (SNNs), which are regarded as the next generation of neural network, have advantages over DNNs: low energy consumption and limited computational complexity. Motivated by these advantages, in this paper, we propose a mechanism to detect a Wi-Fi frame using SNNs and show the feasibility of SNNs for Wi-Fi detection. The mechanism is composed of a preprocessing module for converting collected signals and an SNN module for detecting Wi-Fi frames. The SNN module employs Leaky Integrate-and-Fire (LIF) neurons and Spike-Timing-Dependent Plasticity (STDP) learning rule. To reflect the features of an actual neuromorphic hardware system, our SNN module considers memristive synaptic features such as non-linear weight updates. Our experimental study demonstrates the detection capabilities of the proposed mechanism are comparable to those of previous mechanisms that use DNNs, CNNs and RNNs while consuming definitely less energy than the previous mechanism.

Proton-Assisted Redox-Based Three-Terminal Memristor for Synaptic Device Applications

Emerging technologies, i.e., spintronics, 2D materials, and memristive devices, have been widely investigated as the building block of neuromorphic computing systems. Three-terminal memristor (3TM) is specifically designed to mitigate the challenges encountered by its two-terminal counterpart as it can concurrently execute signal transmission and memory operations. In this work, we present a complementary metal-oxide-semiconductor-compatible 3TM with highly linear weight update characteristics and a dynamic range of ∼15. The switching mechanism is governed by the migration of oxygen ions and protons in and out of the channel under an external gate electric field. The involvement of the protonic defects in the electrochemical reactions is proposed based on the bipolar pulse trains required to initiate the oxidation process and the device electrical characteristics under different humidity levels. For the synaptic operation, an excellent endurance performance with over 256k synaptic weight updates was demonstrated while maintaining a stable dynamic range. Additionally, the synaptic performance of the 3TM is simulated and implemented into a four-layer neural network (NN) model, achieving an accuracy of ∼92% in MNIST handwritten digit recognition. With such desirable conductance modulation characteristics, our proposed 3T-memristor is a promising synaptic device candidate to realize the hardware implementation of the artificial NN.

Interpretable sparse learned weights and their entropy based quantification for online machine health monitoring

Incipient fault detection and diagnosis provide a firm grounding for machine health monitoring. Nevertheless, fault signatures at the time of an incipient fault are extremely weak and easily submerged by interference. Currently, signal processing based approaches and machine learning based approaches have been studied for incipient fault detection and diagnosis. However, the former requires relevant technicians with expert knowledge, while the latter needs a substantial volume of labeled samples without model interpretability. In this study, a physics-informed learning framework that integrates weight-based sparse degradation modeling with entropy based indicators is proposed to realize online incipient fault detection and diagnosis. Firstly, based on available normal baseline data and real-time data, a weight-based sparse degradation model is proposed to continually update physics-informed model weights so that weak fault characteristics indicated by learned weights can be considerably enhanced. Meanwhile, this study introduces a family of entropy based indicators for machine health monitoring and their performances are thoroughly investigated based on simulation and experimental studies, which aims to quantify amplified fault characteristics revealed by the continuously updated model weights for online incipient fault detection. Two case studies show that the proposed methodology has better detection ability and sensitivity than classical health indicators for incipient bearing faults. Since the proposed methodology does not demand fault data for model establishment, it is closer to real engineering applications and has more engineering meanings. Moreover, physics-informed model weights can automatically capture informative frequencies for immediate diagnosis and further analysis.

Research on structural damage identification and localization based on artificial neural network

Abstract Structural health monitoring is a research hotspot in engineering, and structural damage identification is one of the key problems in structural health monitoring research. This paper proposes a study on structural damage identification and localization based on artificial neural networks, and for the problem that the learning convergence speed of the BP neural network is too slow, a genetic algorithm is used to optimize the update of weights and thresholds during training and learning. In the finite element simulation, the structure's primary and secondary damage are taken as input nodes, and the optimized GA-BP neural network is used for training and identification. For the localization recognition of the primary damage of the structure, the maximum recognition relative errors of both the BP neural network and the GA-BP neural network did not exceed 5%, but the latter's accuracy was 2.63% higher than that of the former. For the localization recognition of secondary damage, the GA-BP neural network can effectively recognize 90% of the samples. The artificial neural network-based structural damage recognition localization has high recognition efficiency and accuracy, which is conducive to improving the robustness of the structural damage recognition system and is of significant help to real-time structural health monitoring.