Weight Update Research Articles

Programming Techniques of Resistive Random-Access Memory Devices for Neuromorphic Computing

Neuromorphic computing offers a promising solution to overcome the von Neumann bottleneck, where the separation between the memory and the processor poses increasing limitations of latency and power consumption. For this purpose, a device with analog switching for weight update is necessary to implement neuromorphic applications. In the diversity of emerging devices postulated as synaptic elements in neural networks, RRAM emerges as a standout candidate for its ability to tune its resistance. The learning accuracy of a neural network is directly related to the linearity and symmetry of the weight update behavior of the synaptic element. However, it is challenging to obtain such a linear and symmetrical behavior with RRAM devices. Thus, extensive research is currently devoted at different levels, from material to device engineering, to improve the linearity and symmetry of the conductance update of RRAM devices. In this work, the experimental results based on different programming pulse conditions of RRAM devices are presented, considering both voltage and current pulses. Their suitability for application as analog RRAM-based synaptic devices for neuromorphic computing is analyzed by computing an asymmetric nonlinearity factor.

Adaptive-Dynamic-Programming-Based Robust Control for a Quadrotor UAV with External Disturbances and Parameter Uncertainties

Thiswork addresses the trajectory-tracking-control problem for a quadrotor unmanned aerial vehicle with external disturbances and parameter uncertainties. A novel adaptive-dynamic-programming-based robust control method is proposed to eliminate the effects of lumped uncertainties (including external disturbances and parameter uncertainties) and to ensure the approximate optimal control performance. Its novelty lies in that two radial basis function neural network observers with fixed-time convergence properties were first established to reconstruct the lumped uncertainties. Notably, they tune only the scalar parameters online and have low computational complexities. Subsequently, two actor–critic neural networks were designed to approximate the optimal cost functions and control policies for the nominal system. In this design, two new actor–critic neural network weight update laws are proposed to eliminate the persistent excitation condition. Then, two adaptive-dynamic-programming-based robust control laws were obtained by integrating the observer reconstruction information and the nominal control policies. The uniformly ultimately bounded stability of the closed-loop tracking control systems was ensured using the Lyapunov methodology. Finally, numerical results are shown to verify the effectiveness and superiority of the proposed control scheme.

Formula omitted]-norm minimization for outlier-resistant elliptic positioning in [formula omitted]-stable impulsive interference

This short communication considers the problem of robust elliptic positioning (EP) in α-stable impulsive interference based on the ℓp-norm minimization criterion. We start with converting the associated nonconvex optimization task into the well-established ℓp-minimization framework of iteratively reweighted least squares (IRLS). While the effect of p is taken into account by a weight update step in each IRLS iteration, we employ an assortment of surrogate functions, in order to effectively handle the location update subproblem using the majorization–minimization technique. The superiority of our proposal over a number of existing EP methods in terms of localization accuracy in α-stable impulsive noise is demonstrated via numerical simulations.

Inkjet printing cellulose nanofibril/PVA hybrid dielectrics for large hysteresis and high-performance a-IGZO synaptic transistors

Inkjet printing green nanomaterials for high-performance synaptic transistors is a hot research topic in academic communities. However, it remains challenging to develop printable and sustainable dielectric inks for the devices with desirable properties of nonvolatile behavior and large hysteresis, which are the basis for emulating controllable weight update in neuron computing. To address this challenge, we present a solution by fabricating a biodegradable cellulose nanofibril (CNF) dielectric film through inkjet printing, which serves as the source of hysteresis in amorphous indium–gallium–zinc oxide (a-IGZO) synaptic transistors. By employing suitable additives and meticulously engineered microfluidic flows, we have successfully achieved the high-quality CNF/polyvinyl alcohol (PVA) composite electrolyte films by inkjet-printing. This accomplishment has enabled the emulation of excellent synaptic functionalities, including short-term to long-term potentiation/depression, as well as the implementation of a nearly symmetric and linear weight update protocol. This work holds significant promise in advancing the development of cost-effective and environmentally friendly synaptic electronics.

Reducing communication in federated learning via efficient client sampling

Federated learning (FL) ameliorates privacy concerns in settings where a central server coordinates learning from data distributed across many clients; rather than sharing the data, the clients train locally and report the models they learn to the server. Aggregation of local models requires communicating massive amounts of information between the clients and the server, consuming network bandwidth. We propose a novel framework for updating the global model in communication-constrained FL systems by requesting input only from the clients with informative updates, and estimating the local updates that are not communicated. Specifically, describing the progression of the model’s weights by an Ornstein–Uhlenbeck process allows us to develop sampling strategy for selecting a subset of clients with significant weight updates; model updates of the clients not selected for communication are replaced by their estimates. We test this policy on realistic federated benchmark datasets and show that the proposed framework provides up to 50% reduction in communication while maintaining competitive or achieving superior performance compared to baselines. The proposed method represents a new line of strategies for communication-efficient FL that is orthogonal to the existing user-driven techniques, such as compression, thus complementing rather than aiming to replace those existing methods.

A Deep Learning Framework with an Intermediate Layer Using the Swarm Intelligence Optimizer for Diagnosing Oral Squamous Cell Carcinoma.

One of the most prevalent cancers is oral squamous cell carcinoma, and preventing mortality from this disease primarily depends on early detection. Clinicians will greatly benefit from automated diagnostic techniques that analyze a patient's histopathology images to identify abnormal oral lesions. A deep learning framework was designed with an intermediate layer between feature extraction layers and classification layers for classifying the histopathological images into two categories, namely, normal and oral squamous cell carcinoma. The intermediate layer is constructed using the proposed swarm intelligence technique called the Modified Gorilla Troops Optimizer. While there are many optimization algorithms used in the literature for feature selection, weight updating, and optimal parameter identification in deep learning models, this work focuses on using optimization algorithms as an intermediate layer to convert extracted features into features that are better suited for classification. Three datasets comprising 2784 normal and 3632 oral squamous cell carcinoma subjects are considered in this work. Three popular CNN architectures, namely, InceptionV2, MobileNetV3, and EfficientNetB3, are investigated as feature extraction layers. Two fully connected Neural Network layers, batch normalization, and dropout are used as classification layers. With the best accuracy of 0.89 among the examined feature extraction models, MobileNetV3 exhibits good performance. This accuracy is increased to 0.95 when the suggested Modified Gorilla Troops Optimizer is used as an intermediary layer.

Cov-Fed: Federated learning-based framework for COVID-19 diagnosis using chest X-ray scans

SARS-CoV-2, a member of the coronavirus family, causes COVID-19, which can range from the ordinary cold to the rare but fatal Severe Acute Respiratory Syndrome (SARS) and Middle East Respiratory Disease (MERS). Radiologists can better diagnose COVID-19 using deep learning and computer vision techniques. However, training requires large centralized datasets. In addition, medical data protection prevents centralized server training with medical data. In this study, we propose a federated learning-based framework, Cov-Fed, to facilitate decentralized training of medical data. Cov-Fed is characterized by the proposed Multi-Efficient Channel Attention Network (MECAN) and a multi-model user-server weight update mechanism, respectively. In addition, Cov-Fed leverages PySyft encryption and decryption mechanisms for privacy preservation. Due to the salient nature of features corresponding to viruses, the classification of x-ray scans is challenging. Hence, we propose a multi-ECA attention mechanism embedded in the Cov-Fed to enhance feature maps of chest X-ray scans for classification without compromising performance. Extensive experiments are conducted on several public medical datasets and compared with some state-of-the-art models. Based on several evaluation metrics, Cov-Fed exhibited satisfactory performance with an average accuracy of 87.65%, an inference time of 92.8s, and ROC-AUC of 92.46% compared with some state-of-the-art models.

Comparison of Score-Driven Equity-Gold Portfolios During the COVID-19 Pandemic Using Model Confidence Sets

Abstract Gold may have a hedge, safe haven, or diversifier property when added to stock portfolios. Motivated by the favorable statistical properties and out-of-sample performance of score-driven models, we investigate for equity-gold portfolios whether score-driven mean, volatility, and copula models can improve the performances of DCC (dynamic conditional correlation) portfolios, the naïve portfolio strategy, and the Standard & Poor’s 500 (S&P 500) index. We consider 2880 score-driven portfolio strategies. We use score-driven Clayton, rotated Clayton, Frank, Gaussian, Gumbel, rotated Gumbel, Plackett, and Student’s t copulas. We use several classical and score-driven models of marginal distribution. We use weekly, monthly, quarterly, semi-annual, and annual updates of portfolio weights. We use minimum-variance, maximum Sharpe ratio, and maximum utility function strategies. We use rolling data-windows for portfolio optimization for the COVID-19 investment period of February 2020 to September 2021. We classify competing portfolios by using a new robust multi-step model confidence set (MCS) test approach and provide evidence of the superiority of score-driven portfolios.

A software defect prediction method based on learnable three-line hybrid feature fusion

Software defect prediction (SDP) plays a crucial role in ensuring the security and quality of software systems. However, it faces challenges posed by high-dimensional features present in software defect datasets and the limited effectiveness of traditional nonlinear dimensionality reduction methods in extracting essential feature information. To address these issues, we propose a novel approach called learnable three-line hybrid feature fusion (LTHFFA), which incorporates the principle of three-line hybrid breeding into feature fusion for the first time. In this method, three distinct dimensionality reduction techniques are initially employed to obtain three separate sets of features. Subsequently, a learnable weight factor feature fusion method is proposed to facilitate automatically learn and dynamically update of feature weights. By integrating the three feature sets based on the principle of three-line hybrid breeding, we derive learnable three-line hybrid fusion features. These features are then utilized in the context of software defect prediction. Experimental results demonstrate the superior performance of LTHFFA compared to nine other dimensionality reduction methods across seventeen publicly available software defect datasets. LTHFFA exhibits the ability to effectively integrate multiple feature sets, reduce feature redundancy, and enhance predictive accuracy. Moreover, statistical analysis using Friedman ranking and Holm's post-hoc test confirms the significant advantage of LTHFFA over alternative dimensionality reduction methods.

Discrete Robust Principal Component Analysis via Binary Weights Self-Learning.

Principal component analysis (PCA) is a typical unsupervised dimensionality reduction algorithm, and one of its important weaknesses is that the squared l2 -norm cannot overcome the influence of outliers. Existing robust PCA methods based on paradigm have the following two drawbacks. First, the objective function of PCA based on the l1 -norm has no rotational invariance and limited robustness to outliers, and its solution mostly uses a greedy search strategy, which is expensive. Second, the robust PCA based on the l2,1 -norm and the l2,p -norm is essential to learn probability weights for data, which only weakens the influence of outliers on the learning projection matrix and cannot be completely eliminated. Moreover, the ability to detect anomalies is also very poor. To solve these problems, we propose a novel discrete robust principal component analysis (DRPCA). Through self-learning binary weights, the influence of outliers on the projection matrix and data center estimation can be completely eliminated, and anomaly detection can be directly performed. In addition, an alternating iterative optimization algorithm is designed to solve the proposed problem and realize the automatic update of binary weights. Finally, our proposed model is successfully applied to anomaly detection applications, and experimental results demonstrate that the superiority of our proposed method compared with the state-of-the-art methods.

DESIGN OF HYBRID ADAPTIVE MODEL FOR IMAGE DENOISING

This thesis work proposes a new denoising algorithm based on Particle filter and Wavelet (Curvelet) transform combination, particle filter generates weights through SIR algorithm to cancel the interference of noise present in the image, while curvelet transform is used to shrink the remaining segments of noise, so this method can both remove image blurr and maintain good texture as well. The PF+Clet Image Denoiser is successfully designed and implemented, which is a new approach in image enhancement and Interference cancellation. This thesis concludes that it is quite efficient algorithm among other adaptive filtering techniques. Experimental results also show that proposed algorithm performs extremely well when noise density is increased, as obtained image is completely visible. This approach comprises of generation of particles by performing weight normalization, resampling and update state. Therefore, for large number of particles execution time is more when compared to other adaptive filtering approach. Key Words: Particle Filter(PF), Curvelet(CLet), Wavelet(Wlet), Peak signal to noise Ratio(PSNR), Deblurring.

An Efficient Neuromorphic Implementation of Temporal Coding-Based On-Chip STDP Learning

Fulfilling brain intelligence on the chip has long been a fascinating and challenging endeavor. Numerous neuromorphic computing platforms based on spiking neural networks have simulated the architecture and information processing of the brain, providing new insights to realize artificial intelligence. However, the critical issue still remains how to effectively learn on site. In this brief, we present a neuromorphic processor aimed at performing instant on-chip learning for edge processing purposes. This design adopts a temporal coding based implementation of nearest-neighbor and additive spike-timing-dependent plasticity (STDP) learning method. A pair of spike tracing counters and a reconfigurable look-up table are deployed to perform the STDP weight updating. And a 16-core architecture is also leveraged to enhance learning parallelism and scalability. Integrating up to 2048 neurons and 2M 8-bit synapses, the processor achieves peak performance of 81.92GSOPS and 22.80GSOPS, with energy efficiency of 1.28pJ/SOP and 4.99pJ/SOP in the inference and learning phases, respectively. Validation experiments demonstrate that it achieves recognition accuracies of 93.54%, 59.46%, 72.67% and 79.77% for MNIST, Fashion-MNIST, EMNIST and Chinese-MNIST datasets by on-chip labeling methods, respectively.

Demonstration of Pavlov Associative Memory by Implementation of Rate Coding Using Magnetic Tunnel Junction Neurons

In this paper, a fully functioning Pavlov associative memory system, one of the most famous examples of the biological brain function, is implemented by a basis of rate coding. In order to create a neural signal which is capable of emulating the synaptic weight based on rate coding, a neural signal is constructed by utilizing telegraphic switching of the magnetic tunnel junction (MTJ), a random magnetization reversal phenomenon. Spike rate of the MTJ neuron is controllable by adjusting the STT current bias, creating signals with rate information that is capable of implementing rate coding scheme as a weight update method for the TiO <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sub> memristor synapse. The proposed neuron creates random spike signals containing rate information, which participates in emulating Pavlov associative memory characteristics such as learning and forgetting, as well as rate controlling characteristics. The suggested structure shows a coding scheme which both input and output neurons participate, which can be expanded to larger neuromorphic neural network applications.

Enhancing hyperspectral remote sensing image classification using robust learning technique

Advanced sensor tech integrates into diverse applications, including remote sensing, robotics, and IoT. Combining artificial intelligence (AI) with sensors enhances their capabilities, creating smart sensors, revolutionizing remote sensing and Internet of Things (IoT). This synergy forms a potent technology in the field. This study carries out a comprehensive analysis of the progress made in Hyperspectral sensors and AI-based classification techniques that are employed in remote sensing fields that utilize hyperspectral images. The classification of images obtained from Hyperspectral Sensors (HSS) has emerged as a prominent research subject within the domain of remote sensing. HSS offer a wealth of information across numerous spectral bands, supporting diverse applications such as land cover classification, environmental monitoring, agricultural assessment, change detection, and more. However, the abundance of data present in HSS also poses the challenge called the curse of dimensionality. The reduction of data dimensionality is crucial before applying any machine learning model to achieve optimal results. The present study introduces a new hybrid strategy combining the Back-Propagation algorithm with a variable adaptive momentum (BPVAM) and principal component analysis (PCA) for the purpose of classifying hyperspectral images. PCA is first applied to obtain an optimal set of discriminative features by eliminating highly correlated and redundant features. These features are then fed into the BPVAM model for classification. The addition of the momentum term in the weight update equation of the backpropagation algorithm helped achieve faster convergence with high accuracy. The proposed model was subjected to evaluation through experiments conducted on two benchmark datasets. These results indicated that the hybrid model based on BPVAM with PCA is an efficient technique for HSS classification.

Field-Free Spin-Orbit Torque-Induced Magnetization Switching in the IrMn/CoTb Bilayers with a Spontaneous In-Plane Exchange Bias.

Spin-orbit torque (SOT)-induced magnetization switching in ferrimagnetic materials is promising for application in a new generation of information storage devices. Here, we demonstrate SOT-induced field-free magnetization switching of the perpendicularly magnetized CoTb ferrimagnet layer in the IrMn/CoTb bilayer, in which the in-plane magnetic inversion symmetry is broken by a spontaneous in-plane exchange bias (IEB) established by isothermal crystallization of the IrMn layer. We obtain a significant SOT effective field acting on the CoTb layer and a large effective spin Hall angle in this system, derived by the second harmonic voltage measurement method. Moreover, the IrMn/CoTb bilayer demonstrates multistate magnetic switching behavior with different amplitudes of current pulses at zero field, which can mimic the synaptic weight updates in the neuromorphic network. These findings make the IrMn/CoTb bilayer with spontaneous IEB a promising candidate for potential applications in multilevel storage and neuromorphic computing.

High optical contrast and multi-level storage of the ultracompact plasmonic device based on phase change materials

The combination of photonic devices and chalcogenide phase change materials (PCMs) provides a potential solution for photonic storage and computing. However, there are still some issues that need to be addressed for currently integrated phase-change photonic memory such as low speed, high energy consumption, and large footprint. To enable stronger interaction between light and PCMs, we utilize the plasmonic enhancement effect to amplify the local electric field, resulting in high energy efficiency in photonic storage and computation. This work proposes a nanoscale elliptical plasmonic nanoantenna that dramatically improves switching speed and energy by placing an Ag elliptical ring wrapped with Ge2Sb2Te5 (GST) on a silicon waveguide. The device exhibits crystalline and amorphous states with insertion losses of 0.3 dB and 1.7 dB, respectively, at a wavelength of 1550 nm. Write and erase operations can be completed by optical pulses of 1.5 ns–1 mW and 3 ns–3 mW, respectively. An ultra-small device volume of only 4.465 × 10−4 (µm3) is capable of achieving a switching contrast ratio of 28.5 %. Moreover, the non-volatile storage of multi-level intermediate states facilitates a wide weight-variation range and allows for precise weight updates, enhancing the overall performance and accuracy of the system. In the handwritten digit recognition task of the three-layer perceptron, the device demonstrates a recognition accuracy rate of 95 %, indicating that our approach can enable a more efficient neuromorphic photonic computing network.

Model-based neuroadaptive event-triggered tracking consensus control for nonlinear multiagent systems with input delay

In this article, a neuroadaptive event-triggered control methodology is proposed to tackle the leader-following tracking consensus issue of uncertain nonlinear multiagent systems (MASs) with input delay and external disturbances. The proposed approach incorporates radial bias function neural networks (RBF NNs) and the integral compensation technique, effectively addressing the issues of nonlinear uncertainty and input delay, respectively. For the sake of reducing the transmission of information, we adopt the strategy that the neural network (NN) weight update law is only changed at the event-triggered instant, and a novel adaptive model and related controller are constructed. Simultaneously, this paper establishes an impulsive dynamic system to analyze the stability of the closed-loop system by extending the Lyapunov theorem to continuous and reset dynamics. Sufficient conditions for achieving leader-following tracking consensus are derived without the Zeno behavior. Finally, a numerical example is presented to illustrate the validity of the proposed method.

Classification of Crops through Self-Supervised Decomposition for Transfer Learning

The 2S-DT (Self-Supervised Decomposition for Transfer Learning) model, created for crop categorization using remotely sensed data, is a unique method introduced in this paper. It deals with the difficulty of incorrectly identifying crops with comparable phenology patterns, a problem that frequently arises in agricultural remote sensing. Two datasets from Nanajangudu taluk in the Mysore district, which has a widely varied irrigated agriculture system, are used to assess the model. Using self-supervised learning, the 2S-DT model addresses the misclassification issue that frequently occurs when working with unlabeled classes, especially in high-resolution images. It uses class decomposition (CD) layer and a downstream learning approach. Using the model’s learning and the particulars of each geographical context, this layer improves the information’s arrangement. Our model architecture’s foundation is ResNet, a well-known deep learning framework. Each residual block in our ResNet architecture is made up of two 3x3 convolutional layers. Each convolutional layer is followed by batch normalization and Rectified Linear Unit (ReLU) activation functions, which improve the model’s capacity for learning. We utilized a 7x7 convolutional layer with 64 filters and a stride of 2 for Conv1 in ResNet18, resulting in an output size of 112x112x64. Conv2, which consists of Res2a and Res2b, generated an output with the dimensions 48x48x64. Conv3, which included Res3a and Res3b, produced an output with the dimensions 28x28x128. These architectural selections were made with our experimental needs in mind. The 2S-DT model’s newly added features make it easier to identify classes and update weights, improving the stability of the features’ spatial and spectral data. Extensive tests performed on two datasets show the model’s viability. Overall accuracy has improved significantly, with the 2S-DT model surpassing comparable models like TVSM, 3DCAE, and GAN Model by obtaining 95.65% accuracy for dataset 1 and 88.91% accuracy for dataset 2.

Structure-Aware Low-Rank Adaptation for Parameter-Efficient Fine-Tuning

With the growing scale of pre-trained language models (PLMs), full parameter fine-tuning becomes prohibitively expensive and practically infeasible. Therefore, parameter-efficient adaptation techniques for PLMs have been proposed to learn through incremental updates of pre-trained weights, such as in low-rank adaptation (LoRA). However, LoRA relies on heuristics to select the modules and layers to which it is applied, and assigns them the same rank. As a consequence, any fine-tuning that ignores the structural information between modules and layers is suboptimal. In this work, we propose structure-aware low-rank adaptation (SaLoRA), which adaptively learns the intrinsic rank of each incremental matrix by removing rank-0 components during training. We conduct comprehensive experiments using pre-trained models of different scales in both task-oriented (GLUE) and task-agnostic (Yelp and GYAFC) settings. The experimental results show that SaLoRA effectively captures the structure-aware intrinsic rank. Moreover, our method consistently outperforms LoRA without significantly compromising training efficiency.

Bulk-Switching Memristor-Based Compute-In-Memory Module for Deep Neural Network Training.

The constant drive to achieve higher performance in deep neural networks (DNNs) has led to the proliferation of very large models. Model training, however, requires intensive computation time and energy. Memristor-based compute-in-memory (CIM) modules can perform vector-matrix multiplication (VMM) in place and in parallel, and have shown great promises in DNN inference applications. However, CIM-based model training faces challenges due to non-linear weight updates, device variations, and low-precision. In this work, a mixed-precision training scheme is experimentally implemented to mitigate these effects using a bulk-switching memristor-based CIM module. Low-precision CIM modules are used to accelerate the expensive VMM operations, with high-precision weight updates accumulated in digital units. Memristor devices are only changed when the accumulated weight update value exceeds a pre-defined threshold. The proposed scheme is implemented with a system-onchip of fully integrated analog CIM modules and digital sub-systems, showing fast convergence of LeNet training to 97.73%. The efficacy of training larger models is evaluated using realistic hardware parameters and verifies that CIM modules can enable efficient mix-precision DNN training with accuracy comparable to full-precision software-trained models. Additionally, models trained on chip are inherently robust to hardware variations, allowing direct mapping to CIM inference chips without additional re-training.