MNIST Research Articles

Efficient memristor accelerator for transformer self-attention functionality

The adoption of transformer networks has experienced a notable surge in various AI applications. However, the increased computational complexity, stemming primarily from the self-attention mechanism, parallels the manner in which convolution operations constrain the capabilities and speed of convolutional neural networks (CNNs). The self-attention algorithm, specifically the matrix-matrix multiplication (MatMul) operations, demands a substantial amount of memory and computational complexity, thereby restricting the overall performance of the transformer. This paper introduces an efficient hardware accelerator for the transformer network, leveraging memristor-based in-memory computing. The design targets the memory bottleneck associated with MatMul operations in the self-attention process, utilizing approximate analog computation and the highly parallel computations facilitated by the memristor crossbar architecture. Remarkably, this approach resulted in a reduction of approximately 10 times in the number of multiply-accumulate (MAC) operations in transformer networks, while maintaining 95.47% accuracy for the MNIST dataset, as validated by a comprehensive circuit simulator employing NeuroSim 3.0. Simulation outcomes indicate an area utilization of 6895.7 μm2\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\mu m^2$$\\end{document}, a latency of 15.52 seconds, an energy consumption of 3 mJ, and a leakage power of 59.55 μW\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\mu W$$\\end{document}. The methodology outlined in this paper represents a substantial stride towards a hardware-friendly transformer architecture for edge devices, poised to achieve real-time performance.

Classification with electromagnetic waves

AbstractThe integration of neural networks and machine learning techniques has ushered in a revolution in various fields, including electromagnetic inversion, geophysical exploration, and microwave imaging. While these techniques have significantly improved image reconstruction and the resolution of complex inverse scattering problems, this paper explores a different question: Can near‐field electromagnetic waves be harnessed for object classification? To answer this question, we first create a dataset based on the MNIST dataset, where we transform the grayscale pixel values into relative electrical permittivity values to form scatterers and calculate the electromagnetic waves scattered from these objects using a 2D electromagnetic finite‐difference frequency‐domain solver. Then, we train various machine learning models with this dataset to classify the objects. When we compare the classification accuracy and efficiency of these models, we observe that the neural networks outperform others, achieving a 90% classification accuracy solely from the data without a need for projecting the input data into a latent space. The impacts of the training dataset size, the number of antennas, and the location of antennas on the accuracy and time spent during training are also investigated. These results demonstrate the potential for classifying objects with near‐field electromagnetic waves in a simple setup and lay the groundwork for further research in this exciting direction.

State-of-the-Art Results with the Fashion-MNIST Dataset

In September 2024, the Fashion-MNIST dataset will be 7 years old. Proposed as a replacement for the well-known MNIST dataset, it continues to be used to evaluate machine learning model architectures. This paper describes new results achieved with the Fashion-MNIST dataset using classical machine learning models and a relatively simple convolutional network. We present the state-of-the-art results obtained using the CNN-3-128 convolutional network and data augmentation. The developed CNN-3-128 model containing three convolutional layers achieved an accuracy of 99.65% in the Fashion-MNIST test image set. In addition, this paper presents the results of computational experiments demonstrating the dependence between the number of adjustable parameters of the convolutional network and the maximum acceptable classification quality, which allows us to optimise the computational cost of model training.

Fractional-order spike-timing-dependent gradient descent for multi-layer spiking neural networks

Accumulated detailed knowledge about the neuronal activities in human brains has brought more attention to bio-inspired spiking neural networks (SNNs). In contrast to non-spiking deep neural networks (DNNs), SNNs can encode and transmit spatiotemporal information more efficiently by exploiting biologically realistic and low-power event-driven neuromorphic architectures. However, the supervised learning of SNNs still remains a challenge because the spike-timing-dependent plasticity (STDP) of connected spiking neurons is difficult to implement and interpret in existing backpropagation learning schemes. This paper proposes a fractional-order spike-timing-dependent gradient descent (FO-STDGD) learning model by considering a derived nonlinear activation function that describes the relationship between the quasi-instantaneous firing rate and the temporal membrane potentials of nonleaky integrate-and-fire neurons. The training strategy can be generalized to any fractional orders between 0 and 2 since the FO-STDGD incorporates the fractional gradient descent method into the calculation of spike-timing-dependent loss gradients. The proposed FO-STDGD model is tested on the MNIST and DVS128 Gesture datasets and its accuracy under different network structure and fractional orders is analyzed. It can be found that the classification accuracy increases as the fractional order increases, and specifically, the case of fractional order 1.9 improves by 155 % relative to the case of fractional order 1 (traditional gradient descent). In addition, our scheme demonstrates the state-of-the-art computational efficacy for the same SNN structure and training epochs.

Out-of-Distribution Detection with Memory-Augmented Variational Autoencoder

This paper proposes a novel method capable of both detecting OOD data and generating in-distribution data samples. To achieve this, a VAE model is adopted and augmented with a memory module, providing capacities for identifying OOD data and synthesising new in-distribution samples. The proposed VAE is trained on normal data and the memory stores prototypical patterns of the normal data distribution. At test time, the input is encoded by the VAE encoder; this encoding is used as a query to retrieve related memory items, which are then integrated with the input encoding and passed to the decoder for reconstruction. Normal samples reconstruct well and yield low reconstruction errors, while OOD inputs produce high reconstruction errors as their encodings get replaced by retrieved normal patterns. Prior works use memory modules for OOD detection with autoencoders, but this method leverages a VAE architecture to enable generation abilities. Experiments conducted with CIFAR-10 and MNIST datasets show that the memory-augmented VAE consistently outperforms the baseline, particularly where OOD data resembles normal patterns. This notable improvement is due to the enhanced latent space representation provided by the VAE. Overall, the memory-equipped VAE framework excels in identifying OOD and generating creative examples effectively.

A new perspective on optimizers: leveraging moreau-yosida approximation in gradient-based learning

Machine Learning (ML) heavily relies on optimization techniques built upon gradient descent. Numerous gradient-based update methods have been proposed in the scientific literature, particularly in the context of neural networks, and have gained widespread adoption as optimizers in ML software libraries. This paper introduces a novel perspective by framing gradient-based update strategies using the Moreau-Yosida (MY) approximation of the loss function. Leveraging a first-order Taylor expansion, we demonstrate the concrete exploitability of the MY approximation to generalize the model update process. This enables the evaluation and comparison of regularization properties underlying popular optimizers like gradient descent with momentum, ADAGRAD, RMSprop, and ADAM. The MY-based unifying view opens up possibilities for designing new update schemes with customizable regularization properties. To illustrate this potential, we propose a case study that redefines the concept of closeness in the parameter space using network outputs. We present a proof-of-concept experimental procedure, demonstrating the effectiveness of this approach in continual learning scenarios. Specifically, we employ the well-known permuted MNIST dataset, a progressively-permuted MNIST and CIFAR-10 benchmarks, and a non i.i.d. stream. Additionally, we validate the update scheme’s efficacy in an offline-learning scenario. By embracing the MY-based unifying view, we pave the way for advancements in optimization techniques for machine learning.

All-optical reconfigurable optical neural network chip based on wavelength division multiplexing

Optical computing has become an important way to achieve low power consumption and high computation speed. Optical neural network (ONN) is one of the key branches of optical computing due to its wide range of applications. However, the integrated ONN schemes proposed in previous works have some disadvantages, such as fixed network structure, complex matrix-vector multiplication (MVM) unit, and few all-optical nonlinear activation function (NAF) methods. Moreover, for the most compact MVM schemes based on wavelength division multiplexing (WDM), it is infeasible to employ intrinsic nonlinear effects to implement NAF, which brings frequent O-E-O conversion in ONN chips. Besides, it is also hard to realize a reconfigurable ONN with coherent MVMs, while it is much easier to implement in WDM schemes. We propose for the first time an all-optical silicon-based ONN chip based on WDM by adopting a new adjustment mechanism: optical gradient force (OGF). The proposed scheme is reconfigurable with tunable layers, variable neurons per layer, and adjustable NAF curves. In the task of classification of the MNIST dataset, our chip can realize an accuracy of 85.13% with 4 full-connected layers and only 50 neurons in total. In addition, we analyze the influence of the OGF-based NAF under fabrication errors and propose a calibration method. Compared to the previous works, our scheme has the two-fold advantages of compactness and reconfiguration, and it paves the way for the all-optical ONN based on WDM and opens the path to unblocking the bottleneck of integrated large-dimension ONNs.

Mixed precision quantization of silicon optical neural network chip

In recent years, the field of neural network research has witnessed remarkable advancements in various domains. One of the emerging approaches is the integration of photonic computing, which leverages the unique properties of light for ultra-fast information processing. In this article, we establish a mixed precision quantization model to silicon-based optical neural networks and evaluates their performance on the MNIST and Fashion-MNIST datasets. Through a genetic algorithm-based optimization process, we achieve significant parameter compression while maintaining competitive accuracy. Our findings demonstrate that with an average quantization bitwidth of 4.5 bits on the MNIST dataset, we achieve an impressive 85.94% reduction in parameter size compared to traditional 32-bit networks, with only a marginal accuracy drop of 0.65%. Similarly, on the Fashion-MNIST dataset, we achieve an average quantization bitwidth of 5.67 bits, resulting in an 82.28% reduction in parameter size with a slight accuracy drop of 0.8%.

A complex neural network model by Hilbert Transform

The phase information of the optical wave plays a vital role in processing wave-related signals. In deep learning fields, complex-valued neural networks are put forward on the concept of complex amplitudes for full utilization of phase information. To build a complex-valued neural network, the common way is to exploit Fourier Transform of the observed signal to extract amplitude and phase information. However, this will lead to spectrum waste for a real-valued signal by introducing negative frequencies that have no physical meaning. To this end, we attempt to use Hilbert Transform as an alternative to yield a single sideband spectrum and avoid negative frequencies from interacting with positive ones. On the other hand, Fourier transform is a global analysis thus it tells nothing about the time domain. As our key insight, we further explore the usage of instantaneous frequency calculated by Hilbert Transform and propose a new method of constructing complex input from a time–frequency angle. Simple pixel-wise classification experiments are carried out on two hyperspectral datasets and MNIST dataset. Experimental results have demonstrated that Hilbert Transform with instantaneous frequency performs better by a large margin than Fourier Transform owing to the additional time information.

Intelligent Handwritten Identification Using Novel Hybrid Convolutional Neural Networks – Long-short-term Memory Architecture

Handwritten character identification finds broad applications in document analysis, digital forensics, and human-computer interaction. Conventional methods encounter challenges in accurately deciphering a range of handwriting styles and variations. Consequently, investigating intelligent system for handwriting identification becomes crucial to enhance accuracy and efficiency. This paper introduces an innovative hybrid deep learning architecture, seamlessly integrating the strengths of convolutional neural networks (CNN) and long-short term memory (LSTM) within a one single framework. The combination of these structures enables the model to effectively capture both spatial features and temporal dependencies inherent in handwritten strokes, resulting in improved recognition performance. The proposed 2DCNN-LSTM algorithm has been tested on MNIST dataset. The proposed hybrid CNN-LSTM structure has been compared to conventional intelligent machine learning methods, and the results demonstrate the superior performance of the hybrid CNN-LSTM, showcasing heightened accuracy, sensitivity, specificity, and other evaluation metrics.

VFLGAN: Vertical Federated Learning-based Generative Adversarial Network for Vertically Partitioned Data Publication

In the current artificial intelligence (AI) era, the scale and quality of the dataset play a crucial role in training a high-quality AI model. However, good data is not a free lunch and is always hard to access due to privacy regulations like the General Data Protection Regulation (GDPR). A potential solution is to release a synthetic dataset with a similar distribution to that of the private dataset. Nevertheless, in some scenarios, it has been found that the attributes needed to train an AI model belong to different parties, and they cannot share the raw data for synthetic data publication due to privacy regulations. In PETS 2023, Xue et al. [29] proposed the first generative adversary network-based model, VertiGAN, for vertically partitioned data publication. However, after thoroughly investigating, we found that VertiGAN is less effective in preserving the correlation among the attributes of different parties. This article proposes a Vertical Federated Learning-based Generative Adversarial Network, VFLGAN, for vertically partitioned data publication to address the above issues. Our experimental results show that compared with VertiGAN, VFLGAN significantly improves the quality of synthetic data. Taking the MNIST dataset as an example, the quality of the synthetic dataset generated by VFLGAN is 3.2 times better than that generated by VertiGAN w.r.t. the Frechet Distance. We also designed a more efficient and effective Gaussian mechanism for the proposed VFLGAN to provide the synthetic dataset with a differential privacy guarantee. On the other hand, differential privacy only gives the upper bound of the worst-case privacy guarantee. This article also proposes a practical auditing scheme that applies membership inference attacks to estimate privacy leakage through the synthetic dataset.

Asymmetric Spatiotemporal Online Learning for Deep Spiking Neural Networks

Although the precise symmetric forward and feedback connections between neurons are thought to be impossible in the brain, most existing deep spiking neural networks (SNNs) spatio-temporal learning algorithms still require such a strong architectural constraint to complete tasks. Besides that, as an expected feature for specialized neuromorphic hardware to be deployed, effective online learning of deep SNNs for training spatio-temporal data are still lacking. To address these issues, an effective biologically plausible asymmetric spatio-temporal online learning algorithm called ASTOL is proposed for training deep SNNs in real-time. ASTOL could learn the spatial and temporal features simultaneously in real-time, without the precise weights symmetric backpropagation and the need to know the duration of spatio-temporal data in advance. Experimental results show that the proposed ASTOL algorithm achieves comparable performance in real-time learning on the rate coding MNIST dataset and the temporal coding music MedleyDB dataset and speech TIDIGITS dataset with other state-of-the-art algorithms with simpler learning mechanism way as it avoids transport of synaptic weights. Besides that, the experiment on MNIST datasets also shows that ASTOL can use simpler network structures to achieve good performance faster.

A robust federated learning algorithm for partially trusted environments

Due to the distributed nature of federated learning, it is vulnerable to poisoning attacks during the training process. The model’s resistance to poisoning attacks can be improved using robust aggregation algorithms. Current research on federated learning to resist poisoning attacks is mainly based on two settings: No trust or Byzantine robustness. However, both settings are not close enough to reality in practical scenarios. In many practical applications, some participants in federated learning are trustworthy. For example, participants who have participated in the training of this model before and performed very well, or participants with strong compliance and credibility such as governments and some national agencies participate in the training. In existing research, these trusted participants still have to accept the judgment of the aggregation node, which generates unnecessary computation, increases overhead, and does not take advantage of a trusted environment. Since there is no attack behavior on the trusted client, its training results are used to classify the trustworthiness of other untrusted clients and identify attack nodes with higher accuracy. Therefore, this paper proposes a robust federated learning algorithm for partially trusted environments. The proposed scheme uses the experimental results of trusted clients to judge the behavior of untrustworthy clients by the cosine similarity and the Local Outlier Factor and further identify and detect malicious clients. Experiments are performed on MNIST and CIFAR datasets. Comparison with other six aggregation algorithms under 30% attack scenario. And compared with the other four aggregation algorithms under 70% attack conditions. Our algorithm is more accurate than almost all of the other aggregation algorithms. The paper is the first to conduct robust research on federated learning in a partially trusted environment, and the proposed algorithm can more effectively resist poisoning attacks.

Multi-adversarial autoencoders: Stable, faster and self-adaptive representation learning

The variational autoencoder (VAE) and generative adversarial networks (GAN) are two prominent approaches to achieving a probabilistic generative model by way of an autoencoder and a two-player minimax game. While VAEs often suffer from over-simplified posterior approximations, the adversarial autoencoder (AAE) has shown promise by adopting GAN to match the variational posterior to an arbitrary prior through adversarial training. Both VAEs and GANs face significant challenges such as training stability, mode collapse, and difficulty in extracting meaningful latent representations. In this paper, we propose the Multi-adversarial Autoencoder (MAAE), which extends the AAE framework by incorporating multiple discriminators and enabling soft-ensemble feedback. By adaptively regulating the collective feedback from multiple discriminators, MAAE captures a balance between fitting the data distribution and performing accurate inference and accelerates training stability while extracting meaningful and interpretable latent representations. Experimental evaluations on MNIST, CIFAR10, and CelebA datasets demonstrate significant improvements in latent representation, quality of generated samples, log-likelihood, and a pairwise comparison metric, with comparisons to recent methods.

SecureEI: Proactive intellectual property protection of AI models for edge intelligence

Deploying AI models on edge computing platforms enhances real-time performance, reduces network dependency, and ensures data privacy on terminal devices. However, these advantages come with increased risks of model leakage and misuse due to the vulnerability of edge environments to physical and cyber attacks compared to cloud-based solutions. To mitigate these risks, we propose SecureEI, a proactive intellectual property protection method for AI models that leverages model splitting and data poisoning techniques. SecureEI divides the model into two components: DeviceNet, which processes input data into protected license data, and EdgeNet, which operates on the license data to perform the intended tasks. This method ensures that only the transformed license data yields high model accuracy, while original data remains unrecognizable, even under fine-tuning attacks. We further employ targeted training strategies and weight adjustments to enhance the model’s resistance to potential attacks that aim to restore its recognition capabilities for original data. Evaluations on MNIST, Cifar10, and FaceScrub datasets demonstrate that SecureEI not only maintains high model accuracy on license data but also significantly bolsters defense against fine-tuning attacks, outperforming existing state-of-the-art techniques in safeguarding AI intellectual property on edge platforms.

Supervised structure learning

This paper concerns structure learning or discovery of discrete generative models. It focuses on Bayesian model selection and the assimilation of training data or content, with a special emphasis on the order in which data are ingested. A key move—in the ensuing schemes—is to place priors on the selection of models, based upon expected free energy. In this setting, expected free energy reduces to a constrained mutual information, where the constraints inherit from priors over outcomes (i.e., preferred outcomes). The resulting scheme is first used to perform image classification on the MNIST dataset to illustrate the basic idea, and then tested on a more challenging problem of discovering models with dynamics, using a simple sprite-based visual disentanglement paradigm and the Tower of Hanoi (cf., blocks world) problem. In these examples, generative models are constructed autodidactically to recover (i.e., disentangle) the factorial structure of latent states—and their characteristic paths or dynamics.

A multi‐dimensional incentive mechanism based on age of update in hierarchical federated learning

AbstractFederated learning represents a decentralized approach to machine learning, enabling numerous devices to collaboratively contribute to model training while ensuring the privacy of individual data. However, the existing incentive mechanism of hierarchical federated learning (HFL) only considers the data contribution of a single round, which needs to be revised. For non‐IID data sets, the continuous selection of any end devices will cause the weights to diverge in a specific direction. Therefore, a new metric is needed to avoid continuously selecting a certain end device to ensure the overall effectiveness. We introduce a metric to describe the importance of updates: age of update (AoU), which can help select end devices not selected in the previous round to promote a faster model convergence. We put forward an incentive mechanism based on AoU, reputation, and data quantity in HFL (ARDHFL). We have derived the optimal equilibrium solution for the three‐stage Stackelberg game. Based on this solution, we can ensure maximum edge‐cloud utility while incentivizing end devices to engage actively in HFL tasks and providing superior data to train the HFL model. Finally, we conducted extensive experiments to prove that ARDHFL can effectively improve the performance. Compared with the fixed scheme, random scheme, FMore and InFEDge, the testing accuracy of ARDHFL in the MNIST dataset has been improved by 29.7%, 9.3%, 6.8% and 6.1%, respectively. In the CIFAR‐10 dataset, it has been improved by 40.2%, 33.1%, 16.4% and 14.2%, respectively, and demands fewer communication iterations to achieve the same testing accuracy.

Development of a conditional variational autoencoder for handwritten digit recognition.

This study examines the performance of Conditional Variational Autoencoder (CVAE) in handwritten digit recognition. Using the MNIST dataset, two variants of the CVAE models — convolutional and multilevel architecture — were developed and compared. The research methodology includes comprehensive data preprocessing, architecture design, training, and thorough evaluation processes. The obtained data highlight the better performance of the convolutional model-based CVAE in achieving recognition accuracy compared to its multilayer counterpart. Evaluation metrics include analysis of original and reconstructed images, comparison of hidden layer vector distribution patterns, and visualization of loss function dynamics. In addition, the study highlights the practical implications of CVAEs in various fields, highlighting their performance in digit recognition tasks due to their inherent robustness and extraordinary generalizability.

Comparative analysis of CNN models for handwritten digit recognition

The paper discusses the subject of convolutional neural networks used for handwritten digit classification. The purpose of the research is to evaluate the accuracy, performance, training, and classification time of three OCR networks (VGG-16, VGG-19 and AlexNet) and compare them with each other while selecting the most optimal one. The popular MNIST dataset of 70,000 images was used for the study. For each model, a preliminary study was conducted to determine the optimal parameters in the form of the number of input data and number of training epochs. The result of the work indicates that, despite the longer training and classification time, the AlexNet model achieved the highest precision, recall, and F1-score, indicating its ability to effectively classify images.

Reconfigurable Five-In-One Carbon Nanotube Optoelectronic Transistor for Intelligent Computing and Communication.

Advances in artificial general intelligence (AGI) necessitate the integration of diverse functionalities to address complex tasks. Carbon nanotubes (CNTs), with their unique physical properties, have broad applications in emerging research fields, providing a foundation for next-generation devices that could overcome the limits of Moore's Law. This work demonstrates a novel intelligent device that integrates five functions─sensors, memory, neuromorphic computing, logic, and communication─using CNT field-effect transistors (CNFETs) compatible with CMOS processes. Through passivation and annealing techniques, we have significantly enhanced the optoelectronic performance of CNFETs, leading to the development of multifunctional optoelectronic synaptic transistors. These optimized CNFETs enable dual-mode weight-tunable synaptic functions, including long-term plasticity and multilevel storage. Additionally, a CNT-based neural network has achieved high recognition accuracy on the MNIST data set, showcasing the potential of in-memory computing. This research also innovates by integrating logic functions with optoelectronic communication capabilities, paving the way for next-generation intelligent computing and communication integrated systems.