Handwritten Digit Dataset Research Articles

A Variation-Aware Binary Neural Network Framework for Process Resilient In-Memory Computations

Binary neural networks (BNNs) that use 1-bit weights and activations have garnered interest as extreme quantization provides low power dissipation. By implementing BNNs as computation-in-memory (CIM), which computes multiplication and accumulations on memory arrays in an analog fashion, namely, analog CIM, we can further improve the energy efficiency to process neural networks. However, analog CIMs are susceptible to process variation, which refers to the variability in manufacturing that causes fluctuations in the electrical properties of transistors, resulting in significant degradation in BNN accuracy. Our Monte Carlo simulations demonstrate that in an SRAM-based analog CIM implementing the VGG-9 BNN model, the classification accuracy on the CIFAR-10 image dataset is degraded to below 50% under process variations in a 28 nm FD-SOI technology. To overcome this problem, we present a variation-aware BNN framework. The proposed framework is developed for SRAM-based BNN CIMs since SRAM is most widely used as on-chip memory; however, it is easily extensible to BNN CIMs based on other memories. Our extensive experimental results demonstrate that under process variation of 28 nm FD-SOI, with an SRAM array size of 128×128, our framework significantly enhances classification accuracies on both the MNIST hand-written digit dataset and the CIFAR-10 image dataset. Specifically, for the CONVNET BNN model on MNIST, accuracy improves from 60.24% to 92.33%, while for the VGG-9 BNN model on CIFAR-10, accuracy increases from 45.23% to 78.22%.

Atomic Nb-doping of WS2 for high-performance synaptic transistors in neuromorphic computing

Owing to the controllable growth and large-area synthesis for high-density integration, interest in employing atomically thin two-dimensional (2D) transition-metal dichalcogenides (TMDCs) for synaptic transistors is increasing. In particular, substitutional doping of 2D materials allows flexible modulation of material physical properties, facilitating precise control in defect engineering for eventual synaptic plasticity. In this study, to increase the switch ratio of synaptic transistors, we selectively performed experiments on WS2 and introduced niobium (Nb) atoms to serve as the channel material. The Nb atoms were substitutionally doped at the W sites, forming a uniform distribution across the entire flakes. The synaptic transistor devices exhibited an improved switch ratio of 103, 100 times larger than that of devices prepared with undoped WS2. The Nb atoms in WS2 play crucial roles in trapping and detrapping electrons. The modulation of channel conductivity achieved through the gate effectively simulates synaptic potentiation, inhibition, and repetitive learning processes. The Nb-WS2 synaptic transistor achieves 92.30% recognition accuracy on the Modified National Institute of Standards and Technology (MNIST) handwritten digit dataset after 125 training iterations. This study’s contribution extends to a pragmatic and accessible atomic doping methodology, elucidating the strategies underlying doping techniques for channel materials in synaptic transistors.

Defect-engineered monolayer MoS2 with enhanced memristive and synaptic functionality for neuromorphic computing

Two-dimensional transition metal dichalcogenides (TMDs)-based memristors are promising candidates for realizing artificial synapses in next-generation computing. However, practical implementation faces several challenges, such as high non-linearity and asymmetry in synaptic weight updates, limited dynamic range, and cycle-to-cycle variability. Here, utilizing optimal-power argon plasma treatment, we significantly enhance the performance matrix of memristors fabricated from monolayer MoS2. Our approach not only improves linearity and symmetry in synaptic weight updates but also increases the number of available synaptic weight updates and enhances Spike-Time Dependent Plasticity. Notably, it broadens the switching ratio by two orders, minimizes cycle-to-cycle variability, reduces non-linear factors, and achieves an energy consumption of ~30 fJ per synaptic event. Implementation of these enhancements is demonstrated through Artificial Neural Network simulations, yielding a learning accuracy of ~97% on the MNIST hand-written digits dataset. Our findings underscore the significance of defect engineering as a powerful tool in advancing the synaptic functionality of memristors.

Partial coherence enhances parallelized photonic computing

Advancements in optical coherence control1–5 have unlocked many cutting-edge applications, including long-haul communication, light detection and ranging (LiDAR) and optical coherence tomography6–8. Prevailing wisdom suggests that using more coherent light sources leads to enhanced system performance and device functionalities9–11. Our study introduces a photonic convolutional processing system that takes advantage of partially coherent light to boost computing parallelism without substantially sacrificing accuracy, potentially enabling larger-size photonic tensor cores. The reduction of the degree of coherence optimizes bandwidth use in the photonic convolutional processing system. This breakthrough challenges the traditional belief that coherence is essential or even advantageous in integrated photonic accelerators, thereby enabling the use of light sources with less rigorous feedback control and thermal-management requirements for high-throughput photonic computing. Here we demonstrate such a system in two photonic platforms for computing applications: a photonic tensor core using phase-change-material photonic memories that delivers parallel convolution operations to classify the gaits of ten patients with Parkinson’s disease with 92.2% accuracy (92.7% theoretically) and a silicon photonic tensor core with embedded electro-absorption modulators (EAMs) to facilitate 0.108 tera operations per second (TOPS) convolutional processing for classifying the Modified National Institute of Standards and Technology (MNIST) handwritten digits dataset with 92.4% accuracy (95.0% theoretically).

Integrated sources model: A new space-learning model for heterogeneous multi-view data reduction, visualization, and clustering

In machine learning, multi-view data involve multiple distinct sets of attributes (“views”) for a common set of observations; when each view has the same attributes considered in different contexts, the data are said to contain multiple views of homogeneous format, which can be conceptualized as a tensor. In this article, we describe a novel approach for integrating multiple views of heterogeneous format into a common latent space using a workflow that involves non-negative matrix and tensor factorization (NMF/NTF). This approach, which we refer to as the integrated sources model (ISM), consists of two main steps: Embedding and analysis. In the embedding step, the views are transformed into matrices with common non-negative components. In the analysis step, the transformed views are combined into a tensor and decomposed using NTF. We also present a variant of ISM; the integrated latent sources model (ILSM), which offers significant advantages over ISM in terms of computational power and in cases where the views are highly unbalanced with regard to the number of attributes per view. Noteworthy, ISM can be extended to process multi-omic and multi-view datasets even in the presence of missing views. We provide a proof-of-concept analysis using five examples, including the UCI Digits (the University of California Irvine Pen-Based Recognition of Handwritten Digits) dataset, a public cell-type gene signatures dataset, and a multi-omic single-cell dataset. These examples demonstrate that, in most cases, multi-view clustering is better achieved with ISM or its variant ILSM than with other latent space approaches. We also show how the non-negativity and sparsity of the ISM model components enable straightforward interpretations, in contrast to other approaches that involve latent factors of mixed signs. Finally, we present potential applications to single-cell multi-omics and spatial mapping, including spatial imaging, spatial transcriptomics, and computational biology, which are currently under evaluation. ISM relies on state-of-the-art algorithms invoked through a simple workflow implemented in Python.

WI-TMLEGA: Weight Initialization and Training Method Based on Entropy Gain and Learning Rate Adjustment.

Addressing the issues of prolonged training times and low recognition rates in large model applications, this paper proposes a weight training method based on entropy gain for weight initialization and dynamic adjustment of the learning rate using the multilayer perceptron (MLP) model as an example. Initially, entropy gain was used to replace random initial values for weight initialization. Subsequently, an incremental learning rate strategy was employed for weight updates. The model was trained and validated using the MNIST handwritten digit dataset. The experimental results showed that, compared to random initialization, the proposed initialization method improves training effectiveness by 39.8% and increases the maximum recognition accuracy by 8.9%, demonstrating the feasibility of this method in large model applications.

Resistive switching and synaptic characteristics of Hf-doped ZnO sandwiched between HfO2-based memristors for neuromorphic computing

With the rise of neuromorphic computing, memristors have attracted much attention as a device with potential synaptic properties. In this paper, we investigated the embedding of Hf-doped ZnO (Hf:ZnO) layers with different doping concentrations in HfO2-based memristors to design HfO2/(0 %-10 %)Hf:ZnO/HfO2 tri-layer structured memristors for resistive switching (RS) and synaptic characteristics. Utilizing first-principles calculations, we integrated assessments of both oxygen vacancy migration capability and material conductivity to evaluate the rationale behind our design. The experimental results show that the HfO2/(5 %)Hf:ZnO/HfO2 tri-layer structured memristor presents the best RS performance, with a storage window of 103. We analyze the impact of insertion layers with varying doping concentrations on the RS performance of the devices and elucidate the RS mechanism. In addition, we investigated the synaptic properties of HfO2/(5 %)Hf:ZnO/HfO2 tri-layer structured memristor, and effectively simulated synaptic plasticity akin to the synaptic behavior found in biological neurons. Finally, we also constructed a memristor-based Spiking Neural Network for recognizing the Mixed National Institute of Standards and Technology handwritten digit dataset, achieving a recognition accuracy of 82 %. This comprehensive study shows that the insertion of (5 %)Hf:ZnO into HfO2-based memristors holds significant promise for upcoming applications in the field of neuromorphic computing.

Ultra-fast GST-based optical neuron for the implementation of integrated photonic neural networks

A fast-integrated optical neuron is proposed based on a micro-ring resonator structure. The GST phase change material is used in the MRR structure, and neuron switching is achieved by the electrical actuation of this material. The nonlinear activation function of the neuron is extracted through three-dimensional electrical and optical simulations. The neuron is then employed in a three-layer perceptron network using the extracted activation function, and its performance is examined in the classification of handwritten digits. The overall performance of the proposed neuron is compared with the previous optical neural networks. Simulations show an ultra-fast switching time of 200fs, which is one order of magnitude faster than the best-reported value, and an accuracy of 98.9% (in the classification of MNIST handwritten digits dataset) which is among the best results, while the estimated neuron footprint is relatively small.

Role of spatial coherence in diffractive optical neural networks

Diffractive optical neural networks (DONNs) have emerged as a promising optical hardware platform for ultra-fast and energy-efficient signal processing for machine learning tasks, particularly in computer vision. Previous experimental demonstrations of DONNs have only been performed using coherent light. However, many real-world DONN applications require consideration of the spatial coherence properties of the optical signals. Here, we study the role of spatial coherence in DONN operation and performance. We propose a numerical approach to efficiently simulate DONNs under incoherent and partially coherent input illumination and discuss the corresponding computational complexity. As a demonstration, we train and evaluate simulated DONNs on the MNIST dataset of handwritten digits to process light with varying spatial coherence.

Predictive coding with spiking neurons and feedforward gist signaling.

Predictive coding (PC) is an influential theory in neuroscience, which suggests the existence of a cortical architecture that is constantly generating and updating predictive representations of sensory inputs. Owing to its hierarchical and generative nature, PC has inspired many computational models of perception in the literature. However, the biological plausibility of existing models has not been sufficiently explored due to their use of artificial neurons that approximate neural activity with firing rates in the continuous time domain and propagate signals synchronously. Therefore, we developed a spiking neural network for predictive coding (SNN-PC), in which neurons communicate using event-driven and asynchronous spikes. Adopting the hierarchical structure and Hebbian learning algorithms from previous PC neural network models, SNN-PC introduces two novel features: (1) a fast feedforward sweep from the input to higher areas, which generates a spatially reduced and abstract representation of input (i.e., a neural code for the gist of a scene) and provides a neurobiological alternative to an arbitrary choice of priors; and (2) a separation of positive and negative error-computing neurons, which counters the biological implausibility of a bi-directional error neuron with a very high baseline firing rate. After training with the MNIST handwritten digit dataset, SNN-PC developed hierarchical internal representations and was able to reconstruct samples it had not seen during training. SNN-PC suggests biologically plausible mechanisms by which the brain may perform perceptual inference and learning in an unsupervised manner. In addition, it may be used in neuromorphic applications that can utilize its energy-efficient, event-driven, local learning, and parallel information processing nature.

Deep Learning-Based Dzongkha Handwritten Digit Classification

In computer vision applications, pattern recognition is one of the important fields in artificial intelligence. With the advancement in deep learning technology, many machine learning algorithms were developed to tackle the problem of pattern recognition. The purpose of conducting the research is to create the first-ever Dzongkha handwritten digit dataset and develop a model to classify the digit. In the study, the 3 layer set of CONV → ReLU → POOL, followed by a fully connected layer, dropout layer, and softmax function were used to train the digit. In the dataset, each class (0-9) contains 1500 images which are split into train, validation, and test sets: 70:20:10. The model was trained on three different image dimensions: 28 by 28, 32 by 32, and 64 by 64. Compared to image dimensions 28 by 28 and 32 by 32, 64 by 64 gave the highest train, validation, and test accuracy of 98.66%, 98.9%, and 99.13% respectively. In the future, the sample of digits needs to be increased and use the transfer learning concept to train the model.

Multilevel Ferroelectric Domain Wall Memory for Neuromorphic Computing

AbstractLow‐power and parallel processing Neuromorphic computing that imitates on the human brain's extraordinary data processing and learning capabilities is promising in non‐von Neumann application platforms in the post‐Moore era. Ferroelectric domain walls appearing as nanoscale topological defects in solids exhibit coexisting ordering parameters besides the spontaneous polarization, leading to the emergence of rich physics and electronic effects in the development of neuromorphic electronics that go beyond the conventional CMOS. In this study, the four‐state domain wall memory is developed on LiNbO3 ferroelectric thin films integrated with Si substrates, where each state can be stably manipulated at a specific low voltage, showcasing outstanding fatigue resistance and retention performance. The constructed crossbar array that functions as a kernel for image convolution can be used to implement processing operations like image filtering, sharpness enhancement, and edge detection. Additional simulation from a convolutional neural network model shows 96.98% accuracy on the Modified National Institute of Standards and Technology handwritten digits dataset. The stable multi‐state domain wall memory provides a new approach for computing and promotes research in domain wall electronics to meet future enormous demands of big data.

Distributed quantum neural networks via partitioned features encoding

Quantum neural networks are expected to be a promising application in near-term quantum computing, but face challenges such as vanishing gradients during optimization and limited expressibility by a limited number of qubits and shallow circuits. To mitigate these challenges, an approach using distributed quantum neural networks has been proposed to make a prediction by approximating outputs of a large circuit using multiple small circuits. However, the approximation of a large circuit requires an exponential number of small circuit evaluations. Here, we instead propose to distribute partitioned features over multiple small quantum neural networks and use the ensemble of their expectation values to generate predictions. To verify our distributed approach, we demonstrate ten class classification of the Semeion and MNIST handwritten digit datasets. The results of the Semeion dataset imply that while our distributed approach may outperform a single quantum neural network in classification performance, excessive partitioning reduces performance. Nevertheless, for the MNIST dataset, we succeeded in ten class classification with exceeding 96% accuracy. Our proposed method not only achieved highly accurate predictions for a large dataset but also reduced the hardware requirements for each quantum neural network compared to a large single quantum neural network. Our results highlight distributed quantum neural networks as a promising direction for practical quantum machine learning algorithms compatible with near-term quantum devices. We hope that our approach is useful for exploring quantum machine learning applications.

Secure Aggregation Protocol Based on DC-Nets and Secret Sharing for Decentralized Federated Learning.

In the era of big data, millions and millions of data are generated every second by different types of devices. Training machine-learning models with these data has become increasingly common. However, the data used for training are often sensitive and may contain information such as medical, banking, or consumer records, for example. These data can cause problems in people's lives if they are leaked and also incur sanctions for companies that leak personal information for any reason. In this context, Federated Learning emerges as a solution to the privacy of personal data. However, even when only the gradients of the local models are shared with the central server, some attacks can reconstruct user data, allowing a malicious server to violate the FL principle, which is to ensure the privacy of local data. We propose a secure aggregation protocol for Decentralized Federated Learning, which does not require a central server to orchestrate the aggregation process. To achieve this, we combined a Multi-Secret-Sharing scheme with a Dining Cryptographers Network. We validate the proposed protocol in simulations using the MNIST handwritten digits dataset. This protocol achieves results comparable to Federated Learning with the FedAvg protocol while adding a layer of privacy to the models. Furthermore, it obtains a timing performance that does not significantly affect the total training time, unlike protocols that use Homomorphic Encryption.

Echo State Property upon Noisy Driving Input

The echo state property (ESP) is a key concept for understanding the working principle of the most widely used reservoir computing model, the echo state network (ESN). The ESP is achieved most of the operation time under general conditions, yet the property is lost when a combination of driving input signals and intrinsic reservoir dynamics causes unfavorable conditions for forgetting the initial transient state. A widely used treatment, setting the spectral radius of the weight matrix below the unity, is not sufficient as it may not properly account for the nature of driving inputs. Here, we characterize how noisy driving inputs affect the dynamical properties of an ESN and the empirical evaluation of the ESP. The standard ESN with a hyperbolic tangent activation function is tested using the MNIST handwritten digit datasets at different additive white Gaussian noise levels. The correlations among the neurons, input mapping, and memory capacity of the reservoir nonlinearly decrease with the noise level. These trends agree with the deterioration of the MNIST classification accuracy against noise. In addition, the ESP index for noisy driving input is developed as a tool to help easily assess ESPs in practical applications. Bifurcation analysis explicates how the noise destroys an asymptotical convergence in an ESN and confirms that the proposed index successfully captures the ESP against noise. These results pave the way for developing noise-robust reservoir computing systems, which may promote the validity and utility of reservoir computing for real-world machine learning applications.

Empowering edge devices: FPGA‐based 16‐bit fixed‐point accelerator with SVD for CNN on 32‐bit memory‐limited systems

AbstractConvolutional neural networks (CNNs) are now often used in deep learning and computer vision applications. Its convolutional layer accounts for most calculations and should be computed fast in a local edge device. Field‐programmable gate arrays (FPGAs) have been adequately explored as promising hardware accelerators for CNNs due to their high performance, energy efficiency, and reconfigurability. This paper developed an efficient FPGA‐based 16‐bit fixed‐point hardware accelerator unit for deep learning applications on the 32‐bit low‐memory edge device (PYNQ‐Z2 board). Additionally, singular value decomposition is applied to the fully connected layer for dimensionality reduction of weight parameters. The accelerator unit was designed for all five layers and employed eight processing elements in convolution layers 1 and 2 for parallel computations. In addition, array partitioning, loop unrolling, and pipelining are the techniques used to increase the speed of calculations. The AXI‐Lite interface was also used to communicate between IP and other blocks. Moreover, the design is tested with grayscale image classification on MNIST handwritten digit dataset and color image classification on the Tumor dataset. The experimental results show that the proposed accelerator unit implementation performs faster than the software‐based implementation. Its inference speed is 89.03% more than INTEL 3‐core CPU, 86.12% higher than Haswell 2‐core CPU, and 82.45% more than NVIDIA Tesla K80 GPU. Furthermore, the throughput of the proposed design is 4.33GOP/s, which is better than the conventional CNN accelerator architectures.

Strain-mediated multistate skyrmion for neuron devices.

Magnetic skyrmions are potential candidates for neuromorphic computing because of their inherent topological stability, low drive current density and nanoscale size. However, an artificial neuron device based on current-driven skyrmion motion cannot satisfy the requirement of energy efficiency and integration density due to hundreds of millions of interconnected neurons and synapses present in the deep networks. Here, we present a compact and energy efficient skyrmion-based artificial neuron consisting of ferromagnetic/heavy metal/ferroelectric layers which uses strain-mediated voltage manipulation of skyrmion states to mimic the Integrate-and-Fire (IF) function of biological neurons. By implementation of a spiking neural network (SNN) based on the proposed skyrmionic neuronal devices, it can achieve a high accuracy of 95.08% on a modified National Institute of Standards and Technology (MNIST) handwritten digit dataset, as well as a low power consumption of ∼46.8 fJ per epoch per neuron. The present work suggests a novel way to realize energy-efficient and high-density neuromorphic computing.

A stochastic photo-responsive memristive neuron for an in-sensor visual system based on a restricted Boltzmann machine.

In-sensor computing has gained attention as a solution to overcome the von Neumann computing bottlenecks inherent in conventional sensory systems. This attention is due to the ability of sensor elements to directly extract meaningful information from external signals, thereby simplifying complex data. The advantage of in-sensor computing can be maximized with the sampling principle of a restricted Boltzmann machine (RBM) to extract significant features. In this study, a stochastic photo-responsive neuron is developed using a TiN/In-Ga-Zn-O/TiN optoelectronic memristor and an Ag/HfO2/Pt threshold-switching memristor, which can be configured as an input neuron in an in-sensor RBM. It demonstrates a sigmoidal switching probability depending on light intensity. The stochastic properties allow for the simultaneous exploration of various neuron states within the network, making identifying optimal features in complex images easier. Based on semi-empirical simulations, high recognition accuracies of 90.9% and 95.5% are achieved using handwritten digit and face image datasets, respectively. In addition, the in-sensor RBM effectively reconstructs abnormal face images, indicating that integrating in-sensor computing with probabilistic neural networks can lead to reliable and efficient image recognition under unpredictable real-world conditions.

A General Model for Side Information in Neural Networks

We investigate the utility of side information in the context of machine learning and, in particular, in supervised neural networks. Side information can be viewed as expert knowledge, additional to the input, that may come from a knowledge base. Unlike other approaches, our formalism can be used by a machine learning algorithm not only during training but also during testing. Moreover, the proposed approach is flexible as it caters for different formats of side information, and we do not constrain the side information to be fed into the input layer of the network. A formalism is presented based on the difference between the neural network loss without and with side information, stating that it is useful when adding side information reduces the loss during the test phase. As a proof of concept we provide experimental results for two datasets, the MNIST dataset of handwritten digits and the House Price prediction dataset. For the experiments we used feedforward neural networks containing two hidden layers, as well as a softmax output layer. For both datasets, side information is shown to be useful in that it improves the classification accuracy significantly.

Recognising Image Shapes from Image Parts, not Neural Parts

This paper describes an image processing method that makes use of image parts instead of neural parts. Neural networks excel at image or pattern recognition and they do this by constructing complex networks of weighted values that can cover the complexity of the pattern data. These features however are integrated holistically into the network, which means that they can be difficult to use in an individual sense. A different method might scan individual images and use a more local method to try to recognise the features in it. This paper suggests such a method and it is conjectured that this method is more ‘intelligent’ than a traditional neural network. The image parts that it creates not only have more meaning, but they can also be put into a positional context and allow for an explainable result. Tests show that it can be quite accurate, on some handwritten digit datasets, but not as accurate as a neural network. The fact that it offers an explainable interface however, could make it interesting.