Handwritten Digit Recognition Research Articles

A new pre-conditioned STDP rule and its hardware implementation in neuromorphic crossbar array

This paper proposes a new pre-conditioned spike-timing-dependent plasticity (STDP) learning rule as well as an efficient system-level time-domain circuit modeling and simulation method for the whole crossbar array structure. First, the new pre-conditioned STDP rule is conceived by exercising a slight perturbation to the conventional STDP curve. Software-wise, it can be described in spiking neural network (SNN) codes, while hardware-wise, it is realized by the memristor crossbar array in conjunction with specific spike signals, just as the conventional STDP. The new STDP rule enables the neural network in a neuromorphic chip to achieve better performance both in terms of software and hardware implementation compared with the conventional STDP rule. Second, a time-domain circuit modeling method is proposed for the simulation of the whole crossbar array. Numerical experiments demonstrate that the new pre-conditioned STDP is superior to the conventional STDP both in terms of accuracy and low power consumption. At the software level, a 784 × 100 spiking neural network, trained by the pre-conditioned STDP, for the MNIST handwritten digit recognition achieves an accuracy of 85.90%, which is 3.09% higher than that trained by the conventional STDP. At the hardware level, a downscaled 196 × 10 crossbar array is simulated and trained. The crossbar array trained by the new STDP consumes 21.5% less power than that trained by the conventional STDP. Moreover, it does not compromise the original robustness when subject to non-ideal characteristics such as variation of conductance and resistance of the interconnect in the crossbar array.

Composite Optimization Algorithms for Sigmoid Networks.

In this letter, we use composite optimization algorithms to solve sigmoid networks. We equivalently transfer the sigmoid networks to a convex composite optimization and propose the composite optimization algorithms based on the linearized proximal algorithms and the alternating direction method of multipliers. Under the assumptions of the weak sharp minima and the regularity condition, the algorithm is guaranteed to converge to a globally optimal solution of the objective function even in the case of nonconvex and nonsmooth problems. Furthermore, the convergence results can be directly related to the amount of training data and provide a general guide for setting the size of sigmoid networks. Numerical experiments on Franke's function fitting and handwritten digit recognition show that the proposed algorithms perform satisfactorily and robustly.

Design and Implementation of Handwritten Digit Recognition Accelerator Based on Systolic Array

Due to the rapid development in artificial intelligence, the classic Von Neumann architecture is no longer able to meet the demands of high-computing, high-storage, and high-bandwidth artificial intelligence applications. To address this issue, this paper proposes a handwritten digit recognition accelerator based on a systolic array. First, a 5 convolutional layer handwritten digit recognition network is built. Besides, The FPGA-based convolution accelerator employs im2col technology to convert convolution calculations into matrix multiplications and uses a systolic array to efficiently perform these multiplications. Furthermore, the four convolution processes are pipelined to improve the throughput. Finally, a handwritten digit recognition system is implemented on the PYNQ-Z2. Compared with the software implementation on Arm, the speedup of the accelerator is 598x, the delay is 1.24 s, and the power consumption is 2.51 w.

Defending against Adversarial Attacks using Digital Image Processing

The rapidly maturing deep neural networks are used in self-driving cars, facial recognition payment, security, and other aspects, and are crucial to how we conduct our daily lives. However, past research has shown that adversarial attacks pose a significant danger to deep learning. This paper proposes a defense based on digital image processing to protect the handwritten digit recognition model from adversarial attacks. This method can defend against adversarial attacks simply and effectively using spatial filtering and thresholding for image pre-processing to remove adversarial perturbations. The experimental findings on the MNIST data set show that this technique can increase the average accuracy of the model against adversarial samples to 91.20%.

Enhanced regularization for on-chip training using analog and temporary memory weights

In-memory computing techniques are used to accelerate artificial neural network (ANN) training and inference tasks. Memory technology and architectural innovations allow efficient matrix–vector multiplications, gradient calculations, and updates to network weights. However, on-chip learning for edge devices is quite challenging due to the frequent updates. Here, we propose using an analog and temporary on-chip memory (ATOM) cell with controllable retention timescales for implementing the weights of an on-chip training task. Measurement results for Read–Write timescales are presented for an ATOM cell fabricated in GlobalFoundries’ 45 nm RFSOI technology. The effect of limited retention and its variability is evaluated for training a fully connected neural network with a variable number of layers for the MNIST hand-written digit recognition task. Our studies show that weight decay due to temporary memory can have benefits equivalent to regularization, achieving a ∼33% reduction in the validation error (from 3.6% to 2.4%). We also show that the controllability of the decay timescale can be advantageous in achieving a further ∼26% reduction in the validation error. This strongly suggests the utility of temporary memory during learning before on-chip non-volatile memories can take over for the storage and inference tasks using the neural network weights. We thus propose an algorithm-circuit codesign in the form of temporary analog memory for high-performing on-chip learning of ANNs.

3D trigonal FAPbI3‐based multilevel resistive switching nonvolatile memory for artificial neural synapse

AbstractHybrid perovskites have attracted enormous attention in the next generation resistive switching (RS) memristor for the artificial synapses, owing to their ambipolar charge transport, long diffusion length, and tunable visible bandgap. However, the variable switch, limited reproducibility, and poor endurance are the obstacles to the practical application of the perovskite memristors. Herein, we reported a multilevel RS nonvolatile memory based on a 3D trigonal HC(NH2)2PbI3 (α‐FAPbI3) perovskite layer modified by 1‐cyanobutyl‐3‐methylimidazolium chloride ([CNBmim]Cl) and sandwiched between ITO and Au electrodes (Au/[CNBmim]Cl/α‐FAPbI3/SnO2/ITO). In contrast to the bare memristor with failure switching from low resistance state (LRS) to high resistance state (HRS), the memristor device based on the α‐FAPbI3 modified with [CNBmim]Cl (Target device) shows the retention time over 104 s with On/Off ratio (>102) and endurance up to 550 cycles. The stable RS cycle benefits from the accelerated electrons de‐trapping from the reduced defects and fast charge separation in the interface of α‐FAPbI3/electrode, leading to the rupture of conductive filaments and transition of LRS to HRS. As a two‐terminal analog synaptic device, the target device can realize random handwritten digit recognition with an impressive accuracy of 89.3% on the condition of low learning phases (500 training cycles).

Flexible, Transparent, and Wafer-Scale Artificial Synapse Array Based on TiOx /Ti3 C2 Tx Film for Neuromorphic Computing.

A high-density neuromorphic computing memristor array based on 2Dmaterials paves the way for next-generation information-processing components and in-memory computing systems. However, the traditional 2D-materials-based memristor devices suffer from poor flexibility and opacity, which hinders the application of memristors in flexible electronics. Here, a flexible artificial synapse array based on TiOx /Ti3 C2 Tx film is fabricated by a convenient and energy-efficient solution-processing technique, which realizes high transmittance (≈90%) and oxidation resistance (>30 days). The TiOx /Ti3 C2 Tx memristor shows low device-to-device variability, long memory retention and endurance, a high ON/OFF ratio, and fundamental synaptic behavior. Furthermore, satisfactory flexibility (R = 1.0mm) and mechanical endurance (104 bending cycles) of the TiOx /Ti3 C2 Tx memristor are achieved, which is superior to other film memristors prepared by chemical vapor deposition. In addition, high-precision (>96.44%) MNIST handwritten digits recognition classification simulation indicates that the TiOx /Ti3 C2 Tx artificial synapse array holds promise for future neuromorphic computing applications, and provides excellent high-density neuron circuits for new flexible intelligent electronic equipment.

Photonic convolutional reservoir computing based on VCSEL with multiple optical injections

In this paper, a photonic convolutional time delay reservoir computing (C-TD-RC) system based on vertical cavity surface emitting laser (VCSEL) with multiple optical injections has been proposed. Here, the convolutional neural networks (CNNs) and the time delay reservoir computing (TD-RC) are combined in the system. Using TD-RC instead of the fully connected layer in the traditional CNNs can solve the problem of gradient vanishing and gradient explosion in CNNs. Two classification tasks including handwritten digits recognition and spoken digits classification are solved by the C-TD-RC system. In addition, in order to achieve better classification performance of RC, a new post-processing method combining ridge regression and winner take all strategy has been adopted. Numerical results demonstrate that the minimum Word Error Rate (WER) can reach 0.0135 for handwritten digits recognition and 0.0213 for spoken digits classification tasks by the C-TD-RC system. Furthermore, the effects of the virtual nodes, the injection paths and the number of convolution kernels applied in the first and second convolution layers on the C-TD-RC system are considered. The proposed C-TD-RC scheme is of great significance to the performance enhancement of RC in the future.

Handwritten Digit Recognition using Machine Learning with Python

Handwritten Digit Recognition is one of the essentially significant issues in pattern recognition applications. The main purpose of this project is to build an automatic handwritten digit recognition method for the recognition of handwritten digit strings. This paper proposes a simple convolution neural network approach to handwritten digit recognition. Convolutional Neural Network model is implemented using MNIST dataset. This dataset consists 60,000 small square 28×28pixel grayscale images of handwritten single digits between 0 and 9. The applications of digit recognition include postal mail sorting, check processing, form data entry, etc. The core of the issue exists in the capacity to foster a proficient calculation that can perceive manually written digits and which is put together by clients by the method of a scanner, tablet, and other computerized gadgets.

Handwritten digit recognition using machine learning

Different people are expected to possess varying writing styles that are distinctive to their personalities. There are many aspects that make handwriting differ from person to person, which include spaces, inclination, height, basic patterns, connecting strokes, sizes and widths, markings, and ornaments, among others. However, the challenges come when the handwritten text must be converted into digital form to enhance information sharing and storage. To address this challenge, the recent past has seen the rise of reliance on machines over humans and the subsequent development of machine learning algorithms. Likewise, the recognition of handwritten text is part of the important field of research and development that appear to have a promising future. The handwritten characters recognition has gained considerable attention in the area of machine learning and pattern recognition, most recently. Several techniques have been proposed for recognizing handwritten text. In line with this, many studies have been conducted that explain techniques employed in converting textural substance from paper material to a form that is readable by a machine. Such a digital recognition system has the potential to create a paperless environment by processing existing paper documents and digitization in future. This paper presents two recent techniques based on neural networks, Convolutional Neural Networks (CNNs) and Waikato Environment for Knowledge Analysis (WEKA) and makes a comparative analysis. In this paper, we want to ensure the reliable and effective approaches to the recognition of handwritten digits. The results show that CNN is comparatively more accurate than WEKA, with an accuracy rate as high as 99.59 percent, with the lowest reported degree of accuracy standing at 88.3 percent. On the other hand, the highest report accuracy with regards to WEKA was 82.92 percent, and the lowest stood at 67.45 percent. Although there is a need for more research to be carried out, the information from the two algorithms demonstrates that the existing techniques are increasingly becoming more reliable, and CNN is leading in terms of accuracy.

Proton-Assisted Redox-Based Three-Terminal Memristor for Synaptic Device Applications

Emerging technologies, i.e., spintronics, 2D materials, and memristive devices, have been widely investigated as the building block of neuromorphic computing systems. Three-terminal memristor (3TM) is specifically designed to mitigate the challenges encountered by its two-terminal counterpart as it can concurrently execute signal transmission and memory operations. In this work, we present a complementary metal-oxide-semiconductor-compatible 3TM with highly linear weight update characteristics and a dynamic range of ∼15. The switching mechanism is governed by the migration of oxygen ions and protons in and out of the channel under an external gate electric field. The involvement of the protonic defects in the electrochemical reactions is proposed based on the bipolar pulse trains required to initiate the oxidation process and the device electrical characteristics under different humidity levels. For the synaptic operation, an excellent endurance performance with over 256k synaptic weight updates was demonstrated while maintaining a stable dynamic range. Additionally, the synaptic performance of the 3TM is simulated and implemented into a four-layer neural network (NN) model, achieving an accuracy of ∼92% in MNIST handwritten digit recognition. With such desirable conductance modulation characteristics, our proposed 3T-memristor is a promising synaptic device candidate to realize the hardware implementation of the artificial NN.

Black Phosphorus/Ferroelectric P(VDF-TrFE) Field-Effect Transistors with High Mobility for Energy-Efficient Artificial Synapse in High-Accuracy Neuromorphic Computing.

The neuromorphic system is an attractive platform for next-generation computing with low power and fast speed to emulate knowledge-based learning. Here, we design ferroelectric-tuned synaptic transistors by integrating 2D black phosphorus (BP) with a flexible ferroelectric copolymer poly(vinylidene fluoride-trifluoroethylene) (P(VDF-TrFE)). Through nonvolatile ferroelectric polarization, the P(VDF-TrFE)/BP synaptic transistors show a high mobility value of 900 cm2 V-1 s-1 with a 103 on/off current ratio and can operate with low energy consumption down to the femtojoule level (∼40 fJ). Reliable and programmable synaptic behaviors have been demonstrated, including paired-pulse facilitation, long-term depression, and potentiation. The biological memory consolidation process is emulated through ferroelectric gate-sensitive neuromorphic behaviors. Inspiringly, the artificial neural network is simulated for handwritten digit recognition, achieving a high recognition accuracy of 93.6%. These findings highlight the prospects of 2D ferroelectric field-effect transistors as ideal building blocks for high-performance neuromorphic networks.

Highly sensitive spin-torque diodes based on in-plane magnetized magnetic tunnel junctions

We investigate the highly sensitive spin torque diode (STD) effect in a magnetic tunnel junction (MTJ) with an in-plane polarizer and an in-plane free layer. Under injection locking mechanisms, a high rectification voltage of 12 mV is obtained with an input radio frequency power of 1 μW under direct current bias current and a weak magnetic field, corresponding to a high sensitivity of 12 000 mV/mW. In addition, we use the nonlinear rectification characteristics of STD to mimic a neuron with a ReLU-like activation function to perform the recognition of handwritten digits in the Mixed National Institute of Standards and Technology database, where a produced accuracy of up to 93.53% is obtained. These findings suggest that the MTJ with in-plane magnetized electrodes holds promising potential in developing high sensitivity STDs for Internet of Things applications and neuromorphic computing.

Ensemble Learning Network for Handwritten Digit Recognition Based on Fusion Optimized CNN

<p>Handwritten digit recognition is an active research field. These recognition systems are faced with many challenges, including accuracy, speed and automatic extraction of complex handwriting features. In this paper, a Stacking ensemble learning model based on fusion optimized CNN is proposed, which can be effectively used for handwritten digit recognition. To better extract the features of complex handwritten digital images and maximize the reliability of the model, the Bagging strategy combined with six CNNs is used for feature extraction for the first time, and SVM is used for classification. This not only improves the accuracy and stability of the model, but also effectively avoids over-fitting. In addition, a fusion optimization algorithm based on Adam and SGD is proposed to solve the problem that CNN falls into local optimum due to a large number of iterations. During the process of training, ASCNN can not only speed up the convergence rate in the early stage, but also reduce the oscillation phenomenon in the late stage. Extensive experimental results on the well-known MNIST and USPS handwriting image datasets demonstrate the effectiveness of the proposed model.</p> <p> </p>

A Novel YOLOv5 Deep Learning Model for Handwriting Detection and Recognition

Computer Vision (CV) has become an essential field in Artificial Intelligence applications. Object detection and recognition (ODR) is one of the fundamental tasks of computer vision implementations. However, developing an efficient ODR model is still a significant problem. The model’s execution time and speed are the most critical features during the inference or detection and recognition process, which need to be improved using the latest object detection architectures. In this paper, the handwritten detection and recognition (HDR) model is developed based on previously known algorithms with their efficiency, such as Faster R-CNN and YOLOv4 in the first hand. On the other hand, two new models capable of detecting and recognizing handwritten digits using the latest ODR algorithm are proposed, one based on the latest YOLO family architecture (YOLOv5-HDR) with high speed and accuracy and the other using the transformers architecture (DETR). To the best of our knowledge, this is the first study to achieve a details comparison between YOLOv5 and transformers-based models in handwritten digit detection. Finally, the detailed performance analysis achieved by the paper proves that the YOLOv4-based model achieved the testing inference 13% faster than Faster R-CNN. However, the proposed YOLOv5-based model outperformed the YOLOv4 and the transformers-based one as it increased the testing execution time 25% faster than the YOLOv4, three times faster than the DETR model. A further adversarial attack test has been conducted to ensure the robust performance of the proposed model. Furthermore, numerical experiment results and their analyses demonstrate the robustness and effectiveness of the proposed YOLOv5-based model being the most stable for handwritten digit detection and recognition tasks.

Fourier-based type-2 fuzzy neural network: Simple and effective for high dimensional problems

The main contribution of this study is to introduce a simple and effective deep learning Fourier-based type-2 fuzzy neural network for high-dimensional problems. The rules are directly constructed by fast Fourier transformation. The input matrix/vector is segmented, and each segment represents a fuzzy rule. The upper/lower bounds of rule firings are obtained by the Fourier transformation approach. The output is computed by a simple type-reduction method. All antecedent and consequent parameters are optimized by simple gradient descent and fuzzy correntropy-based extended Kalman filter. The kernel size of a conventional correntropy-based filters is determined by a fuzzy system. The convergence of the learning method is proved by the Lyapunov method. The effectiveness of the suggested approach is verified by the face recognition problem (1024 input variables), English handwriting digit recognition (1024 input variables), and modeling problem with real-world data set (32 input variables). The simulations and comparisons demonstrate the superiority of the introduced scheme.

Memristive artificial synapses based on Au–TiO2 composite thin film for neuromorphic computing

Neuromorphic computing architecture is considered to be a highly desirable next-generation computing architecture as it simulates the way the brain processes information. The basic device supporting such an architecture is called an artificial synapse, which possesses synapse-like functionalities. Here in this work, an Au–TiO2 composite thin film (Au nanoparticles embedding into TiO2 matrix) based memristive artificial synapse has been fabricated with excellent interface-type resistive switching (RS) characteristics. The conductivity of the device can be continuously tuned by applying different sequences of pulses, which could be analogous to the weight change of synapses. Various synaptic behaviors have been emulated, such as long-term potentiation/depression, short-term/long-term memory, learning-forgetting process, and paired-pulse facilitation. Finally, an artificial neural network for hand-written digits recognition has been constructed with an accuracy level as high as ∼90%. The excellent performance of the Au–TiO2 based device demonstrates the availability of incorporating the second phase to tune RS properties and shows its potential in a memristor for artificial synapses and neuromorphic computing with enhanced performance.

Hand Written Digit Recognition using CNN

Abstract: Digit Recognition is an interesting and significant topic. Because the manually written digits are not all the same size, thickness, position, and direction, and to determine the issue of handwritten digit recognition, different obstacles must be considered. The variety and distinctiveness of different people's composition styles also influence the appearance and presence of the digits. It is a method of perceiving and organizing transcribed digits. It offers a wide range of uses, such as programmed bank checks, postal locations, and tax paperwork, among others. The goal of this project is to create a classification algorithm that can recognize handwritten digits. The outcomes of some of the most widely used Machine Learning Algorithms, such as SVM, KNN, and RFC, as well as Deep Learning calculations, such as multilayer CNN, using Keras, Theano, and TensorFlow. Using these, an accuracy of 98.70% was reached using CNN (Keras) when compared to 97.91% using SVM, 96.67% using KNN, and 96.89% using RFC.

A Posit based Handwritten Digits Recognition System

A Posit based Handwritten Digits Recognition System

Universum parametric $$\nu $$-support vector regression for binary classification problems with its applications

AbstractUniversum data sets, a collection of data sets that do not belong to any specific class in a classification problem, give previous information about data in the mathematical problem under consideration to enhance the classifiers’ generalization performance. Recently, some researchers have integrated Universum data into the existing models to propose new models which result in improved classification performance. Inspired by these Universum models, an efficient parametric$$ \nu $$ν-support vector regression with Universum data ($$ {\mathfrak {U}} $$UPar-$$ \nu $$ν-SVR) is proposed in this work. This method, which finds two non-parallel hyperplanes by solving one optimization problem and considers heteroscedastic noise, overcomes some common disadvantages of the previous methods. The$$ {\mathfrak {U}} $$UPar-$$ \nu $$ν-SVR includes unlabeled samples that don’t belong to any class in the training process, which results in a quadratic programming problem. Two approaches are proposed to solve this problem. The first approach derives the dual formulation using the Lagrangian function and KKT conditions. Furthermore, a least squares parametric$$ \nu $$ν-support vector regression with Universum data (named LS-$$ {\mathfrak {U}} $$UPar-$$ \nu $$ν-SVR) is suggested to further increase the generalization performance. The LS-$$ {\mathfrak {U}} $$UPar-$$ \nu $$ν-SVR solves a single system of linear equations, instead of addressing a quadratic programming problem in the dual formulation. Numerical experiments on artificial, UCI, credit card, NDC, and handwritten digit recognition data sets show that the suggested Universum model and its solving methodologies improve prediction accuracy.