MNIST Test Set Research Articles

Stable attractors for neural networks classification via ordinary differential equations (SA-nODE)

Abstract A novel approach for supervised classification is presented which sits at the intersection of machine learning and dynamical systems theory. At variance with other methodologies that employ ordinary differential equations for classification purposes, the untrained model is a priori constructed to accommodate for a set of pre-assigned stationary stable attractors. Classifying amounts to steer the dynamics towards one of the planted attractors, depending on the specificity of the processed item supplied as an input. Asymptotically the system will hence converge on a specific point of the explored multi-dimensional space, flagging the category of the object to be eventually classified. Working in this context, the inherent ability to perform classification, as acquired ex post by the trained model, is ultimately reflected in the shaped basin of attractions associated to each of the target stable attractors. The performance of the proposed method is here challenged against simple toy models crafted for the purpose, as well as by resorting to well established reference standards. More precisely, we achieved an accuracy of 98.06 % on the MNIST test set and 88.21% on the Fashion MNIST test set. Although this method does not reach the performance of state-of-the-art deep learning algorithms, it illustrates that continuous dynamical systems with closed analytical interaction terms can serve as high-performance classifiers.

Implementing handwritten text recognition using deep learning with TensorFlow: An MNIST dataset approach

Handwritten text recognition (HTR) is a pivotal technology with extensive applications in document digitization, postal automation, and educational tools. This paper delves into the implementing a deep learning-based system for recognizing handwritten digits using TensorFlow and the MNIST dataset. The MNIST dataset, a widely-used benchmark, comprises 60,000 training images and 10,000 testing images of handwritten digits, each standardized to a 28x28 pixel grayscale format. Leveraging the power of Convolutional Neural Networks (CNNs), our model effectively extracts features and classifies digits with high accuracy. The model architecture consists of multiple convolutional layers for feature extraction, pooling layers for dimensionality reduction, and fully connected layers for classification. Preprocessing steps include normalizing pixel values and one-hot encoding the labels, ensuring the data is optimally formatted for training. The TensorFlow framework, known for its robustness and scalability, facilitates the development and deployment of the model. Through a series of experiments, the model demonstrates impressive performance, achieving high accuracy on the MNIST test set. This paper underscores the potential of deep learning in handwritten text recognition. It sets the stage for future enhancements, such as recognizing more complex handwritten texts and optimizing the system for practical applications. The results highlight the effectiveness of deep learning techniques in overcoming the challenges associated with handwritten text recognition, paving the way for advanced, real-world implementations.

Photonic Binary Convolutional Neural Network Based on Microring Resonator Array

We propose the photonic architecture based on microring resonator (MRR) arrays for binary convolutional neural networks (BCNNs) accelerated computing. The MRR crossbar unit is used for computing weight {−1, 1} and the single MRR is for input {0, 1}. The computing parallelism is improved through wavelength division multiplexing. The photonic BCNN achieves 97.29% classification accuracy on the MNIST test set which is only 1.94% lower than the accuracy of the 32-bit neural network, and saves <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$32\times $ </tex-math></inline-formula> at memory usage. We analyze effects of input and weight encoding errors on the photonic BCNN. When the input or weight error rate is less than 0.01%, the test accuracy remains unchanged. We evaluate the performance of the photonic BCNN architecture considering optical loss, inter-channel crosstalk, operation frequency and device power consumption. The energy efficiency of the designed photonic BCNN architecture is 1.72 pJ/MAC, which is <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$4.80\times $ </tex-math></inline-formula> and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$61.32\times $ </tex-math></inline-formula> better than the 8-bit and 16-bit architecture respectively. The photonic BCNN is promising to be used for edge computing.

A Hybrid Improved Neural Networks Algorithm Based on L2 and Dropout Regularization

Small samples are prone to overfitting in the neural network training process. This paper proposes an optimization approach based on L2 and dropout regularization called a hybrid improved neural network algorithm to overcome this issue. The proposed model was evaluated based on the Modified National Institute of Standards and Technology (MNIST, grayscale-28 × 28 × 1) and Canadian Institute for Advanced Research 10 (CIFAR10, RGB - 32 × 32 × 3) as the training data sets and data applied to the LeNet-5 and Autoencoder neural network architectures. The evaluation is conducted based on cross-validation; the result of the model prediction is used as the final measure to evaluate the quality of the model. The results show that the proposed hybrid algorithm can perform more effectively, avoid overfitting, improve the accuracy of network model prediction in classification tasks, and reduce the reconstruction error in the unsupervised domain. In addition, employing the proposed algorithm without increasing the time complexity can reduce the effect of noisy data and bias and improve the training time of neural network models. Quantitative and qualitative experimental results show that the accuracy of using the proposed algorithm in this paper with the MNIST test set has an improvement of 2.3% and 0.9% compared to L2 regularization and dropout regularization, respectively, and based on the CIFAR10 data set, the accuracy improvement of 0.92% compared with L2 regularization and 1.31% concerning dropout regularization. The reconstruction error of using the proposed algorithm in this paper with the MNIST data set has an improvement of 0.00174 and 0.00398 compared to L2 regularization and dropout regularization, respectively, and based on the CIFAR10 data set, the accuracy improvement of 0.00078 compared with L2 regularization and 0.00174 concerning dropout regularization.

Multicore Photonic Complex-Valued Neural Network with Transformation Layer

Photonic neural network chips have been widely studied because of their low power consumption, high speed and large bandwidth. Using amplitude and phase to encode, photonic chips can accelerate complex-valued neural network computations. In this article, a photonic complex-valued neural network (PCNN) chip is designed. The scale of the single-core PCNN chip is limited because of optical losses, and the multicore architecture of the chip is used to improve computing capability. Further, for improving the performance of the PCNN, we propose the transformation layer, which can be implemented by the designed photonic chip to transform real-valued encoding to complex-valued encoding, which has richer information. Compared with real-valued input, the transformation layer can effectively improve the classification accuracy from 93.14% to 97.51% of a 64-dimensional input on the MNIST test set. Finally, we analyze the multicore computation of the PCNN. Compared with the single-core architecture, the multicore architecture can improve the classification accuracy by implementing larger neural networks and has better phase noise robustness. The proposed architecture and algorithms are beneficial to promote the accelerated computing of photonic chips for complex-valued neural networks and are promising for use in many applications, such as image recognition and signal processing.

A 2D2R ReRAM CIM accelerator for multilayer perceptron in visual classification applications

Computing-in-memory (CIM) suffers from remarkable area and performance costs from large memory cell and inefficient peripheral circuits for MLP acceleration. In this work, a 2D2R ReRAM CIM accelerator is proposed with circuit improvements for MLP in visual classification applications. The 2D2R ReRAM cell is organized as crossbar array for dense weight storage. The multi-level input circuit is proposed for high precision input with compensation for the nonlinearity of 2D2R cell. The successive-approximation output circuit is employed with all-0 detecting for fast and energy-efficient conversions. Besides, the weight-aware configurable computing is proposed to dynamically configure the input count for partial sum for improved performance. Compared with the traditional 2T2R cell, the area of 2D2R cell can be reduced by >85%. The accuracy of MNIST test set is 97.76% for the accuracy of 3b input, output and weight, and the energy efficiency can reach 174.5TOP/W under the working frequency of 200 MHz, realizing ∼2.6× improvement over previous work.

Neuron Circuits for Low-Power Spiking Neural Networks Using Time-To-First-Spike Encoding

Hardware-based spiking neural networks (SNNs) are regarded as promising candidates for the cognitive computing system due to low power consumption and highly parallel operation. In this work, we train the SNN in which the firing time carries information using temporal backpropagation. The temporally encoded SNN with 512 hidden neurons showed an accuracy of 96.90% for the MNIST test set. Furthermore, the effect of the device variation on the accuracy in temporally encoded SNN is investigated and compared with that of the rate-encoded network. In a hardware configuration of our SNN, NOR-type analog memory having an asymmetric floating gate is used as a synaptic device. In addition, we propose a neuron circuit including a refractory period generator for temporally encoded SNN. The performance of the 2-layer neural network consisting of synapses and proposed neurons is evaluated through circuit simulation using SPICE. The network with 128 hidden neurons showed an accuracy of 94.9%, a 0.1% reduction compared to that of the system simulation of the MNIST dataset. Finally, the latency and power consumption of each block constituting the temporal network is analyzed and compared with those of the rate-encoded network depending on the total time step. Assuming that the total time step number of the network is 256, the temporal network consumes 15.12 times lower power than the rate-encoded network and can make decisions 5.68 times faster.

Handwritten Digit Recognition Scheme Based on Noise Reduction and Trestle Self-Coding

In this paper, in order to improve the accuracy of handwritten number recognition, a convolutional neural network based on noise reduction and stack auto coding is formed by combining the denoising auto coding algorithm with stack auto coding algorithm. In this paper, the MNIST dataset is preprocessed first, then the noise reduction and stack auto coding network layer are constructed, and finally the stack auto coding algorithm based on the improved denoising auto coding is used to train and test the dataset. Experimental data show that the proposed method achieves low loss value and effectively improves the recognition accuracy in the MNIST test set.

A Low-bit And Data-conversion-free Memristive Spiking Computing Network

Memristor crossbar holds promising potential in massively parallel data processing, as computing tasks beyond vector-matrix multiplication typically requires additional complex data conversion modules. In this work, we report a memristive spiking computing network (MSCN) without data conversion. Two memristor crossbars are used to perform vector-matrix multiplication, while spiking neurons are used to integrate the signals from each column of the memristor crossbars. Input and output signals are binary spikes (spiking or non-spiking); thus data conversions between the analog and digital domains are avoided. The MSCN is suitable for accelerating artificial neural network (ANN). A multi-layer perception (MLP) consisting of four layers is simulated based on the analog MSCN, and recognition accuracy of 98.55% under the MNIST test set is achieved. However, in the practical memristor, it is challenging to adjust conductance analogously and even challenging to implement multi-bit conductance states. Therefore, the memristor conductance states are further reduced to 3 bits and 1 bit. Encouragingly, recognition accuracies just slightly decrease to 98.36% and 97.96% for 3-bit and 1-bit conductance states, respectively, which is cost-effective considering the hardware reliability improvement.

Design and Development of Image Recognition Toolkit Based on Deep Learning

Deep learning algorithms have shown superior performance than traditional algorithms when dealing with computationally intensive tasks in many fields. The algorithm model based on deep learning has good performance and can improve the recognition accuracy in relevant applications in the field of computer vision. TensorFlow is a flexible opensource machine learning platform proposed by Google, which can run on a variety of platforms, such as CPU, GPU, and mobile devices. TensorFlow platform can also support current popular deep learning models. In this paper, an image recognition toolkit based on TensorFlow is designed and developed to simplify the development process of more and more image recognition applications. The toolkit uses convolutional neural networks to build a training model, which consists of two convolutional layers: one batch normalization layer before each convolutional layer, and the other pooling layer after each convolutional layer. The last two layers of the model use the full connection layer to output recognition results. Batch gradient descent algorithm is adopted in the optimization algorithm, and it integrates the advantages of both the gradient descent algorithm and the stochastic gradient descent algorithm, which greatly reduces the number of convergence iterations and has little influence on the convergence effect. The total training parameters of the toolkit model reach 1.7 million. In order to prevent overfitting problems, the dropout layer before each full connection layer is added and the threshold of 0.5 is set in the design. The convolution neural network model is trained and tested by the MNIST set on TensorFlow. The experimental result shows that the toolkit achieves the recognition accuracy of 99% on the MNIST test set. The development of the toolkit provides powerful technical support for the development of various image recognition applications, reduces its difficulty, and improves the efficiency of resource utilization.

STBNN: Hardware-friendly spatio-temporal binary neural network with high pattern recognition accuracy

In recent years, the weight binarized neural network (BNN) technology has made great progress. However, neural networks with binarized inputs and binarized weights suffer from low accuracy in pattern recognition or inefficiency in hardware implementation. This work proposes a spatio-temporal binary neural network (STBNN) to solve this problem. STBNN has binary network input/output, binary neuron input/output, and binarized weights, and it integrates the computationally expensive batch normalization (BN) operation widely used in previous BNNs into the neuron threshold. STBNN can largely save computing resources and storage space while maintaining high accuracy (e.g., 98.0% on the MNIST test set). Using binary input (0 or 1) and binarized weight (±1), the product of input and weight can be realized by a 1-bit Signed AND operation instead of multiplication operation in hardware implementation, thus significantly reducing computing resources, memory requirements, and power consumption. The results show that compared with a 32-bit multi-layer perceptron (MLP)-based hardware design, the STBNN-based hardware design typically reduces these three indicators by 84.2%, 96.4%, and 96.7%, respectively. This work provides an effective method to construct hardware-friendly neural network models and a guide for designing an extremely hardware-saving neural network processor.

Teaching machines to write like humans using L-attributed grammar

Reading and writing are easy for humans. The automatic reading of handwritten characters has been studied for several decades. Machine learning algorithms for reading tasks often require a huge amount of data to perform with similar accuracy to humans, yet it is also difficult to gain sufficient meaningful data. Automatic writing tasks have not been studied as extensively. In this paper, we teach machines to write like teaching a child by telling the machine the method for writing each character using L-attributed grammar. With the aid of the proposed TMTW (Teaching Machines To Write) interacting system, a human as a teacher only needs to provide the writing sequence of parts and control lines. The proposed system automatically perceives the relationships between control lines and parts, and constructs the grammars. Top-down derivation and the stroke generation method are applied to generate varying characters based on the learned grammars. For as long as a machine can write, it can be applied in robot control or training sample generation for automatic reading tasks. The MNIST and CASIA datasets are used to demonstrate the effectiveness of the proposed system on different languages. The machine written samples are used to train a network, which is evaluated on the MNIST test set. A test error rate of 1.23% is achieved using only approximately 20 grammars on average for each digit. Using the generated and handwritten samples together as a training set can reduce the test error rate to 0.61%. Similar experiments are conducted using the CASIA data set, and the results demonstrated that the proposed method is effective in generating characters with a complex structure. The source codes and grammars used in this paper have been made publicly available in https://github.com/step123456789/TMTW.

Noise-boosted bidirectional backpropagation and adversarial learning

Bidirectional backpropagation trains a neural network with backpropagation in both the backward and forward directions using the same synaptic weights. Special injected noise can then improve the algorithm’s training time and accuracy because backpropagation has a likelihood structure. Training in each direction is a form of generalized expectation–maximization because backpropagation itself is a form of generalized expectation–maximization. This requires backpropagation invariance in each direction: The gradient log-likelihood in each direction must give back the original update equations of the backpropagation algorithm. The special noise makes the current training signal more probable as bidirectional backpropagation climbs the nearest hill of joint probability or log-likelihood. The noise for injection differs for classification and regression even in the same network because of the constraint of backpropagation invariance. The backward pass in a bidirectionally trained classifier estimates the centroid of the input pattern class. So the feedback signal that arrives back at the input layer of a classifier tends to estimate the local pattern-class centroid. Simulations show that noise speeded convergence and improved the accuracy of bidirectional backpropagation on both the MNIST test set of hand-written digits and the CIFAR-10 test set of images. The noise boost further applies to regular and Wasserstein bidirectionally trained adversarial networks. Bidirectionality also greatly reduced the problem of mode collapse in regular adversarial networks.

Attention Inspired Network: Steep learning curve in an invariant pattern recognition model

Hubel and Wiesel’s study about low areas of the visual cortex (VC) inspired deep models for invariant pattern recognition. In such models, simple and complex layers alternate local feature extraction with subsampling to add invariance to distortion or transformations. However, it was shown that to tolerate large changes between examples of the same category, the subsampling operation has to discard so much information that the model loses the capability to discriminate between categories. So, in practice, small changes are tolerated by these layers and, afterwards, a powerful classifier is introduced to do the rest. By incorporating insights from higher areas of the VC, we add to the already used retinotopic step an object-centered step which increases invariance capabilities without losing so much information. By doing so, we reduce the need for a powerful, data hungry classification layer and, thus, are able to introduce a simple classification mechanism which is based on selective attention. The resulting model is tested with an invariant pattern recognition task in the MNIST and ETL-1 datasets. We verify that the model is able to achieve better accuracies with less training examples. More specifically, on the MNIST test set, the model achieves a 100% accuracy when trained with little more than 10% of the training set.

DNN Engine: A 28-nm Timing-Error Tolerant Sparse Deep Neural Network Processor for IoT Applications

This paper presents a 28-nm system-on-chip (SoC) for Internet of things (IoT) applications with a programmable accelerator design that implements a powerful fully connected deep neural network (DNN) classifier. To reach the required low energy consumption, we exploit the key properties of neural network algorithms: parallelism, data reuse, small/sparse data, and noise tolerance. We map the algorithm to a very large scale integration (VLSI) architecture based around an single-instruction, multiple-data data path with hardware support to exploit data sparsity by completely eliding unnecessary computation and data movement. This approach exploits sparsity, without compromising the parallel computation. We also exploit the inherent algorithmic noise-tolerance of neural networks, by introducing circuit-level timing violation detection to allow worst case voltage guard-bands to be minimized. The resulting intermittent timing violations may result in logic errors, which conventionally need to be corrected. However, in lieu of explicit error correction, we cope with this by accentuating the noise tolerance of neural networks. The measured test chip achieves high classification accuracy (98.36% for the MNIST test set), while tolerating aggregate timing violation rates > 10−1. The accelerator achieves a minimum energy of 0.36 $\mu \text{J}$ /inference at 667 MHz; maximum throughput at 1.2 GHz and 0.57 $\mu \text{J}$ /inference; or a 10% margined operating point at 1 GHz and 0.58 $\mu \text{J}$ /inference.

Deep Belief Network Training Improvement Using Elite Samples Minimizing Free Energy

Nowadays, it is very popular to use deep architectures in machine learning. Deep Belief Networks (DBNs) are deep architectures that use stack of Restricted Boltzmann Machines (RBMs) to create a powerful generative model using the training data. In this paper, we present an improvement in a common method that is usually used in training RBMs. The new method uses free energy as a criterion to obtain elite samples from generative model. We argue that these samples can compute gradient of log probability of training data more accurately. According to the results, an error rate of 0.99% was achieved on MNIST test set. This result shows that the proposed method outperforms the method presented in the first paper introducing DBN (1.25% error rate) and general classification methods such as SVM (1.4% error rate) and KNN (1.6% error rate). In another test using ISOLET dataset, letter classification error dropped to 3.59% compared to 5.59% error rate achieved in the papers using this dataset.

Combination of the assembly neural network with a perceptron for recognition of handwritten digits arranged in numeral strings

The purpose of the paper is to design and test neural network structures and mechanisms for making use of the information that is contained in the character strings for more correct recognition of the characters constituting these strings. Two neural networks are considered in the paper; both networks are combined into a joint recognition system. The first is the assembly neural network and the second is the neural network of a perceptron type. A computer simulation of the system is performed. The combined system solves the task of recognition of handwritten digits of the MNIST test set provided that the digits have been arranged in the numeral strings memorized in the system. During a recognition process of an input numeral string, the assembly neural network executes intermediate recognition of the digits basing on which a perceptron type network accomplishes the final choice among the limited combinations of strings memorized in the network. The experiments have demonstrated that the combined system is able to make use of the information that is contained in the strings for more correct recognition of digits of the MNIST test set. In particular, the experiments have shown that the combined system commits no errors in the recognition of MNIST test set on the condition that the digits of this set had been organized in the strings of more than 5 digits each.

A FAST SVM TRAINING ALGORITHM

A fast support vector machine (SVM) training algorithm is proposed under SVM's decomposition framework by effectively integrating kernel caching, digest and shrinking policies and stopping conditions. Kernel caching plays a key role in reducing the number of kernel evaluations by maximal reusage of cached kernel elements. Extensive experiments have been conducted on a large handwritten digit database MNIST to show that the proposed algorithm is much faster than Keerthi et al.'s improved SMO, about nine times. Combined with principal component analysis, the total training for ten one-against-the-rest classifiers on MNIST took less than an hour. Moreover, the proposed fast algorithm speeds up SVM training without sacrificing the generalization performance. The 0.6% error rate on MNIST test set has been achieved. The promising scalability of the proposed scheme paves a new way to solve more large-scale learning problems in other domains such as data mining.