MNIST Database Of Handwritten Digits Research Articles

Photonic convolution accelerator based on a hybrid integrated multi-wavelength laser array by photonic wire bonding for real-time image classification.

We propose and experimentally demonstrate a compact and efficient photonic convolution accelerator based on a hybrid integrated multi-wavelength DFB laser array by photonic wire bonding. The photonic convolution accelerator operates at 60.12 GOPS for one 3 × 3 kernel with a convolution window vertical sliding stride of 1 and generates 500 images of real-time image classification. Furthermore, real-time image classification on the MNIST database of handwritten digits with a prediction accuracy of 93.86% is achieved. This work provides a novel, to the best of our knowledge, compact hybrid integration platform to realize the optical convolutional neural networks.

Research and evaluation of the efficiency of handwritten character recognition methods using convulsional neural networks

The paper analyzes the possibilities of using deep learning neural networks for the implementation of text processing information systems, substantiates the prospects of this technology and the possibilities of its improvement. The possibility of using the MNIST database of handwritten digits on, as well as the EMNIST database of handwritten letters and numbers, namely the EMNIST Letters set, was considered. The research object was modelled in IDEF0 and IDEF3 notations for the «AS-IS» option. It was found that «Building a dataset for model training» and «Choosing a convolutional neural network architecture» are the most frequently asked questions. Reengineering measures are proposed, namely, the expediency of using, in addition to the well-known EMNIST and MNIST sets, a set of handwritten and italic fonts with Ukrainian glyphs, additionally created as part of the work, is shown. For this purpose, modern IT tools such as the Pillow library, Image Data Generator and the Scikit-Learn package were used to select training and test samples. Also, in addition to the initially proposed simplest CNN architectures of the Lenet type, the use of more complex architectures of the AlexNet and VGG-16 types is proposed. Taking into account the proposed reengineering measures, IDEF0 and IDEF3 diagrams were constructed for the «TO-BE» option. The paper analyzes in detail the results of recognizing handwritten Ukrainian letters and Arabic numerals using 6 different CNN architectures using a synthetic data set for training. The research presented in the work was carried out using a software application developed in the Python programming language using the Scikit Learn package, which provides the user with the ability to recognize handwritten text using a multilayer perceptron. carried out precisely using the developed program. It is justified that, in contrast to simple Lenet-type architectures, it is more appropriate to use more complex options, namely the VGG-16 type architecture. Experimental studies of the influence of the number of CNN parameters of different architectures on the recognition accuracy and training time of the neural network have been carried out. Also analyzed are the results of character recognition when recognizing images that do not belong to the training or test sample

Neural networks: Studying their decision-making rules

The question of a better understanding of the behavior of neural networks is quite relevant, especially in industries with a high level of risks. To solve this problem, the possibilities of the new DeepRED decomposition algorithm, capable of extracting decision-making rules by deep neural networks with several hidden layers, are explored in the paper. The study of the DeepRED algorithm was carried out on the example of extracting the rules of an experimental neural network during the classification of images of the MNIST database of handwritten digits, which made it possible to reveal a number of limitations of the DeepRED algorithm.

Unconstrained neighbor selection for minimum reconstruction error-based K-NN classifiers

It is essential to define more convincing and applicable classifiers for small datasets. In this paper, a minimum reconstruction error-based K-nearest neighbors (K-NN) classifier is proposed. We propose a new neighbor selection method. In the proposed neighbor selection method, a subset of data that minimize the reconstruction error is assigned as the neighbors of the query sample. Also, there is not any constraint on the distance of the neighbors from the query sample. An {{l}}_{0}-based sparse representation problem is introduced for selecting the neighbors. These neighbors are assigned as neighbors of the minimum reconstruction error-based K-NN classifiers. Three {{l}}_{0}-based minimum reconstruction error-based K-NN classifiers are introduced. These classifiers are less sensitive to the reconstruction coefficients in minimum reconstruction error-based K-NN classifiers and reconstruct the query sample with less error. The results on UCI machine learning repository, UCR time-series archive datasets, and a small subset (16%) of MNIST handwritten digit database demonstrate the suitable performance of the proposed method. The recognition precision increases up to more than 3% in some evaluations.

Experimental demonstration of a photonic convolutional accelerator based on a monolithically integrated multi-wavelength distributed feedback laser.

We propose and experimentally demonstrate a simple and energy-efficient photonic convolutional accelerator based on a monolithically integrated multi-wavelength distributed feedback semiconductor laser using the superimposed sampled Bragg grating structure. The photonic convolutional accelerator operates at 44.48 GOPS for one 2 × 2 kernel with a convolutional window vertical sliding stride of 2 and generates 100 images of real-time recognition. Furthermore, a real-time recognition task on the MNIST database of handwritten digits with a prediction accuracy of 84% is achieved. This work provides a compact and low-cost way to realize photonic convolutional neural networks.

Photonics-enabled spiking timing-dependent convolutional neural network for real-time image classification.

A photonics-enabled spiking timing-dependent convolutional neural network (CNN) is proposed by manipulating photonics multidimensional parameters in terms of wavelength, temporal and spatial, which breaks the traditional CNN architecture mapping from a spatially parallel to a time-dependent series structure. The proposed CNN with the application of real-time image recognition comprises a photonics convolution processor to accelerate the computing and an involved electronic full connection to execute the classification task. A timing-dependent series of matrix-matrix operations is conducted in the photonics convolution processor that can be achieved based on multidimensional multiplexing by the accumulation of carriers from an active mode-locked laser, dispersion latency induced by a dispersion compensation fiber, and wavelength spatial separation via a waveshaper. Incorporated with the electronic full connection, a photonics-enabled CNN is proven to perform a real-time recognition task on the MNIST database of handwritten digits with a prediction accuracy of 90.04%. Photonics enables conventional neural networks to accelerate machine learning and neuromorphic computing and has the potential to be widely used in information processing and computing, such as goods classification, vowel recognition, and speech identification.

An improved method for text detection using Adam optimization algorithm

Optical Character Recognition (OCR) is an automatic identification technique which is applied in different application areas to translate documents or images into analysable and editable data. Printed or typed characters are easy to recognize as they have well defined shape and size, but this is not true in case of handwritten text. Handwriting of every individual is different so OCR face difficulty to recognize the characters. In past, researchers have been used different Machine Learning and Artificial Intelligence tools and techniques to analyse handwritten and printed documents and also worked to create an electronic format file from them. It is difficult to reuse this information as it is very difficult to search the content from these documents by lines or words. To solve this problem, OpenCV technique is used in this research work which focuses on training and testing of neural network model to conduct Document Image Analysis. The proposed model is named as J&M model for Text Detection from Hand written images. Implementation of research work is done in Python on MNIST database of handwritten digits. From this research work, 99.5% of training accuracy and 99% of testing accuracy was achieved along with training loss of 1.5%.

Comparison of classical machine learning algorithms in the task of handwritten digits classification

The purpose of this paper is to compare classical machine learning algorithms for handwritten number classification. The following algorithms were chosen for comparison: Logistic Regression, SVM, Decision Tree, Random Forest and k-NN. MNIST handwritten digit database is used in the task of training and testing the above algorithms. The dataset consists of 70,000 images of numbers from 0 to 9. The algorithms are compared considering such criteria as the learning speed, prediction construction speed, host machine load, and classification accuracy. Each algorithm went through the training and testing phases 100 times, with the desired KPIs retained at each iteration. The results were averaged to reach reliable outcomes.

Power Side-Channel Attacks on BNN Accelerators in Remote FPGAs

To lower cost and increase the utilization of Cloud Field-Programmable Gate Arrays (FPGAs), researchers have recently been exploring the concept of multi-tenant FPGAs, where multiple independent users simultaneously share the same remote FPGA. Despite its benefits, multi-tenancy opens up the possibility of malicious users co-locating on the same FPGA as a victim user, and extracting sensitive information. This issue becomes especially serious when the user is running a machine learning algorithm that is processing sensitive or private information. To demonstrate the dangers, this paper presents a remote, power-based side-channel attack on a deep neural network accelerator running in a variety of Xilinx FPGAs and also on Cloud FPGAs using Amazon Web Services (AWS) F1 instances. This work in particular shows how to remotely obtain voltage estimates as a deep neural network inference circuit executes, and how the information can be used to recover the inputs to the neural network. The attack is demonstrated with a binarized convolutional neural network used to recognize handwriting images from the MNIST handwritten digit database. With the use of precise time-to-digital converters for remote voltage estimation, the MNIST inputs can be successfully recovered with a maximum normalized cross-correlation of 79% between the input image and the recovered image on local FPGA boards and 72% on AWS F1 instances. The attack requires no physical access nor modifications to the FPGA hardware.

Design of classifiers based on ANN approximations of traditional methods

AbstractIn this paper, we suggest basing the development of classification methods on traditional techniques but approximating them in whole or in part with artificial neural networks (ANNs). Compared to a direct ANN approach, the underlying traditional method is often easier to analyse. Classification failures can be better understood and corrected for, while at the same time faster execution can be obtained due to the parallel ANN structure. Furthermore, such a two‐step design philosophy partly eliminates the ‘guesswork’ associated with the design of ANNs. The expected gain is thus that the benefits from the traditional field, and the ANN field can be obtained by combining the best features from both fields. We illustrate our approach by working through an explicit example, namely, a Nearest Neighbour classifier applied to a subset of the MNIST database of handwritten digits. Two different approaches are discussed for how to translate the traditional method into an ANN. The first approach is based on a constructive implementation which directly reflects the original algorithm. The second approach uses ANNs to approximate the whole, or part of, the original method. An important part of the approach is to show how improvements can be introduced. In line with the presented philosophy, this is done by extending the traditional method in several ways followed by ANN approximation of the modified algorithms. The extensions are based on a windowed version of the nearest neighbour algorithm. We show that the improvements carry over to the ANN implementations. We further investigate the stability of the solutions by modifying the training set. It is shown that the errors do not change significantly. This also holds true for the ANN approximations providing confidence that the two‐step strategy is robust.

Pattern Recognition of Spiking Neural Networks Based on Visual Mechanism and Supervised Synaptic Learning.

Electrophysiological studies have shown that mammalian primary visual cortex are selective for the orientations of visual stimuli. Inspired by this mechanism, we propose a hierarchical spiking neural network (SNN) for image classification. Grayscale input images are fed through a feed-forward network consisting of orientation-selective neurons, which then projected to a layer of downstream classifier neurons through the spiking-based supervised tempotron learning rule. Based on the orientation-selective mechanism of the visual cortex and tempotron learning rule, the network can effectively classify images of the extensively studied MNIST database of handwritten digits, which achieves 96% classification accuracy based on only 2000 training samples (traditional training set is 60000). Compared with other classification methods, our model not only guarantees the biological plausibility and the accuracy of image classification but also significantly reduces the needed training samples. Considering the fact that the most commonly used deep learning neural networks need big data samples and high power consumption in image recognition, this brain-inspired computational neural network model based on the layer-by-layer hierarchical image processing mechanism of the visual cortex may provide a basis for the wide application of spiking neural networks in the field of intelligent computing.

A novel versatile window function for memristor model with application in spiking neural network

The memristor is a nano-size electrical device that is widely investigated in memory, logic circuits, and neuromorphic systems. This paper puts forward a general window function for Hewlett-Packard (HP) TiO2 memristor model, which covers almost all typical window functions and resolves the boundary lock problem, the inflexible parameter problem, and the distorted pinched hysteresis loop problem. Several adjustable parameters are introduced in the proposed window function in order to make it more flexible and scalable. A comparison with some classic window functions is given. Meanwhile, the range of these parameters is deduced with the aim of finding suitable parameters to simulate memristor devices. This TiO2 memristor model based on the novel window function is further applied to the spiking neural network and tested with the MNIST database of handwritten digits. The result demonstrates that the memristor-based neural network has higher recognition accuracy and better classification effect, indicating the availability of the entire scheme.

Energy-Efficient Pattern Recognition Hardware With Elementary Cellular Automata

The development of power-efficient Machine Learning Hardware is of high importance to provide Artificial Intelligence (AI) characteristics to those devices operating at the Edge. Unfortunately, state-of-the-art data-driven AI techniques such as deep learning are too costly in terms of hardware and energy requirements for Edge Computing (EC) devices. Recently, Cellular Automata (CA) have been proposed as a feasible way to implement Reservoir Computing (RC) systems in which the automaton rule is fixed and the training is performed using a linear regression model. In this work we show that Reservoir Computing based on CA may arise as a promising AI alternative for devices operating at the edge due to its intrinsic simplicity. For this purpose, a new low-power CA-based reservoir hardware is proposed and implemented in a FPGA (known as ReCA circuitry). The use of Elementary Cellular Automata (ECA) is able to further simplify the RC structure to implement a power efficient AI system suitable to be implemented in EC applications. Experiments have been conducted on the well-known MNIST handwritten digits database, obtaining competitive results in terms of processing time, circuit area, power and inference accuracy.

Rotation scaling and translation invariants by a remediation of Hu’s invariant moments

Due to the invariance to translation, rotation and scaling, the seven invariant moments presented by Hu (Visual pattern recognition by moment invariants, IRE Transactions on Information Theory, vol. 8, February 1962, pp. 179–187) are widely used in the field of pattern recognition. The set of these moments is finite; therefore, they do not comprise a complete set of image descriptors. To solve this problem, we introduce in this paper a new set of invariant moments of infinite order. The non-orthogonal property causes the redundancy of information. For this reason, we propose a new set of orthogonal polynomials in two variables, and we present a set of orthogonal moments, which are invariant to rotation, scale and translation. The presented approaches are tested by the invariability of the moments, the image retrieval and the classification of the objects. In this framework, using the proposed orthogonal moments, we present two classification systems. The first based on the Fuzzy C-Means Clustering algorithm (FCM) and the second based on the Radial Basis Functions Neural Network (RBF). The performance of our invariant moments is compared with Legendre invariant moments, Tchebichef-Krawtchouk (TKIM), Tchebichef-Hahn (THIM), Krawtchouk-Hahn (KHIM), Hu invariant moments, the descriptor of histogram of oriented gradients (HOG), the adaptive hierarchical density histogram features (AHDH) and with descriptors of color and texture Hist, HSV, FOS and SGLD. The experimental tests are performed on seven image databases: Columbia Object Image Library (COIL-20) database, MPEG7-CE shape database, MNIST handwritten digit database, MNIST fashion image database, ImageNet database, COIL-100 database and ORL database. The obtained results show the efficiency and superiority of our orthogonal invariant moments.

Deep Convolutional Self-Organizing Map Network for Robust Handwritten Digit Recognition

Deep Convolutional Neural Networks (DCNN) are currently the predominant technique commonly used to learn visual features from images. However, the complex structure of most recent DCNNs impose two major requirements namely, huge labeled dataset and high computational resources. In this paper, we develop a new efficient deep unsupervised network to learn invariant image representation from unlabeled visual data. The proposed Deep Convolutional Self-organizing Maps (DCSOM) network comprises a cascade of convolutional SOM layers trained sequentially to represent multiple levels of features. The 2D SOM grid is commonly used for either data visualization or feature extraction. However, this work employs high dimensional map size to create a new deep network. The N-Dimensional SOM (ND-SOM) grid is trained to extract abstract visual features using its classical competitive learning algorithm. The topological order of the features learned from ND-SOM helps to absorb local transformation and deformation variations exhibited in the visual data. The input image is divided into an overlapped local patches where each local patch is represented by the N-coordinates of the winner neuron in the ND-SOM grid. Each dimension of the ND-SOM can be considered as a non-linear principal component and hence it can be exploited to represent the input image using N-Feature Index Image (FII) bank. Multiple convolutional SOM layers can be cascaded to create a deep network structure. The output layer of the DCSOM network computes local histograms of each FII bank in the final convolutional SOM layer. A set of experiments using MNIST handwritten digit database and all its variants are conducted to evaluate the robust representation of the proposed DCSOM network. Experimental results reveal that the performance of DCSOM outperforms state-of-the-art methods for noisy digits and achieve a comparable performance with other complex deep learning architecture for other image variations.

An energy efficient time-mode digit classification neural network implementation.

This paper presents the design of an ultra-low energy neural network that uses time-mode signal processing). Handwritten digit classification using a single-layer artificial neural network (ANN) with a Softmin-based activation function is described as an implementation example. To realize time-mode operation, the presented design makes use of monostable multivibrator-based multiplying analogue-to-time converters, fixed-width pulse generators and basic digital gates. The time-mode digit classification ANN was designed in a standard CMOS 0.18 μm IC process and operates from a supply voltage of 0.6 V. The system operates on the MNIST database of handwritten digits with quantized neuron weights and has a classification accuracy of 88%, which is typical for single-layer ANNs, while dissipating 65.74 pJ per classification with a speed of 2.37 k classifications per second. This article is part of the theme issue 'Harmonizing energy-autonomous computing and intelligence'.

On-chip learning for domain wall synapse based Fully Connected Neural Network

Spintronic devices are considered as promising candidates in implementing neuromorphic systems or hardware neural networks, which are expected to perform better than other existing computing systems for certain data classification and regression tasks. In this paper, we have designed a feedforward Fully Connected Neural Network (FCNN) with no hidden layer using spin orbit torque driven domain wall devices as synapses and transistor based analog circuits as neurons. A feedback circuit is also designed using transistors, which at every iteration computes the change in weights of the synapses needed to train the network using Stochastic Gradient Descent (SGD) method. Subsequently it sends write current pulses to the domain wall based synaptic devices which move the domain walls and updates the weights of the synapses. Through a combination of micromagnetic simulations, analog circuit simulations and numerically solving FCNN training equations, we demonstrate "on-chip" training of the designed FCNN on the MNIST database of handwritten digits in this paper. We report the training and test accuracies, energy consumed in the synaptic devices for the training and possible issues with hardware implementation of FCNN that can limit its test accuracy.

The improvement of classification accuracy with denoising class autoencoder

Denoising autoencoder is a data compression algorithm in deep learning, and it can successfully extract features which contain main information of the input data. However, this method is trained in an unsupervised manner and cannot aim at extracting the specified information. Therefore, the input data will be classified into a class label by main but not necessary information using traditional denoising autoencoder, and redundant information reduces the classification accuracy. To solve this problem, denoising class autoencoder presented in this paper is adopted to train a classification function and extract class features. Both the proposed approach and traditional methods are trained on MNIST handwritten digit database with mini-batch gradient descent, moreover the recognition result of the proposed approach demonstrates the effectiveness in comparison with conventional methods.

Machine Learning with Partially Homomorphic Encrypted Data

Machine learning had been widely used to analyze various kinds of data, including sensitive data such as medical and financial data. A trained machine learning model can be wrapped in a web application so that people can access it easily via internet. However, if the data to be analyzed is private or confidential, this will cause a problem; the application administrator may read the input. As shown by Dowlin et al. in their remarkable paper, this kind of problem can be solved with homomorphic encryption scheme. Paillier encryption scheme is one kind of encryption scheme that has homomorphic property. In this research, we will show that one type of machine learning model can take an input encrypted by Paillier encryption scheme and produce an encrypted output that shares the same key. A machine learning model will be trained with the MNIST database of hand-written digits. This model will be tested with the test data encrypted with Paillier encryption scheme. The experiment shows that the model achieved 92.92% accuracy on the test set.

Similarity Learning and Generalization with Limited Data: A Reservoir Computing Approach

We investigate the ways in which a machine learning architecture known as Reservoir Computing learns concepts such as “similar” and “different” and other relationships between image pairs and generalizes these concepts to previously unseen classes of data. We present two Reservoir Computing architectures, which loosely resemble neural dynamics, and show that a Reservoir Computer (RC) trained to identify relationships between image pairs drawn from a subset of training classes generalizes the learned relationships to substantially different classes unseen during training. We demonstrate our results on the simple MNIST handwritten digit database as well as a database of depth maps of visual scenes in videos taken from a moving camera. We consider image pair relationships such as images from the same class; images from the same class with one image superposed with noise, rotated 90°, blurred, or scaled; images from different classes. We observe that the reservoir acts as a nonlinear filter projecting the input into a higher dimensional space in which the relationships are separable; i.e., the reservoir system state trajectories display different dynamical patterns that reflect the corresponding input pair relationships. Thus, as opposed to training in the entire high-dimensional reservoir space, the RC only needs to learns characteristic features of these dynamical patterns, allowing it to perform well with very few training examples compared with conventional machine learning feed-forward techniques such as deep learning. In generalization tasks, we observe that RCs perform significantly better than state-of-the-art, feed-forward, pair-based architectures such as convolutional and deep Siamese Neural Networks (SNNs). We also show that RCs can not only generalize relationships, but also generalize combinations of relationships, providing robust and effective image pair classification. Our work helps bridge the gap between explainable machine learning with small datasets and biologically inspired analogy-based learning, pointing to new directions in the investigation of learning processes.