Modified National Institute Of Standards And Technology Research Articles

Effect of weight overlap region on neuromorphic system with memristive synaptic devices

Recently, hardware-based neural network using memristive devices, so called neuromorphic system, has been extensively studied. Especially, on-chip (in situ) learning methods where training occurs inside hardware structure itself have been proposed and optimized based on memristor crossbar arrays regarding the linearity of weight-update characteristics. In this study, we analyze the effect of conductance overlap region of memristor on the recognition accuracy for on-chip learning simulation. The effect of conductance overlap region on recognition accuracy for modified national institute of standards and technology (MNIST) dataset is studied with an identical potentiation/depression pulse applied to Pt/Al2O3/TiOx/Ti/Pt stacked memristor. The overlap range can be varied by different pulse amplitude, and the training characteristics of memristive neural network is significantly dependent on the weight-update overlap region.

Determination of the influence of the choice of the pruning procedure parameters on the learning quality of a multilayer perceptron

Pruning connections in a fully connected neural network allows to remove redundancy in the structure of the neural network and thus reduce the computational complexity of its implementation while maintaining the resulting characteristics of the classification of images entering its input. However, the issues of choosing the parameters of the pruning procedure have not been sufficiently studied at the moment. The choice essentially depends on the configuration of the neural network. However, in any neural network configuration there is one or more multilayer perceptrons. For them, it is possible to develop universal recommendations for choosing the parameters of the pruning procedure. One of the most promising methods for practical implementation is considered – the iterative pruning method, which uses preprocessing of input signals to regularize the learning process of a neural network. For a specific configuration of a multilayer perceptron and the MNIST (Modified National Institute of Standards and Technology) dataset, a database of handwritten digit samples proposed by the US National Institute of Standards and Technology as a standard when comparing image recognition methods, dependences of the classification accuracy of handwritten digits and learning rate were obtained on the learning step, pruning interval, and the number of links removed at each pruning iteration. It is shown that the best set of parameters of the learning procedure with pruning provides an increase in the quality of classification by about 1 %, compared with the worst set in the studied range. The convex nature of these dependencies allows a constructive approach to finding a neural network configuration that provides the highest classification accuracy with the minimum amount of computational costs during implementation.

In-depth analysis on electrical parameters of floating gate IGZO synaptic transistor affecting pattern recognition accuracy

The reliable conductance modulation of synaptic devices is key when implementing high-performance neuromorphic systems. Herein, we propose a floating gate indium gallium zinc oxide (IGZO) synaptic device with an aluminum trapping layer to investigate the correlation between its diverse electrical parameters and pattern recognition accuracy. Basic synaptic properties such as excitatory postsynaptic current, paired pulse facilitation, long/short term memory, and long-term potentiation/depression are demonstrated in the IGZO synaptic transistor. The effects of pulse tuning conditions associated with the pulse voltage magnitude, interval, duration, and cycling number of the applied pulses on the conductance update are systematically investigated. It is discovered that both the nonlinearity of the conductance update and cycle-to-cycle variation should be critically considered using an artificial neural network simulator to ensure the high pattern recognition accuracy of Modified National Institute of Standards and Technology (MNIST) handwritten digit images. The highest recognition rate of the MNIST handwritten dataset is 94.06% for the most optimized pulse condition. Finally, a systematic study regarding the synaptic parameters must be performed to optimize the developed synapse device.

In-depth analysis on electrical parameters of floating gate IGZO synaptic transistor affecting pattern recognition accuracy.

The reliable conductance modulation of synaptic devices is key when implementing high-performance neuromorphic systems. Herein, we propose a floating gate IGZO synaptic device with an aluminum trapping layer to investigate the correlation between its diverse electrical parameters and pattern recognition accuracy. Basic synaptic properties such as excitatory postsynaptic current, paired pulse facilitation, long/short term memory, and long-term potentiation/depression are demonstrated in the IGZO synaptic transistor. The effects of pulse tuning conditions associated with the pulse voltage magnitude, interval, duration, and cycling number of the applied pulses on the conductance update are systematically investigated. It is discovered that both the nonlinearity of the conductance update and cycle-to-cycle variation should be critically considered using an artificial neural network simulator to ensure the high pattern recognition accuracy of Modified National Institute of Standards and Technology (MNIST) handwritten digit images. The highest recognition rate of the MNIST handwritten dataset is 94.06% for the most optimized pulse condition. Finally, a systematic study regarding the synaptic parameters must be performed to optimize the developed synapse device.

Coexistence of non-volatile and volatile characteristics of the Pt/TaOx/TiN device

Here, we have identified the volatile and non-volatile traits of Pt/TaOx/TiN device through the various electrical experiments by controlling the compliance current (CC) and determined the availability of this device as a memristor. First of all, the configuration analysis is performed to ensure the structure and thickness of the device. We monitored the volatile resistive switching, which can be facilitated as short-term memory (STM) at a low current level. But in the higher CC, the forming process at the negative bias needs to be accompanied and shows the bipolar resistive switching (BRS), which can be used as non-volatile memory (NVM) within the neuromorphic system. Also, we investigated the conduction mechanism and proved that Space-Charge-Limited-Current (SCLC) dominates the high resistive state in the reset mechanism. The distinct difference between volatile and non-volatile characteristics that appear depending on the CC is shown in the retention test. Finally, we conducted the pattern recognition of the Modified National Institute of Standards and Technology (MNIST) and obtained two pattern recognition accuracy using the potentiation/depression data measured from different BRS types.

HANDWRITTEN MATHEMATICAL EQUATION SOLVER

With recent developments in Artificial intelligence and deep learning every major field which is using computers for any type of work is trying to ease the work using deep learning methods. Deep learning is used in a wide range of fields due to its diverse range of applications like health, sports, robotics, education, etc. In deep learning, a Convolutional neural network (CNN) is being used in image classification, pattern recognition, Text classification, face recognition, live monitoring systems, handwriting recognition, Digit recognition, etc. In this paper, we propose a system for educational use where the recognition and solving process of mathematical equations will be done by machine. In this system for recognition of equations, we use a Convolutional neural network (CNN) model. The proposed system can recognize and solve mathematical equations with basic operations (-,+,/,*) of multiple digits as well as polynomial equations. The model is trained with Modified National Institute of Standards and Technology (MNIST) dataset as well as a manually prepared dataset of operator symbols (“-”,”+”, “/”, “*”, “(“, “)” ). Further, the system uses the RNN model to solve the recognized operations.

NOR-Type 3-D Synapse Array Architecture Based on Charge-Trap Flash Memory

In this work, we proposed a three-dimensional (3-D) channel stacked array architecture based on charge-trap flash (CTF) memory for an artificial neural network accelerator. The proposed synapse array architecture could be a promising solution for implementing efficiently a large-size artificial neural network on a limited-size hardware chip. We designed a full array architecture including a stacked layer selection circuit. In addition, we investigated the synaptic characteristics of CTF device by using technology computer-aided design (TCAD) simulation. We demonstrated the feasibility of the synapse array for neural network accelerators through a system-level MATLAB simulation with the Modified National Institute of Standards and Technology (MNIST) database.

Hyper-Parameter Optimization of Semi-Supervised GANs Based-Sine Cosine Algorithm for Multimedia Datasets

Generative Adversarial Networks (GANs) are neural networks that allow models to learn deep representations without requiring a large amount of training data. Semi-Supervised GAN Classifiers are a recent innovation in GANs, where GANs are used to classify generated images into real and fake and multiple classes, similar to a general multi-class classifier. However, GANs have a sophisticated design that can be challenging to train. This is because obtaining the proper set of parameters for all models-generator, discriminator, and classifier is complex. As a result, training a single GAN model for different datasets may not produce satisfactory results. Therefore, this study proposes an SGAN model (Semi-Supervised GAN Classifier). First, a baseline model was constructed. The model was then enhanced by leveraging the Sine-Cosine Algorithm and Synthetic Minority Oversampling Technique (SMOTE). SMOTE was used to address class imbalances in the dataset, while Sine Cosine Algorithm (SCA) was used to optimize the weights of the classifier models. The optimal set of hyperparameters (learning rate and batch size) were obtained using grid manual search. Four well-known benchmark datasets and a set of evaluation measures were used to validate the proposed model. The proposed method was then compared against existing models, and the results on each dataset were recorded and demonstrated the effectiveness of the proposed model. The proposed model successfully showed improved test accuracy scores of 1%, 2%, 15%, and 5% on benchmarking multimedia datasets; Modified National Institute of Standards and Technology (MNIST) digits, Fashion MNIST, Pneumonia Chest X-ray, and Facial Emotion Detection Dataset, respectively.

Exploring the operation of a neural network using the example of handwritten digits recognition in an advanced informatics course using the python 3.8 programming language

The article discusses the possibility of including artificial intelligence topics in the informatics course at the level of secondary general education on the demonstration example of handwritten digits recognition using the Python 3.8 programming language and the TensorFlow package. A hands-on example of a classification problem using the classic MNIST (Modified National Institute of Standards and Technology) training example set is dealt with step-by-step.Most of the tools required for the work (language interpreter, basic libraries, and shell) are downloaded in the form of a single software distribution kit of Anaconda. The network is trained using different methods. When the model is trained, the optimization method, the loss function, and the metric for estimation are specified. Image processing by convolutional nets containing special layers, which "convolve" the image the way a bitmap filter does, is also demonstrated

Hyper-parameter optimization of convolutional neural network based on particle swarm optimization algorithm

Deep neural networks have accomplished enormous progress in tackling many problems. More specifically, convolutional neural network (CNN) is a category of deep networks that have been a dominant technique in computer vision tasks. Despite that these deep neural networks are highly effective; the ideal structure is still an issue that needs a lot of investigation. Deep Convolutional Neural Network model is usually designed manually by trials and repeated tests which enormously constrain its application. Many hyper-parameters of the CNN can affect the model performance. These parameters are depth of the network, numbers of convolutional layers, and numbers of kernels with their sizes. Therefore, it may be a huge challenge to design an appropriate CNN model that uses optimized hyper-parameters and reduces the reliance on manual involvement and domain expertise. In this paper, a design architecture method for CNNs is proposed by utilization of particle swarm optimization (PSO) algorithm to learn the optimal CNN hyper-parameters values. In the experiment, we used Modified National Institute of Standards and Technology (MNIST) database of handwritten digit recognition. The experiments showed that our proposed approach can find an architecture that is competitive to the state-of-the-art models with a testing error of 0.87%.

Gradient Decomposition Methods for Training Neural Networks With Non-ideal Synaptic Devices.

While promising for high-capacity machine learning accelerators, memristor devices have non-idealities that prevent software-equivalent accuracies when used for online training. This work uses a combination of Mini-Batch Gradient Descent (MBGD) to average gradients, stochastic rounding to avoid vanishing weight updates, and decomposition methods to keep the memory overhead low during mini-batch training. Since the weight update has to be transferred to the memristor matrices efficiently, we also investigate the impact of reconstructing the gradient matrixes both internally (rank-seq) and externally (rank-sum) to the memristor array. Our results show that streaming batch principal component analysis (streaming batch PCA) and non-negative matrix factorization (NMF) decomposition algorithms can achieve near MBGD accuracy in a memristor-based multi-layer perceptron trained on the MNIST (Modified National Institute of Standards and Technology) database with only 3 to 10 ranks at significant memory savings. Moreover, NMF rank-seq outperforms streaming batch PCA rank-seq at low-ranks making it more suitable for hardware implementation in future memristor-based accelerators.

ReSe2-Based RRAM and Circuit-Level Model for Neuromorphic Computing

Resistive random-access memory (RRAM) devices have drawn increasing interest for the simplicity of its structure, low power consumption and applicability to neuromorphic computing. By combining analog computing and data storage at the device level, neuromorphic computing system has the potential to meet the demand of computing power in applications such as artificial intelligence (AI), machine learning (ML) and Internet of Things (IoT). Monolayer rhenium diselenide (ReSe2), as a two-dimensional (2D) material, has been reported to exhibit non-volatile resistive switching (NVRS) behavior in RRAM devices with sub-nanometer active layer thickness. In this paper, we demonstrate stable multiple-step RESET in ReSe2 RRAM devices by applying different levels of DC electrical bias. Pulse measurement has been conducted to study the neuromorphic characteristics. Under different height of stimuli, the ReSe2 RRAM devices have been found to switch to different resistance states, which shows the potentiation of synaptic applications. Long-term potentiation (LTP) and depression (LTD) have been demonstrated with the gradual resistance switching behaviors observed in long-term plasticity programming. A Verilog-A model is proposed based on the multiple-step resistive switching behavior. By implementing the LTP/LTD parameters, an artificial neural network (ANN) is constructed for the demonstration of handwriting classification using Modified National Institute of Standards and Technology (MNIST) dataset.

Memory Model for Morphological Semantics of Visual Stimuli Using Sparse Distributed Representation

Recent achievements on CNN (convolutional neural networks) and DNN (deep neural networks) researches provide a lot of practical applications on computer vision area. However, these approaches require construction of huge size of training data for learning process. This paper tries to find a way for continual learning which does not require prior high-cost training data construction by imitating a biological memory model. We employ SDR (sparse distributed representation) for information processing and semantic memory model, which is known as a representation model of firing patterns on neurons in neocortex area. This paper proposes a novel memory model to reflect remembrance of morphological semantics of visual input stimuli. The proposed memory model considers both memory process and recall process separately. First, memory process converts input visual stimuli to sparse distributed representation, and in this process, morphological semantic of input visual stimuli can be preserved. Next, recall process can be considered by comparing sparse distributed representation of new input visual stimulus and remembered sparse distributed representations. Superposition of sparse distributed representation is used to measure similarities. Experimental results using 10,000 images in MNIST (Modified National Institute of Standards and Technology) and Fashion-MNIST data sets show that the sparse distributed representation of the proposed model efficiently keeps morphological semantic of the input visual stimuli.

A robust neuromorphic vision sensor with optical control of ferroelectric switching

The rapid development of the artificial intelligence field has increased the demand for retina-inspired neuromorphic vision sensors with integrated sensing, memory, and processing functions. Here, we present a neuromorphic vision sensor with an optoelectronic transistor structure consisting of monolayer molybdenum disulfide and barium titanate ferroelectric film. Beyond conventional electrical tuning of ferroelectric polarization, the optoelectronic transistor can exhibit a light-dosage tunable synaptic behavior with a high switching ratio and good non-volatility, enabled by photo-induced ferroelectric polarization reversal. The wavelength-dependent optical sensing and multi-level optical memory properties are utilized to achieve the in-sensor neuromorphic visual pre-processing. A simulated artificial neural network built from the proposed vision sensors with neuromorphic pre-processing function demonstrated that the image recognition rate for the Modified National Institute of Standards and Technology (MNIST) handwritten dataset could be significantly improved by reducing redundant data. The obtained results suggest that 2D semiconductor/ferroelectric optoelectronic transistors can provide a promising hardware implementation towards constructing high-performance neuromorphic visual systems

Hardware-Based Spiking Neural Networks Using Capacitor-Less Positive Feedback Neuron Devices

In this article, hardware-based spiking neural networks (SNNs) using capacitor-less positive feedback (PF) neuron devices are designed. It was reported that the PF device can simultaneously process the excitatory and inhibitory signals. The PF device shows very steep subthreshold slope (SS < 1 mV/dec) due to the PF opertaion, leading to low-power and reliable neuron device. The PF devices also show the behavior of leaky integrate and fire (LIF) neuron, which is the most popular neuron model in SNNs. For hardware configuration, the neuron characteristics of PF device are investigated with the transient behavior of the anode current. Based on the PF neuron devices, the SNN shows the accuracy of 98.19% for the Modified National Institute of Standards and Technology (MNIST) database classification in four-hidden layer, fully-connected neural network, which is near the accuracy (98.46%) of the artificial neural networks using rectified linear unit (ReLU) activation function.

All-optical image identification with programmable matrix transformation.

An optical neural network is proposed and demonstrated with programmable matrix transformation and nonlinear activation function of photodetection (square-law detection). Based on discrete phase-coherent spatial modes, the dimensionality of programmable optical matrix operations is 30∼37, which is implemented by spatial light modulators. With this architecture, all-optical classification tasks of handwritten digits, objects and depth images are performed. The accuracy values of 85.0% and 81.0% are experimentally evaluated for MNIST (Modified National Institute of Standards and Technology) digit and MNIST fashion tasks, respectively. Due to the parallel nature of matrix multiplication, the processing speed of our proposed architecture is potentially as high as 7.4∼74 T FLOPs per second (with 10∼100 GHz detector).

Automatic, Qualitative Scoring of the Clock Drawing Test (CDT) Based on U-Net, CNN and Mobile Sensor Data.

The Clock Drawing Test (CDT) is a rapid, inexpensive, and popular screening tool for cognitive functions. In spite of its qualitative capabilities in diagnosis of neurological diseases, the assessment of the CDT has depended on quantitative methods as well as manual paper based methods. Furthermore, due to the impact of the advancement of mobile smart devices imbedding several sensors and deep learning algorithms, the necessity of a standardized, qualitative, and automatic scoring system for CDT has been increased. This study presents a mobile phone application, mCDT, for the CDT and suggests a novel, automatic and qualitative scoring method using mobile sensor data and deep learning algorithms: CNN, a convolutional network, U-Net, a convolutional network for biomedical image segmentation, and the MNIST (Modified National Institute of Standards and Technology) database. To obtain DeepC, a trained model for segmenting a contour image from a hand drawn clock image, U-Net was trained with 159 CDT hand-drawn images at 128 × 128 resolution, obtained via mCDT. To construct DeepH, a trained model for segmenting the hands in a clock image, U-Net was trained with the same 159 CDT 128 × 128 resolution images. For obtaining DeepN, a trained model for classifying the digit images from a hand drawn clock image, CNN was trained with the MNIST database. Using DeepC, DeepH and DeepN with the sensor data, parameters of contour (0–3 points), numbers (0–4 points), hands (0–5 points), and the center (0–1 points) were scored for a total of 13 points. From 219 subjects, performance testing was completed with images and sensor data obtained via mCDT. For an objective performance analysis, all the images were scored and crosschecked by two clinical experts in CDT scaling. Performance test analysis derived a sensitivity, specificity, accuracy and precision for the contour parameter of 89.33, 92.68, 89.95 and 98.15%, for the hands parameter of 80.21, 95.93, 89.04 and 93.90%, for the numbers parameter of 83.87, 95.31, 87.21 and 97.74%, and for the center parameter of 98.42, 86.21, 96.80 and 97.91%, respectively. From these results, the mCDT application and its scoring system provide utility in differentiating dementia disease subtypes, being valuable in clinical practice and for studies in the field.

Robust Binary Neural Network Operation From 233 K to 398 K via Gate Stack and Bias Optimization of Ferroelectric FinFET Synapses

A synergistic approach for optimizing devices, circuits, and neural network architectures was used to abate junction-temperature-change-induced performance degradation of an Fe-FinFET-based artificial neural network. We demonstrated that the digital nature of the binarized neural network, with the “0” state programmed deep in the subthreshold and the “1” state in strong inversion, is crucial for robust deep neural network inference. The performance of a purely software-based binary neural network (BNN), with 96.1% accuracy for Modified National Institute of Standards and Technology (MNIST) handwritten digit recognition, was used as a baseline. The Fe-FinFET-based BNN (including device-to-device variation at 300 K) achieved 95.7% inference accuracy on the MNIST dataset. Although substantial inference accuracy degradation with temperature change was observed in a nonbinary neural network, the BNN with optimized Fe-FinFETs as synaptic devices had excellent resistance to temperature change effects, and maintained a minimum inference accuracy of 95.2% within a temperature range of -40 to 125 °C after gate stack and bias optimization. However, reprogramming to adjust device conductance was necessary for temperatures higher than 125 °C.

Core-Shell Dual-Gate Nanowire Charge-Trap Memory for Synaptic Operations for Neuromorphic Applications

This work showcases the physical insights of a core-shell dual-gate (CSDG) nanowire transistor as an artificial synaptic device with short/long-term potentiation and long-term depression (LTD) operation. Short-term potentiation (STP) is a temporary potentiation of a neural network, and it can be transformed into long-term potentiation (LTP) through repetitive stimulus. In this work, floating body effects and charge trapping are utilized to show the transition from STP to LTP while de-trapping the holes from the nitride layer shows the LTD operation. Furthermore, linearity and symmetry in conductance are achieved through optimal device design and biases. In a system-level simulation, with CSDG nanowire transistor a recognition accuracy of up to 92.28% is obtained in the Modified National Institute of Standards and Technology (MNIST) pattern recognition task. Complementary metal-oxide-semiconductor (CMOS) compatibility and high recognition accuracy makes the CSDG nanowire transistor a promising candidate for the implementation of neuromorphic hardware.

Fully Unsupervised Spike-Rate-Dependent Plasticity Learning With Oxide- Based Memory Devices

Artificial synapses are fundamental building blocks for neuromorphic computing hardware. In this work, we demonstrate fully unsupervised neuromorphic learning using analog weight change properties of bilayer nonfilamentry oxide-based resistive memory (OxRAM) devices as synaptic elements. Essential functions of a biological synapse, such as potentiation, depression, and spike-rate-dependent plasticity (SRDP), are experimentally demonstrated. Furthermore, we show the tune-ability of SRDP with preneuron/postneuron spike-train parameters (frequency and amplitude). Through simulations, we show a two-layer fully connected neuromorphic network powered by SRDP and OxRAM synapses. Our network achieves state-of-the-art best classification accuracy results compared to the literature (for SRDP-based purely unsupervised spiking neural networks (SNNs), without the use of any convolutional layer) on full Modified National Institute of Standards and Technology (MNIST) dataset: 92.07% (training) and 90.76% (test). Detailed impact of device properties, such as variability, retention, and endurance, is analyzed. The impact of introducing sparsity during training and weight pruning during inference was also investigated.