MNIST Data Research Articles

Low-power flexible organic memristor based on PEDOT:PSS/pentacene heterojunction for artificial synapse.

Organic synaptic memristors are of considerable interest owing to their attractive characteristics and potential applications to flexible neuromorphic electronics. In this work, an organic type-II heterojunction consisting of poly(3,4-ethylenedioxythiophene): polystyrene sulfonate (PEDOT:PSS) and pentacene was adopted for low-voltage and flexible memristors. The conjugated polymer PEDOT:PSS serves as the flexible resistive switching (RS) layer, while the thin pentacene layer plays the role of barrier adjustment. This heterojunction enabled the memristor device to be triggered with low-energy RS operations (V < ± 1.0 V and I < 9.0 μA), and simultaneously providing high mechanical bending stability (bending radius of ≈2.5 mm, bending times = 1,000). Various synaptic properties have been successfully mimicked. Moreover, the memristors presented good potentiation/depression stability with a low cycle-to-cycle variation (CCV) of less than 8%. The artificial neural network consisting of this flexible memristor exhibited a high accuracy of 89.0% for the learning with MNIST data sets, even after 1,000 tests of 2.5% stress-strain. This study paves the way for developing low-power and flexible synaptic devices utilizing organic heterojunctions.

Image Denoising Based on GAN with Optimization Algorithm

Image denoising has been a knotty issue in the computer vision field, although the developing deep learning technology has brought remarkable improvements in image denoising. Denoising networks based on deep learning technology still face some problems, such as in their accuracy and robustness. This paper constructs a robust denoising network based on a generative adversarial network (GAN). Since the neural network has the phenomena of gradient dispersion and feature disappearance, the global residual is added to the autoencoder in the generator network, to extract and learn the features of the input image, so as to ensure the stability of the network. On this basis, we proposed an optimization algorithm (OA), to train and optimize the mean and variance of noise on each node of the generator. Then the robustness of the denoising network was improved through back propagation. Experimental results showed that the model’s denoising effect is remarkable. The accuracy of the proposed model was over 99% in the MNIST data set and over 90% in the CIFAR10 data set. The peak signal to noise ratio (PSNR) and structural similarity (SSIM) values of the proposed model were better than the state-of-the-art models in the BDS500 data set. Moreover, an anti-interference test of the model showed that the defense capacities of both the fast gradient sign method (FGSM) and project gradient descent (PGD) attacks were significantly improved, with PSNR and SSIM values decreased by less than 2%.

Feature Learning-Based Generative Adversarial Network Data Augmentation for Class-Based Few-Shot Learning

As training deep neural networks enough requires a large amount of data, there have been a lot of studies to deal with this problem. Data augmentation techniques are basic solutions to increase training data using existing data. Geometric transformations and color space augmentations are well-known augmentation techniques, but they still require some manual work and can generate limited types of data only. Therefore, there are many interests in generative-model-based augmentation lately, which can learn the distribution of data. This study proposes a set of GAN-based data augmentation methods that can generate good quality training data. The proposed networks, f-DAGAN (data augmentation generative adversarial networks), have been motivated by the DAGAN that learns data distribution from two real data. The basic f-DAGAN uses dual discriminators handling both generated data and generated feature spaces for better learning the given data. The other versions of f-DAGANs have been proposed for generating hard or easy data that have additional dual classifiers for both generated data and feature spaces to control the generator. Hard data is useful for optimized training to increase the target performance such as classification accuracy. Easy data generation can be used especially in few-shot learning. The quality of generated data has been validated in two ways: using t-SNE visualization of generated data and classification accuracy by training with generated data using the MNIST data set. The t-SNE representations show that data generated by f-DAGAN are evenly distributed for every class better than the exiting generative model-based augmentation methods. The f-DAGAN also shows the best classification accuracy by training with generated data. The f-DAGAN version for easy and hard data generation generates data well from five-shot learning and performs well in sample data generation experiments.

Enhancing deep learning algorithm accuracy and stability using multicriteria optimization: an application to distributed learning with MNIST digits

The training phase is the most crucial stage during the machine learning process. In the case of labeled data and supervised learning, machine learning entails minimizing the loss function under various constraints. We provide an innovative model for learning with numerous data sets, resulting from the application of multicriteria optimization techniques to existing deep learning algorithms. Data fitting is formulated as a multicriteria model in which each criterion measures the data fitting error on a specific data set. This is an optimization model involving a vector-valued function, and it has to be analyzed using the notion of Pareto efficiency. We present stability results for efficient solutions in the presence of input and output data perturbations. The multiple data set environment comes into play to eliminate the bias caused by the selection of a specific training set. To apply this concept, we present a scalarization strategy as well as numerical experiments in digit classification using MNIST data.

An Improved and Customized Hybrid of Deep and Machine Learning Technique Model for Handwritten Digit Recognition

In modern research, pattern recognition is a vast field for the academicians and researchers to contribute their work. Various kinds of patterns like images, character, handwritten digit, etc can be recognized and classify with the help of the intelligent techniques. This research work is concentrated on the classification of handwritten digit recognition. In the world, peoples’ handwriting is different from each other’s and uses different languages, so it is necessary to develop a model that is able to recognize handwritten digits with high accuracy. Various deep and machine learning methods have been implemented by various authors and achieved satisfactory results. This research work has proposed a hybrid technique that combines deep learning and machine learning algorithms for classification of handwritten digit characters. The proposed model has two steps, first feature extraction, and second is classification of handwritten digits. The Convolutional neural network (CNN) has been used as feature extractor, and support vector machine (SVM), K-nearest neighbors (KNN), and Random Forest Classifier (RFC) algorithms have been used to recognize and classify handwritten digits. We have used very famous handwritten digit dataset as MNIST that contains 70000 samples of 0-9 digits, and EMNIST is an extended version of the MNIST dataset, that contains 280000 handwritten digit samples. The contribution of this research work is to enhance the recognition rate by our proposed hybrid models as CNN with SVM named as CNN-SVM, CNN with RFC named as CNN-RFC, and CNN with KNN named as CNN-KNN model, and achieved excellent recognition rate with both MNIST and EMNIST data samples. We have achieved the testing accuracy as 99.45% by CNN-RFC, 99.48% by CNN-KNN, and 99.55% by CNN-SVM with MNIST dataset while 99.54% by CNN-RFC, 99.46% by CNN-KNN, and 99.66% by CNN-SVM with EMNIST dataset. The testing accuracy achieved by our proposed model is tremendous and higher than previous research work.

Sensitivity of Sparse Codes to Image Distortions.

Sparse coding has been proposed as a theory of visual cortex and as an unsupervised algorithm for learning representations. We show empirically with the MNIST data set that sparse codes can be very sensitive to image distortions, a behavior that may hinder invariant object recognition. A locally linear analysis suggests that the sensitivity is due to the existence of linear combinations of active dictionary elements with high cancellation. A nearest-neighbor classifier is shown to perform worse on sparse codes than original images. For a linear classifier with a sufficiently large number of labeled examples, sparse codes are shown to yield higher accuracy than original images, but no higher than a representation computed by a random feedforward net. Sensitivity to distortions seems to be a basic property of sparse codes, and one should be aware of this property when applying sparse codes to invariant object recognition.

A Spike Neural Network Model for Lateral Suppression of Spike-Timing-Dependent Plasticity with Adaptive Threshold

Aiming at the practical constraints of high resource occupancy and complex calculations in the existing Spike Neural Network (SNN) image classification model, in order to seek a more lightweight and efficient machine vision solution, this paper proposes an adaptive threshold Spike Neural Network (SNN) model of lateral inhibition of Spike-Timing-Dependent Plasticity (STDP). The conversion from grayscale image to pulse sequence is completed by convolution normalization and first pulse time coding. The network self-classification is realized by combining the classical Spike-Timing-Dependent Plasticity algorithm (STDP) and lateral suppression algorithm. The occurrence of overfitting is effectively suppressed by introducing an adaptive threshold. The experimental results on the MNIST data set show that compared with the traditional SNN classification model, the complexity of the weight update algorithm is reduced from O(n2) to O(1), and the accuracy rate can still remain stable at about 96%. The provided model is conducive to the migration of software algorithms to the bottom layer of the hardware platform, and can provide a reference for the realization of edge computing solutions for small intelligent hardware terminals with high efficiency and low power consumption.

Ambient Stable All Inorganic CsCu2I3 Artificial Synapses for Neurocomputing.

In resistive switching memories or artificial synaptic devices, halide perovskites have attracted attention for their unusual features such as rapid ion migration, adjustable composition, and facile synthesis. Herein, the environmentally friendly and highly air stable CsCu2I3 perovskite films are used as the active layer in the Au/CsCu2I3/ITO/glass artificial synapses. The device shows variable synaptic plasticities such as long-term and short-term synaptic plasticity, paired-pulse facilitation, and spike-timing-dependent plasticity by combining potentiation and depression along the formation of conductive filaments. The performances of the devices are maintained for 160 days under ambient conditions. Additionally, the accuracy evaluation of the CsCu2I3-based artificial synapses performs exceptionally well with the MNIST and Fashion MNIST data sets, demonstrating high learning accuracy in deep neural networks. Using the novel B-site engineered halide perovskite material with extreme air stability, this study paves the way for artificial synaptic devices for next-generation in-memory hardware.

CRFL: A novel federated learning scheme of client reputation assessment via local model inversion

Federated learning (FL) is gradually becoming a key learning paradigm in Privacy-preserving Machine Learning (ML) systems. In FL, a large number of clients cooperate with a central server to learn a shared model without sharing their own data sets. However, since there is a great disparity between the client data sets, standard FL is often hard to tune and suffers from performance degradation due to the inharmony among local models. To this end, in this paper we propose a novel FL scheme, termed client reputation federated learning (CRFL), which dynamically assesses the reputation of the clients participating in FL. Our method leverages techniques from model explanation, and aims at precisely measure each client's impact to the global model. To be specific, we first calculate the saliency-weighted variance on pixelwise relevance scores as the quality factor of a single sample. Then we extract activation function values at the last hidden layer to compute the divergence factor of individual data set. Finally, the server integrates these two factors as an assessment of the client reputation. By leveraging such assessment, CRFL can dynamically adjust the weights of the clients in each aggregation round, thus leading to a significant improvement over the baseline method in terms of model accuracy and convergence rate. Intensive experiments are conducted on the MNIST and CIFAR-10 data sets, and experimental results demonstrate the efficacy of the proposed method.

Uncertainty estimation based adversarial attack in multi-class classification

Model uncertainty has gained popularity in machine learning due to the overconfident predictions derived from standard neural networks which are not trustworthy. Recently, Monte-Carlo based adversarial attack (MC-AA) has been proposed as a simple uncertainty estimation method which is powerful in capturing data points that lie in the overlapping distribution of the decision boundary. MC-AA produces uncertainties by performing back-and-forth perturbations of a given data point towards the decision boundary using the idea of adversarial attacks. Despite its efficacy against other uncertainty estimation methods, this method has been only examined on binary classification problems. Thus, we present and examine MC-AA with multi-class classification tasks. We point out the limitation of this method with multiple classes which we tackle by converting multiclass problem into ‘one-versus-all’ classification. We compare MC-AA against other recent model uncertainty methods on Cora – a graph structured dataset – and MNIST – an image dataset. Thus, the conducted experiments are performed using a variety of deep learning algorithms to perform the classification. Consequently, we discuss the best results of model uncertainty with Cora data using LEConv model of AUC-score 0.889 and MNIST data using CNN of AUC-score 0.98 against other uncertainty estimation methods.

Feature autoencoder for detecting adversarial examples

Deep neural networks (DNNs) have gained widespread adoption in computer vision. Unfortunately, state-of-the-art DNNs are vulnerable to adversarial example (AE) attacks, where an adversary introduces imperceptible perturbations to a test example for defrauding DNNs. The obstacles have urged intensive research on improving the DNN robustness via adversarial training, that is, the clean data set is blended with adversarial examples to carry out training. However, the adversarial example attack technologies are open-ended, and the adversarial training is insufficient to focus on improving robustness performance. To circumvent this limitation, we mitigate adversarial example attacks from another perspective, which aims at detecting adversarial examples. Feature autoencoder detector (FADetector), a novel defense framework that exploits feature knowledge is proposed. One of the hallmarks of FADetector is to not involve adversarial examples to train the detector. Our extensive evaluation on MNIST and CIFAR-10 data sets demonstrates that our defense outperforms the conventional autoencoder detectors in terms of detection accuracy.

Distributional barycenter problem through data-driven flows

A new method is proposed for the solution of the data-driven optimal transport barycenter problem and of the more general distributional barycenter problem that the article introduces. The distributional barycenter problem provides a conceptual and computational toolbox for central problems in pattern recognition, such as the simulation of conditional distributions, the construction of a representative for a family of distributions indexed by a covariate and a new class of data-based generative models. The method proposed improves on previous approaches based on adversarial games, by slaving the discriminator to the generator and minimizing the need for parameterizations. It applies not only to a discrete family of distributions, but to more general distributions conditioned to factors z of any cardinality and type. The methodology is applied to numerical examples, including an analysis of the MNIST data set with a new cost function that penalizes non-isometric maps.

Federated Learning With Soft Clustering

In this article, we consider the problem of federated learning (FL) with training data that are non independent and identically distributed (non-IID) across the clients. To cope with data heterogeneity, an iterative federated clustering algorithm (IFCA) has been proposed. IFCA partitions the clients into a number of clusters and lets the clients in the same cluster optimize a shared model. However, in IFCA, the clusters are nonoverlapping, which leads to an inefficient utilization of the local information since the knowledge of a client is used by only one cluster during each round. To capture the complex nature of real-world data, soft clustering methods with overlapping clusters have been proposed that attain superior performance over the hard ones. Motivated by this, we propose a new algorithm named FL with soft clustering (FLSC) by combining the strengths of soft clustering and IFCA, where the clients are partitioned into overlapping clusters and the information of each participating client is used by multiple clusters simultaneously during each round. The experimental results show that FLSC achieves better learning performance on the classification tasks on the MNIST and Fashion-MNIST data sets, compared with the state-of-the-art baseline methods, i.e., the global model method and IFCA.

Using brain inspired principles to unsupervisedly learn good representations for visual pattern recognition

Although deep learning has solved difficult problems in visual pattern recognition, it is mostly successful in tasks where there are lots of labeled training data available. Furthermore, the global back-propagation based training rule and the amount of employed layers represents a departure from biological inspiration. The brain is able to perform most of these tasks in a very general way from limited to no labeled data. For these reasons it is still a key research question to look into computational principles in the brain that can help guide models to unsupervisedly learn good representations which can then be used to perform tasks like classification. To that end, we start by recalling four key brain-inspired principles that relate to simple vision: modeling ”whats” and ”wheres” separately; including a time component; context dependency; and layer-wise learning. Then, we take these principles and use them to convey an a priori structure to our model that makes the learning problem easier. With that, our model is able to generate such high quality representations for the MNIST data set. We compare the obtained results with similar recent works and verify extremely competitive results.

An automatic learning rate decay strategy for stochastic gradient descent optimization methods in neural networks

Stochastic Gradient Descent (SGD) series optimization methods play the vital role in training neural networks, attracting growing attention in science and engineering fields of the intelligent system. The choice of learning rates affects the convergence rate of SGD series optimization methods. Currently, learning rate adjustment strategies mainly face the following problems: (1) The traditional learning rate decay method mainly adopts manual manner during training iterations, the small learning rate produced from which causes slow convergence in training neural networks. (2) Adaptive method (e.g., Adam) has poor generalization performance. To alleviate the above issues, we propose a novel automatic learning rate decay strategy for SGD optimization methods in neural networks. On the basis of the observation that the convergence rate's upper bound enjoys minimization in a specific iteration concerning the current learning rate, we first present the expression of the current learning rate determined by historical learning rates. And merely one extra parameter is initialized to generate automatic decreasing learning rates during the training process. Our proposed approach is applied to SGD and Momentum SGD optimization algorithms, and concrete theoretical proof explains its convergence. Numerical simulations are conducted on the MNIST and Cifar-10 data sets with different neural networks. Experimental results show that our algorithm outperforms existing classical ones, achieving faster convergence rate, better stability, and generalization performance in neural network training. It also lays a foundation for large-scale parallel search of initial parameters in intelligent systems.

Potential Flow Generator With L2 Optimal Transport Regularity for Generative Models.

We propose a potential flow generator with L2 optimal transport regularity, which can be easily integrated into a wide range of generative models, including different versions of generative adversarial networks (GANs) and normalizing flow models. With only a slight augmentation to the original generator loss functions, our generator not only tries to transport the input distribution to the target one but also aims to find the one with minimum L2 transport cost. We show the effectiveness of our method in several 2-D problems and illustrate the concept of "proximity" due to the L2 optimal transport regularity. Subsequently, we demonstrate the effectiveness of the potential flow generator in image translation tasks with unpaired training data from the MNIST data set and the CelebA data set with a comparison against vanilla Wasserstein GAN with gradient penalty (WGAN-GP) and CycleGAN.

Toward a Brain-Inspired Developmental Neural Network Based on Dendritic Spine Dynamics.

Neural networks with a large number of parameters are prone to overfitting problems when trained on a relatively small training set. Introducing weight penalties of regularization is a promising technique for solving this problem. Taking inspiration from the dynamic plasticity of dendritic spines, which plays an important role in the maintenance of memory, this letter proposes a brain-inspired developmental neural network based on dendritic spine dynamics (BDNN-dsd). The dynamic structure changes of dendritic spines include appearing, enlarging, shrinking, and disappearing. Such spine plasticity depends on synaptic activity and can be modulated by experiences-in particular, long-lasting synaptic enhancement/suppression (LTP/LTD), coupled with synapse formation (or enlargement)/elimination (or shrinkage), respectively. Subsequently, spine density characterizes an approximate estimate of the total number of synapses between neurons. Motivated by this, we constrain the weight to a tunable bound that can be adaptively modulated based on synaptic activity. Dynamic weight bound could limit the relatively redundant synapses and facilitate the contributing synapses. Extensive experiments demonstrate the effectiveness of our method on classification tasks of different complexity with the MNIST, Fashion MNIST, and CIFAR-10 data sets. Furthermore, compared to dropout and L2 regularization algorithms, our method can improve the network convergence rate and classification performance even for a compact network.

CHE: Channel-Wise Homomorphic Encryption for Ciphertext Inference in Convolutional Neural Network

Privacy-preserving deep learning (PPDL), which leverages the Homomorphic Encryption (HE), has attracted attention as a promising approach to ensure the privacy of deep learning applications’ data. While recent studies have developed and evaluated the HE-based PPDL algorithms, the achieved performances, such as accuracy and latency, need improvement to make the applications practical. This work aims to improve the performance of the image classification of HE-based PPDL by combining two approaches—Channel-wise Homomorphic Encryption (CHE) and Batch Normalization (BN) with coefficient merging. Although these are commonly used schemes, their detailed algorithms and formulations have not been clearly described. The main contribution of current study is to provide complete and reproducible descriptions of these schemes. We evaluate our CHE and BN implementation by targeting the Cheon-Kim-Kim-Song scheme as an HE scheme and Convolution Neural Network (CNN) as a machine learning scheme while using the MNIST and CIFAR-10 as the datasets. In addition, we compare the results with the five state-of-the-art neural network architectures. Our experiments demonstrate that the CHE can serve as a tool for empirically achieving shorter latency (the shortest 7.76 seconds) and higher accuracy (the highest 99.32%) compared with the previous studies that aimed to establish the classification of the encrypted MNIST data with CNN. Our approach can aid in designing a more robust and flexible PPDL.

Data Disclosure With Non-Zero Leakage and Non-Invertible Leakage Matrix

We study a statistical signal processing privacy problem, where an agent observes useful data <inline-formula> <tex-math notation="LaTeX">$Y$ </tex-math></inline-formula> and wants to reveal the information to a user. Since the useful data is correlated with the private data <inline-formula> <tex-math notation="LaTeX">$X$ </tex-math></inline-formula>, the agent employs a privacy mechanism to generate data <inline-formula> <tex-math notation="LaTeX">$U$ </tex-math></inline-formula> that can be released. We study the privacy mechanism design that maximizes the revealed information about <inline-formula> <tex-math notation="LaTeX">$Y$ </tex-math></inline-formula> while satisfying a strong <inline-formula> <tex-math notation="LaTeX">$\ell _{1}$ </tex-math></inline-formula>-privacy criterion. When a sufficiently small leakage is allowed, we show that the optimizer distributions of the privacy mechanism design problem have a specific geometry, i.e., they are perturbations of fixed vector distributions. This geometrical structure allows us to use a local approximation of the conditional entropy. By using this approximation the original optimization problem can be reduced to a linear program so that an approximate solution for the optimal privacy mechanism can be easily obtained. The main contribution of this work is to consider a non-invertible leakage matrix with non-zero leakage. In our first example, inspired by a watermark application, we first demonstrate the accuracy of the approximation. Then, we employ different measures for utility and privacy leakage to compare the privacy-utility trade-off using our approach with other methods. In particular, we show that by allowing small leakage, significant utility can be achieved using our method compared to the case where no leakage is allowed. In the second and third examples which are based on the MNIST data set and medical applications, we illustrate the suggested design for disclosed data <inline-formula> <tex-math notation="LaTeX">$U$ </tex-math></inline-formula>. It has been shown that the letters of <inline-formula> <tex-math notation="LaTeX">$Y$ </tex-math></inline-formula> which are disclosing more information about <inline-formula> <tex-math notation="LaTeX">$X$ </tex-math></inline-formula> are combined (randomized) to produce a new letter of <inline-formula> <tex-math notation="LaTeX">$U$ </tex-math></inline-formula>.

RELU-Function and Derived Function Review

The activation function plays an important role in training and improving performance in deep neural networks (dnn). The rectified linear unit (relu) function provides the necessary non-linear properties in the deep neural network (dnn). However, few papers sort out and compare various relu activation functions. Most of the paper focuses on the efficiency and accuracy of certain activation functions used by the model, but does not pay attention to the nature and differences of these activation functions. Therefore, this paper attempts to organize the RELU-function and derived function in this paper. And compared the accuracy of different relu functions (and its derivative functions) under the Mnist data set. From the experimental point of view, the relu function performs the best, and the selu and elu functions perform poorly.