Fashion-MNIST Datasets Research Articles

A Lightweight Binarized Convolutional Neural Network Model for Small Memory and Low-Cost Mobile Devices

In recent years, the high cost of implementing deep neural networks due to their large model size and parameter complexity has made it a challenging problem to design lightweight models that reduce application costs. The existing binarized neural networks suffer from both the large memory occupancy and the big number of trainable params they use. We propose a lightweight binarized convolutional neural network (CBCNN) model to address the multiclass classification/identification problem. We use both binary weights and activation. We show experimentally that a model using only 0.59 M trainable params is sufficient to reach about 92.94% accuracy on the GTSRB dataset, and it has similar performances compared to other methods on MNIST and Fashion-MNIST datasets. Accordingly, most arithmetic operations with bitwise operations are simplified, thus both used memory size and memory accesses are reduced by 32 times. Moreover, the color information was removed, which also reduced drastically the training time. All these together allow our architecture to run effectively and in real time on simple CPUs (rather than GPUs). Through the results we obtained, we show that despite simplifications and color information removal, our network achieves similar performances compared to classical CNNs with lower costs in both in training and embedded deployment.

MP-CLF: An effective Model-Preserving Collaborative deep Learning Framework for mitigating data leakage under the GAN

The development of Internet of Things (IoT) communication technology has accelerated the data transmission between IoT devices, thus facilitating collaborative data processing based on the cloud, such as collaborative deep learning. The collaborative deep learning framework allows local devices to cooperate on training models without sharing private data, which resolves the contradiction of the availability and privacy of data. However, the emergence of the Generative Adversarial Network (GAN) attack has shown that poorly protected local data is vulnerable to being learned by adversaries. In this paper, we aim to address the threat GAN attacks pose to collaborative deep learning. We propose a Model-Preserving Collaborative deep Learning Framework, called MP-CLF, which can effectively resist the GAN attack. Based on fully connected neural network learning, MP-CLF employs a matrix blinding technology to break the local modeling of the GAN attack by blinding specific model parameters and trainers’ data, which is easily implementable and has strong security. Besides, MP-CLF builds a user partition model pre-training to improve training quality and strengthen model protection. Using the MNIST dataset and Fashion-MNIST dataset, we experimentally demonstrate that MP-CLF can completely resist the GAN attack with good computational efficiency.

CHARLES: A C++ fixed-point library for Photonic-Aware Neural Networks

In this paper we present CHARLES (C++ pHotonic Aware neuRaL nEtworkS), a C++ library aimed at providing a flexible tool to simulate the behavior of Photonic-Aware Neural Network (PANN). PANNs are neural network architectures aware of the constraints due to the underlying photonic hardware, mostly in terms of low equivalent precision of the computations. For this reason, CHARLES exploits fixed-point computations for inference, while it supports both floating-point and fixed-point numerical formats for training. In this way, we can compare the effects due to the quantization in the inference phase when the training phase is performed on a classical floating-point model and on a model exploiting high-precision fixed-point numbers.To validate CHARLES and identify the most suited numerical format for PANN training, we report the simulation results obtained considering three datasets: Iris, MNIST, and Fashion-MNIST. Fixed-training is shown to outperform floating-training when executing inference on bitwidths suitable for photonic implementation. Indeed, performing the training phase in the floating-point domain and then quantizing to lower bitwidths results in a very high accuracy loss. Instead, when fixed-point numbers are exploited in the training phase, the accuracy loss due to quantization to lower bitwidths is significantly reduced.In particular, we show that for Iris dataset, fixed-training achieves a performance similar to floating-training. Fixed-training allows to obtain an accuracy of 90.4% and 68.1% with the MNIST and Fashion-MNIST datasets using only 6 bits, while the floating-training reaches an accuracy of just 25.4% and 50.0% when exploiting the same bitwidths.

Multi-Layered Projected Entangled Pair States for Image Classification

Tensor networks have been recognized as a powerful numerical tool; they are applied in various fields, including physics, computer science, and more. The idea of a tensor network originates from quantum physics as an efficient representation of quantum many-body states and their operations. Matrix product states (MPS) form one of the simplest tensor networks and have been applied to machine learning for image classification. However, MPS has certain limitations when processing two-dimensional images, meaning that it is preferable for an projected entangled pair states (PEPS) tensor network with a similar structure to the image to be introduced into machine learning. PEPS tensor networks are significantly superior to other tensor networks on the image classification task. Based on a PEPS tensor network, this paper constructs a multi-layered PEPS (MLPEPS) tensor network model for image classification. PEPS is used to extract features layer by layer from the image mapped to the Hilbert space, which fully utilizes the correlation between pixels while retaining the global structural information of the image. When performing classification tasks on the Fashion-MNIST dataset, MLPEPS achieves a classification accuracy of 90.44%, exceeding tensor network models such as the original PEPS. On the COVID-19 radiography dataset, MLPEPS has a test set accuracy of 91.63%, which is very close to the results of GoogLeNet. Under the same experimental conditions, the learning ability of MLPEPS is already close to that of existing neural networks while having fewer parameters. MLPEPS can be used to build different network models by modifying the structure, and as such it has great potential in machine learning.

Chaotic Whale Optimization Algorithm in Hyperparameter Selection in Convolutional Neural Network Algorithm

In several previous studies, metaheuristic methods were used to search for CNN hyperparameters. However, this research only focuses on searching for CNN hyperparameters in the type of network architecture, network structure, and initializing network weights. Therefore, in this article, we only focus on searching for CNN hyperparameters with network architecture type, and network structure with additional regularization. In this article, the CNN hyperparameter search with regularization uses CWOA on the MNIST and FashionMNIST datasets. Each dataset consists of 60,000 training data and 10,000 testing data. Then during the research, the training data was only taken 50% of the total data, then the data was divided again by 10% for data validation and the rest for training data. The results of the research on the MNIST CWOA dataset have an error value of 0.023 and an accuracy of 99.63. Then the FashionMNIST CWOA dataset has an error value of 0.23 and an accuracy of 91.36.

Convolutional networks with short-term memory effects

The scale expansion and depth increase of convolutional neural network lead to the increase in the number of devices and chip size, which makes the hardware implementation of neural network more and more difficult. To solve this problem, we design a convolutional neural network with short-term memory to save the number of devices and reduce the complexity of the hardware implementation. Memristor, as a natural nanoscale synapse, is one of the important devices to realize convolutional neural network chip. However, most current memristor-based neural networks use non-volatile memristors. Non-volatile memristors can only store long-term weights because of the stability of resistance, but volatile memristor can store short-term and long-term weights due to its forgetting property. This paper designs a LeNet-5 convolutional neural network by using forgetting memristor bridges composed of volatile memristors. Due to the short-term memory of volatile memristors, 49% of the number of memristors can be effectively saved compared to the neural networks using non-volatile memristors. By writing different weights for the forgetting memristor bridges, a recognition accuracy of about 97% is achieved on the MNIST dataset, and a good recognition accuracy is still achieved at 40,000 images if a recovery signal is given. What is more, in the FASHION-MNIST and ORL datasets, we also achieve 85% and 91% recognition accuracy respectively with the same network.

A Federated Learning Framework against Data Poisoning Attacks on the Basis of the Genetic Algorithm

Obtaining the balance between information loss and training accuracy is crucial in federated learning. Nevertheless, inadequate data quality will affect training accuracy. Here, to improve the training accuracy without affecting information loss, we propose a malicious data detection model using the genetic algorithm to resist model poisoning attack. Specifically, the model consists of three modules: (1) Participants conduct single point training on data and upload accuracy to the third-party server; (2) Formulate data scoring formula based on data quantity and quality; (3) Use the genetic algorithm to obtain the threshold which makes the score highest. Data with accuracy which exceeds this threshold can participate in cooperative training of federated learning. Before participating in training, participants’ data is optimized to oppose data poisoning attacks. Experiments on two datasets validated the effectiveness of the proposed model. It was also verified in the fashion-MNIST data set and cifar10 data set that the training accuracy of GAFL is 7.45% higher than that of the federated learning model in the fashion-MNIST data set and 8.18% in the cifar10 data set.

Variational quantum one-class classifier

One-class classification (OCC) is a fundamental problem in pattern recognition with a wide range of applications. This work presents a semi-supervised quantum machine learning algorithm for such a problem, which we call a variational quantum one-class classifier (VQOCC). The algorithm is suitable for noisy intermediate-scale quantum computing because the VQOCC trains a fully-parameterized quantum autoencoder with a normal dataset and does not require decoding. The performance of the VQOCC is compared with that of the one-class support vector machine (OC-SVM), the kernel principal component analysis (PCA), and the deep convolutional autoencoder (DCAE) using handwritten digit and Fashion-MNIST datasets. The numerical experiment examined various structures of VQOCC by varying data encoding, the number of parameterized quantum circuit layers, and the size of the latent feature space. The benchmark shows that the classification performance of VQOCC is comparable to that of OC-SVM and PCA, although the number of model parameters grows only logarithmically with the data size. The quantum algorithm outperformed DCAE in most cases under similar training conditions. Therefore, our algorithm constitutes an extremely compact and effective machine learning model for OCC.

A Novel Method for Fashion Clothing Image Classification Based on Deep Learning

Image recognition and classification is a significant research topic in computational vision and widely used computer technology. Themethods often used in image classification and recognition tasks are based on deep learning, like Convolutional Neural Networks(CNNs), LeNet, and Long Short-Term Memory networks (LSTM). Unfortunately, the classification accuracy of these methods isunsatisfactory. In recent years, using large-scale deep learning networks to achieve image recognition and classification canimprove classification accuracy, such as VGG16 and Residual Network (ResNet). However, due to the deep network hierarchyand complex parameter settings, these models take more time in the training phase, especially when the sample number is small, which can easily lead to overfitting. This paper suggested a deep learning-based image classification technique based on a CNN model and improved convolutional and pooling layers. Furthermore, the study adopted the approximate dynamic learning rate update algorithm in the model training to realize the learning rate’s self-adaptation, ensure the model’s rapid convergence, and shorten the training time. Using the proposed model, an experiment was conducted on the Fashion-MNIST dataset, taking 6,000 images as the training dataset and 1,000 images as the testing dataset. In actual experiments, the classification accuracy of the suggested method was 93 percent, 4.6 percent higher than that of the basic CNN model. Simultaneously, the study compared the influence of the batch size of model training on classification accuracy. Experimental outcomes showed this model is very generalized in fashion clothing image classification tasks.

Demonstration of Synaptic Behavior in a Heavy-Metal-Ferromagnetic-Metal-Oxide-Heterostructure-Based Spintronic Device for On-Chip Learning in Crossbar-Array-Based Neural Networks

Nanomagnetic and spintronic devices, which make use of physical phenomena in materials and interfaces like perpendicular magnetic anisotropy (PMA) and spin–orbit torque (SOT) to exhibit multiple electrically readable and controllable states, have been widely considered as synaptic elements in analog crossbar arrays for on-chip learning of analog neural networks (ANN). Here, in such a heavy-metal-ferromagnetic-metal-oxide-heterostructure-based (Pt/Co/SiO2) spintronic device, multiple mixed states are first demonstrated experimentally. These mixed states correspond to different magnetization configurations between two saturated states: all magnetic moments vertically up and all moments vertically down (the ferromagnetic layer exhibits PMA). These mixed states are then modulated through in-plane-current pulses, which result in SOT at the heavy-metal–ferromagnet interface, and thus long-term potentiation (LTP) and long-term depression (LTD) are demonstrated in the device (synaptic behavior). The experimentally obtained LTP-and-LTD behavior is explained qualitatively through micromagnetic simulations, which model the interface phenomena phenomenologically. The synaptic bit resolution is then determined experimentally to be 5 (≈ 30 distinguishable states) by measuring the stability of the mixed states. The nonlinearity and asymmetry in the obtained LTP and LTD are quantified experimentally. Next, a crossbar-array-based ANN is simulated using spintronic-synapse-device models based on the experimentally obtained LTP and LTD, and reasonably high classification accuracy is predicted for MNIST and Fashion-MNIST data sets, despite the synaptic nonidealities like nonlinearity, asymmetry, and cycle-to-cycle variations. The impact of nonlinearity and asymmetry on classification accuracy is found to be much higher than that due to limited bit resolution (we do not go below 10 bits per synapse cell though) and cycle-to-cycle variations.

An optimized optical diffractive deep neural network with OReLU function based on genetic algorithm

The diffractive deep neural network (D2NN) has demonstrated advantages in artificial intelligence image processing tasks, such as classification and lens-free imaging. Due to the lack of nonlinear layers, the nonlinear representation and generalization capability of D2NN needs to be improved. To solve this problem, we introduce an Optical Rectified Linear Units (OReLU) function utilizing beam splitting, sensing, and modulation of optoelectronic devices. We experimentally verified the functionality of the proposed OReLU and analyzed the impact of the actual error on the whole network. The simulation result shows that such a nonlinear layer increases the sparsity of the network, which reduces the interdependence between parameters and also improves the classification accuracy of the network. Since the model's convergence speed and classification accuracy are influenced by the truncation threshold associated with the nonlinear function, we also optimize the threshold factor to achieve the highest classification accuracy of 97.97% and 87.85% on the MNIST and Fashion-MNIST datasets using the genetic algorithm.

Network Pathway Extraction Focusing on Object Level

In this paper, I propose an efficient method of identifying important neurons that are related to an object’s concepts by mainly considering the relationship between these neurons and their object concept or class. I first quantify the activation values among neurons, based on which histograms of each neuron are generated. Then, the obtained histograms are clustered to identify the neurons’ importance. A network-wide holistic approach is also introduced to efficiently identify important neurons and their influential connections to reveal the pathway of a given class. The influential connections as well as their important neurons are carefully evaluated to reveal the sub-network of each object’s concepts. The experimental results on the MNIST and Fashion MNIST datasets show the effectiveness of the proposed method.

A Supervised Learning Algorithm for Multilayer Spiking Neural Networks Based on Temporal Coding Toward Energy-Efficient VLSI Processor Design.

Spiking neural networks (SNNs) are brain-inspired mathematical models with the ability to process information in the form of spikes. SNNs are expected to provide not only new machine-learning algorithms but also energy-efficient computational models when implemented in very-large-scale integration (VLSI) circuits. In this article, we propose a novel supervised learning algorithm for SNNs based on temporal coding. A spiking neuron in this algorithm is designed to facilitate analog VLSI implementations with analog resistive memory, by which ultrahigh energy efficiency can be achieved. We also propose several techniques to improve the performance on recognition tasks and show that the classification accuracy of the proposed algorithm is as high as that of the state-of-the-art temporal coding SNN algorithms on the MNIST and Fashion-MNIST datasets. Finally, we discuss the robustness of the proposed SNNs against variations that arise from the device manufacturing process and are unavoidable in analog VLSI implementation. We also propose a technique to suppress the effects of variations in the manufacturing process on the recognition performance.

Federated Learning Resource Optimization and Client Selection for Total Energy Minimization Under Outage, Latency, and Bandwidth Constraints With Partial or No CSI

We consider the problem of minimizing the total energy consumption due to the computation and communication tasks of federated learning (FL) under bandwidth and latency constraints. To avoid channel state information (CSI) feedback to the transmitter, we adopt outage probability as an additional constraint in the energy minimization problem. First, we define a feasibility metric based on the system design parameters to exclude slow clients (stragglers). Then, we propose a novel client selection algorithm, after excluding stragglers, based on dividing the remaining clients into clusters, where clients within the same cluster collaborate in a communication round to train their local models. For each communication round, one client cluster is selected in a round-robin fashion. Furthermore, we formulate and solve a resource allocation problem to optimize the transmit power, clock frequency, allocated bandwidth, and communication latency of clients within each cluster to minimize the total energy subject to total bandwidth, latency, and outage constraints. Moreover, we extend our FL design framework to the case of no CSI at both the client and server ends using differential transmission to eliminate CSI estimation pilot overhead and complexity at comparable total energy consumption and learning accuracy to coherent transmission. We test our proposed algorithms over MNIST and Fashion-MNIST datasets in iid and non-iid settings. Our proposed client selection algorithm reduces the number of participating clients per communication round by 41% compared to the baseline while maintaining the same learning accuracy. Moreover, our results demonstrate that increasing the number of receive antennas at the server from one to four can reduce the number of communication rounds required to reach a predetermined testing accuracy level by up to 53% for the Fashion-MNIST dataset.

Construction method of hybrid quantum long-short term memory neural network for image classification

Long-short term memory (LSTM) neural network solves the problems of long-term dependence, gradient disappearance and gradient explosion by introducing memory units, and is widely used in time series analysis and prediction. Combining quantum computing with LSTM neural network will help to improve its computational efficiency and reduce the number of model parameters, thus significantly improving the performance of traditional LSTM neural network. This paper proposes a hybrid quantum LSTM (hybrid quantum long-short term memory, HQLSTM) network model that can be used to realize the image classification. It uses variable quantum circuits to replace the nerve cells in the classical LSTM network to realize the memory function of the quantum network. At the same time, it introduces Choquet integral operator to enhance the degree of aggregation between data. The memory cells in the HQLSTM network are composed of multiple variation quantum circuits (VQC) that can realize different functions. Each VQC consists of three parts: the coding layer, which uses angle coding to reduce the complexity of network model design; the variation layer, which is designed with quantum natural gradient optimization algorithm, so that the gradient descent direction does not target specific parameters, thereby optimizing the parameter update process and improving the generalization and convergence speed of the network model; the measurement layer, which uses the Pauli Z gate to measure, and the expected value of the measurement result is input to the next layer to extract useful information from the quantum circuit. The experimental results on the MNIST, FASHION-MNIST and CIFAR datasets show that the HQLSTM model achieves higher image classification accuracy and lower loss value than the classical LSTM model and quantum LSTM model. At the same time, the network space complexity of HQLSTM and quantum LSTM are significantly reduced compared with the classical LSTM network.

Boosting Accuracy of Differentially Private Federated Learning in Industrial IoT With Sparse Responses

Empowered by 5G, it has been extensively explored by existing works on the deployment of differentially private federated learning (DPFL) in the Industrial Internet of Things (IIoT). Through federated learning, decentralized IIoT devices can collaboratively train a machine learning model by merely exchanging model gradients with a parameter server (PS) for multiple global iterations. Differentially private (DP) mechanisms will be incorporated by IIoT devices (also called clients) to prevent the leakage of privacy due to the exposure of gradients because original gradients will be distorted DP noises. Yet, learning with distorted gradients can seriously deteriorate model accuracy, making DPFL unusable in reality. To address this problem, we propose a novel DPFL with sparse responses (DPFL-SR) algorithm, which applies the sparse vector technique to reduce the privacy budget consumption in each global iteration. Specifically, DPFL-SR evaluates the value of each gradient, and only distorts and uploads significant gradients to the PS because significant gradients are more essential for model training. Since insignificant gradients are not disclosed, the reserved privacy budget can be used to return significant gradients for more iterations so that DPFL-SR can achieve higher model accuracy without lowering the privacy protection level. Extensive experiments are conducted with the MNIST and Fashion-MNIST datasets to demonstrate the practicability and superiority of DPFL-SR in IIoT systems.

Quantum phase recognition using quantum tensor networks

Machine learning (ML) has recently facilitated many advances in solving problems related to many-body physical systems. Given the intrinsic quantum nature of these problems, it is natural to speculate that quantum-enhanced machine learning will enable us to unveil even greater details than we currently have. With this motivation, this paper examines a quantum machine learning approach based on shallow variational ansatz inspired by tensor networks for supervised learning tasks. In particular, we first look at the standard image classification tasks using the Fashion-MNIST dataset and study the effect of repeating tensor network layers on ansatz’s expressibility and performance. Finally, we use this strategy to tackle the problem of quantum phase recognition for the transverse-field Ising and Heisenberg spin models in one and two dimensions, where we were able to reach \(\ge 98\%\) test-set accuracies with both multi-scale entanglement renormalization ansatz (MERA) and tree tensor network (TTN) inspired parametrized quantum circuits.

ИССЛЕДОВАНИЕ КВАНТОВОЙ ВЫЧИСЛИТЕЛЬНОЙ СИСТЕМЫ И РЕАЛИЗАЦИЯ КВАНТОВОГО ЯДРА НА ПЛИС

The quantum core method is one of the most important methods in quantum machine learning.However, the number of features used for quantum nuclei is limited to a few dozen features.The block product state structure is used as a quantum feature map and the implementation ofprogrammable gate matrices is demonstrated. The relevance of these studies lies in the mathematicaland software modeling and implementation of a quantum computing system as part of thedevelopment of the implementation of a quantum core on FPGA for solving classes of problems ofa classical nature. The scientific novelty of this research area is the development of a hybrid simulatorof the quantum cores of a central processing unit (CPU) and a programmable logic integratedcircuit (FPGA) several orders of magnitude faster than a conventional quantum computingsimulator. This joint development of the implemented quantum core and its efficient FPGA implementation allowed numerical simulation of the quantum core based on gates in terms of inputfeatures, up to 780-dimensional features using 4000 samples. We applied the quantum kernel toimage classification problems using the Fashion-MNIST dataset and showed that the quantumkernel is comparable to Gaussian kernels with optimized throughput. The analysis of the work inthis field has shown that a new qualitative level has now been reached, opening up promising opportunitiesfor the implementation of multi-qubit quantum computing. The prospects for implementationand development are connected not only with technological capabilities, but also with solvingthe issues of building effective quantum systems for solving actual mathematical problems,cryptography problems and control (optimization) problems.

Implementation of CNN and ANN for Fashion-MNIST-Dataset using Different Optimizers

Objectives: The paper presents application of convolution neural network and artificial neural network for image classification problem for clothing dataset along with their performance comparison against different optimizers. The major objective of this paper is to perform image classification on fashion-mnist clothing dataset images. Methods: The methods used here are, the traditional ANN and CNN. Here image classification is performed on Fashion-mnist, clothing dataset using CNN and ANN with different optimizers. The performance of the working of ANN and CNN in classifying images from fashion-mnist dataset is compared against different optimizers namely stochastic gradient Descent, Adagrad, RMS prop and Adam optimizer. Findings: The study found that CNN worked better than ANN yielding training accuracy of 95%, 93% and testing accuracy of 91%, 89% when used with Adam and RmsProp respectively. Novelty: The novelty of this work is to present a comparative study of image classification using CNN, ANN using different optimizers, since not many studies or research articles showed the performance comparison of traditional and convolution neural networks in image classification along with different optimizers. Since the real-world scenarios of today require enormous data to be processed, CNN can fit well to diversify applications since they highly reduce the number of parameters to be trained that speeds up the training process. Moreover, to be specific on image classification problems they require the best and most prominent features to be detected and uncovered; this can be achieved using CNN since it has the concept of convolution using filters at its Core. Hence, CNN is highly recommended for such image classification applications than the traditional artificial-neural-networks because of the aforementioned reasons. Keywords: ANN (Artificial NeuralNetworks); CNN (ConvolutionNeural Networks); Optimization; SGD (Stochastic Gradient Descent); RmsProp (Root mean Square propagation); Adam (Adaptive moment estimation); AdaGrad (Adaptive Gradient)

Improving Network Training on Resource-Constrained Devices via Habituation Normalization

As a technique for accelerating and stabilizing training, the batch normalization (BN) is widely used in deep learning. However, BN cannot effectively estimate the mean and the variance of samples when training/fine-tuning with small batches of data on resource-constrained devices. It will lead to a decrease in the accuracy of the deep learning model. In the fruit fly olfactory system, the algorithm based on the "negative image" habituation model can filter redundant information and improve numerical stability. Inspired by the circuit mechanism, we propose a novel normalization method, the habituation normalization (HN). HN first eliminates the "negative image" obtained by habituation and then calculates the statistics for normalizing. It solves the problem of accuracy degradation of BN when the batch size is small. The experiment results show that HN can speed up neural network training and improve the model accuracy on vanilla LeNet-5, VGG16, and ResNet-50 in the Fashion MNIST and CIFAR10 datasets. Compared with four standard normalization methods, HN keeps stable and high accuracy in different batch sizes, which shows that HN has strong robustness. Finally, the applying HN to the deep learning-based EEG signal application system indicates that HN is suitable for the network fine-tuning and neural network applications under limited computing power and memory.