Fashion-MNIST Datasets Research Articles

A multi-objective approach for communication reduction in federated learning under devices heterogeneity constraints

Federated learning is a paradigm that proposes protecting data privacy by sharing local models instead of raw data during each iteration of model training. However, these models can be large, with many parameters, provoking a substantial communication cost and having a notable environmental impact. Reducing communication overhead is paramount but conflictual to maintaining the model’s accuracy. Most research has dealt with the different factors influencing communication reduction separately without addressing their correlations. Moreover, most of them do not consider the heterogeneity of clients’ hardware. Finding the optimal configuration to fulfil all these training aspects can become intractable for classical techniques. This work explores the add-in that multi-objective evolutionary algorithms can provide for solving the communication overhead problem while achieving high accuracy. We do this by 1) realistically modelling and formulating this task as a multi-objective problem by considering the devices’ heterogeneity, 2) including all the communication-triggering aspects, and 3) applying a multi-objective evolutionary algorithm with an intensification operator to solve the problem. A simulated client–server architecture of four devices with four different processing speeds is studied. Both fully connected and convolutional neural network models are investigated with 33,400 and 887,530 weights, respectively. The experiments are performed using the MNIST and Fashion-MNIST datasets. A comparison is made between three approaches using an extensive set of metrics. Results prove that our approach obtains solutions with better accuracy than the full-communication setting and other methods while getting reductions in communications by around 1,000 times in most cases and up to 10,000 times in some cases compared to the maximum communication setting.

The experiment of federated learning algorithm

As technology advances, concerns regarding data privacy and security have become prominent challenges in machine learning applications. Nevertheless, the introduction of federated learning technology has effectively tackled this concern by concurrently enhancing model performance and preserving data privacy, thereby presenting a more secure and efficient solution within our digital realm. Utilizing discussions about the background of federated learning technology, coupled with pertinent algorithmic procedures and logic, this paper proficiently implements the FedMA algorithm. Additionally, the study performs a comparative analysis of the accuracy and efficiency of the FedMA, FedAvg, FedDyn, and MOON algorithms utilizing the Fashion-MNIST dataset. Moreover, the investigation not only optimizes parameter tuning for the MOON algorithm but also extends experiments to the CIFAR-10 and AGNews datasets, thereby providing additional comparisons of performance and strengths among various federated learning algorithms. In conclusion, the paper provides a comprehensive summary and outlines potential avenues for future research. These insights enhance the comprehension of federated learning and offer valuable guidance to advance and refine its practical applications.

PMONN: an optical neural network for photonic integrated circuits based on micro-resonator.

We propose an improved optical neural network (ONN) circuit architecture based on conventional micro-resonator ONNs, called the Phase-based Micro-resonator Optical Neural Network (PMONN). PMONN's core architecture features a Convolutions and Batch Normalization (CB) unit, comprising a phase-based (PB) convolutional layer, a Depth-Point-Wise (DPW) convolutional layer, and a reconstructed Batch Normalization (RBN) layer. The PB convolution kernel uses modulable phase shifts of Add-drop MRRs as learnable parameters and their optical transfer function as convolution weights. The DPW convolution kernel amplifies PB convolution weights by learning the amplification factors. To address the internal covariate shift during training, the RBN layer normalizes DPW outputs by reconstructing the BN layer of the electronic neural network, which is then merged with the DPW layer in the test stage. We employ the tunable DAs in the architecture to implement the merged layer. PMONN achieves 99.15% and 91.83% accuracy on MNIST and Fashion-MNIST datasets, respectively. This work presents a method for implementing an optical neural network on the improved architecture based on MRRs and increases the flexibility and reusability of the architecture. PMONN has potential applications as the backbone for future optical object detection neural networks.

Hybrid Quantum Neural Network Image Anti-Noise Classification Model Combined with Error Mitigation

In this study, we present an innovative approach to quantum image classification, specifically designed to mitigate the impact of noise interference. Our proposed method integrates key technologies within a hybrid variational quantum neural network architecture, aiming to enhance image classification performance and bolster robustness in noisy environments. We utilize a convolutional autoencoder (CAE) for feature extraction from classical images, capturing essential characteristics. The image information undergoes transformation into a quantum state through amplitude coding, replacing the coding layer of a traditional quantum neural network (QNN). Within the quantum circuit, a variational quantum neural network optimizes model parameters using parameterized quantum gate operations and classical–quantum hybrid training methods. To enhance the system’s resilience to noise, we introduce a quantum autoencoder for error mitigation. Experiments conducted on FashionMNIST datasets demonstrate the efficacy of our classification model, achieving an accuracy of 92%, and it performs well in noisy environments. Comparative analysis with other quantum algorithms reveals superior performance under noise interference, substantiating the effectiveness of our method in addressing noise challenges in image classification tasks. The results highlight the potential advantages of our proposed quantum image classification model over existing alternatives, particularly in noisy environments.

SHE-MTJ based ReLU-max pooling functions for on-chip training of neural networks

We present a detailed investigation of various routes to optimize the power consumption of the spintronic-based devices for implementing rectified linear activation (ReLU) and max-pooling functions. We examine the influence of various spin Hall effect layers, and their input resistances on the power consumption of the ReLU-max pooling functions, we also access the impact of the thermal stability factor of the free-ferromagnet layer on the power consumption and accuracy of the device. The design for ReLU-max pooling relies on the continuous rotation of magnetization, which is accomplished by applying orthogonal spin current to the free-FM layer. We also demonstrate the non-trivial power-resistance relation, where the power consumption decreases with an increase in SHE resistance. We utilize the hybrid spintronic-CMOS simulation platform that combines Keldysh non-equilibrium Green’s function (NEGF) with Landau-Lifshitz-Gilbert-Slonzewski (LLGS) equations and the HSPICE circuit simulator to evaluate our network. Our design takes 0.343 μW of power for ReLU emulation and 17.86 μW of power for ReLU-max pooling network implementation at a thermal stability factor of 4.58, all while maintaining reliable results. We validate the efficiency of our design by implementing a convolutional neural network that classifies the handwritten-MNIST and fashion-MNIST datasets. This implementation illustrates that the classification accuracies achieved are on par with those attained using the ideal software ReLU-max pooling functions, with an energy consumption of 167.31 pJ per sample.

Time regularization in optimal time variable learning

AbstractRecently, optimal time variable learning in deep neural networks was introduced in Antil et al. In this manuscript we extend the concept by introducing a regularization term that directly relates to the time horizon in discrete dynamical systems. Furthermore, we propose an adaptive pruning approach for Residual Neural Networks (ResNets), which reduces network complexity without compromising expressiveness, while simultaneously decreasing training time. The results are illustrated by applying the proposed concepts to classification tasks on the well known MNIST and Fashion MNIST data sets. Our PyTorch code is available on https://github.com/frederikkoehne/time_variable_learning.

Exploring the Impact of Denoising Autoencoder Architectures on Image Retrieval

In today’s busy world, people are shifting to online shopping for their basic needs like groceries, clothes, etc. As a result, modern applications are incorporating image search as part of their application. However, retrieving accurate fashion images is challenging due to the low inter-class separability. In this paper, we aim to propose and implement three deep learning denoising autoencoder image retrieval models trained on Fashion-MNIST dataset and MNIST dataset. Fashion-MNIST is widely used datasets to evaluate model because of its low inter-class separability. It is difficult to get a good accuracy on Fashion-MNIST dataset. The proposed models train to learn significant features of fashion images and improve the precision of image retrieval. These models are compared based on the Label Ranking Average Precision (LRAP) values, with the third model bagging the highest LRAP value. These models also trained and tested on MNIST dataset, which have similar configuration as Fashion-MNIST dataset but with high inter-class separability. Intriguingly, the second and third models trained on MNIST dataset demonstrate neck-to-neck performance, both achieving comparable accuracy. This choice of dataset ensures that the models’ accuracy is more noticeable and significant. The outcomes of this study validate the efficacy of denoising autoencoders in fashion image retrieval and emphasize their prospective in improving online shopping experiences.

DT-SCNN: dual-threshold spiking convolutional neural network with fewer operations and memory access for edge applications.

The spiking convolutional neural network (SCNN) is a kind of spiking neural network (SNN) with high accuracy for visual tasks and power efficiency on neuromorphic hardware, which is attractive for edge applications. However, it is challenging to implement SCNNs on resource-constrained edge devices because of the large number of convolutional operations and membrane potential (Vm) storage needed. Previous works have focused on timestep reduction, network pruning, and network quantization to realize SCNN implementation on edge devices. However, they overlooked similarities between spiking feature maps (SFmaps), which contain significant redundancy and cause unnecessary computation and storage. This work proposes a dual-threshold spiking convolutional neural network (DT-SCNN) to decrease the number of operations and memory access by utilizing similarities between SFmaps. The DT-SCNN employs dual firing thresholds to derive two similar SFmaps from one Vm map, reducing the number of convolutional operations and decreasing the volume of Vms and convolutional weights by half. We propose a variant spatio-temporal back propagation (STBP) training method with a two-stage strategy to train DT-SCNNs to decrease the inference timestep to 1. The experimental results show that the dual-thresholds mechanism achieves a 50% reduction in operations and data storage for the convolutional layers compared to conventional SCNNs while achieving not more than a 0.4% accuracy loss on the CIFAR10, MNIST, and Fashion MNIST datasets. Due to the lightweight network and single timestep inference, the DT-SCNN has the least number of operations compared to previous works, paving the way for low-latency and power-efficient edge applications.

An adjustable multistage resistance switching behavior of a photoelectric artificial synaptic device with a ferroelectric diode effect for neuromorphic computing.

Neuromorphic computing, which mimics biological neural networks, is widely regarded as the optimal solution for addressing the limitations of traditional von Neumann computing architecture. In this work, an adjustable multistage resistance switching ferroelectric Bi2FeCrO6 diode artificial synaptic device was fabricated using a sol-gel method with a simple process. The device exhibits nonlinearity in its electrical characteristics, demonstrating tunable multistage resistance switching behavior and a strong ferroelectric diode effect through the manipulation of ferroelectric polarization. One of its salient advantages resides in its capacity to dynamically regulate its polarization state in response to an external electric field, thereby facilitating the fine-tuning of synaptic connection strength while maintaining synaptic stability. The device is capable of accurately simulating the fundamental properties of biological synapses, including long/short-term plasticity, paired-pulse facilitation, and spike-timing-dependent plasticity. Additionally, the device exhibits a distinctive photoelectric response and is capable of inducing synaptic plasticity by light signal activation. The utilization of a femtosecond laser for the scrutiny of carrier transport mechanisms imparts profound insights into the intricate dynamics governing the optical memory effect. Furthermore, utilizing a convolutional neural network (CNN) architecture, the recognition accuracy of the MNIST and fashion MNIST datasets was improved to 95.6% and 78%, respectively, through the implementation of improved random adaptive algorithms. These findings present a new opportunity for utilizing Bi2FeCrO6 materials in the development of artificial synapses for neuromorphic computation.

Efficient Image Annotation and Caption System Using Deep Convolutional Neural Networks

This study aims to develop an annotation and image annotation system using the Fashion MNIST dataset, which consists of 70,000 grayscale images of ten clothing categories. The system uses a long short-term memory (LSTM) network to generate captions and a convolutional neural network (CNN) to extract image features. Performance evaluation metrics such as Precision, Recall, F1 score, BLEU score, METEOR score, CIDEr score, and ROUGE-L score are used where the accuracy of each clothing category is calculated to evaluate the performance of the model across different categories. Visual analysis of the generated captions is performed to gain insight into the effectiveness of the model and potential areas for improvement. The results indicate the model's success in classifying clothing items, as evidenced by its high accuracy on the test set. The qualitative study reveals the model's ability to identify different types of clothing by providing relevant captions, where the feature representation layer (normalization) plays a crucial role in transforming the detected features. to a flattened row which is then passed to a fully connected layer to learn the relationships and make final decisions with the output layer using a softmax activation function to assign probabilities to each image class, with the class with the highest probability selected as the predicted image class.

ResNet-Lite: On Improving Image Classification with a Lightweight Network

Deep learning methods, specifically convolutional neural networks (CNNs), have achieved state-of-the-art in various tasks including image classification. However, the computational and memory requirements of advanced CNNs like ResNet-50 pose deployment challenges, particularly in resource-constrained environments. Our study introduces a new lightweight approach, namely ResNet-Lite, for image classification. It combines knowledge distillation and network tuning along with hyperparameter tuning techniques to overcome deployment barriers. ResNet-Lite involves the extracting of knowledge from a pre-trained ResNet-50 network and transferring it to a smaller network, creating a more compact and effective approach. After that, hyperparameter and network tuning have been applied to determine the optimal combination of parameters that maximizes the generalization and performance of the model. Our experimental results demonstrate that ResNet-Lite achieves a significantly reduced model size while maintaining competitive classification performance. Specifically, it outperforms the original ResNet-50 model by 5.40% and by 7.13% in accuracy on the CIFAR-10 and Fashion-MNIST datasets, respectively. In summary, our study provides a practical solution for developing high-performance image classification models, even in resource-constrained environments, contributing to the field of advanced deep learning.

Graph-Based Semi-Supervised Deep Image Clustering With Adaptive Adjacency Matrix.

Image clustering is a research hotspot in machine learning and computer vision. Existing graph-based semi-supervised deep clustering methods suffer from three problems: 1) because clustering uses only high-level features, the detailed information contained in shallow-level features is ignored; 2) most feature extraction networks employ the step odd convolutional kernel, which results in an uneven distribution of receptive field intensity; and 3) because the adjacency matrix is precomputed and fixed, it cannot adapt to changes in the relationship between samples. To solve the above problems, we propose a novel graph-based semi-supervised deep clustering method for image clustering. First, the parity cross-convolutional feature extraction and fusion module is used to extract high-quality image features. Then, the clustering constraint layer is designed to improve the clustering efficiency. And, the output layer is customized to achieve unsupervised regularization training. Finally, the adjacency matrix is inferred by actual network prediction. A graph-based regularization method is adopted for unsupervised training networks. Experimental results show that our method significantly outperforms state-of-the-art methods on USPS, MNIST, street view house numbers (SVHN), and fashion MNIST (FMNIST) datasets in terms of ACC, normalized mutual information (NMI), and ARI.

An encoding framework for binarized images using hyperdimensional computing.

Hyperdimensional Computing (HDC) is a brain-inspired and lightweight machine learning method. It has received significant attention in the literature as a candidate to be applied in the wearable Internet of Things, near-sensor artificial intelligence applications, and on-device processing. HDC is computationally less complex than traditional deep learning algorithms and typically achieves moderate to good classification performance. A key aspect that determines the performance of HDC is encoding the input data to the hyperdimensional (HD) space. This article proposes a novel lightweight approach relying only on native HD arithmetic vector operations to encode binarized images that preserves the similarity of patterns at nearby locations by using point of interest selection and local linear mapping. The method reaches an accuracy of 97.92% on the test set for the MNIST data set and 84.62% for the Fashion-MNIST data set. These results outperform other studies using native HDC with different encoding approaches and are on par with more complex hybrid HDC models and lightweight binarized neural networks. The proposed encoding approach also demonstrates higher robustness to noise and blur compared to the baseline encoding.

LMFRNet: A Lightweight Convolutional Neural Network Model for Image Analysis

Convolutional neural networks (CNNs) have transformed the landscape of image analysis and are widely applied across various fields. With their widespread adoption in fields like medical diagnosis and autonomous driving, CNNs have demonstrated powerful capabilities. Despite their success, existing models face challenges in deploying and operating in resource-constrained environments, limiting their practicality in real-world scenarios. We introduce LMFRNet, a lightweight CNN model. Its innovation resides in a multi-feature block design, effectively reducing both model complexity and computational load. Achieving an exceptional accuracy of 94.6% on the CIFAR-10 dataset, this model showcases remarkable performance while demonstrating parsimonious resource utilization. We further validate the performance of the model on the CIFAR-100, MNIST, and Fashion-MNIST datasets, demonstrating its robustness and generalizability across diverse datasets. Furthermore, we conducted extensive experiments to investigate the influence of critical hyperparameters. These experiments provided valuable insights for effective model training.

WOx channel engineering of Cu-ion-driven synaptic transistor array for low-power neuromorphic computing

The multilevel current states of synaptic devices in artificial neural networks enable next-generation computing to perform cognitive functions in an energy-efficient manner. Moreover, considering large-scale synaptic arrays, multiple states programmed in a low-current regime may be required to achieve low energy consumption, as demonstrated by simple numerical calculations. Thus, we propose a three-terminal Cu-ion-actuated CuOx/HfOx/WO3 synaptic transistor array that exhibits analogously modulated channel current states in the range of tens of nanoamperes, enabled by WO3 channel engineering. The introduction of an amorphous stoichiometric WO3 channel formed by reactive sputtering with O gas significantly lowered the channel current but left it almost unchanged with respect to consecutive gate voltage pulses. An additional annealing process at 450 °C crystallized the WO3, allowing analog switching in the range of tens of nanoamperes. The incorporation of N gas during annealing induced a highly conductive channel, making the channel current modulation negligible as a function of the gate pulse. Using this optimized gate stack, Poole–Frenkel conduction was identified as a major transport characteristic in a temperature-dependent study. In addition, we found that the channel current modulation is a function of the gate current response, which is related to the degree of progressive movement of the Cu ions. Finally, the synaptic characteristics were updated using fully parallel programming and demonstrated in a 7 × 7 array. Using the CuOx/HfOx/WO3 synaptic transistors as weight elements in multilayer neural networks, we achieved a 90% recognition accuracy on the Fashion-MNIST dataset.

Addressing posterior collapse by splitting decoders in variational recurrent autoencoders

Variational recurrent autoencoder model (VRAE) is an appealing technique for capturing the variabilities underlying complex sequential data, which is realized by introducing high-level latent random variables as hidden states. Existing models suffer from the well-known ‘posterior collapse’ problem, meaning that a powerful autoregressive decoder equipped in the model itself could capture all the variabilities and hence leave the latent variables learning nothing from the data. From the perspective of model training, the posterior collapse problem can result in a very low Kullback–Leibler divergence (KL-divergence) value, which means the posteriors of the latent variables tend to be just the priors. In this paper, we address this problem by proposing a Bayesian variational recurrent neural network (BVRNN) model, in which two additional decoders are added into the original VRAE. These extra decoders can force the latent variables to learn meaningful knowledge during the training process. We conduct experiments on MNIST and Fashion-MNIST dataset. The experimental results show that the proposed model outperforms several baseline models. We further adapt the proposed model to a very challenging task in natural language processing, namely Named-Entity Recognition (NER). Experimental results show that our model is competitive to the state-of-the-art models on NER.

A surface-normal photodetector as nonlinear activation function in diffractive optical neural networks

Optical neural networks (ONNs) enable high speed, parallel, and energy efficient processing compared to their conventional digital electronic counterparts. However, realizing large scale ONN systems is an open problem. Among various integrated and non-integrated ONNs, free-space diffractive ONNs benefit from a large number of pixels of spatial light modulators to realize millions of neurons. However, a significant fraction of computation time and energy is consumed by the nonlinear activation function that is typically implemented using a camera sensor. Here, we propose a novel surface-normal photodetector (SNPD) with an optical-in–electrical-out (O–E) nonlinear response to replace the camera sensor that enables about three orders of magnitude faster (5.7 µs response time) and more energy efficient (less than 10 nW/pixel) response. Direct efficient vertical optical coupling, polarization insensitivity, inherent nonlinearity with no control electronics, low optical power requirements, and the possibility of implementing large scale arrays make the SNPD a promising O–E nonlinear activation function for diffractive ONNs. To show the applicability of the proposed neural nonlinearity, successful classification simulations of the MNIST and Fashion MNIST datasets using the measured response of SNPD with accuracy comparable to that of an ideal ReLU function are demonstrated.

Optimization of post-annealing temperature for high-performance synaptic transistors based on In–Ga–Zn–O channel and trap layers

Synaptic transistors using an indium–gallium–zinc oxide (IGZO) charge trap layer (CTL) have great potential for high-performance neuromorphic applications due to their excellent stability and high linearity in conductance modulation. Since the high conductivity of the IGZO CTL enhances charge de-trapping performance, the post-annealing process, which directly affects the conductivity of the film, should be carefully conducted. In this study, we investigated the effect of post-annealing temperature (TPA) on the charge de-trapping performance of synaptic transistors. The synaptic transistor showed superior charge de-trapping performance at a TPA of 250 °C, resulting from the increased conductivity of the CTL. Through various electrical and chemical analysis methods, it was revealed that the hydrogen decrease within the CTL and the oxidation–reduction reaction at the interface between the tunneling layer and the CTL contributed to the significant increase in the conductivity of the CTL. The synaptic transistor with the optimal TPA exhibited high linearity of potentiation (αp: 0.56) and depression (αd: −0.65) with 64 conductance levels, enabling the spiking neural network simulation to achieve a high accuracy of 88.75% using a Fashion-MNIST dataset. This work shows that TPA is one of the critical process strategies for the high charge de-trapping performance of the synaptic transistor by inducing the chemical reaction at the CTL interface and hydrogen content change within the CTL.

Self-Organizing Democratized Learning: Toward Large-Scale Distributed Learning Systems.

Emerging cross-device artificial intelligence (AI) applications require a transition from conventional centralized learning systems toward large-scale distributed AI systems that can collaboratively perform complex learning tasks. In this regard, democratized learning (Dem-AI) lays out a holistic philosophy with underlying principles for building large-scale distributed and democratized machine learning systems. The outlined principles are meant to study a generalization in distributed learning systems that go beyond existing mechanisms such as federated learning (FL). Moreover, such learning systems rely on hierarchical self-organization of well-connected distributed learning agents who have limited and highly personalized data and can evolve and regulate themselves based on the underlying duality of specialized and generalized processes. Inspired by Dem-AI philosophy, a novel distributed learning approach is proposed in this article. The approach consists of a self-organizing hierarchical structuring mechanism based on agglomerative clustering, hierarchical generalization, and corresponding learning mechanism. Subsequently, hierarchical generalized learning problems in recursive forms are formulated and shown to be approximately solved using the solutions of distributed personalized learning problems and hierarchical update mechanisms. To that end, a distributed learning algorithm, namely DemLearn, is proposed. Extensive experiments on benchmark MNIST, Fashion-MNIST, FE-MNIST, and CIFAR-10 datasets show that the proposed algorithm demonstrates better results in the generalization performance of learning models in agents compared to the conventional FL algorithms. The detailed analysis provides useful observations to further handle both the generalization and specialization performance of the learning models in Dem-AI systems.

Implementation of Convolutional Neural Network Method in Identifying Fashion Image

The fashion industry has changed a lot over the years, which makes it hard for people to compare different kinds of fashion. To make it easier, different styles of clothing are tried out to find the exact and precise look desired. So, we opted to employ the Convolutional Neural Network (CNN) method for fashion classification. This approach represents one of the methodologies employed to utilize computers for the purpose of recognizing and categorizing items. The goal of this research is to see how well the Convolutional Neural Network method classifies the Fashion-MNIST dataset compared to other methods, models, and classification processes used in previous research. The information in this dataset is about different types of clothes and accessories. These items are divided into 10 categories, which include ankle boots, bags, coats, dresses, pullovers, sandals, shirts, sneakers, t-shirts, and trousers. The new classification method worked better than before on the test dataset. It had an accuracy value of 95. 92%, which is higher than in previous research. This research also uses a method called image data generator to make the Fashion MNIST image better. This method helps prevent too much focus on certain details and makes the results more accurate.