Fashion-MNIST Datasets Research Articles

SHE-MTJ based ReLU-max pooling functions for on-chip training of neural networks

We present a detailed investigation of various routes to optimize the power consumption of the spintronic-based devices for implementing rectified linear activation (ReLU) and max-pooling functions. We examine the influence of various spin Hall effect layers, and their input resistances on the power consumption of the ReLU-max pooling functions, we also access the impact of the thermal stability factor of the free-ferromagnet layer on the power consumption and accuracy of the device. The design for ReLU-max pooling relies on the continuous rotation of magnetization, which is accomplished by applying orthogonal spin current to the free-FM layer. We also demonstrate the non-trivial power-resistance relation, where the power consumption decreases with an increase in SHE resistance. We utilize the hybrid spintronic-CMOS simulation platform that combines Keldysh non-equilibrium Green’s function (NEGF) with Landau-Lifshitz-Gilbert-Slonzewski (LLGS) equations and the HSPICE circuit simulator to evaluate our network. Our design takes 0.343 μW of power for ReLU emulation and 17.86 μW of power for ReLU-max pooling network implementation at a thermal stability factor of 4.58, all while maintaining reliable results. We validate the efficiency of our design by implementing a convolutional neural network that classifies the handwritten-MNIST and fashion-MNIST datasets. This implementation illustrates that the classification accuracies achieved are on par with those attained using the ideal software ReLU-max pooling functions, with an energy consumption of 167.31 pJ per sample.

Time regularization in optimal time variable learning

AbstractRecently, optimal time variable learning in deep neural networks was introduced in Antil et al. In this manuscript we extend the concept by introducing a regularization term that directly relates to the time horizon in discrete dynamical systems. Furthermore, we propose an adaptive pruning approach for Residual Neural Networks (ResNets), which reduces network complexity without compromising expressiveness, while simultaneously decreasing training time. The results are illustrated by applying the proposed concepts to classification tasks on the well known MNIST and Fashion MNIST data sets. Our PyTorch code is available on https://github.com/frederikkoehne/time_variable_learning.

Exploring the Impact of Denoising Autoencoder Architectures on Image Retrieval

In today’s busy world, people are shifting to online shopping for their basic needs like groceries, clothes, etc. As a result, modern applications are incorporating image search as part of their application. However, retrieving accurate fashion images is challenging due to the low inter-class separability. In this paper, we aim to propose and implement three deep learning denoising autoencoder image retrieval models trained on Fashion-MNIST dataset and MNIST dataset. Fashion-MNIST is widely used datasets to evaluate model because of its low inter-class separability. It is difficult to get a good accuracy on Fashion-MNIST dataset. The proposed models train to learn significant features of fashion images and improve the precision of image retrieval. These models are compared based on the Label Ranking Average Precision (LRAP) values, with the third model bagging the highest LRAP value. These models also trained and tested on MNIST dataset, which have similar configuration as Fashion-MNIST dataset but with high inter-class separability. Intriguingly, the second and third models trained on MNIST dataset demonstrate neck-to-neck performance, both achieving comparable accuracy. This choice of dataset ensures that the models’ accuracy is more noticeable and significant. The outcomes of this study validate the efficacy of denoising autoencoders in fashion image retrieval and emphasize their prospective in improving online shopping experiences.

DT-SCNN: dual-threshold spiking convolutional neural network with fewer operations and memory access for edge applications.

The spiking convolutional neural network (SCNN) is a kind of spiking neural network (SNN) with high accuracy for visual tasks and power efficiency on neuromorphic hardware, which is attractive for edge applications. However, it is challenging to implement SCNNs on resource-constrained edge devices because of the large number of convolutional operations and membrane potential (Vm) storage needed. Previous works have focused on timestep reduction, network pruning, and network quantization to realize SCNN implementation on edge devices. However, they overlooked similarities between spiking feature maps (SFmaps), which contain significant redundancy and cause unnecessary computation and storage. This work proposes a dual-threshold spiking convolutional neural network (DT-SCNN) to decrease the number of operations and memory access by utilizing similarities between SFmaps. The DT-SCNN employs dual firing thresholds to derive two similar SFmaps from one Vm map, reducing the number of convolutional operations and decreasing the volume of Vms and convolutional weights by half. We propose a variant spatio-temporal back propagation (STBP) training method with a two-stage strategy to train DT-SCNNs to decrease the inference timestep to 1. The experimental results show that the dual-thresholds mechanism achieves a 50% reduction in operations and data storage for the convolutional layers compared to conventional SCNNs while achieving not more than a 0.4% accuracy loss on the CIFAR10, MNIST, and Fashion MNIST datasets. Due to the lightweight network and single timestep inference, the DT-SCNN has the least number of operations compared to previous works, paving the way for low-latency and power-efficient edge applications.

An adjustable multistage resistance switching behavior of a photoelectric artificial synaptic device with a ferroelectric diode effect for neuromorphic computing.

Neuromorphic computing, which mimics biological neural networks, is widely regarded as the optimal solution for addressing the limitations of traditional von Neumann computing architecture. In this work, an adjustable multistage resistance switching ferroelectric Bi2FeCrO6 diode artificial synaptic device was fabricated using a sol-gel method with a simple process. The device exhibits nonlinearity in its electrical characteristics, demonstrating tunable multistage resistance switching behavior and a strong ferroelectric diode effect through the manipulation of ferroelectric polarization. One of its salient advantages resides in its capacity to dynamically regulate its polarization state in response to an external electric field, thereby facilitating the fine-tuning of synaptic connection strength while maintaining synaptic stability. The device is capable of accurately simulating the fundamental properties of biological synapses, including long/short-term plasticity, paired-pulse facilitation, and spike-timing-dependent plasticity. Additionally, the device exhibits a distinctive photoelectric response and is capable of inducing synaptic plasticity by light signal activation. The utilization of a femtosecond laser for the scrutiny of carrier transport mechanisms imparts profound insights into the intricate dynamics governing the optical memory effect. Furthermore, utilizing a convolutional neural network (CNN) architecture, the recognition accuracy of the MNIST and fashion MNIST datasets was improved to 95.6% and 78%, respectively, through the implementation of improved random adaptive algorithms. These findings present a new opportunity for utilizing Bi2FeCrO6 materials in the development of artificial synapses for neuromorphic computation.

Efficient Image Annotation and Caption System Using Deep Convolutional Neural Networks

This study aims to develop an annotation and image annotation system using the Fashion MNIST dataset, which consists of 70,000 grayscale images of ten clothing categories. The system uses a long short-term memory (LSTM) network to generate captions and a convolutional neural network (CNN) to extract image features. Performance evaluation metrics such as Precision, Recall, F1 score, BLEU score, METEOR score, CIDEr score, and ROUGE-L score are used where the accuracy of each clothing category is calculated to evaluate the performance of the model across different categories. Visual analysis of the generated captions is performed to gain insight into the effectiveness of the model and potential areas for improvement. The results indicate the model's success in classifying clothing items, as evidenced by its high accuracy on the test set. The qualitative study reveals the model's ability to identify different types of clothing by providing relevant captions, where the feature representation layer (normalization) plays a crucial role in transforming the detected features. to a flattened row which is then passed to a fully connected layer to learn the relationships and make final decisions with the output layer using a softmax activation function to assign probabilities to each image class, with the class with the highest probability selected as the predicted image class.

Graph-Based Semi-Supervised Deep Image Clustering With Adaptive Adjacency Matrix.

Image clustering is a research hotspot in machine learning and computer vision. Existing graph-based semi-supervised deep clustering methods suffer from three problems: 1) because clustering uses only high-level features, the detailed information contained in shallow-level features is ignored; 2) most feature extraction networks employ the step odd convolutional kernel, which results in an uneven distribution of receptive field intensity; and 3) because the adjacency matrix is precomputed and fixed, it cannot adapt to changes in the relationship between samples. To solve the above problems, we propose a novel graph-based semi-supervised deep clustering method for image clustering. First, the parity cross-convolutional feature extraction and fusion module is used to extract high-quality image features. Then, the clustering constraint layer is designed to improve the clustering efficiency. And, the output layer is customized to achieve unsupervised regularization training. Finally, the adjacency matrix is inferred by actual network prediction. A graph-based regularization method is adopted for unsupervised training networks. Experimental results show that our method significantly outperforms state-of-the-art methods on USPS, MNIST, street view house numbers (SVHN), and fashion MNIST (FMNIST) datasets in terms of ACC, normalized mutual information (NMI), and ARI.

An encoding framework for binarized images using hyperdimensional computing.

Hyperdimensional Computing (HDC) is a brain-inspired and lightweight machine learning method. It has received significant attention in the literature as a candidate to be applied in the wearable Internet of Things, near-sensor artificial intelligence applications, and on-device processing. HDC is computationally less complex than traditional deep learning algorithms and typically achieves moderate to good classification performance. A key aspect that determines the performance of HDC is encoding the input data to the hyperdimensional (HD) space. This article proposes a novel lightweight approach relying only on native HD arithmetic vector operations to encode binarized images that preserves the similarity of patterns at nearby locations by using point of interest selection and local linear mapping. The method reaches an accuracy of 97.92% on the test set for the MNIST data set and 84.62% for the Fashion-MNIST data set. These results outperform other studies using native HDC with different encoding approaches and are on par with more complex hybrid HDC models and lightweight binarized neural networks. The proposed encoding approach also demonstrates higher robustness to noise and blur compared to the baseline encoding.

LMFRNet: A Lightweight Convolutional Neural Network Model for Image Analysis

Convolutional neural networks (CNNs) have transformed the landscape of image analysis and are widely applied across various fields. With their widespread adoption in fields like medical diagnosis and autonomous driving, CNNs have demonstrated powerful capabilities. Despite their success, existing models face challenges in deploying and operating in resource-constrained environments, limiting their practicality in real-world scenarios. We introduce LMFRNet, a lightweight CNN model. Its innovation resides in a multi-feature block design, effectively reducing both model complexity and computational load. Achieving an exceptional accuracy of 94.6% on the CIFAR-10 dataset, this model showcases remarkable performance while demonstrating parsimonious resource utilization. We further validate the performance of the model on the CIFAR-100, MNIST, and Fashion-MNIST datasets, demonstrating its robustness and generalizability across diverse datasets. Furthermore, we conducted extensive experiments to investigate the influence of critical hyperparameters. These experiments provided valuable insights for effective model training.

WOx channel engineering of Cu-ion-driven synaptic transistor array for low-power neuromorphic computing

The multilevel current states of synaptic devices in artificial neural networks enable next-generation computing to perform cognitive functions in an energy-efficient manner. Moreover, considering large-scale synaptic arrays, multiple states programmed in a low-current regime may be required to achieve low energy consumption, as demonstrated by simple numerical calculations. Thus, we propose a three-terminal Cu-ion-actuated CuOx/HfOx/WO3 synaptic transistor array that exhibits analogously modulated channel current states in the range of tens of nanoamperes, enabled by WO3 channel engineering. The introduction of an amorphous stoichiometric WO3 channel formed by reactive sputtering with O gas significantly lowered the channel current but left it almost unchanged with respect to consecutive gate voltage pulses. An additional annealing process at 450 °C crystallized the WO3, allowing analog switching in the range of tens of nanoamperes. The incorporation of N gas during annealing induced a highly conductive channel, making the channel current modulation negligible as a function of the gate pulse. Using this optimized gate stack, Poole–Frenkel conduction was identified as a major transport characteristic in a temperature-dependent study. In addition, we found that the channel current modulation is a function of the gate current response, which is related to the degree of progressive movement of the Cu ions. Finally, the synaptic characteristics were updated using fully parallel programming and demonstrated in a 7 × 7 array. Using the CuOx/HfOx/WO3 synaptic transistors as weight elements in multilayer neural networks, we achieved a 90% recognition accuracy on the Fashion-MNIST dataset.

Addressing posterior collapse by splitting decoders in variational recurrent autoencoders

Variational recurrent autoencoder model (VRAE) is an appealing technique for capturing the variabilities underlying complex sequential data, which is realized by introducing high-level latent random variables as hidden states. Existing models suffer from the well-known ‘posterior collapse’ problem, meaning that a powerful autoregressive decoder equipped in the model itself could capture all the variabilities and hence leave the latent variables learning nothing from the data. From the perspective of model training, the posterior collapse problem can result in a very low Kullback–Leibler divergence (KL-divergence) value, which means the posteriors of the latent variables tend to be just the priors. In this paper, we address this problem by proposing a Bayesian variational recurrent neural network (BVRNN) model, in which two additional decoders are added into the original VRAE. These extra decoders can force the latent variables to learn meaningful knowledge during the training process. We conduct experiments on MNIST and Fashion-MNIST dataset. The experimental results show that the proposed model outperforms several baseline models. We further adapt the proposed model to a very challenging task in natural language processing, namely Named-Entity Recognition (NER). Experimental results show that our model is competitive to the state-of-the-art models on NER.

A surface-normal photodetector as nonlinear activation function in diffractive optical neural networks

Optical neural networks (ONNs) enable high speed, parallel, and energy efficient processing compared to their conventional digital electronic counterparts. However, realizing large scale ONN systems is an open problem. Among various integrated and non-integrated ONNs, free-space diffractive ONNs benefit from a large number of pixels of spatial light modulators to realize millions of neurons. However, a significant fraction of computation time and energy is consumed by the nonlinear activation function that is typically implemented using a camera sensor. Here, we propose a novel surface-normal photodetector (SNPD) with an optical-in–electrical-out (O–E) nonlinear response to replace the camera sensor that enables about three orders of magnitude faster (5.7 µs response time) and more energy efficient (less than 10 nW/pixel) response. Direct efficient vertical optical coupling, polarization insensitivity, inherent nonlinearity with no control electronics, low optical power requirements, and the possibility of implementing large scale arrays make the SNPD a promising O–E nonlinear activation function for diffractive ONNs. To show the applicability of the proposed neural nonlinearity, successful classification simulations of the MNIST and Fashion MNIST datasets using the measured response of SNPD with accuracy comparable to that of an ideal ReLU function are demonstrated.

Optimization of post-annealing temperature for high-performance synaptic transistors based on In–Ga–Zn–O channel and trap layers

Synaptic transistors using an indium–gallium–zinc oxide (IGZO) charge trap layer (CTL) have great potential for high-performance neuromorphic applications due to their excellent stability and high linearity in conductance modulation. Since the high conductivity of the IGZO CTL enhances charge de-trapping performance, the post-annealing process, which directly affects the conductivity of the film, should be carefully conducted. In this study, we investigated the effect of post-annealing temperature (TPA) on the charge de-trapping performance of synaptic transistors. The synaptic transistor showed superior charge de-trapping performance at a TPA of 250 °C, resulting from the increased conductivity of the CTL. Through various electrical and chemical analysis methods, it was revealed that the hydrogen decrease within the CTL and the oxidation–reduction reaction at the interface between the tunneling layer and the CTL contributed to the significant increase in the conductivity of the CTL. The synaptic transistor with the optimal TPA exhibited high linearity of potentiation (αp: 0.56) and depression (αd: −0.65) with 64 conductance levels, enabling the spiking neural network simulation to achieve a high accuracy of 88.75% using a Fashion-MNIST dataset. This work shows that TPA is one of the critical process strategies for the high charge de-trapping performance of the synaptic transistor by inducing the chemical reaction at the CTL interface and hydrogen content change within the CTL.

Self-Organizing Democratized Learning: Toward Large-Scale Distributed Learning Systems.

Emerging cross-device artificial intelligence (AI) applications require a transition from conventional centralized learning systems toward large-scale distributed AI systems that can collaboratively perform complex learning tasks. In this regard, democratized learning (Dem-AI) lays out a holistic philosophy with underlying principles for building large-scale distributed and democratized machine learning systems. The outlined principles are meant to study a generalization in distributed learning systems that go beyond existing mechanisms such as federated learning (FL). Moreover, such learning systems rely on hierarchical self-organization of well-connected distributed learning agents who have limited and highly personalized data and can evolve and regulate themselves based on the underlying duality of specialized and generalized processes. Inspired by Dem-AI philosophy, a novel distributed learning approach is proposed in this article. The approach consists of a self-organizing hierarchical structuring mechanism based on agglomerative clustering, hierarchical generalization, and corresponding learning mechanism. Subsequently, hierarchical generalized learning problems in recursive forms are formulated and shown to be approximately solved using the solutions of distributed personalized learning problems and hierarchical update mechanisms. To that end, a distributed learning algorithm, namely DemLearn, is proposed. Extensive experiments on benchmark MNIST, Fashion-MNIST, FE-MNIST, and CIFAR-10 datasets show that the proposed algorithm demonstrates better results in the generalization performance of learning models in agents compared to the conventional FL algorithms. The detailed analysis provides useful observations to further handle both the generalization and specialization performance of the learning models in Dem-AI systems.

Implementation of Convolutional Neural Network Method in Identifying Fashion Image

The fashion industry has changed a lot over the years, which makes it hard for people to compare different kinds of fashion. To make it easier, different styles of clothing are tried out to find the exact and precise look desired. So, we opted to employ the Convolutional Neural Network (CNN) method for fashion classification. This approach represents one of the methodologies employed to utilize computers for the purpose of recognizing and categorizing items. The goal of this research is to see how well the Convolutional Neural Network method classifies the Fashion-MNIST dataset compared to other methods, models, and classification processes used in previous research. The information in this dataset is about different types of clothes and accessories. These items are divided into 10 categories, which include ankle boots, bags, coats, dresses, pullovers, sandals, shirts, sneakers, t-shirts, and trousers. The new classification method worked better than before on the test dataset. It had an accuracy value of 95. 92%, which is higher than in previous research. This research also uses a method called image data generator to make the Fashion MNIST image better. This method helps prevent too much focus on certain details and makes the results more accurate.

Enhancing Fashion Classification with Vision Transformer (ViT) and Developing Recommendation Fashion Systems Using DINOVA2

As e-commerce platforms grow, consumers increasingly purchase clothes online; however, they often need clarification on clothing choices. Consumers and stores interact through the clothing recommendation system. A recommendation system can help customers to find clothing that they are interested in and can improve turnover. This work has two main goals: enhancing fashion classification and developing a fashion recommendation system. The main objective of fashion classification is to apply a Vision Transformer (ViT) to enhance performance. ViT is a set of transformer blocks; each transformer block consists of two layers: a multi-head self-attention layer and a multilayer perceptron (MLP) layer. The hyperparameters of ViT are configured based on the fashion images dataset. CNN models have different layers, including multi-convolutional layers, multi-max pooling layers, multi-dropout layers, multi-fully connected layers, and batch normalization layers. Furthermore, ViT is compared with different models, i.e., deep CNN models, VGG16, DenseNet-121, Mobilenet, and ResNet50, using different evaluation methods and two fashion image datasets. The ViT model performs the best on the Fashion-MNIST dataset (accuracy = 95.25, precision = 95.20, recall = 95.25, F1-score = 95.20). ViT records the highest performance compared to other models in the fashion product dataset (accuracy = 98.53, precision = 98.42, recall = 98.53, F1-score = 98.46). A recommendation fashion system is developed using Learning Robust Visual Features without Supervision (DINOv2) and a nearest neighbor search that is built in the FAISS library to obtain the top five similarity results for specific images.

A Homomorphic Encryption Framework for Privacy-Preserving Spiking Neural Networks

Machine learning (ML) is widely used today, especially through deep neural networks (DNNs); however, increasing computational load and resource requirements have led to cloud-based solutions. To address this problem, a new generation of networks has emerged called spiking neural networks (SNNs), which mimic the behavior of the human brain to improve efficiency and reduce energy consumption. These networks often process large amounts of sensitive information, such as confidential data, and thus privacy issues arise. Homomorphic encryption (HE) offers a solution, allowing calculations to be performed on encrypted data without decrypting them. This research compares traditional DNNs and SNNs using the Brakerski/Fan-Vercauteren (BFV) encryption scheme. The LeNet-5 and AlexNet models, widely-used convolutional architectures, are used for both DNN and SNN models based on their respective architectures, and the networks are trained and compared using the FashionMNIST dataset. The results show that SNNs using HE achieve up to 40% higher accuracy than DNNs for low values of the plaintext modulus t, although their execution time is longer due to their time-coding nature with multiple time steps.

Residual Tensor Train: A Quantum-Inspired Approach for Learning Multiple Multilinear Correlations

States of quantum many-body systems are defined in a high-dimensional Hilbert space, where rich and complex interactions among subsystems can be modelled. In machine learning, complex multiple multilinear correlations may also exist within input features. In this paper, we present a quantum-inspired multilinear model, named Residual Tensor Train (ResTT), to capture the multiple multilinear correlations of features, from low to high orders, within a single model. ResTT is able to build a robust decision boundary in a high-dimensional space for solving fitting and classification tasks. In particular, we prove that the fully-connected layer and the Volterra series can be taken as special cases of ResTT. Furthermore, we derive the rule for weight initialization that stabilizes the training of ResTT based on a mean-field analysis. We prove that such a rule is much more relaxed than that of TT, which means ResTT can easily address the vanishing and exploding gradient problem that exists in the existing TT models. Numerical experiments demonstrate that ResTT outperforms the state-of-the-art tensor network and benchmark deep learning models on MNIST and Fashion-MNIST datasets. Moreover, ResTT achieves better performance than other statistical methods on two practical examples with limited data which are known to have complex feature interactions.

Transformer in optronic neural networks for image classification

Transformer, with its self-attention mechanism, can acquire global features while parallelizing training. This study proposes an Optronic Vision Transformer (OPViT), which utilizes spatial light modulators (SLMs) and lens group to realize matrix multiplication. This is the first time that Transformer has been introduced into optical neural network, fully exploiting the advantages of photonic computing to minimize its computational cost. Based on the advantages of OPViT, it is reconfigurable, scalable, and has low space complexity. We achieve excellent classification results directly with only one layer of Transformer and two or three layers of optical convolution, achieving 98.70% and 88.93% test accuracy on the MNIST and Fashion-MNIST datasets, respectively. The classification results outperform existing optical convolutional neural networks, and are even comparable to previous optical architectures connected to electronic networks.

CNNTuner: Image Classification with A Novel CNN Model Optimized Hyperparameters

Today, the impact of deep learning in computer vision applications is growing every day. Deep learning techniques apply in many areas such as clothing search, automatic product recommendation. The main task in these applications is to perform the classification process automatically. But, high similarities between multiple apparel objects make classification difficult. In this paper, a new deep learning model based on convolutional neural networks (CNNs) is proposed to solve the classification problem. These networks can extract features from images using convolutional layers, unlike traditional machine learning algorithms. As the extracted features are highly discriminative, good results can be obtained in terms of classification performance. Performance results vary according to the number of filters and window sizes in the convolution layers that extract the features. Considering that there is more than one parameter that influences the performance result, the parameter that gives the best result can be determined after many experimental studies. The specified parameterization process is a difficult and laborious process. To address this issue, the parameters of a newly proposed CNN-based deep learning model were optimized using the Keras Tuner tool on the Fashion MNIST (F-MNIST) dataset containing multi-class fashion images. The performance results of the model were obtained using the data separated according to the cross-validation technique 5. At the same time, to measure the impact of the optimized parameters on classification, the performance results of the proposed model, called CNNTuner, are compared with state-of-the-art (SOTA) studies.