Fashion-MNIST Datasets Research Articles

The backpropagation algorithm implemented on spiking neuromorphic hardware

The capabilities of natural neural systems have inspired both new generations of machine learning algorithms as well as neuromorphic, very large-scale integrated circuits capable of fast, low-power information processing. However, it has been argued that most modern machine learning algorithms are not neurophysiologically plausible. In particular, the workhorse of modern deep learning, the backpropagation algorithm, has proven difficult to translate to neuromorphic hardware. This study presents a neuromorphic, spiking backpropagation algorithm based on synfire-gated dynamical information coordination and processing implemented on Intel’s Loihi neuromorphic research processor. We demonstrate a proof-of-principle three-layer circuit that learns to classify digits and clothing items from the MNIST and Fashion MNIST datasets. To our knowledge, this is the first work to show a Spiking Neural Network implementation of the exact backpropagation algorithm that is fully on-chip without a computer in the loop. It is competitive in accuracy with off-chip trained SNNs and achieves an energy-delay product suitable for edge computing. This implementation shows a path for using in-memory, massively parallel neuromorphic processors for low-power, low-latency implementation of modern deep learning applications.

Artificial photoelectric synaptic devices with ferroelectric diode effect for high-performance neuromorphic computing

With the rapid advancement of artificial intelligence and machine learning, the demand for neuromorphic computing systems has intensified. High-performance artificial synaptic devices are crucial to achieving this objective. Ferroelectric diode artificial synaptic devices were prepared using aluminum-doped BaTiO3 (BTAO) thin films by a low-cost sol-gel method. These devices mimic the basic properties of biological synapses, such as long-term plasticity (LTP) and short-term plasticity (STP), under electrical stimulation. Additionally, the devices exhibit an excellent UV light response, enabling the transition from STP to LTP by adjusting the light pulse intensity, width, and number of pulses. Aluminum doping significantly enhances the ferroelectricity of BaTiO3 films, increasing the Pr from 2.31 µC/cm² to 9.08 µC/cm². By modulating the Schottky barrier through polarization, the BTAO films exhibit a switchable diode effect, which facilitates fine-tuning of the synaptic connection strength while maintaining synaptic stability. Recognition accuracies of 97.32% for the MNIST dataset and 87.48% for the Fashion-MNIST dataset were achieved by convolutional neural network simulations. These results suggest new possibilities for brain-like processing with ferroelectric diodes.

Integrated convolutional kernel based on two-dimensional photonic crystals.

Optical neural networks (ONNs) exhibit significant potential for accelerating artificial intelligence task processing due to their low latency, high bandwidth, and parallel processing capabilities. Photonic crystals (PhCs) are extensively utilized in integrated optoelectronics because of their unique photonic bandgap properties and precise control of light waves. In this study, we propose an optical reconfigurable convolutional kernel based on PhCs. This kernel can perform convolutional operations on weights by constructing a PhC weight bank. The convolutional kernel demonstrates exceptional performance within the developed optical convolutional neural network framework, successfully realizing various image edge processing tasks. It achieves blind recognition accuracies of 97.81% for the MNIST dataset and 80.31% for the Fashion-MNIST dataset. This study not only demonstrates the feasibility of constructing optical neural networks based on PhCs but to our knowledge, also offers new avenues for the future development of optical computing.

Enhancing federated learning with dynamic weight adjustment based on particle swarm optimization

Federated learning (FL) stands as a promising distributed machine learning approach today, allowing model training on local clients without data sharing. However, varied contributions among clients often impact the global model’s performance, while FL confronts challenges like communication overhead, data heterogeneity, and privacy concerns. In this paper, we introduce a novel federated learning server aggregation algorithm: the Federated Learning Algorithm with Optimized Weight Aggregation via Particle Swarm Optimization Algorithm (AdpFedPSO). This approach dynamically adjusts client model contribution weights based on their performance and stability, aiming to enhance global model accuracy, convergence speed, and make model aggregation smarter and more adaptable. Through experimental validation on real datasets, we find that AdpFedPSO enhances accuracy by about 15% on the 0.6-Dirichlet MNIST dataset, 7.3% on the FashionMNIST dataset, and 13.4% on the CIFAR-10 dataset compared to the traditional FL aggregation method, FedAvg. These results indicate that AdpFedPSO not only enhances the accuracy of the global model but also expedites the model’s convergence speed. Additionally, it demonstrates resilience across various levels of client numbers, participation rates, and client heterogeneity, providing valuable reference and guidance for the further development of FL technology. Moreover, the concept of employing the particle swarm optimization algorithm for FL model aggregation also offers new insights and directions for future research.

Structure-Based Training: A Training Method Aimed at Pixel Errors for a Correlation-Coefficient-Based Neural Network.

In research on building a one-shot learning neural network without pre-training using mass data, the limitation on the information obtained from a single training sample downgrades the performance of the network. In order to improve performance and take full advantage of the support set, in this study, we design three kinds of shadow nodes and propose a structure-based training method for a correlation-coefficient-based neural network. This training strategy focuses on branches that are not activated or inactivated as expected. In contrast to existing networks that optimize the parameters using back-propagation, the training method proposed in this paper optimizes the structure of the correlation-coefficient-based network by correcting its pixel errors. For the shadow nodes and training process based on this strategy, the intersection over union (IOU) of a detected target increases by 4.83% in the experiments when using the Fashion-Mnist dataset, increases by 4.02% when using the Omniglot dataset, and increases by 3.89% when using the Cifar-10 dataset. The samples in category "7" wrongly classified as "1" decreased by 27.32% when using the Mnist dataset after training. This training strategy, along with shadow nodes, makes the correlation-coefficient-based network a more practical model and enables the network to develop during the accumulation of reliable samples, thus making it more suitable for simple target detection projects that collect samples over time. Moreover, the shadow nodes and training method proposed in this paper supplement the non-gradient-based parameter-gaining strategy. Additionally, it is a new attempt to explore the imitation of a human's ability to learn a new pattern from a low number of references.

Leveraging Tunability of Localized-Interfacial Memristors for Efficient Handling of Complex Neural Networks.

A scalable (<130 nm) resistive switching memristor that features both filamentary and interfacial switching aimed at neuromorphic computing is developed in this study. The typically perceived noise or volatility was effectively harnessed as a controlled mechanism for interfacial switching. The multilayer structure for the proposed memristor enhances switching stability by curbing ionic overmigration and mitigating leakage paths. Furthermore, the memristors showcased their reliability by demonstrating more than 15 M cycles in the filamentary mode and 1 M pulses in the interfacial mode. Additionally, retention tests at 85 °C for 104 s confirmed the stability across different states, affirming its reliability as a nonvolatile CMOS-compatible element. While many studies validate performance solely on the MNIST data set, this work also evaluates more complex data sets, demonstrating the robustness of the demonstrated memristor in supervised learning. Specifically, supervised learning simulations on MNIST and fashion MNIST data sets indicated a high learning rate with <4% deviations from numerical training, while offline inference trained on CIFAR-10 and CIFAR-100 data sets revealed <2.5% and <7% deviations caused by programing error accumulation, even with increased memristor counts for these highly complex data sets. Unsupervised learning via spike-timing-dependent plasticity further highlights the potential of the developed memristor in bridging artificial and biological paradigms, offering a significant advance toward efficient and biologically inspired computing architectures.

Pruning convolution neural networks using filter clustering based on normalized cross-correlation similarity

ABSTRACT Despite all the recent development and success of deep neural networks, deployment of a deep model onto the resource-constrained devices still remains challenging. However, model pruning can resolve this issue for Convolutional Neural Networks (CNNs), since it is one of the most popular approaches to reducing computational complexities. Therefore, this article presents a pruning model for convolutional neural networks. The proposed method classifies and arranges similar filters into the same cluster where the similarity is calculated using a three-dimensional normalized cross-correlation. Moreover, these steps can be completed entirely based on the filter values while not requiring a set of test images as well as the acquisition of any filter activation. In the research, the performances of the proposed model pruning method have been evaluated, where it is observed that the proposed approach is computationally light and requires significantly less time and resources compared to ML and activation-based approaches. In the experiments, using the VGG16 model on the Cifar10 dataset, the proposed approach results in the pruned model(s) which are comparable in performance with models found using activation-based methods and expensive ML-based methods. Similar results are found when pruning a custom CNN on the MNIST and Fashion MNIST datasets as well.

State-of-the-Art Results with the Fashion-MNIST Dataset

In September 2024, the Fashion-MNIST dataset will be 7 years old. Proposed as a replacement for the well-known MNIST dataset, it continues to be used to evaluate machine learning model architectures. This paper describes new results achieved with the Fashion-MNIST dataset using classical machine learning models and a relatively simple convolutional network. We present the state-of-the-art results obtained using the CNN-3-128 convolutional network and data augmentation. The developed CNN-3-128 model containing three convolutional layers achieved an accuracy of 99.65% in the Fashion-MNIST test image set. In addition, this paper presents the results of computational experiments demonstrating the dependence between the number of adjustable parameters of the convolutional network and the maximum acceptable classification quality, which allows us to optimise the computational cost of model training.

Imbalanced Data Parameter Optimization of Convolutional Neural Networks Based on Analysis of Variance

Classifying imbalanced data is important due to the significant practical value of accurately categorizing minority class samples, garnering considerable interest in many scientific domains. This study primarily uses analysis of variance (ANOVA) to investigate the main and interaction effects of different parameters on imbalanced data, aiming to optimize convolutional neural network (CNN) parameters to improve minority class sample recognition. The CIFAR-10 and Fashion-MNIST datasets are used to extract samples with imbalance ratios of 25:1, 15:1, and 1:1. To thoroughly assess model performance on imbalanced data, we employ various evaluation metrics, such as accuracy, recall, F1 score, P-mean, and G-mean. In highly imbalanced datasets, optimizing the learning rate significantly affects all performance metrics. The interaction between the learning rate and kernel size significantly impacts minority class samples in moderately imbalanced datasets. Through parameter optimization, the accuracy of the CNN model on the 25:1 highly imbalanced CIFAR-10 and Fashion-MNIST datasets improves by 14.20% and 5.19% compared to the default model and by 8.21% and 3.87% compared to the undersampling model, respectively, while also enhancing other evaluation metrics for minority classes.

Mixed precision quantization of silicon optical neural network chip

In recent years, the field of neural network research has witnessed remarkable advancements in various domains. One of the emerging approaches is the integration of photonic computing, which leverages the unique properties of light for ultra-fast information processing. In this article, we establish a mixed precision quantization model to silicon-based optical neural networks and evaluates their performance on the MNIST and Fashion-MNIST datasets. Through a genetic algorithm-based optimization process, we achieve significant parameter compression while maintaining competitive accuracy. Our findings demonstrate that with an average quantization bitwidth of 4.5 bits on the MNIST dataset, we achieve an impressive 85.94% reduction in parameter size compared to traditional 32-bit networks, with only a marginal accuracy drop of 0.65%. Similarly, on the Fashion-MNIST dataset, we achieve an average quantization bitwidth of 5.67 bits, resulting in an 82.28% reduction in parameter size with a slight accuracy drop of 0.8%.

Single-pixel object classification using ordered illumination patterns

The fixed-ordering illumination patterns lack an effective information interchange with objects. This leads to the light intensity values carrying random feature information, affecting the model's building and recognition capability. Furthermore, this method needs numerous illumination pattern projection counts, which results in low sampling efficiency. Therefore, we propose a new optimized-ordering illumination patterns method. In this paper, we first integrate fixed-ordering illumination patterns with category-representative images to obtain light intensity values that carry category feature information. Subsequently, the mechanisms of intra-category feature importance and inter-category feature differences are introduced to reorder the light intensity values and eliminate index numbers to generate an optimized-ordering index array. Finally, based on this index array, illumination patterns are extracted from the original fixed ordering method to form the new optimized-ordering illumination patterns. Experimental results show that in simulation experiments using the MNIST, Fashion MNIST, TRI, and Multi-category datasets, the optimized ordering method achieved recognition accuracies of 90.00%, 80.00%, 80.00%, and 80.00% with only 30 sampling times. Actual tests with physical objects such as mini numbers and clothing also confirmed these findings. The method offers a new research approach by enhancing recognition accuracy and sampling efficiency in single-pixel object classification.

APCSMA: Adaptive Personalized Client-Selection and Model-Aggregation Algorithm for Federated Learning in Edge Computing Scenarios.

With the rapid advancement of the Internet and big data technologies, traditional centralized machine learning methods are challenged when dealing with large-scale datasets. Federated Learning (FL), as an emerging distributed machine learning paradigm, enables multiple clients to collaboratively train a global model while preserving privacy. Edge computing, also recognized as a critical technology for handling massive datasets, has garnered significant attention. However, the heterogeneity of clients in edge computing environments can severely impact the performance of the resultant models. This study introduces an Adaptive Personalized Client-Selection and Model-Aggregation Algorithm, APCSMA, aimed at optimizing FL performance in edge computing settings. The algorithm evaluates clients' contributions by calculating the real-time performance of local models and the cosine similarity between local and global models, and it designs a ContriFunc function to quantify each client's contribution. The server then selects clients and assigns weights during model aggregation based on these contributions. Moreover, the algorithm accommodates personalized needs in local model updates, rather than simply overwriting with the global model. Extensive experiments were conducted on the FashionMNIST and Cifar-10 datasets, simulating three data distributions with parameters dir = 0.1, 0.3, and 0.5. The accuracy improvements achieved were 3.9%, 1.9%, and 1.1% for the FashionMNIST dataset, and 31.9%, 8.4%, and 5.4% for the Cifar-10 dataset, respectively.

High-performance deep spiking neural networks with 0.3 spikes per neuron

Communication by rare, binary spikes is a key factor for the energy efficiency of biological brains. However, it is harder to train biologically-inspired spiking neural networks than artificial neural networks. This is puzzling given that theoretical results provide exact mapping algorithms from artificial to spiking neural networks with time-to-first-spike coding. In this paper we analyze in theory and simulation the learning dynamics of time-to-first-spike-networks and identify a specific instance of the vanishing-or-exploding gradient problem. While two choices of spiking neural network mappings solve this problem at initialization, only the one with a constant slope of the neuron membrane potential at threshold guarantees the equivalence of the training trajectory between spiking and artificial neural networks with rectified linear units. For specific image classification architectures comprising feed-forward dense or convolutional layers, we demonstrate that deep spiking neural network models can be effectively trained from scratch on MNIST and Fashion-MNIST datasets, or fine-tuned on large-scale datasets, such as CIFAR10, CIFAR100 and PLACES365, to achieve the exact same performance as that of artificial neural networks, surpassing previous spiking neural networks. Our approach accomplishes high-performance classification with less than 0.3 spikes per neuron, lending itself for an energy-efficient implementation. We also show that fine-tuning spiking neural networks with our robust gradient descent algorithm enables their optimization for hardware implementations with low latency and resilience to noise and quantization.

Ensuring Toplogical Data-Structure Preservation under Autoencoder Compression Due to Latent Space Regularization in Gauss–Legendre Nodes

We formulate a data-independent latent space regularization constraint for general unsupervised autoencoders. The regularization relies on sampling the autoencoder Jacobian at Legendre nodes, which are the centers of the Gauss–Legendre quadrature. Revisiting this classic allows us to prove that regularized autoencoders ensure a one-to-one re-embedding of the initial data manifold into its latent representation. Demonstrations show that previously proposed regularization strategies, such as contractive autoencoding, cause topological defects even in simple examples, as do convolutional-based (variational) autoencoders. In contrast, topological preservation is ensured by standard multilayer perceptron neural networks when regularized using our approach. This observation extends from the classic FashionMNIST dataset to (low-resolution) MRI brain scans, suggesting that reliable low-dimensional representations of complex high-dimensional datasets can be achieved using this regularization technique.

Efficient sparse spiking auto-encoder for reconstruction, denoising and classification

Auto-encoders are capable of performing input reconstruction, denoising, and classification through an encoder-decoder structure. Spiking Auto-Encoders (SAEs) can utilize asynchronous sparse spikes to improve power efficiency and processing latency on neuromorphic hardware. In our work, we propose an efficient SAE trained using only Spike-Timing-Dependant Plasticity (STDP) learning. Our auto-encoder uses the Time-To-First-Spike (TTFS) encoding scheme and needs to update all synaptic weights only once per input, promoting both training and inference efficiency due to the extreme sparsity. We showcase robust reconstruction performance on the Modified National Institute of Standards and Technology (MNIST) and Fashion-MNIST datasets with significantly fewer spikes compared to state-of-the-art SAEs by 1–3 orders of magnitude. Moreover, we achieve robust noise reduction results on the MNIST dataset. When the same noisy inputs are used for classification, accuracy degradation is reduced by 30%–80% compared to prior works. It also exhibits classification accuracies comparable to previous STDP-based classifiers, while remaining competitive with other backpropagation-based spiking classifiers that require global learning through gradients and significantly more spikes for encoding and classification of MNIST/Fashion-MNIST inputs. The presented results demonstrate a promising pathway towards building efficient sparse spiking auto-encoders with local learning, making them highly suited for hardware integration.

Federated learning: Impact of different algorithms and models on prediction results based on fashion-MNIST data set

The realm of federated learning is rapidly advancing amid the era of big data. Therefore, how to select a suitable federated learning algorithm to achieve realistic tasks has become particularly critical. In this study, we explore the impact of different algorithms and models on the prediction results of Federated Learning (FL) using the Fashion-MNIST data set. Federated Learning enhances data privacy and reduces latency by training models directly on local devices since it is a decentralized machine learning approach. We analyze the performance of several FL algorithms including Federated Averaging (FedAvg), Federated Stochastic Gradient Descent (FedSGD), Federated Proximal (FedProx), and SCAFFOLD. Our experiments reveal significant differences in accuracy and stability among these algorithms, highlighting their strengths and weaknesses in handling non-IID (Non-Independent and Identically Distributed) data. FedProx demonstrate superior performance in terms of accuracy and robustness, making them suitable for complex federated learning environments. These discoveries offer crucial insights for choosing suitable FL algorithms and models in practical applications.

Comparative analysis of federated learning algorithms under non-IID data

With the increasing global attention to data privacy and security issues, how to effectively utilize distributed data while protecting personal privacy has become an important research topic. Federated learning (FL) aims to address data silos and privacy issues by enabling multiple devices or servers to train shared models in collaboration without submitting raw data to a central server. Although a variety of federated learning algorithms have been proposed, there is still a gap in the research on their performance differences under the same model. The goal of this study is to compare and analyze the performance differences of different federated learning algorithms on the same model through experiments. Based on the Fashion-MNIST dataset, this paper compares four commonly used federated learning algorithms in detail: Federal Averaging (FedAvg), Federated Stochastic Gradient Descent (FedSGD), Stochastic Controlled Averaging for Federated Learning (SCAFFOLD), and Federated Proximal (FedProx). The experimental results show that FedProx performs best in all evaluation indicators, followed by SCAFFOLD and FedAvg, while FedSGD performs the worst. These insights into algorithm performance with non-IID data inform practical application suitability and guide future research.

A Fair Contribution Measurement Method for Federated Learning.

Federated learning is an effective approach for preserving data privacy and security, enabling machine learning to occur in a distributed environment and promoting its development. However, an urgent problem that needs to be addressed is how to encourage active client participation in federated learning. The Shapley value, a classical concept in cooperative game theory, has been utilized for data valuation in machine learning services. Nevertheless, existing numerical evaluation schemes based on the Shapley value are impractical, as they necessitate additional model training, leading to increased communication overhead. Moreover, participants' data may exhibit Non-IID characteristics, posing a significant challenge to evaluating participant contributions. Non-IID data have greatly affected the accuracy of the global model, weakened the marginal effect of the participants, and led to the underestimated contribution measurement results of the participants. Current work often overlooks the impact of heterogeneity on model aggregation. This paper presents a fair federated learning contribution measurement scheme that addresses the need for additional model computations. By introducing a novel aggregation weight, it enhances the accuracy of the contribution measurement. Experiments on the MNIST and Fashion MNIST dataset show that the proposed method can accurately compute the contributions of participants. Compared to existing baseline algorithms, the model accuracy is significantly improved, with a similar time cost.

HQC-MCDCNN: a novel hybrid quantum–classical multi-path denoising convolutional neural network

Image denoising is a longstanding and enduring visual problem, and with the continuous rise of quantum computing in the field of machine learning, its role in image processing has become increasingly important. This paper introduces for the first time the use of multiscale variational quantum circuits in the field of image denoising, aiming to enhance the performance of classical convolutional neural networks and explore the potential advantages of combining quantum and classical approaches. In this work, we propose a novel Hybrid Quantum-Classical Multi-Path Denoising Convolutional Neural Network, abbreviated as HQC-MCDCNN. The HQC-MCDCNN is composed of a hybrid of quantum and classical elements, with the quantum part using multiscale variational quantum circuits instead of classical convolutional layers for feature extraction, and the classical part employing a newly designed multi-path denoising convolutional neural network for supervised learning. Together, these components synergistically achieve image denoising. It is worth noting that this paper aims to build readers’ intuition for quantum computing, presenting all internal details of this work with rich images and visualizations. To demonstrate the denoising capability of HQC-MCDCNN, we conducted rigorous comparative experiments. Due to the constraints of Noisy Intermediate-Scale Quantum (NISQ) devices and the limited number of quantum bits, the experiments were based on the MNIST and Fashion-MNIST datasets with varying degrees of noise (noise factors ranging from 0.3 to 0.7), employing a 6-fold stratified sampling strategy for cross-validation. The experimental results indicate that HQC-MCDCNN is promising across all evaluation metrics, particularly outperforming other models by 56.5% in the average UIQ index. This suggests that our hybrid model exhibits outstanding feature extraction capabilities and excellent denoising performance, providing a promising path for addressing image denoising challenges.

Direct electromagnetic information processing with planar diffractive neural network.

Diffractive neural network in electromagnetic wave-driven system has attracted great attention due to its ultrahigh parallel computing capability and energy efficiency. However, recent neural networks based on the diffractive framework still face the bottlenecks of misalignment and relatively large size limiting their further applications. Here, we propose a planar diffractive neural network (pla-NN) with a highly integrated and conformal architecture to achieve direct signal processing in the microwave frequency. On the basis of printed circuit fabrication process, the misalignment could be effectively circumvented while enabling flexible extension for multiple conformal and stacking designs. We first conduct validation on the fashion-MNIST dataset and experimentally build up a system using the proposed network architecture for direct recognition of different geometry structures in the electromagnetic space. We envision that the presented architecture, once combined with the advanced dynamic maneuvering techniques and flexible topology, would exhibit unlimited potentials in the areas of high-performance computing, wireless sensing, and flexible wearable electronics.