MNIST Research Articles

Effective theory of collective deep learning

Unraveling the emergence of collective learning in systems of coupled artificial neural networks points to broader implications for physics, machine learning, neuroscience, and society. Here we introduce a minimal model of interacting deep neural nets that condenses several recent decentralized algorithms by considering a competition between two terms: the local learning dynamics in the parameters of each neural network unit, and a diffusive coupling among units that tends to homogenize the parameters of the ensemble. We derive an effective theory for linear networks to show that the coarse-grained behavior of our system is equivalent to a deformed Ginzburg-Landau model with quenched disorder. This framework predicts depth-dependent disorder-order-disorder phase transitions in the parameters' solutions that reveal a depth-delayed onset of a collective learning phase and a low-rank microscopic learning path. We validate the theory in coupled ensembles of realistic neural networks trained on MNIST and CIFAR-10 datasets under privacy constraints. Interestingly, experiments confirm that individual networks–trained on private data–can fully generalize to unseen data classes when the collective learning phase emerges. Our work elucidates the physics of collective learning and contributes to the mechanistic interpretability of deep learning in decentralized settings. Published by the American Physical Society 2024

The backpropagation algorithm implemented on spiking neuromorphic hardware

The capabilities of natural neural systems have inspired both new generations of machine learning algorithms as well as neuromorphic, very large-scale integrated circuits capable of fast, low-power information processing. However, it has been argued that most modern machine learning algorithms are not neurophysiologically plausible. In particular, the workhorse of modern deep learning, the backpropagation algorithm, has proven difficult to translate to neuromorphic hardware. This study presents a neuromorphic, spiking backpropagation algorithm based on synfire-gated dynamical information coordination and processing implemented on Intel’s Loihi neuromorphic research processor. We demonstrate a proof-of-principle three-layer circuit that learns to classify digits and clothing items from the MNIST and Fashion MNIST datasets. To our knowledge, this is the first work to show a Spiking Neural Network implementation of the exact backpropagation algorithm that is fully on-chip without a computer in the loop. It is competitive in accuracy with off-chip trained SNNs and achieves an energy-delay product suitable for edge computing. This implementation shows a path for using in-memory, massively parallel neuromorphic processors for low-power, low-latency implementation of modern deep learning applications.

Nitrogen-doped carbon quantum dot-decorated In2O3 synaptic transistors for neuromorphic computing

Nitrogen-doped carbon quantum dots (N-CQDs) are promising materials for electronic devices due to their variable bandgap and structural stability. Here, we integrate N-CQDs into In2O3 synaptic transistors with electrolyte gating, resulting in a hybrid structure. The surface functional groups and defects of N-CQDs empower the charge trapping mechanism, permitting controlled conduction and charge regulation, which are crucial for emulating linear and symmetric artificial synaptic devices. Devices incorporating N-CQDs demonstrate enhanced stability and memory characteristics, low energy consumption, consistent retention, and a significant hysteresis window across multiple voltage cycles. Finally, the study emulates biological synapses and cognitive functions, achieving an energy consumption of 10 fJ per synaptic event and a pattern recognition accuracy of 91.2% on the MNIST dataset in hardware neural networks. This work demonstrates the potential of well-manipulating charge trapping in N-CQDs to develop high-performance, nonvolatile synaptic devices.

Permutation-equivariant quantum convolutional neural networks

Abstract The Symmetric group $S_{n}$ manifests itself in large classes of quantum systems as the invariance of certain characteristics of a quantum state with respect to permuting the qubits. Subgroups of $S_{n}$ arise, among many other contexts, to describe label symmetry of classical images with respect to spatial transformations, such as reflection or rotation. Equipped with the formalism of geometric quantum machine learning, in this study we propose the architectures of equivariant quantum convolutional neural networks (EQCNNs) adherent to $S_{n}$ and its subgroups. We demonstrate that a careful choice of pixel-to-qubit embedding order can facilitate easy construction of EQCNNs for small subgroups of $S_{n}$. Our novel EQCNN architecture corresponding to the full permutation group $S_{n}$ is built by applying all possible QCNNs with equal probability, which can also be conceptualized as a dropout strategy in quantum neural networks. For subgroups of $S_{n}$, our numerical results using MNIST datasets show better classification accuracy than non-equivariant QCNNs. The $S_{n}$-equivariant QCNN architecture shows significantly improved training and test performance than non-equivariant QCNN for classification of connected and non-connected graphs. When trained with sufficiently large number of data, the $S_{n}$-equivariant QCNN shows better average performance compared to $S_{n}$-equivariant QNN . These results contribute towards building powerful quantum machine learning architectures in permutation-symmetric systems.

Artificial photoelectric synaptic devices with ferroelectric diode effect for high-performance neuromorphic computing

With the rapid advancement of artificial intelligence and machine learning, the demand for neuromorphic computing systems has intensified. High-performance artificial synaptic devices are crucial to achieving this objective. Ferroelectric diode artificial synaptic devices were prepared using aluminum-doped BaTiO3 (BTAO) thin films by a low-cost sol-gel method. These devices mimic the basic properties of biological synapses, such as long-term plasticity (LTP) and short-term plasticity (STP), under electrical stimulation. Additionally, the devices exhibit an excellent UV light response, enabling the transition from STP to LTP by adjusting the light pulse intensity, width, and number of pulses. Aluminum doping significantly enhances the ferroelectricity of BaTiO3 films, increasing the Pr from 2.31 µC/cm² to 9.08 µC/cm². By modulating the Schottky barrier through polarization, the BTAO films exhibit a switchable diode effect, which facilitates fine-tuning of the synaptic connection strength while maintaining synaptic stability. Recognition accuracies of 97.32% for the MNIST dataset and 87.48% for the Fashion-MNIST dataset were achieved by convolutional neural network simulations. These results suggest new possibilities for brain-like processing with ferroelectric diodes.

Analysis of ensemble machine learning classification comparison on the skin cancer MNIST dataset

This study aims to analyze the performance of various ensemble machine learning methods, such as Adaboost, Bagging, and Stacking, in the context of skin cancer classification using the skin cancer MNIST dataset. We also evaluate the impact of handling dataset imbalance on the classification model’s performance by applying imbalanced data methods such as random under sampling (RUS), random over sampling (ROS), synthetic minority over-sampling technique (SMOTE), and synthetic minority over-sampling technique with edited nearest neighbor (SMOTEENN). The research findings indicate that Adaboost is effective in addressing data imbalance, while imbalanced data methods can significantly improve accuracy. However, the selection of imbalanced data methods should be carefully tailored to the dataset characteristics and clinical objectives. In conclusion, addressing data imbalance can enhance skin cancer classification accuracy, with Adaboost being an exception that shows a decrease in accuracy after applying imbalanced data methods.

An Empirical Study of WGAN and WGAN-GP for Enhanced Image Generation

This paper aims to advance the Wasserstein Generative Adversarial Networks (WGANs) and their enhancements, particularly focusing on the gradient penalty. Generative Adversarial Networks (GANs), introduced by Goodfellow et al. in 2014, have revolutionized the domain of image generation. To address the limitations of GANs, the WGAN was proposed. However, WGANs rely on weight clipping, which introduces its own set of issues such as slow convergence and potential gradient vanishing. The inefficiency and instability of WGANs have troubled its users. To solve these problems, WGAN with Gradient Penalty (WGAN-GP) was developed to address these challenges. It provides more stable gradients and reduces the risk of mode collapse by using a gradient penalty to enforce the necessary constraints. In this paper, the author implemented both WGAN and WGAN with Gradient Penalty (WGAN-GP) and evaluated them using the CIFAR-10 and MNIST datasets. The results show that WGAN-GP's outputs are more stable and efficient in the early rounds, confirming the effectiveness of the gradient penalty in training image datasets.

Leveraging multiplexed metasurfaces for multi-task learning with all-optical diffractive processors

Abstract Diffractive Neural Networks (DNNs) leverage the power of light to enhance computational performance in machine learning, offering a pathway to high-speed, low-energy, and large-scale neural information processing. However, most existing DNN architectures are optimized for single tasks and thus lack the flexibility required for the simultaneous execution of multiple tasks within a unified artificial intelligence platform. In this work, we utilize the polarization and wavelength degrees of freedom of light to achieve optical multi-task identification using the MNIST, FMNIST, and KMNIST datasets. Employing bilayer cascaded metasurfaces, we construct dual-channel DNNs capable of simultaneously classifying two tasks, using polarization and wavelength multiplexing schemes through a meta-atom library. Numerical evaluations demonstrate performance accuracies comparable to those of individually trained single-channel, single-task DNNs. Extending this approach to three-task parallel recognition reveals an expected performance decline yet maintains satisfactory classification accuracies of greater than 80 % for all tasks. We further introduce a novel end-to-end joint optimization framework to redesign the three-task classifier, demonstrating substantial improvements over the meta-atom library design and offering the potential for future multi-channel DNN designs. Our study could pave the way for the development of ultrathin, high-speed, and high-throughput optical neural computing systems.

Federated Learning's Dynamic Defense Against Byzantine Attacks: Integrating SIFT-Wavelet and Differential Privacy for Byzantine Grade Levels Detection

Federated learning, which enables decentralized training across multiple devices while maintaining data privacy, is susceptible to Byzantine poisoning attacks. This paradigm reduces the need for centralized data storage and transmission, thereby mitigating privacy risks associated with traditional data aggregation methods. However, FL introduces new challenges, notably susceptibility to Byzantine poisoning attacks, where rogue participants can tamper with model updates, threatening the consistency and security of the aggregated model. Our approach addresses this vulnerability by implementing robust aggregation methods, sophisticated pre-processing techniques, and a novel Byzantine grade-level detection mechanism. We introduce a federated aggregation operator designed to mitigate the impact of malicious clients. Our pre-processing includes data loading and transformation, data augmentation, and feature extraction using SIFT and wavelet transforms. Additionally, we employ differential privacy and model compression to improve the robustness and performance of the federated learning framework. Our approach is assessed using a tailored neural network model applied to the MNIST dataset, achieving 97% accuracy in detecting Byzantine attacks. Our results demonstrate that robust aggregation significantly improves the resilience and performance. This comprehensive approach ensures the integrity of the federated learning process, effectively filtering out adversarial influences and sustaining high accuracy even when faced with adversarial Byzantine clients.

Integrated convolutional kernel based on two-dimensional photonic crystals.

Optical neural networks (ONNs) exhibit significant potential for accelerating artificial intelligence task processing due to their low latency, high bandwidth, and parallel processing capabilities. Photonic crystals (PhCs) are extensively utilized in integrated optoelectronics because of their unique photonic bandgap properties and precise control of light waves. In this study, we propose an optical reconfigurable convolutional kernel based on PhCs. This kernel can perform convolutional operations on weights by constructing a PhC weight bank. The convolutional kernel demonstrates exceptional performance within the developed optical convolutional neural network framework, successfully realizing various image edge processing tasks. It achieves blind recognition accuracies of 97.81% for the MNIST dataset and 80.31% for the Fashion-MNIST dataset. This study not only demonstrates the feasibility of constructing optical neural networks based on PhCs but to our knowledge, also offers new avenues for the future development of optical computing.

Enhancing federated learning with dynamic weight adjustment based on particle swarm optimization

Federated learning (FL) stands as a promising distributed machine learning approach today, allowing model training on local clients without data sharing. However, varied contributions among clients often impact the global model’s performance, while FL confronts challenges like communication overhead, data heterogeneity, and privacy concerns. In this paper, we introduce a novel federated learning server aggregation algorithm: the Federated Learning Algorithm with Optimized Weight Aggregation via Particle Swarm Optimization Algorithm (AdpFedPSO). This approach dynamically adjusts client model contribution weights based on their performance and stability, aiming to enhance global model accuracy, convergence speed, and make model aggregation smarter and more adaptable. Through experimental validation on real datasets, we find that AdpFedPSO enhances accuracy by about 15% on the 0.6-Dirichlet MNIST dataset, 7.3% on the FashionMNIST dataset, and 13.4% on the CIFAR-10 dataset compared to the traditional FL aggregation method, FedAvg. These results indicate that AdpFedPSO not only enhances the accuracy of the global model but also expedites the model’s convergence speed. Additionally, it demonstrates resilience across various levels of client numbers, participation rates, and client heterogeneity, providing valuable reference and guidance for the further development of FL technology. Moreover, the concept of employing the particle swarm optimization algorithm for FL model aggregation also offers new insights and directions for future research.

Image classification with deconvolution operation and augmentation

Abstract Several image classification approaches have been evolved over the years utilizing convolutional neural network (CNN). In convolution operation of CNN, the shifting of kernels to overlapping regions of the image learns redundant data as the images are strongly correlated in reality. The redundant data make the neural network training a challenging task. Again, Deep Learning methods evaluated on small dataset yields degraded performance. To deal with these issues, a proposal is made in this paper that uses deconvolution operation to minimize correlations from images and data augmentation technique to increase the size of datasets. Plant Village, Tomato, and Covid-19 datasets were used for evaluating the performance of the proposed method. 70% of the datasets were used for training, 10% for validation, and 20% for testing purposes. The CIFAR10, MNIST, and Mini-ImageNet datasets were also considered for performance evaluation. The proposed method performed better than other existing methods in terms of classification accuracy.

Phase Change Memory Drift Compensation in Spiking Neural Networks Using a Non-Linear Current Scaling Strategy

The non-ideality aspects of phase change memory (PCM) such as drift and resistance variability can pose significant obstacles in neuromorphic hardware implementations. A unique drift and variability compensation strategy is demonstrated and implemented in an FD-SOI SNN hardware unit composed of embedded phase change memories (ePCMs), current attenuators, and spiking neurons. The effect of drift and variability compensation on inference accuracy is tested on the MNIST dataset to show that our drift and variability mitigation strategy is effective in sustaining its accuracy over time. The variability is reduced by up to 5% while the drift coefficient is reduced by up to 57.8%. The drift is compensated and the SNN classification accuracy is sustained for up to 2 years with intrinsic control-free hardware that tracks the ePCM current over time and consumes less than 30 µW. The results are based on ePCM chip experimental data and pos-layout simulation of a test chip comprising the proposed circuit solution.

Ferrimagnet-Based Neuromorphic Device Mimicking the Ventral Visual Pathway for High-Accuracy Target Recognition.

The ventral visual pathway (VVP) of the human brain efficiently implements target recognition by employing a deep hierarchical structure to build complex visual concepts from simple features. Artificial neural networks (ANNs) based on spintronic devices are capable of target recognition, but their poor interpretability and limited network depth hinder ANNs from mimicking the VVP. Hardware implementation of the VVP requires a biorealistic spintronic device as well as the corresponding interpretable and deep network structure, which have not been reported so far. Here, we report a ferrimagnetic neuron with a continuously differentiable exponential linear unit (CeLu) activation function, which is closer to biological neurons and could mitigate the issue of limited network depth. Meanwhile, we also demonstrate that a ferrimagnet can construct artificial synapses with high linearity and symmetry to meet the requirements of weight update algorithms. Based on these neurons and synapses, we propose an all-spin convolutional neural network (CNN) with a high interpretability and deep neural network, to mimic the VVP. Compared to the state-of-the-art spintronic-based neuromorphic computing model, the CNN with bionic function, using experimentally derived device parameters, achieves high recognition accuracies of over 91% and 98% on the CIFAR-10 datasets and the MNIST datasets, respectively, showing improvements of 1.13% and 1.76%. Our work provides a promising method to improve the bionic performance of spintronic device-based neural networks.

Structure-Based Training: A Training Method Aimed at Pixel Errors for a Correlation-Coefficient-Based Neural Network.

In research on building a one-shot learning neural network without pre-training using mass data, the limitation on the information obtained from a single training sample downgrades the performance of the network. In order to improve performance and take full advantage of the support set, in this study, we design three kinds of shadow nodes and propose a structure-based training method for a correlation-coefficient-based neural network. This training strategy focuses on branches that are not activated or inactivated as expected. In contrast to existing networks that optimize the parameters using back-propagation, the training method proposed in this paper optimizes the structure of the correlation-coefficient-based network by correcting its pixel errors. For the shadow nodes and training process based on this strategy, the intersection over union (IOU) of a detected target increases by 4.83% in the experiments when using the Fashion-Mnist dataset, increases by 4.02% when using the Omniglot dataset, and increases by 3.89% when using the Cifar-10 dataset. The samples in category "7" wrongly classified as "1" decreased by 27.32% when using the Mnist dataset after training. This training strategy, along with shadow nodes, makes the correlation-coefficient-based network a more practical model and enables the network to develop during the accumulation of reliable samples, thus making it more suitable for simple target detection projects that collect samples over time. Moreover, the shadow nodes and training method proposed in this paper supplement the non-gradient-based parameter-gaining strategy. Additionally, it is a new attempt to explore the imitation of a human's ability to learn a new pattern from a low number of references.

Efficient training of energy-based models using Jarzynski equality**This article is an updated version of: Carbone D, Hua M, Coste S and Vanden-Eijnden E 2023 Efficient training of energy-based models using Jarzynski equality Advances in Neural Information Processing Systems vol 36, ed A Oh, T Naumann, A Globerson, K Saenko, M Hardt and S Levine (Curran Associates, Inc.) pp 52583–614.

Abstract Energy-based models (EBMs) are generative models inspired by statistical physics with a wide range of applications in unsupervised learning. Their performance is well measured by the cross-entropy (CE) of the model distribution relative to the data distribution. Using the CE as the objective for training is, however, challenging because the computation of its gradient with respect to the model parameters requires sampling of the model distribution. Here, we show how the results for nonequilibrium thermodynamics based on the Jarzynski equality together with tools from sequential Monte Carlo sampling can be used to perform this computation efficiently and avoid the uncontrolled approximations made using the standard contrastive divergence algorithm. Specifically, we introduce a modification of the unadjusted Langevin algorithm (ULA), in which each walker acquires a weight that enables the estimation of the gradient of the CE at any step during gradient descent, thereby bypassing sampling biases induced by slow mixing of the ULA. We illustrate these results with numerical experiments on Gaussian mixture distributions as well as the MNIST and CIFAR-10 datasets. We show that the proposed approach outperforms methods based on the contrastive divergence algorithm in all the considered situations.

A step function based recursion method for 0/1 deep neural networks

The deep neural network with step function activation (0/1 DNNs) is a fundamental composite model in deep learning which has high efficiency and robustness to outliers. However, due to the discontinuity and lacking subgradient information of the 0/1 DNNs model, prior researches are largely focused on designing continuous functions to approximate the step activation and developing continuous optimization methods. In this paper, by introducing two sets of network node variables into the 0/1 DNNs and by exploring the composite structure of the resulted model, the 0/1 DNNs is decomposed into a unary optimization model associated with the step function and three derivational optimization subproblems associated with the other variables. For the unary optimization model and two derivational optimization subproblems, we present a closed form solution, and for the third derivational optimization subproblem, we propose an efficient proximal method. Based on this, a globally convergent step function based recursion method for the 0/1 DNNs is developed. The efficiency and performance of the proposed algorithm are validated via theoretical analysis as well as some illustrative numerical examples on classifying MNIST, FashionMNIST and Cifar10 datasets.

ZnO-based artificial synaptic diodes with zero-read voltage for neural network computing

Brain-inspired neuromorphic sensory devices play a crucial role in addressing the limitations of von Neumann systems in contemporary computing. Currently, synaptic devices rely on memristors and thin-film transistors, requiring the establishment of a read voltage. A built-in electric field exists within the p–n junction, enabling the operation of zero-read-voltage synaptic devices. In this study, we propose an artificial synapse utilizing a ZnO diode. Typical rectification curves characterize the formation of ZnO diodes. ZnO diodes demonstrate distinct synaptic properties, including paired-pulse facilitation, paired-pulse depression, long-term potentiation, and long-term depression modulations, with a read voltage of 0 V. An artificial neural network is constructed to simulate recognition tasks using MNIST and Fashion-MNIST databases, achieving test accuracy values of 92.36% and 76.71%, respectively. This research will pave the way for advancing zero-read-voltage artificial synaptic diodes for neural network computing.

Leveraging Tunability of Localized-Interfacial Memristors for Efficient Handling of Complex Neural Networks.

A scalable (<130 nm) resistive switching memristor that features both filamentary and interfacial switching aimed at neuromorphic computing is developed in this study. The typically perceived noise or volatility was effectively harnessed as a controlled mechanism for interfacial switching. The multilayer structure for the proposed memristor enhances switching stability by curbing ionic overmigration and mitigating leakage paths. Furthermore, the memristors showcased their reliability by demonstrating more than 15 M cycles in the filamentary mode and 1 M pulses in the interfacial mode. Additionally, retention tests at 85 °C for 104 s confirmed the stability across different states, affirming its reliability as a nonvolatile CMOS-compatible element. While many studies validate performance solely on the MNIST data set, this work also evaluates more complex data sets, demonstrating the robustness of the demonstrated memristor in supervised learning. Specifically, supervised learning simulations on MNIST and fashion MNIST data sets indicated a high learning rate with <4% deviations from numerical training, while offline inference trained on CIFAR-10 and CIFAR-100 data sets revealed <2.5% and <7% deviations caused by programing error accumulation, even with increased memristor counts for these highly complex data sets. Unsupervised learning via spike-timing-dependent plasticity further highlights the potential of the developed memristor in bridging artificial and biological paradigms, offering a significant advance toward efficient and biologically inspired computing architectures.

Pruning convolution neural networks using filter clustering based on normalized cross-correlation similarity

ABSTRACT Despite all the recent development and success of deep neural networks, deployment of a deep model onto the resource-constrained devices still remains challenging. However, model pruning can resolve this issue for Convolutional Neural Networks (CNNs), since it is one of the most popular approaches to reducing computational complexities. Therefore, this article presents a pruning model for convolutional neural networks. The proposed method classifies and arranges similar filters into the same cluster where the similarity is calculated using a three-dimensional normalized cross-correlation. Moreover, these steps can be completed entirely based on the filter values while not requiring a set of test images as well as the acquisition of any filter activation. In the research, the performances of the proposed model pruning method have been evaluated, where it is observed that the proposed approach is computationally light and requires significantly less time and resources compared to ML and activation-based approaches. In the experiments, using the VGG16 model on the Cifar10 dataset, the proposed approach results in the pruned model(s) which are comparable in performance with models found using activation-based methods and expensive ML-based methods. Similar results are found when pruning a custom CNN on the MNIST and Fashion MNIST datasets as well.