MNIST Research Articles

Trainable Reference Spikes Improve Temporal Information Processing of SNNs With Supervised Learning.

Spiking neural networks (SNNs) are the next-generation neural networks composed of biologically plausible neurons that communicate through trains of spikes. By modifying the plastic parameters of SNNs, including weights and time delays, SNNs can be trained to perform various AI tasks, although in general not at the same level of performance as typical artificial neural networks (ANNs). One possible solution to improve the performance of SNNs is to consider plastic parameters other than just weights and time delays drawn from the inherent complexity of the neural system of the brain, which may help SNNs improve their information processing ability and achieve brainlike functions. Here, we propose reference spikes as a new type of plastic parameters in a supervised learning scheme in SNNs. A neuron receives reference spikes through synapses providing reference information independent of input to help during learning, whose number of spikes and timings are trainable by error backpropagation. Theoretically, reference spikes improve the temporal information processing of SNNs by modulating the integration of incoming spikes at a detailed level. Through comparative computational experiments, we demonstrate using supervised learning that reference spikes improve the memory capacity of SNNs to map input spike patterns to target output spike patterns and increase classification accuracy on the MNIST, Fashion-MNIST, and SHD data sets, where both input and target output are temporally encoded. Our results demonstrate that applying reference spikes improves the performance of SNNs by enhancing their temporal information processing ability.

Proton-gated organic thin-film transistors for leaky integrate-and-fire convolutional spiking neural networks

Artificial spiking neurons, integral to the functionality of spiking neural networks, are designed to mimic the information transmission via discrete spikes in biological nervous systems. Traditional approaches that necessitate the charging of capacitors and the inclusion of discharge circuits for neuron membrane potential integration and leakage, present challenges in terms of cost and space efficiency. To overcome the challenges, this work proposes a hardware leaky integrate-and-fire neuron based on organic thin-film transistors. Under the electric field, the ion dynamics in the gate electrolyte can mimic the processes of membrane potential integration, leakage, and reset in spiking neurons. The convolutional spiking neural networks composed of such organic spiking neurons achieves excellent recognition rates (∼97.26 %) on the MNIST dataset. This indicates that the organic spiking neuron has enormous potential in next-generation non-von Neumann neuromorphic computing.

Uncertainty-Aware Federated Reinforcement Learning for Optimizing Accuracy and Energy in Heterogeneous Industrial IoT

The Internet of Things (IoT) technology has revolutionized various industries by allowing data collection, analysis, and decision-making in real time through interconnected devices. However, challenges arise in implementing Federated Learning (FL) in heterogeneous industrial IoT environments, such as maintaining model accuracy with non-Independent and Identically Distributed (non-IID) datasets and straggler IoT devices, ensuring computation and communication efficiency, and addressing weight aggregation issues. In this study, we propose an Uncertainty-Aware Federated Reinforcement Learning (UA-FedRL) method that dynamically selects epochs of individual clients to effectively manage heterogeneous industrial IoT devices and improve accuracy, computation, and communication efficiency. Additionally, we introduce the Predictive Weighted Average Aggregation (PWA) method to tackle weight aggregation issues in heterogeneous industrial IoT scenarios by adjusting the weights of individual models based on their quality. The UA-FedRL addresses the inherent complexities and challenges of implementing FL in heterogeneous industrial IoT environments. Extensive simulations in complex IoT environments demonstrate the superior performance of UA-FedRL on both MNIST and CIFAR-10 datasets compared to other existing approaches in terms of accuracy, communication efficiency, and computation efficiency. The UA-FedRL algorithm attain an accuracy of 96.83% on the MNIST dataset and 62.75% on the CIFAR-10 dataset, despite the presence of 90% straggler IoT devices, attesting to its robust performance and adaptability in different datasets.

Defense against Adversarial Attacks in Image Recognition Based on Multilayer Filters

The security and privacy of a system are urgent issues in achieving secure and efficient learning-based systems. Recent studies have shown that these systems are susceptible to subtle adversarial perturbations applied to inputs. Although these perturbations are difficult for humans to detect, they can easily mislead deep learning classifiers. Noise injection, as a defense mechanism, can offer a provable defense against adversarial attacks by reducing sensitivity to subtle input changes. However, these methods face issues of computational complexity and limited adaptability. We propose a multilayer filter defense model, drawing inspiration from filter-based image denoising techniques. This model inserts a filtering layer after the input layer and before the convolutional layer, and incorporates noise injection techniques during the training process. This model substantially enhances the resilience of image classification systems to adversarial attacks. We also investigated the impact of various filter combinations, filter area sizes, standard deviations, and filter layers on the effectiveness of defense. The experimental results indicate that, across the MNIST, CIFAR10, and CIFAR100 datasets, the multilayer filter defense model achieves the highest average accuracy when employing a double-layer Gaussian filter (filter area size of 3×3, standard deviation of 1). We compared our method with two filter-based defense models, and the experimental results demonstrated that our method attained an average accuracy of 71.9%, effectively enhancing the robustness of the image recognition classifier against adversarial attacks. This method not only performs well on small-scale datasets but also exhibits robustness on large-scale datasets (miniImageNet) and modern models (EfficientNet and WideResNet).

Bidirectional Decoupled Distillation for Heterogeneous Federated Learning.

Federated learning enables multiple devices to collaboratively train a high-performance model on the central server while keeping their data on the devices themselves. However, due to the significant variability in data distribution across devices, the aggregated global model's optimization direction may differ from that of the local models, making the clients lose their personality. To address this challenge, we propose a Bidirectional Decoupled Distillation For Heterogeneous Federated Learning (BDD-HFL) approach, which incorporates an additional private model within each local client. This design enables mutual knowledge exchange between the private and local models in a bidirectional manner. Specifically, previous one-way federated distillation methods mainly focused on learning features from the target class, which limits their ability to distill features from non-target classes and hinders the convergence of local models. To solve this limitation, we decompose the network output into target and non-target class logits and distill them separately using a joint optimization of cross-entropy and decoupled relative-entropy loss. We evaluate the effectiveness of BDD-HFL through extensive experiments on three benchmarks under IID, Non-IID, and unbalanced data distribution scenarios. Our results show that BDD-HFL outperforms state-of-the-art federated distillation methods across five baselines, achieving at most 3% improvement in average classification accuracy on the CIFAR-10, CIFAR-100, and MNIST datasets. The experiments demonstrate the superiority and generalization capability of BDD-HFL in addressing personalization challenges in federated learning.

Tackling Non-IID Data and Data Poisoning in Federated Learning using Adversarial Synthetic Data

Federated learning (FL) involves joint model training by various devices while preserving the privacy of their data. However, it presents a challenge of dealing with heterogeneous data located on participating devices. This issue can further be complicated by the appearance of malicious clients, aiming to sabotage the training process by poisoning local data. In this context, a problem of differentiating between poisoned and non-IID data appears. To address it, a technique utilizing data-free synthetic data generation is proposed, using a reverse concept of adversarial attack. Adversarial inputs allow for improving the training process by measuring clients' coherence, favoring trustworthy participants. Experimental results, obtained from the image classification tasks for MNIST, EMNIST, and CIFAR-10 datasets are reported and analyzed.

Sparse Personalized Federated Learning.

Federated learning (FL) is a collaborative machine learning technique to train a global model (GM) without obtaining clients' private data. The main challenges in FL are statistical diversity among clients, limited computing capability among clients' equipment, and the excessive communication overhead between the server and clients. To address these challenges, we propose a novel sparse personalized FL scheme via maximizing correlation (FedMac). By incorporating an approximated l1 -norm and the correlation between client models and GM into standard FL loss function, the performance on statistical diversity data is improved and the communicational and computational loads required in the network are reduced compared with nonsparse FL. Convergence analysis shows that the sparse constraints in FedMac do not affect the convergence rate of the GM, and theoretical results show that FedMac can achieve good sparse personalization, which is better than the personalized methods based on the l2 -norm. Experimentally, we demonstrate the benefits of this sparse personalization architecture compared with the state-of-the-art personalization methods (e.g., FedMac, respectively, achieves 98.95%, 99.37%, 90.90%, 89.06%, and 73.52% accuracy on the MNIST, FMNIST, CIFAR-100, Synthetic, and CINIC-10 datasets under non-independent and identically distributed (i.i.d.) variants).

Identifying alternately poisoning attacks in federated learning online using trajectory anomaly detection method

Implicit poisoning in federated learning is a significant threat, with malicious nodes subtly altering gradient parameters each round, making detection difficult. This study investigates this problem, revealing that temporal analysis alone struggles to identify such covert attacks, which can bypass online methods like cosine similarity and clustering. Common detection methods rely on offline analysis, resulting in delayed responses. However, recalculating gradient updates reveals distinct characteristics of malicious clients. Based on this finding, we designed a privacy-preserving detection algorithm using trajectory anomaly detection. Singular values of matrices are used as features, and an improved Isolation Forest algorithm processes these to detect malicious behavior. Experiments on MNIST, FashionMNIST, and CIFAR-10 datasets show our method achieves 94.3% detection accuracy and a false positive rate below 1.2%, indicating its high accuracy and effectiveness in detecting implicit model poisoning attacks.

Implementing handwritten text recognition using deep learning with TensorFlow: An MNIST dataset approach

Handwritten text recognition (HTR) is a pivotal technology with extensive applications in document digitization, postal automation, and educational tools. This paper delves into the implementing a deep learning-based system for recognizing handwritten digits using TensorFlow and the MNIST dataset. The MNIST dataset, a widely-used benchmark, comprises 60,000 training images and 10,000 testing images of handwritten digits, each standardized to a 28x28 pixel grayscale format. Leveraging the power of Convolutional Neural Networks (CNNs), our model effectively extracts features and classifies digits with high accuracy. The model architecture consists of multiple convolutional layers for feature extraction, pooling layers for dimensionality reduction, and fully connected layers for classification. Preprocessing steps include normalizing pixel values and one-hot encoding the labels, ensuring the data is optimally formatted for training. The TensorFlow framework, known for its robustness and scalability, facilitates the development and deployment of the model. Through a series of experiments, the model demonstrates impressive performance, achieving high accuracy on the MNIST test set. This paper underscores the potential of deep learning in handwritten text recognition. It sets the stage for future enhancements, such as recognizing more complex handwritten texts and optimizing the system for practical applications. The results highlight the effectiveness of deep learning techniques in overcoming the challenges associated with handwritten text recognition, paving the way for advanced, real-world implementations.

Image reconstruction through a nonlinear scattering medium via deep learning

Image reconstruction through the opaque medium has great significance in fields of biophotonics, optical imaging, mesoscopic physics, and optical communications. Previous researches are limited in the simple linear scattering process. Here, we develop a nonlinear speckle decoder network, which can reconstruct the phase information of the fundamental frequency wave via the nonlinear scattering signal. Further, we validate the ability of our model to recover simple and complex structures by using MNIST and CIFAR data sets, respectively. We then show that the model is able to restore the image information through different sets of nonlinear diffusers and reconstruct the image of a kind of completely unseen object category. The proposed method paves the way to nonlinear scattering imaging and information encryption.

Unrecognizable yet identifiable: Image distortion with preserved embeddings

Biometric authentication systems play a crucial role in modern security systems. However, maintaining the balance of privacy and integrity of stored biometrics derivative data while achieving high recognition accuracy is often challenging. Addressing this issue, we introduce an innovative image transformation technique that effectively renders facial images unrecognizable to the eye while maintaining their identifiability by neural network models, which allows the distorted photo version to be stored for further verification. While initially intended for biometrics systems, the proposed methodology can be used in various artificial intelligence applications to distort the visual data and keep the derived features close. By experimenting with widely used datasets LFW and MNIST, we show that it is possible to build the distortion that changes the image content by more than 70% while maintaining the same recognition accuracy. We compare our method with previously state-of-the-art approaches. We publically release the source code.11https://github.com/ZamDimon/distortion-generator/tree/v1.0.0.

Forecasting the Right Crop Nutrients for Specific Crops Based on Collected Data Using an Artificial Neural Network (ANN)

In farming technologies, it is difficult to properly provide the accurate crop nutrients for respective crops. For this reason, farmers are experiencing enormous problems. Although various types of machine learning (deep learning and convolutional neural networks) have been used to identify crop diseases, as has crop classification-based image processing, they have failed to forecast accurate crop nutrients for various crops, as crop nutrients are numerical instead of visual. Neural networks represent an opportunity for the precision agriculture sector to more accurately forecast crop nutrition. Recent technological advancements in neural networks have begun to provide greater precision, with an array of opportunities in pattern recognition. Neural networks represent an opportunity to effectively solve numerical data problems. The aim of the current study is to estimate the right crop nutrients for the right crops based on the data collected using an artificial neural network. The crop data were collected from the MNIST dataset. To forecast the precise nutrients for the crops, ANN models were developed. The entire system was simulated in a MATLAB environment. The obtained results for forecasting accurate nutrients were 99.997%, 99.996%, and 99.997% for validation, training, and testing, respectively. Therefore, the proposed algorithm is suitable for forecasting accurate crop nutrients for the crops.

Single-pixel object classification using ordered illumination patterns

The fixed-ordering illumination patterns lack an effective information interchange with objects. This leads to the light intensity values carrying random feature information, affecting the model's building and recognition capability. Furthermore, this method needs numerous illumination pattern projection counts, which results in low sampling efficiency. Therefore, we propose a new optimized-ordering illumination patterns method. In this paper, we first integrate fixed-ordering illumination patterns with category-representative images to obtain light intensity values that carry category feature information. Subsequently, the mechanisms of intra-category feature importance and inter-category feature differences are introduced to reorder the light intensity values and eliminate index numbers to generate an optimized-ordering index array. Finally, based on this index array, illumination patterns are extracted from the original fixed ordering method to form the new optimized-ordering illumination patterns. Experimental results show that in simulation experiments using the MNIST, Fashion MNIST, TRI, and Multi-category datasets, the optimized ordering method achieved recognition accuracies of 90.00%, 80.00%, 80.00%, and 80.00% with only 30 sampling times. Actual tests with physical objects such as mini numbers and clothing also confirmed these findings. The method offers a new research approach by enhancing recognition accuracy and sampling efficiency in single-pixel object classification.

Design of Diffractive Neural Networks for Solving Different Classification Problems at Different Wavelengths

We consider the problem of designing a diffractive neural network (DNN) consisting of a set of sequentially placed phase diffractive optical elements (DOEs) and intended for the optical solution of several given classification problems at different operating wavelengths, so that each classification problem is solved at the corresponding wavelength. The problem of calculating the DNN is formulated as the problem of minimizing a functional that depends on the functions of the diffractive microrelief height of the DOEs constituting the DNN and represents the error in solving the given classification problems at the operating wavelengths. We obtain explicit and compact expressions for the derivatives of this functional, and using them, we formulate a gradient method for the DNN calculation. Using this method, we design DNNs for solving the following three classification problems at three different wavelengths: the problem of classifying handwritten digits from the MNIST database, the problem of classifying fashion products from the Fashion MNIST database, and the problem of classifying ten handwritten letters from the EMNIST database. The presented simulation results of the designed DNNs demonstrate the high performance of the proposed method.

Semantic memory-based dynamic neural network using memristive ternary CIM and CAM for 2D and 3D vision.

The brain is dynamic, associative, and efficient. It reconfigures by associating the inputs with past experiences, with fused memory and processing. In contrast, AI models are static, unable to associate inputs with past experiences, and run on digital computers with physically separated memory and processing. We propose a hardware-software co-design, a semantic memory-based dynamic neural network using a memristor. The network associates incoming data with the past experience stored as semantic vectors. The network and the semantic memory are physically implemented on noise-robust ternary memristor-based computing-in-memory (CIM) and content-addressable memory (CAM) circuits, respectively. We validate our co-designs, using a 40-nm memristor macro, on ResNet and PointNet++ for classifying images and three-dimensional points from the MNIST and ModelNet datasets, which achieves not only accuracy on par with software but also a 48.1 and 15.9% reduction in computational budget. Moreover, it delivers a 77.6 and 93.3% reduction in energy consumption.

Ultralow Power In-Sensor Neuronal Computing with Oscillatory Retinal Neurons for Frequency-Multiplexed, Parallel Machine Vision.

In-sensor and near-sensor computing architectures enable multiply accumulate operations to be carried out directly at the point of sensing. In-sensor architectures offer dramatic power and speed improvements over traditional von Neumann architectures by eliminating multiple analog-to-digital conversions, data storage, and data movement operations. Current in-sensor processing approaches rely on tunable sensors or additional weighting elements to perform linear functions such as multiply accumulate operations as the sensor acquires data. This work implements in-sensor computing with an oscillatory retinal neuron device that converts incident optical signals into voltage oscillations. A computing scheme is introduced based on the frequency shift of coupled oscillators that enables parallel, frequency multiplexed, nonlinear operations on the inputs. An experimentally implemented 3 × 3 focal plane array of coupled neurons shows that functions approximating edge detection, thresholding, and segmentation occur in parallel. An example of inference on handwritten digits from the MNIST database is also experimentally demonstrated with a 3 × 3 array of coupled neurons feeding into a single hidden layer neural network, approximating a liquid-state machine. Finally, the equivalent energy consumption to carry out image processing operations, including peripherals such as the Fourier transform circuits, is projected to be <20 fJ/OP, possibly reaching as low as 15 aJ/OP.

Content-based image retrieval by classification with reinforcement optimisation evolutionary machine learning with applications

ABSTRACT Content Based Image Retrieval (CBIR) plays a significant role in identifying the similarity of images with large datasets. It is identified based on the size, colour, and texture features of the image. But in such conditions, it is complex to determine the features of query images in large datasets and does not show accurate similarity when compared with every image in the retrieval process. In order to perform an efficient similarity of images, a novel Machine Learning (ML) approach Kernelized Radial Basis Auto-Encoder Function Neural Network (Ker_RadBAEFNN) technique is proposed that performs the individual image classification in the retrieval process. Moreover, the neural networks are optimised based on the reinforcement process and perform the extraction process regarding individual images. Further,reinforcement-based optimisation estimates the images in neural networks for undertaking an automatic feature extraction of query images. The performance of the classification process is validated based on MNIST, METU, and COCO datasets that determined the efficiency of the recognition and classification process of image retrieval. The experimental analysis is carried out based on various measures such as accuracy, precision, recall, F1-score, RMSE, and MAPE for the proposed and existing GLCM-ABC, PSO-ANN, IRB-CNN, FAGWO, and OCAM methods. The analysis shows that the performance of the proposed attained better effectiveness with attained accuracy by 98% and diminished for state-of-the-art techniques as 92%, 95%, 94%, 96.8%, as well as 96%, respectively. Compared to existing methods, the accuracy rate of the proposed method is maximised by 1.3%.

High-performance deep spiking neural networks with 0.3 spikes per neuron

Communication by rare, binary spikes is a key factor for the energy efficiency of biological brains. However, it is harder to train biologically-inspired spiking neural networks than artificial neural networks. This is puzzling given that theoretical results provide exact mapping algorithms from artificial to spiking neural networks with time-to-first-spike coding. In this paper we analyze in theory and simulation the learning dynamics of time-to-first-spike-networks and identify a specific instance of the vanishing-or-exploding gradient problem. While two choices of spiking neural network mappings solve this problem at initialization, only the one with a constant slope of the neuron membrane potential at threshold guarantees the equivalence of the training trajectory between spiking and artificial neural networks with rectified linear units. For specific image classification architectures comprising feed-forward dense or convolutional layers, we demonstrate that deep spiking neural network models can be effectively trained from scratch on MNIST and Fashion-MNIST datasets, or fine-tuned on large-scale datasets, such as CIFAR10, CIFAR100 and PLACES365, to achieve the exact same performance as that of artificial neural networks, surpassing previous spiking neural networks. Our approach accomplishes high-performance classification with less than 0.3 spikes per neuron, lending itself for an energy-efficient implementation. We also show that fine-tuning spiking neural networks with our robust gradient descent algorithm enables their optimization for hardware implementations with low latency and resilience to noise and quantization.

A Gastrointestinal Image Classification Method Based on Improved Adam Algorithm

In this study, a gastrointestinal image classification method based on the improved Adam algorithm is proposed. Gastrointestinal image classification is of great significance in the field of medical image analysis, but it presents numerous challenges, including slow convergence, susceptibility to local minima, and the complexity and imbalance of medical image data. Although the Adam algorithm is widely used in stochastic gradient descent, it tends to suffer from overfitting and gradient explosion issues when dealing with complex data. To address these problems, this paper proposes an improved Adam algorithm, AdamW_AGC, which combines the weight decay and Adaptive Gradient Clipping (AGC) strategies. Weight decay is a common regularization technique used to prevent machine learning models from overfitting. Adaptive gradient clipping avoids the gradient explosion problem by restricting the gradient to a suitable range and helps accelerate the convergence of the optimization process. In order to verify the effectiveness of the proposed algorithm, we conducted experiments on the HyperKvasir dataset and validation experiments on the MNIST and CIFAR10 standard datasets. Experimental results on the HyperKvasir dataset demonstrate that the improved algorithm achieved a classification accuracy of 75.8%, compared to 74.2% for the traditional Adam algorithm, representing an improvement of 1.6%. Furthermore, validation experiments on the MNIST and CIFAR10 datasets resulted in classification accuracies of 98.69% and 71.7%, respectively. These results indicate that the AdamW_AGC algorithm has advantages in handling complex, high-dimensional medical image classification tasks, effectively improving both classification accuracy and training stability. This study provides new ideas and expansions for future optimizer research.

S-Tune: SOT-MTJ manufacturing parameters tuning for securing the next generation of computing

Hardware-based acceleration approaches for Machine Learning (ML) workloads have been embracing the significant potential of post-CMOS switching devices to attain reduced footprint and/or energy-efficient execution relative to transistor-based GPU and/or TPU-based accelerator architectures. Meanwhile, the promulgation of fabless IC chip manufacturing paradigms has heightened the hardware security concerns inherent in such approaches. Namely, unauthorized access to various supply chain stages may expose significant vulnerabilities resulting in malfunctions including subtle adversarial outcomes via the malicious generation of differentially-corrupted outputs. Whereas the Spin-Orbit Torque Magnetic Tunnel Junction (SOT-MTJ) is a leading spintronic device for use in ML accelerators, as well as holding security tokens, their manufacturing-only security exposures are identified and evaluated herein. Results indicate a novel vulnerability profile whereby an adversary without access to the circuit netlist could differentially-influence the machine learning application’s behavior. Specifically, ML recognition outputs can be significantly swayed via a global modification of oxide thickness (Tox) resulting in bit-flips of the weights in the crossbar array, thus corrupting the recognition of selected digits in MNIST dataset differentially creating an opportunity for an adversary. With just 0.05% of bits in crossbar having a flipped resistance state, digits “4” and “5” show the highest overall error rates, and digit “9” exhibit the lowest impact, with recognition accuracy of digits “2,” “3,” and “8” unaffected by changing the oxide thickness of SOT-MTJs uniformly from 0.75 nm to 1.2 nm without modifying the netlist nor even having access to the circuit design itself. Exposures and mitigation approaches to such novel and potentially damaging manufacturing-side intrusions are identified, postulated, and quantitatively assessed.