On-device Training Research Articles

A Joint Survey in Decentralized Federated Learning and TinyML: A Brief Introduction to Swarm Learning

TinyML/DL is a new subfield of ML that allows for the deployment of ML algorithms on low-power devices to process their own data. The lack of resources restricts the aforementioned devices to running only inference tasks (static TinyML), while training is handled by a more computationally efficient system, such as the cloud. In recent literature, the focus has been on conducting real-time on-device training tasks (Reformable TinyML) while being wirelessly connected. With data processing being shift to edge devices, the development of decentralized federated learning (DFL) schemes becomes justified. Within these setups, nodes work together to train a neural network model, eliminating the necessity of a central coordinator. Ensuring secure communication among nodes is of utmost importance for protecting data privacy during edge device training. Swarm Learning (SL) emerges as a DFL paradigm that promotes collaborative learning through peer-to-peer interaction, utilizing edge computing and blockchain technology. While SL provides a robust defense against adversarial attacks, it comes at a high computational expense. In this survey, we emphasize the current literature regarding both DFL and TinyML/DL fields. We explore the obstacles encountered by resource-starved devices in this collaboration and provide a brief overview of the potential of transitioning to Swarm Learning.

A Platform of Federated Learning Management for Enhanced Mobile Collaboration

Federated learning (FL) has emerged as a crucial technology in today’s data-centric environment, enabling decentralized machine learning while safeguarding user privacy. This study introduces “Federated Learning ML Operations (FedOps) Mobile”, a novel FL framework optimized for the dynamic and heterogeneous ecosystem of mobile devices. FedOps Mobile addresses the inherent challenges of FL—such as system scalability, device heterogeneity, and operational efficiency—through advanced on-device training using TensorFlow Lite and CoreML. The framework’s innovative approach includes sophisticated client selection mechanisms that assess device readiness and capabilities, ensuring equitable and efficient participation across the network. Additionally, FedOps Mobile leverages remote device control for seamless task management and continuous learning, all without compromising the user experience. The main contribution of this study is the demonstration that federated learning across heterogeneous devices, especially those using different operating systems, can be both practical and efficient using the FedOps Mobile framework. This was validated through experiments that evaluated three key areas: operational efficiency, model personalization, and resource optimization in multi-device settings. The results showed that the proposed method excels in client selection, energy consumption, and model optimization, establishing a new benchmark for federated learning in diverse and complex environments.

On-device Training: A First Overview on Existing Systems

The recent breakthroughs in machine learning (ML) and deep learning (DL) have catalyzed the design and development of various intelligent systems over wide application domains. While most existing machine learning models require large memory and computing power, efforts have been made to deploy some models on resource-constrained devices as well. A majority of the early application systems focused on exploiting the inference capabilities of ML and DL models, where data captured from different mobile and embedded sensing components are processed through these models for application goals such as classification and segmentation. More recently, the concept of exploiting the mobile and embedded computing resources for ML/DL model training has gained attention, as such capabilities allow (i) the training of models via local data without the need to share data over wireless links, thus enabling privacy-preserving computation by design, (ii) model personalization and environment adaptation, and (iii) deployment of accurate models in remote and hardly accessible locations without stable internet connectivity. This work summarizes and analyzes state-of-the-art systems research that allows such on-device model training capabilities and provides a survey of on-device training from a systems perspective.

An adaptive binary classifier for highly imbalanced datasets on the Edge

Edge machine learning brings intelligence to low-power devices at the periphery of a network. By running machine learning algorithms on the Edge, classification can be performed faster without the need to transmit large data volumes across a network. However, on-device training is often not feasible since Edge devices have limited computing and storage resources. Improved, Scalable, Efficient, and Fast classifieR (iSEFR) is a classifier that performs both training and testing on low-power devices using linearly separable balanced datasets. The novelty of this work is the improvement of the iSEFR accuracy by fine-tuning the algorithm with datasets having an uneven class distribution. Three adaptive linear function transformation techniques were proposed to improve the decision threshold which is in the form of a linear function. Experiments using stratified sampling with 5-fold cross-validation demonstrate that one of the proposed techniques significantly improved F1-score, Recall and Matthews Correlation Coefficient (MCC) by an average of 23%, 35% and 21% compared to iSEFR. Further evaluation of this technique in a Fog environment using highly imbalanced datasets such as credit card fraud, network intrusion and diabetic retinopathy also showed a significant increase of 38%, 44% and 30% in F1-score, Recall and MCC with a Precision of 97%. The adaptive binary classifier maintained the time complexity of iSEFR without altering the class imbalance.

On-device training of artificial intelligence models on microcontrollers

<p><span>Numerous studies are currently training artificial intelligence (AI) models on tiny devices constrained by computing power and memory limitations by implementing model optimization algorithms. The question arises whether implementing traditional AI models directly on small devices like micro-controller units (MCUs) is feasible. In this study, a library has been developed to train and predict the artificial neural network (ANN) model on common MCUs. The evaluation results on the regression problem indicate that, despite the extensive training time, when combined with multitasking programming on multi-core MCUs, the training does not adversely affect the system's execution. This research contributes an additional solution that enables the direct construction of ANN models on MCU systems with limited resources.</span></p>

Dynamic Bandwidth Slicing in Passive Optical Networks to Empower Federated Learning.

Federated Learning (FL) is a decentralized machine learning method in which individual devices compute local models based on their data. In FL, devices periodically share newly trained updates with the central server, rather than submitting their raw data. The key characteristics of FL, including on-device training and aggregation, make it interesting for many communication domains. Moreover, the potential of new systems facilitating FL in sixth generation (6G) enabled Passive Optical Networks (PON), presents a promising opportunity for integration within this domain. This article focuses on the interaction between FL and PON, exploring approaches for effective bandwidth management, particularly in addressing the complexity introduced by FL traffic. In the PON standard, advanced bandwidth management is proposed by allocating multiple upstream grants utilizing the Dynamic Bandwidth Allocation (DBA) algorithm to be allocated for an Optical Network Unit (ONU). However, there is a lack of research on studying the utilization of multiple grant allocation. In this paper, we address this limitation by introducing a novel DBA approach that efficiently allocates PON bandwidth for FL traffic generation and demonstrates how multiple grants can benefit from the enhanced capacity of implementing PON in carrying out FL flows. Simulations conducted in this study show that the proposed solution outperforms state-of-the-art solutions in several network performance metrics, particularly in reducing upstream delay. This improvement holds great promise for enabling real-time data-intensive services that will be key components of 6G environments. Furthermore, our discussion outlines the potential for the integration of FL and PON as an operational reality capable of supporting 6G networking.

Energy-Efficient Edge and Cloud Image Classification with Multi-Reservoir Echo State Network and Data Processing Units.

In an era dominated by Internet of Things (IoT) devices, software-as-a-service (SaaS) platforms, and rapid advances in cloud and edge computing, the demand for efficient and lightweight models suitable for resource-constrained devices such as data processing units (DPUs) has surged. Traditional deep learning models, such as convolutional neural networks (CNNs), pose significant computational and memory challenges, limiting their use in resource-constrained environments. Echo State Networks (ESNs), based on reservoir computing principles, offer a promising alternative with reduced computational complexity and shorter training times. This study explores the applicability of ESN-based architectures in image classification and weather forecasting tasks, using benchmarks such as the MNIST, FashionMnist, and CloudCast datasets. Through comprehensive evaluations, the Multi-Reservoir ESN (MRESN) architecture emerges as a standout performer, demonstrating its potential for deployment on DPUs or home stations. In exploiting the dynamic adaptability of MRESN to changing input signals, such as weather forecasts, continuous on-device training becomes feasible, eliminating the need for static pre-trained models. Our results highlight the importance of lightweight models such as MRESN in cloud and edge computing applications where efficiency and sustainability are paramount. This study contributes to the advancement of efficient computing practices by providing novel insights into the performance and versatility of MRESN architectures. By facilitating the adoption of lightweight models in resource-constrained environments, our research provides a viable alternative for improved efficiency and scalability in modern computing paradigms.

Memory-Efficient Batch Normalization by One-Pass Computation for On-Device Training

Memory-Efficient Batch Normalization by One-Pass Computation for On-Device Training

Biological plausible algorithm for seizure detection: Toward AI-enabled electroceuticals at the edge

Nearly 1% of people worldwide suffer from epilepsy. Electroencephalogram (EEG)-based diagnostics and monitoring tools, such as scalp EEG, subscalp EEG, stereo EEG, or sub/epi-dural EEG recordings [also known as electrocorticography (ECoG)], are widely used in different settings as the gold standard techniques to perform seizure identification, localization, and more primarily in epilepsy or suspected epilepsy in patients. Techniques such as subscalp EEG and ECoG offer long-term brain interaction, potentially replacing traditional electroceuticals with smart closed-loop therapies. However, these systems require continuous on-device training due to real-time demands and high power consumption. Inspired by the brain architecture, biologically plausible algorithms, such as some neuromorphic computing, show promise in addressing these challenges. In our research, we utilized liquid time-constant spiking neural networks with forward propagation through time to detect seizures in scalp-EEG. We trained and validated our model on the Temple University Hospital dataset and tested its generalization on out-of-sample data from the Royal Prince Alfred Hospital (RPAH) and EPILEPSIAE datasets. Our model achieved high area under the receiver operating characteristic curve (AUROC) scores of 0.83 in both datasets. We assessed the robustness by decreasing the memory size by 90% and obtained an overall AUROC of 0.82 in the RPAH dataset and 0.83 in the EPILEPSIAE dataset. Our model showed outstanding results of 3.1 μJ power consumption per inference and a 20% firing rate during training. This allows for incorporating bio-inspired efficient algorithms for on-device training, tackling challenges such as memory, power consumption, and efficiency.

On-device edge-learning for cardiac abnormality detection using a bio-inspired and spiking shallow network

This work introduces on-device edge learning for cardiac abnormality detection by merging spiking 2D Convolutional Long-Short-Term Memory (ConvLSTM2D) with a bio-inspired shallow neural network, referred to as Closed-form Continuous-time (CfC), to form the sCCfC model. The model achieves an F1 score and AUROC of 0.82 and 0.91 in cardiac abnormalities detection. These results are comparable to the non-spiking ConvLSTM2D–CfC (ConvCfC) model [Huang et al., J. Cardiovasc. Transl. Res. (published online, 2024)]. Notably, the sCCfC model demonstrates a significantly higher energy efficiency with an estimated power consumption of 4.68 μJ/Inf (per inference) on an emulated Loihi’s neuromorphic chip architecture, in contrast to ConvCfC model’s consumption of 450 μJ/Inf on a conventional processor. In addition, as a proof-of-concept, we deployed the sCCfC model on the conventional and relatively resource-constrained Radxa Zero, which is equipped with an Amlogic S905Y2 processor for on-device training, which resulted in performance improvements. After initial training of two epochs on a conventional Graphics Processing Unit, the F1 score and AUROC improved from 0.46 and 0.65 to 0.56 and 0.73, respectively, with five additional epochs of on-device training. Furthermore, when presented with a new dataset, the sCCfC model showcases strong out-of-sample generalization capabilities that can constitute a pseudo-perspective test, achieving an F1 score and AUROC of 0.71 and 0.86, respectively. The spiking sCCfC also outperforms the non-spiking ConvCfC model in robustness regarding effectively handling missing electrocardiogram (ECG) channels during inference. The model’s efficacy extends to single-lead ECG analysis, demonstrating reasonable accuracy in this context, while the focus of our work has been on the computational and memory complexities of the model.

Energy-Efficient Federated Training on Mobile Device

On-device deep learning technology has attracted increasing interest recently. CPUs are the most common commercial hardware on devices and many training libraries have been developed and optimized for them. However, CPUs still suffer from poor training performance (i.e., training time) due to the specific asymmetric multiprocessor. Moreover, the energy constraint imposes restrictions on battery-powered devices. With federated training, we expect the local training to be completed rapidly therefore the global model converges fast. At the same time, energy consumption should be minimized to avoid compromising the user experience. To this end, we consider energy and training time and propose a novel framework with a machine learning-based adaptive configuration allocation strategy, which chooses optimal configuration combinations for efficient ondevice training. We carry out experiments on the popular library MNN and the experimental results show that the adaptive allocation algorithm reduces substantial energy consumption, compared to all batches with fixed configurations on off-the-shelf CPUs.

Personalized Human Activity Recognition: Real-time On-device Training and Inference

Personalized Human Activity Recognition: Real-time On-device Training and Inference

On-Device Training of Fully Quantized Deep Neural Networks on Cortex-M Microcontrollers

On-Device Training of Fully Quantized Deep Neural Networks on Cortex-M Microcontrollers

Toward Smart and Efficient Service Systems: Computational Layered Federated Learning Framework

As increasing concerns have arisen on privacy leakage in data-driven smart services, federated learning (FL) has been introduced to collaboratively learn an efficient model across multiple participants without sharing raw data. However, existing FL schemes hinge on participating nodes to perform intensive ondevice training and network communication, which is a significant burden for energy-constrained mobile devices. In this work, we present the computational layered FL (CLFL) framework to enable resource-constrained devices to perform computationefficient on-device training and lightweight message transmitting. We first introduce the network structure and key aspects of the entities in the framework. Then, we give the implementation principles of CLFL, and present two instance schemes that allow devices to participate in joint training without the need for direct gradient computation or continuous data transmission. In order to more intuitively reflect the performance and efficiency of the proposed methods, we carry out a preliminary implementation and give the comparison with traditional FL. Finally, for the future exploration, we present four related research challenges of CLFL and offer possible solutions.

PePTM: An Efficient and Accurate Personalized P2P Learning Algorithm for Home Thermal Modeling

Nowadays, the integration of home automation systems with smart thermostats is a common trend, designed to enhance resident comfort and conserve energy. The introduction of smart thermostats that can run machine learning algorithms has opened the door for on-device training, enabling customized thermal experiences in homes. However, leveraging the flexibility offered by on-device learning has been hindered by the absence of a tailored learning scheme that allows for accurate on-device training of thermal models. Traditional centralized learning (CL) and federated learning (FL) schemes rely on a central server that controls the learning experience, compromising the home’s privacy and requiring significant energy to operate. To address these challenges, we propose PePTM, a personalized peer-to-peer thermal modeling algorithm that generates tailored thermal models for each home, offering a controlled learning experience with a minimal training energy footprint while preserving the home’s privacy, an aspect difficult to achieve in both CL and FL. PePTM consists of local and collaborative learning phases that enable each home to train its thermal model and collaboratively improve it with a set of similar homes in a peer-to-peer fashion. To showcase the effectiveness of PePTM, we use a year’s worth of data from US homes to train thermal models using the RNN time-series model and compare the data across three learning schemes: CL, FL, and PePTM, in terms of model performance and the training energy footprint. Our experimental results show that PePTM is significantly energy-efficient, requiring 695 and 40 times less training energy than CL and FL, respectively, while maintaining comparable performance. We believe that PePTM sets the stage for new avenues for on-device thermal model training, providing a personalized thermal experience with reduced energy consumption and enhanced privacy.

RedMule: A mixed-precision matrix–matrix operation engine for flexible and energy-efficient on-chip linear algebra and TinyML training acceleration

The increasing interest in TinyML, i.e., near-sensor machine learning on power budgets of a few tens of mW, is currently pushing toward enabling TinyML-class training as opposed to inference only. Current training algorithms, based on various forms of error and gradient backpropagation, rely on floating-point matrix operations to meet the precision and dynamic range requirements. So far, the energy and power cost of these operations has been considered too high for TinyML scenarios. This paper addresses the open challenge of near-sensor training on a few mW power budget and presents RedMulE — Reduced-Precision Matrix Multiplication Engine, a low-power specialized accelerator conceived for multi-precision floating-point General Matrix–Matrix Operations (GEMM-Ops) acceleration, supporting FP16, as well as hybrid FP8 formats, with {sign,exponent,mantissa}=({1,4,3},{1,5,2}). We integrate RedMule into a Parallel Ultra-Low-Power (PULP) cluster containing eight energy-efficient RISC-V cores sharing a tightly-coupled data memory and implement the resulting system in a 22 nm technology. At its best efficiency point (@ 470 MHz, 0.65 V), the RedMulE-augmented PULP cluster achieves 755 GFLOPS/W and 920 GFLOPS/W during regular General Matrix–Matrix Multiplication (GEMM), and up to 1.19 TFLOPS/W and 1.67 TFLOPS/W when executing GEMM-Ops, respectively, for FP16 and FP8 input/output tensors. In its best performance point (@ 613 MHz, 0.8 V), RedMulE achieves up to 58.5 GFLOPS and 117 GFLOPS for FP16 and FP8, respectively, with 99.4% utilization of the array of Computing Elements and consuming less than 60 mW on average, thus enabling on-device training of deep learning models in TinyML application scenarios while retaining the flexibility to tackle other classes of common linear algebra problems efficiently.

Skin Lesion Intelligent Diagnosis in Edge Computing Networks: An FCL Approach

In recent years, automatic skin lesion diagnosis methods based on artificial intelligence have achieved great success. However, the lack of labeled data, visual similarity between skin diseases, and restriction on private data sharing remain the major challenges in skin lesion diagnosis. In this article, first, we propose a federated contrastive learning framework to break down data silos and enhance the generalizability of diagnostic model to unseen data. Subsequently, by combining data features from different participated nodes, the proposed framework can improve the performance of contrastive training. To extract discriminative features during on-device training, we propose a contrastive learning based intelligent skin lesion diagnosis scheme in edge computing networks. Specifically, a contrastive learning based dual encoder network is designed to overcome training sample scarcity by fully leveraging unlabeled samples for performance improvement. Meanwhile, we devise a maximum mean discrepancy based supervised contrastive loss function, which can efficiently explore complex intra-class and inter-class variances of samples. Finally, the diagnosis simulations demonstrate that compared with existing methods, our proposed scheme can achieve superior accuracy in both on-device training and distributed training scenarios.

Embedded federated learning over a LoRa mesh network

In on-device training of machine learning models on microcontrollers a neural network is trained on the device. A specific approach for collaborative on-device training is federated learning. In this paper, we propose embedded federated learning on microcontroller boards using the communication capacity of a LoRa mesh network. We apply a dual board design: The machine learning application that contains a neural network is trained for a keyword spotting task on the Arduino Portenta H7. For the networking of the federated learning process, the Portenta is connected to a TTGO LORA32 board that operates as a router within a LoRa mesh network. We experiment the federated learning application on the LoRa mesh network and analyze the network, system, and application level performance. The results from our experimentation suggest the feasibility of the proposed system and exemplify an implementation of a distributed application with re-trainable compute nodes, interconnected over LoRa, entirely deployed at the tiny edge.

ACBN: Approximate Calculated Batch Normalization for Efficient DNN On-Device Training Processor

Batch normalization (BN) has been established as a very effective component in deep learning, largely helping accelerate the convergence of deep neural network (DNN) training. Nevertheless, its hardware architecture has not received much attention in the field of DNN on-device training processors. Several previous designs incur either high off-chip memory traffic or high circuit complexity, and hence have deficiencies in terms of hardware efficiency and performance. This article proposes approximately calculated BN (ACBN) to achieve a much better tradeoff between hardware efficiency and performance for DNN on-device training processors. The accuracy and convergence rate of the proposed ACBN have been extensively evaluated using four typical DNN models. Compared with the state-of-the-art reference design, the hardware simulation results show the proposed ACBN can at least reduce floating point operations by 22.2% and save external memory access by 33.3% on average. Moreover, the proposed ACBN introduces 63.6% data sparsity for the backward propagation of BN layers of VGG16 on average. To the best of our knowledge, we are the first to introduce data sparsity for the backward propagation of BN layers. The ACBN module is implemented on Zynq UltraScale <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$+$</tex-math> </inline-formula> ZCU102 system-on-chip (SoC) field-programmable gate array (FPGA), and the results show that the implementation of ACBN hardware module saves 33.9% look-up table (LUT), 49.4% flip-flop (FF), 75% digital signal processor (DSP), and reduces the power by 12.4% compared with the reference design while achieving better performance.

Accelerating On-Device DNN Training Workloads via Runtime Convergence Monitor

With the growing demand for processing deep learning applications on edge devices, on-device DNN training has become a major workload to execute a variety of vision tasks suited for users. Therefore, architectures employed with algorithm co-design to accelerate the training process have been steadily studied. However, previous solutions are mostly supported by extended versions of the inference studies, such as sparsity, data flow, quantization, etc. Moreover, most works examine their schemes on from-the-scratch training that cannot tolerate inaccurate computing. Accordingly, there are still factors that hinder the overall speed of the DNN training process that has not been addressed in practical workloads. In this work, we propose a runtime convergence monitor to achieve massive computational savings in the practical on-device training workloads (i.e., transfer learning-based task adaptation). By monitoring the network output data, we determine the training intensity of incoming tasks and adaptively detect the convergence in iteration intervals for training diverse datasets. Furthermore, we enable computation skip of converged images determined by the monitored prediction probability to enhance the training speed within an iteration. As a result, we perform an accurate but fast convergence in model training for the task adaptation with minimal overhead. Unlike the previous approximation methods, our monitoring system enables runtime optimization and can be easily applicable to any type of accelerator attaining significant speedup. Evaluation results on various datasets show geomean of 2.2× speedup when applied in any systolic architectures and further enhancement of 3.6× when applied in accelerators dedicated for on-device training.