Clustered and Multi-Tasked Federated Distillation for Heterogeneous and Resource Constrained Industrial IoT Applications

Abstract

Service heterogeneity, the diversity of services provided by different devices and systems in the Industrial Internet of Things (IIoT), makes communication and data exchange difficult and affects the IIoT performance. Artificial intelligence, particularly machine learning (ML), can address this challenge by analyzing data, predicting behavior, and developing self-configuring autonomic systems. However, incorporating ML into IIoT faces challenges such as high data complexity and variability, communication costs, lack of data privacy and security, and scalability. Federated multitask learning (FML) is a promising solution to tackle most of these challenges by training ML models locally and exchanging only the updated parameters. However, it still faces challenges in device and system compatibility, data heterogeneity, and scalability, especially for hierarchical heterogeneous IIoT environments. To address all these challenges, this article proposes a novel hybrid framework dubbed Clustered Multitask Federated Distillation (CMFD). In conjunction with the FML, CMFD combines clustered FL to address the data heterogeneity and knowledge distillation to address the system compatibility and scalability. To show the efficiency of CMFD for intelligent IIoT applications, we present a case study of heterogeneous IIoT environments using a realistic dataset. We conclude with useful insights emerging from our work and highlight promising future research directions.