Client Clusters Research Articles

Adaptive Client Clustering for Efficient Federated Learning Over Non-IID and Imbalanced Data

Federated learning (FL) is an emerging distributed and privacy-preserving machine learning framework. However, the performance of traditional FL methods is seriously impaired by the real-world data, which appear to be non-IID. The recent clustered federated learning (CFL) methods eliminate the impact of non-IID data by grouping clients with similar data distribution into the same cluster. Unfortunately, existing CFL methods heavily rely on the pre-setting of the cluster number, failing to achieve adaptive client clustering. We also experimentally observe that imbalanced data largely degrade their correctness of client clustering. In this paper, we present a novel CFL method without manual intervention, named AutoCFL, which can eliminate both effects of non-IID and imbalanced data simultaneously. To deal with imbalanced data, the local training adjustment strategy adaptively adjusts the number of local training epochs for each client. To further improve the clustering correctness and adaptability, the weighted voting-based client clustering strategy automatically groups each client into an appropriate cluster. Extensive experiments are conducted to evaluate the design of AutoCFL with three popular datasets under various data settings. Experimental results demonstrate that AutoCFL outperforms state-of-the-art methods, e.g., on average improving model accuracy by 9.24%, while reducing communication costs by 4.67 in an adaptive manner.

SoFL: Clustered Federated Learning Based on Dual Clustering for Heterogeneous Data

Federated Learning (FL) is an emerging privacy-preserving technology that enables training a global model beneficial to all participants without sharing their data. However, differences in data distributions among participants may undermine the stability and accuracy of the global model. To address this challenge, recent research proposes client clustering based on data distribution similarity, generating independent models for each cluster in order to enhance FL performance. Nevertheless, due to the uncertainty of participant identities, FL struggles to rapidly and accurately determine the clusters. Most of the existing algorithms distinguish clients by iterative clustering, which not only increases the computing cost of the server but also affects the convergence speed of the federation model. To address these shortcomings, in this paper, we propose a novel clustering-based FL method, SoFL. SoFL introduces SOM networks, improves the quality of cluster data, and eliminates redundant categories through secondary clustering, encouraging more similar clients to train together. Through this mechanism, SoFL completes the clustering task in one round of communication and speeds up the convergence of federated model training. Simulation results demonstrate that SoFL accurately and swiftly adapts to determine the clusters. In different non-IID settings, SoFL’s model accuracy improvements ranged from 9 to 18% compared to FedAvg and FedProx.

Federated two-stage decoupling with adaptive personalization layers

Federated learning has gained significant attention due to its groundbreaking ability to enable distributed learning while maintaining privacy constraints. However, as a consequence of data heterogeneity among decentralized devices, it inherently experiences significant learning degradation and slow convergence speed. Therefore, it is natural to employ the concept of clustering homogeneous clients into the same group, allowing only the model weights within each group to be aggregated. While most existing clustered federated learning methods employ either model gradients or inference outputs as metrics for client partitioning to group similar devices together, heterogeneity may still exist within each cluster. Moreover, there is a scarcity of research exploring the underlying reasons for determining the appropriate timing for clustering, resulting in the common practice of assigning each client to its own individual cluster, particularly in the context of highly non-independent and identically distributed (Non-IID) data. In this paper, we introduce a two-stage decoupling federated learning algorithm with adaptive personalization layers named FedTSDP, where client clustering is performed twice according to inference outputs and model weights, respectively. Hopkins amended sampling is adopted to determine the appropriate timing for clustering and the sampling weight of public unlabeled data. In addition, a simple yet effective approach is developed to adaptively adjust the personalization layers based on varying degrees of data skew. Experimental results show that our proposed method has reliable performance on both IID and non-IID scenarios.

Enhancing Federated Learning With Pattern-Based Client Clustering

Enhancing Federated Learning With Pattern-Based Client Clustering

A simpler method for understanding emergency shelter access patterns

ABSTRACT The Simplified Access Metric (SAM) is a new approach for characterizing patterns of homelessness. The goal of SAM is to provide emergency shelter operators and housing staff with an intuitive way to understand a person or group’s pattern of homelessness and/or shelter access that can be implemented by non-technical staff using spreadsheet operations. Client data from a large North American shelter will be used to demonstrate that SAM produces similar results to traditional transitional, episodic and chronic client cluster analysis. Since SAM requires less data than cluster analysis, it is also able to generate a timeline of homelessness patterns that can be updated in real time. Using nine years of shelter data, a shelter access timeline is presented that includes the introduction of Housing First programming and the COVID-19 lockdown. Finally, SAM allows shelter staff to move beyond assigning transitional, episodic and chronic labels and instead use the “soft” output of SAM directly to better understand a person’s experience of homelessness.

A Greedy Agglomerative Framework for Clustered Federated Learning

Federated learning (FL) has received widespread attention for decentralized training of deep learning models across devices while preserving privacy. Industrial big data in key applications like healthcare, smart manufacturing, autonomous driving, and robotics is inherently multi-source and heterogeneous. Recent studies have shown that the quality of the global FL model deteriorates in the presence of such non-IID data. To address this, we present a novel clustered FL framework called Federated Learning via Agglomerative Client Clustering (FLACC). FLACC greedily agglomerates clients or groups of clients based on their gradient updates while learning the global FL model. Once the clustering is complete, each cluster is separated into its own federation, allowing clients with similar underlying distributions to train together. In contrast with existing methods, FLACC does not require the number of clusters to be specified <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">a priori</i> , can handle client fractions, and is robust to hyperparameter tuning. We demonstrate the efficacy of this framework through extensive experiments on three benchmark FL datasets and an FL case study simulated using industrial mixed fault classification data. Qualitative clustering results show that FLACC accurately identifies clusters in the presence of various statistical heterogeneities in the client data. Quantitative results show that FLACC outperforms vanilla FL and state-of-the-art personalized and clustered FL methods, even when the underlying clustering structure is not apparent.

FedSW: Federated learning with adaptive sample weights

Federated Learning (FL) is a machine learning approach in which a cluster of clients collaboratively trains a model without sharing the data of any clients. As the datasets of each client's task are typically noisy and statistically diverse, they are not easily learned by a global model, especially deep networks that are prone to overfitting on biased training data. This study proposes an adaptive sample weighting algorithm based on self-paced learning (SPL). Instead of weighting all samples equally, the algorithm assigns weights to each sample based on its impact on the global model. To achieve this, the client objectives are defined as the sum of the weighted empirical risks and the regularizer with respect to the weight of each sample. The final global model is obtained by alternately optimizing the model parameters and sample weights. By applying the implicit SPL regularizer, we derive an analytic formula for the optimal sample weights used in the experiments. We demonstrate that the algorithm converges and that our method is more stable and accurate than the series of federated averaging algorithms. Specifically, when 30% of the training data is corrupted, the test accuracy in our method is up to 50% higher than that achieved by the federated averaging algorithm.

Federated Learning With Taskonomy for Non-IID Data.

Classical federated learning approaches incur significant performance degradation in the presence of non-independent and identically distributed (non-IID) client data. A possible direction to address this issue is forming clusters of clients with roughly IID data. Most solutions following this direction are iterative and relatively slow, also prone to convergence issues in discovering underlying cluster formations. We introduce federated learning with taskonomy (FLT) that generalizes this direction by learning the task relatedness between clients for more efficient federated aggregation of heterogeneous data. In a one-off process, the server provides the clients with a pretrained (and fine-tunable) encoder to compress their data into a latent representation and transmit the signature of their data back to the server. The server then learns the task relatedness among clients via manifold learning and performs a generalization of federated averaging. FLT can flexibly handle a generic client relatedness graph, when there are no explicit clusters of clients, as well as efficiently decompose it into (disjoint) clusters for clustered federated learning. We demonstrate that FLT not only outperforms the existing state-of-the-art baselines in non-IID scenarios but also offers improved fairness across clients. Our codebase can be found at: https://github.com/hjraad/FLT/.

Classification of customer churn prediction model for telecommunication industry using analysis of variance

Customer predictive analytics has shown great potential for effective churn models. Thriving in today's telecommunications industry, discerning between consumers who are likely to migrate to a competitor is enormous. Having reliable predictive client behavior in the future is required. Machine learning algorithms are essential to predict customer turnovers, and researchers have proposed various techniques. Churn prediction is a problem due to the unequal dispersal of classes. Most traditional machine learning algorithms are ineffective in classifying data. Client cluster with a higher risk has been discovered. A support vector machine is employed as the foundational learner, and a churn prediction model is constructed based on each analysis of variance. The separation of churn data revealed by experimental assessment is recommended for churn prediction analysis. Customer attrition is high, but an instantaneous support can ensure that customer needs are addressed and assess an employee's capacity to achieve customer satisfaction. This study uses an analysis of variance (ANOVA) with a support vector machine (SVM), classification in analyzing risks in telecom systems It may be determined that SVM provides the most accurate forecast of customer turnover (95%). The projected outcomes will allow other organizations to assess possible client turnover and collect customer feedback.

Clustered Scheduling and Communication Pipelining for Efficient Resource Management of Wireless Federated Learning

This paper proposes using communication pipelining to enhance the convergence speed of federated learning in mobile edge computing applications. Due to limited wireless sub-channels, a subset of the total clients is scheduled in each iteration of federated learning algorithms. On the other hand, the scheduled clients wait for the slowest client to finish its computation. We propose to first cluster the clients based on the time they need per iteration to compute the local gradients of the federated learning model. Then, we schedule a mixture of clients from all clusters to send their local updates in a pipelined manner. In this way, instead of just waiting for the slower clients to finish their computations, more clients can participate in each iteration. While the time duration of a single iteration does not change, the proposed method can significantly reduce the number of required iterations to achieve a target accuracy. We provide a generic formulation for optimal client clustering under different settings, and we analytically derive an efficient algorithm for obtaining the optimal solution. We also provide numerical results to demonstrate the gains of the proposed method for different datasets and deep learning architectures.

A dynamic adaptive iterative clustered federated learning scheme

Clustered federated learning (CFL), as an important research branch of personalized federated learning (FL), can better cope with the highly statistically heterogeneous federated learning environment and provide higher quality services to clients. However, existing CFL schemes have difficulties in adapting to real-time data distribution changes due to disadvantages such as relatively fixed cluster structure. This poses a great challenge to the practical deployment of CFL schemes. To address the common problems of existing CFL schemes, we propose a more flexible dynamic adaptive cluster federated learning scheme (AICFL). AICFL uses the mutual sensitivity between models and data as intuition to perform cluster identity estimation, cluster addition, cluster model updating, and cluster deletion in the early iterations of FL to find the optimal client cluster partitioning. Firstly, this process does not require a priori estimation of the number of clusters and does not require the online participation of all clients. Secondly, during cluster partitioning, AICFL is able to adjust the cluster structure in real time based on the overall data distribution. Moreover, AICFL has the same ability to adapt to changes in the system environment in the middle and late stages of FL. The experimental results show that our scheme gives the most reasonable cluster partitioning results in all cases which indicates that AICFL is able to cope with the above-mentioned distribution changes well, and has better adaptability and better flexibility than other schemes.

Contrastive encoder pre-training-based clustered federated learning for heterogeneous data

Federated learning (FL) is a promising approach that enables distributed clients to collaboratively train a global model while preserving their data privacy. However, FL often suffers from data heterogeneity problems, which can significantly affect its performance. To address this, clustered federated learning (CFL) has been proposed to construct personalized models for different client clusters. One effective client clustering strategy is to allow clients to choose their own local models from a model pool based on their performance. However, without pre-trained model parameters, such a strategy is prone to clustering failure, in which all clients choose the same model. Unfortunately, collecting a large amount of labeled data for pre-training can be costly and impractical in distributed environments. To overcome this challenge, we leverage self-supervised contrastive learning to exploit unlabeled data for the pre-training of FL systems. Together, self-supervised pre-training and client clustering can be crucial components for tackling the data heterogeneity issues of FL. Leveraging these two crucial strategies, we propose contrastive pre-training-based clustered federated learning (CP-CFL) to improve the model convergence and overall performance of FL systems. In this work, we demonstrate the effectiveness of CP-CFL through extensive experiments in heterogeneous FL settings, and present various interesting observations.

MRFM-analysis for customer segmentation in the industrial equipment market

It is of high importance for enterprises to identify, group and prioritize customers with similar needs in order to develop an individual approach to each of these groups. The article aims to segment B2B consumers based on the analysis of their purchasing behaviour. The theoretical framework of the study is the postulates of behavioural marketing. The research method involves МRFM-analysis (Modified Recency-Frequency-Monetary Analysis) that allows determining homogeneous groups of clients, examining the evolution of their behaviour, and formulating targeted interaction strategies for each group. The paper demonstrates the benefits of the Orange Data Mining machine learning and data mining complex, these are the capability to statistically correctly identify client clusters and the visual clarity of results analysis. The empirical evidence is industrial equipment sales data provided by a large Russian security systems manufacturer for the period of 2015–2022. A relationship is found between the segmentation performed in the study and the Reinartz–Kumar approach applied to decide on a strategy for forming customer loyalty. The authors distinguish between six groups of customers and establish those generating the greatest profit for the company and those having the minimum effects on its turnover. The group of trading firms (about 20% of all the clients) turned out to be the priority one, which, due to the specificity of their activities, have long-term relationships with the manufacturer and high client reliability. It is the client group for which devising targeted strategies stimulating an increase in their demand is most reasonable. For the rest of the consumer groups, it is expedient to use standard marketing strategies.

Personalized federated learning with model interpolation among client clusters and its application in smart home

The proliferation of high-performance personal devices and the widespread deployment of machine learning (ML) applications have led to two consequences: the volume of private data from individuals or groups has exploded over the past few years; and the traditional central servers for training ML models have experienced communication and performance bottlenecks in the face of massive amounts of data. However, this reality also provides the possibility of keeping data local for ML training and fusing models on a broader scale. As a new branch of ML application, Federated Learning (FL) aims to solve the problem of multi-party joint learning on the premise of protecting personal data privacy. However, due to the heterogeneity of devices, including network connection, network bandwidth, computing resources, etc., it is unrealistic to train, update and aggregate models in all devices in parallel, while personal data is often not independent and identically distributed (Non-IID) due to multiple reasons. This reality poses a challenge to the speed and convergence of FL. In this paper, we propose the pFedCAM algorithm, which aims to improve the robustness of the FL system to device heterogeneity and Non-IID data, while achieving some degree of federation model personalization. pFedCAM is based on the idea of clustering and model interpolation by classifying heterogeneous clients and performing FedAvg algorithm in parallel, and then combining them into personalized federated global models by inter-cluster model interpolation. Experiments show that the accuracy of pFedCAM improves 10.3% on Fashion-MNIST and 11.3% on CIFAR-10 compared to the benchmark in the case of Non-IID data. In the end, we applied pFedCAM in HomeProtect, a smart home privacy protection framework we designed, and achieved good practical results in the case of flame recognition.

Federated Learning Resource Optimization and Client Selection for Total Energy Minimization Under Outage, Latency, and Bandwidth Constraints With Partial or No CSI

We consider the problem of minimizing the total energy consumption due to the computation and communication tasks of federated learning (FL) under bandwidth and latency constraints. To avoid channel state information (CSI) feedback to the transmitter, we adopt outage probability as an additional constraint in the energy minimization problem. First, we define a feasibility metric based on the system design parameters to exclude slow clients (stragglers). Then, we propose a novel client selection algorithm, after excluding stragglers, based on dividing the remaining clients into clusters, where clients within the same cluster collaborate in a communication round to train their local models. For each communication round, one client cluster is selected in a round-robin fashion. Furthermore, we formulate and solve a resource allocation problem to optimize the transmit power, clock frequency, allocated bandwidth, and communication latency of clients within each cluster to minimize the total energy subject to total bandwidth, latency, and outage constraints. Moreover, we extend our FL design framework to the case of no CSI at both the client and server ends using differential transmission to eliminate CSI estimation pilot overhead and complexity at comparable total energy consumption and learning accuracy to coherent transmission. We test our proposed algorithms over MNIST and Fashion-MNIST datasets in iid and non-iid settings. Our proposed client selection algorithm reduces the number of participating clients per communication round by 41% compared to the baseline while maintaining the same learning accuracy. Moreover, our results demonstrate that increasing the number of receive antennas at the server from one to four can reduce the number of communication rounds required to reach a predetermined testing accuracy level by up to 53% for the Fashion-MNIST dataset.

WSCC: A Weight-Similarity-Based Client Clustering Approach for Non-IID Federated Learning

The fast development of the Internet of Things (IoT) and deep learning enables learning useful patterns from the massive amount of collected data with sporadic nodes in IoT systems. Federated learning has received increasing attention in distributed machine learning where only intermediate parameters are exchanged with training samples that resided at local nodes. Nonetheless, most of the existing federated learning schemes assume a homogeneous distribution of data. The assumption, however, does not apply to IoT systems because of the heterogeneity of the IoT architecture. The nonindependent and identical distribution (non-IID) property in data volume and statistical distribution of IoT nodes can impact the performance of an aggregated global model that fits all nodes. Existing federated learning solutions for non-IID data sets either have to train additional models or require extra data exchange to check the node distribution. However, due to resource constraints in IoT systems, these approaches will increase the burden on limited computation capacity and cause network overhead. To address the issue, in this article, a novel weight-similarity-based client clustering (WSCC) approach is proposed, in which clients are split into different groups based on their data set distributions. An affinity-propagation-based method with the cosine distance of the client’s weight parameters is designed to iteratively and automatically determine dynamic clusters. The proposed approach is ideal for IoT systems since there are no auxiliary models and extra data transmissions are needed. Through the theoretical convergence analysis and empirical results, we show that our proposed WSCC scheme outperforms the representative federated learning schemes under different non-IID settings, achieving up to 20% improvements in accuracy.

K-Means Customers Clustering by their RFMT and Score Satisfaction Analysis

Businesses derive more revenue from building and maintaining long-term relationships with their customers. Therefore, it is essential to build refined strategies based on customer relationship management, with the purpose of increasing their turnover and profits while retaining their customers. In this context, customer segmentation, which is at the heart of marketing strategy, makes it possible to determine the answers to questions relating to the number of investments to be released, the marketing campaigns to be organized, and the development strategy to be implemented. This paper develops an extended RFMT (Recency, Frequency, Monetary, and Interpurchase Time) model, namely the RFMTS model, by introducing a new dimension as satisfaction ‘S’. The aim of this model is to analyze online consumer satisfaction over time and discern changes to implement customer segmentation. This article proposes an approach to a segmentation, by client clustering along the unsupervised machine learning method k-means based on data generated using the proposed RFMTS model, in order to improve the customer relationship and develop more effective personalized marketing strategies. The study shows that including satisfaction to the existing RFM model for customer clustering has a major impact and helps identify customers who are satisfied and those who are not, unlike previous attempts to develop new RFM models. By ignoring the “satisfaction” indicator, what went well and what didn't went well cannot be understood. Consequently, the business loses its unsatisfied, loyal, and profitable customers and either fails or relies only on the satisfied ones to continue making profits for an indefinite period of time.

Federated learning with hyperparameter-based clustering for electrical load forecasting

Electrical load prediction has become an integral part of power system operation. Deep learning models have found popularity for this purpose. However, to achieve a desired prediction accuracy, they require huge amounts of data for training. Sharing electricity consumption data of individual households for load prediction may compromise user privacy and can be expensive in terms of communication resources. Therefore, edge computing methods, such as federated learning, are gaining more importance for this purpose. These methods can take advantage of the data without centrally storing it. This paper evaluates the performance of federated learning for short-term forecasting of individual house loads as well as the aggregate load. It discusses the advantages and disadvantages of this method by comparing it to centralized and local learning schemes. Moreover, a new client clustering method is proposed to reduce the convergence time of federated learning. The results show that federated learning has a good performance with a minimum root mean squared error (RMSE) of 0.117 kWh for individual load forecasting.

Business incubators as international knowledge intermediaries: Exploring their role in the internationalization of start-ups from an emerging market

This paper investigates how business incubators facilitate the international growth of start-up clients originating from emerging markets. Drawing key insights from a network perspective, we explore the processes by which business incubators create international linkages between the networks of knowledge creation and knowledge application. Our analysis of interviews and archival data on five Chinese high-tech business incubators reveals that clients-interface, market-interface, and knowledge recombination practices of the business incubators create international knowledge linkages, which in turn facilitate the international growth of their start-up clients. This role of incubators as knowledge intermediaries is achieved through several networking and learning mechanisms, including clustering and coaching of international clients, upstream and downstream networking in international markets, and client-market matchmaking internally. This study demonstrates business incubators as an efficient modality of internationalization for locally bound international knowledge and network scarce start-up clients.

Business Incubators as International Knowledge Intermediaries: An Evolutionary Perspective

This research investigates how business incubators perform the role of knowledge intermediary to facilitate the international evolution of start-ups. The research adopts the evolutionary perspective of the firm, by employing the concepts of variation, selection, and retention. We collected and analysed interviews and archival data from five incubators from China that have international operations. The findings show that client and market-interface activities and the internal knowledge recombination practices of the business incubators facilitate the international evolution of start-ups. Those incubators as a knowledge intermediary provide start-ups with several mechanisms, including clustering and coaching of international clients, upstream and downstream networking in international markets, and client-market match-making internally. Business incubators serve as an efficient modality of internationalization for locally bound international knowledge and network scarce start-ups.