Distributed Learning Research Articles

RNEP: Random Node Entropy Pairing for Efficient Decentralized Training with Non-IID Local Data

The proliferation of edge devices and advancements in Internet of Things technology have created a vast array of distributed data sources, necessitating machine learning models that can operate closer to the point of data generation. Traditional centralized machine learning approaches often struggle with real-time big data applications, such as climate prediction and traffic simulation, owing to high communication costs. In this study, we introduce random node entropy pairing, a novel distributed learning method for artificial neural networks tailored to distributed computing environments. This method reduces communication overhead and addresses data imbalance by enabling nodes to exchange only the weights of their local models with randomly selected peers during each communication round, rather than sharing the entire dataset. Our findings indicate that this approach significantly reduces communication costs while maintaining accuracy, even when learning from non-IID local data. Furthermore, we explored additional learning strategies that enhance system accuracy by leveraging the characteristics of this method. The results demonstrate that random node entropy pairing is an effective and efficient solution for distributed learning in environments where communication costs and data distribution present significant challenges.

EXPRESS: Individual differences in distributional statistical learning: better frequency 'discriminators' are better 'estimators'.

People can easily extract and encode statistical information from their environment. However, research has primarily focused on conditional statistical learning (i.e. the ability to learn joint and conditional relationships between stimuli) and has largely neglected distributional statistical learning (i.e. the ability to learn the frequency and variability of distributions). For example, learning that 'E' is more common in the English alphabet than 'Z.' In this paper, we investigate how distributional learning can be measured by exploring the relationship between, and psychometric properties of, four different measures of distributional learning - from the ability to discriminate relative frequencies to the ability to estimate frequencies. We identified moderate relationships between four distributional learning measures and these tasks accounted for a substantial portion of the variance in performance across tasks (44.3%). A measure of divergent validity (intrinsic motivation) did not significantly correlate with any statistical learning measure and accounted for a separate portion of the variance across tasks. More complex measures also showed better reliability and internal consistency. Our results suggest that distributional statistical learning encompasses both the ability to discriminate between relative frequencies and directly estimating them.

Backdoor Attacks in Peer-to-Peer Federated Learning

Most machine learning applications rely on centralized learning processes, opening up the risk of exposure of their training datasets. While federated learning (FL) mitigates to some extent these privacy risks, it relies on a trusted aggregation server for training a shared global model. Recently, new distributed learning architectures based on Peer-to-Peer Federated Learning (P2PFL) offer advantages in terms of both privacy and reliability. Still, their resilience to poisoning attacks during training has not been investigated. In this paper, we propose new backdoor attacks for P2PFL that leverage structural graph properties to select the malicious nodes, and achieve high attack success, while remaining stealthy. We evaluate our attacks under various realistic conditions, including multiple graph topologies, limited adversarial visibility of the network, and clients with non-IID data. Finally, we show the limitations of existing defenses adapted from FL and design a new defense that successfully mitigates the backdoor attacks, without an impact on model accuracy.

A place to call our home: innovative rural physical therapy training in Canada

BackgroundTo promote rural practice and increase enrollment in the entry-level Master of Science Physical Therapy program at the University of Alberta, the Department of Physical Therapy developed a rural physical therapy satellite campus located in a farming region in central Alberta. A distributed learning format was used to connect the rural cohort to the main urban campus. Real time video conferencing was used to connect the two campuses for all lectures, seminars and clinical skills classes. This evaluation aimed to describe a unique rural training program for physical therapy students and its effectiveness in promoting work in rural communities after graduation.MethodsPhysical Therapy students in the first three years (2012–2015) of commencing the rural satellite program (n = 280) were surveyed, and six focus groups were held to capture student experiences, satisfaction and engagement. Data were collected on employment locations of the 2012–2019 graduates’ first physical therapy position and current employment.ResultsSurvey results suggested comparable levels of satisfaction and engagement for all physical therapy students regardless of campus. Focus group data revealed that students quickly accepted the distributed learning technological interface, enjoyed their local campuses, and felt connected to instructors and student colleagues. Compared to the overall physical therapy workforce, a higher percentage of physical therapists graduating from the rural campus reported working in rural centers for both their first and current jobs.ConclusionRegardless of campus, students were satisfied and equally engaged in the physical therapy program. Students who completed the physical therapy program in a rural setting tended to work rurally after graduation. A distributed learning model may be useful for other healthcare training programs to promote engagement in rural health.

Privacy-preserving distributed learning: collaborative training on principal components and orthogonal projections of datapoints

Privacy-preserving distributed learning: collaborative training on principal components and orthogonal projections of datapoints

Distributional learning drives statistical suppression and deafening.

Organisms as diverse as honeybees and humans pick up on probabilities in the world around them. People implicitly learn the likelihood of a color, price range, or even syntactic structure. How does statistical learning affect how we detect events and make decisions, especially when probabilities are completely irrelevant to the task at hand, and can change without warning? We find that people learn and track changes in perceptual probabilities irrelevant to a task and that this learning drives dynamic shifts in perception characterized by graded effects of enhancement and primarily - suppression across acoustic frequency. This can result in a remarkably long-lived 'statistical deafening' that seems maladaptive but may instead reflect use of likelihood to guide and sharpen perception.

Efficient Privacy-Preserving Machine Learning with Lightweight Trusted Hardware

In this paper, we propose a new secure machine learning inference platform assisted by a small dedicated security processor, which will be easier to protect and deploy compared to today's TEEs integrated into high-performance processors. Our platform provides three main advantages over the state-of-the-art: (i) We achieve significant performance improvements compared to state-of-the-art distributed Privacy-Preserving Machine Learning (PPML) protocols, with only a small security processor that is comparable to a discrete security chip such as the Trusted Platform Module (TPM) or on-chip security subsystems in SoCs similar to the Apple enclave processor. In the semi-honest setting with WAN/GPU, our scheme is 4X-63X faster than Falcon (PoPETs'21) and AriaNN (PoPETs'22) and 3.8X-12X more communication efficient. We achieve even higher performance improvements in the malicious setting. (ii) Our platform guarantees security with abort against malicious adversaries under honest majority assumption. (iii) Our technique is not limited by the size of secure memory in a TEE and can support high-capacity modern neural networks like ResNet18 and Transformer. While previous work investigated the use of high-performance TEEs in PPML, this work represents the first to show that even tiny secure hardware with very limited performance can be leveraged to significantly speed-up distributed PPML protocols if the protocol can be carefully designed for lightweight trusted hardware.

Joint antenna selection and resource allocation for mm‐wave directional D2D communications using distributed deep reinforcement learning

AbstractIn this paper, with the promising assumption of using adaptive directional microstrip antenna on user equipment, the problem of joint antenna selection, spectrum assignment, and transmit power allocation for mm‐wave device‐to‐device communications underlying cellular networks is tackled, with the goal of enhancing system throughput and energy efficiency. To address the complexity of this problem, a method based on multi‐agent distributed deep reinforcement learning in which an autonomous intelligent agent is deployed for each user equipment is proposed. The performance evaluation demonstrates its superiority over existing strategies, resulting in improved system performance, reduced outage probability, and enhanced energy efficiency.

FedDSS: A data-similarity approach for client selection in horizontal federated learning

Background and objectiveFederated learning (FL) is an emerging distributed learning framework allowing multiple clients (hospitals, institutions, smart devices, etc.) to collaboratively train a centralized machine learning model without disclosing personal data. It has the potential to address several healthcare challenges, including a lack of training data, data privacy, and security concerns. However, model learning under FL is affected by non-i.i.d. data, leading to severe model divergence and reduced performance due to the varying client's data distributions. To address this problem, we propose FedDSS, Federated Data Similarity Selection, a framework that uses a data-similarity approach to select clients, without compromising client data privacy. MethodsFedDSS comprises a statistical-based data similarity metric, a N-similar-neighbor network, and a network-based selection strategy. We assessed FedDSS' performance against FedAvg's in i.i.d. and non-i.i.d. settings with two public pediatric sepsis datasets (PICD and MIMICIII). Selection fairness was measured using entropy. Simulations were repeated five times to evaluate average loss, true positive rate (TPR), and entropy. ResultsIn i.i.d setting on PICD, FedDSS achieved a higher TPR starting from the 9th round and surpassing 0.6 three rounds earlier than FedAvg. On MIMICIII, FedDSS's loss decreases significantly from the 13th round, with TPR > 0.8 by the 2nd round, two rounds ahead of FedAvg (at the 4th round). In the non-i.i.d. setting, FedDSS achieved TPR > 0.7 by the 4th and > 0.8 by the 7th round, earlier than FedAvg (at the 5th and 11th rounds). In both settings, FedDSS showed reasonable fairness (entropy of 2.2 and 2.1). ConclusionWe demonstrated that FedDSS contributes to improved learning in FL by achieving faster convergence, reaching the desired TPR with fewer communication rounds, and potentially enhancing sepsis prediction (TPR) over FedAvg.

FedDA: Resource-adaptive federated learning with dual-alignment aggregation optimization for heterogeneous edge devices

Federated learning (FL) is an emerging distributed learning paradigm that allows multiple clients to collaborate on training a global model without sharing their local data. However, in practical heterogeneous edge device scenarios, FL faces the challenges of system heterogeneity and data heterogeneity, which leads to unfair participation and degraded global model performance. In this paper, we introduce FedDA, a resource-adaptive FL framework, which adapts to the client’s computing resources by assigning heterogeneous models of different sizes. To improve the performance of heterogeneous model aggregation and adjust to non-independent and identically distributed (non-i.i.d.) data, we propose a dual-alignment aggregation optimization method, i.e., parameter feature space alignment and output space alignment. Specifically, FedDA exploits the permutation symmetry property of weight space to permutate the model parameter positions via an adaptive layer-wise matching method, thereby aligning models with significant deviations in parameter feature space. FedDA mitigates the imbalance in parameter quantity between smaller and larger models through parameter expansion. Additionally, FedDA maps client labels into a uniform embedding space through output space alignment, thus reducing model parameter deviations due to non-i.i.d. data without additional client-side computing overhead. We evaluate the performance of FedDA on benchmark datasets, including FashionMNIST, CIFAR10, CIFAR100 and AGNews. Experimental results demonstrate that FedDA achieves up to 8.71% improvement in model accuracy compared to baseline methods, highlighting its effectiveness in addressing the challenges of heterogeneity.

Federated Incremental Learning algorithm based on Topological Data Analysis

Federated learning is a distributed learning approach aimed at preserving user’s data privacy, while incremental learning is an adaptive machine learning method that enables continuous learning of new data. The combination of these two approaches into Federated Incremental Learning (FIL) algorithms brings together their respective advantages. However, the existing federated incremental learning still faces two main challenges: (1) catastrophic forgetting of previous knowledge by local models when adapting to incremental tasks, and (2) limited capability of the server to capture critical features during federated aggregation. To address these challenges, this paper proposes a federated incremental learning algorithm based on Topological Data Analysis (TDA). Firstly, the algorithm extracts topological features from the input information and designs a Topological Stability Loss (TSL) to mitigate catastrophic forgetting of previous knowledge by local models. Secondly, a feature attention mechanism is utilized to select appropriate attention weights for each local model, enhancing the recognition performance of the global model. The experimental results on publicly available datasets, namely CIFAR10, CIFAR100, and ImageNet, demonstrate that the proposed approach in this paper achieves global accuracies of 67.23%, 65.75%, and 62.41% respectively for the federated incremental learning model. These accuracies surpass the existing Icarl incremental algorithm by improvements of 3.21%, 2.87%, and 1.34% respectively. Therefore, the algorithm presented in this paper achieves better performance, indicating its superiority over existing methods.

Noise-resistant fuzzy multineighbourhood rough set-based feature selection with label enhancement and its application for multilabel classification

Noise pollution in process of feature selection greatly reduces the classification efficacy of multilabel data, and the estimation of label descriptions is insufficient because of the lack of label enhancement for label distribution learning. To overcome the limitations, this work presents a noise-resistant fuzzy multineighbourhood rough set-based feature selection methodology with label enhancement and its application for multilabel classification. First, under the fuzzy T-equivalence relation, to construct new adaptive fuzzy neighbourhood class, the multineighbourhood radius set on feature space is integrated with fuzzy neighbourhood radius via sample interval on label space; then, the adaptive fuzzy multineighbourhood granules are obtained. Second, by integrating the parameterized fuzzy decision of labels and adaptive fuzzy multineighbourhood granules, noise-resistant lower and upper approximations via fuzzy multineighbourhood are constructed. A noise-resistant multilabel fuzzy multineighbourhood rough set model and its noise-resistant approximate accuracy can be proposed. Third, the description degree of label is computed to design label proportion after label enhancement. Certain fuzzy multineighbourhood entropy measures are developed to obtain label enhancement-based mutual information. When the algebraic and information perspectives are fused, label enhancement-based mutual information with fuzzy multineighbourhood via approximate accuracy is presented to evaluate the final association relationship between the features and label sets. Finally, a multilabel feature selection strategy via label enhancement will be constructed to obtain the best feature subset. The experimental results applied to 14 multilabel datasets indicate that this algorithm is significant.

The influence of e-learning on exam performance and the role of achievement goals in shaping learning patterns

Digital learning environments provide opportunities to support learning in higher education. However, it is yet unclear why and how learners use these opportunities. We propose that learners’ achievement goals and their beliefs regarding the instrumentality of e-learning tools to achieve those goals are predictive for learning behavior within digital learning environments. Furthermore, we assume learning behavior characterized by longer overall learning time, more distributed learning, and less learning delay predicts higher exam performance. To test these hypotheses, we analyzed log-file data of 91 university students who had used an intelligent tutoring system as exam preparation in a pre-registered study. Beyond the overall predictive validity of the intelligent tutoring system, we found a negative association between learning delay and exam performance. Achievement goals predicted learning time and time distribution, an association that was partly moderated by perceived instrumentality. This suggests that goals and beliefs are important puzzle pieces for understanding e-learning (behavior).

A personality-guided preference aggregator for ephemeral group recommendation

Ephemeral group recommendation (EGR) aims to suggest items for a group of users who come together for the first time. Existing work typically consider individual preferences as the sole factor in aggregating group preferences. However, they neglect to take into account the importance of the individual inherent factors, such as personality, and thus fail to accurately simulate the group decision-making process. Additionally, these methods often struggle due to insufficient interactive records. To tackle these issues, a Personality-Guided Preference Aggregator (PEGA) is proposed, which guides the preference aggregation of group members based on their personalities, rather than relying solely on their preferences. Specifically, implicit personalities are first extracted from user reviews. Hyper-rectangles are then used to aggregate individual personalities to obtain the “Group Personality”, which allows for the learning of personality distributions within the group. Subsequently, a personality attention mechanism is employed to aggregate group preferences, and a preference-based fine-tuning module is used to balance the weights of personality and preferences. The role of personality in this approach is twofold: (1) To estimate the importance of individual users in a group and provide explainability; (2) To alleviate the data sparsity issue encountered in ephemeral groups. Experimental results demonstrate that, on four real-world datasets, the PEGA model significantly outperforms related baseline models in terms of classification accuracy and interpretability. Moreover, empirical evidence supports the idea that personality plays a pivotal role in enhancing the performance of EGR tasks.

Node and relevant data selection in distributed predictive analytics: A query-centric approach

Distributed Predictive Analytics (DPA) refers to constructing predictive models based on data distributed across nodes. DPA reduces the need for data centralization, thus, alleviating concerns about data privacy, decreasing the load on central servers, and minimizing communication overhead. However, data collected by nodes are inherently different; each node can have different distributions, volumes, access patterns, and features space. This heterogeneity hinders the development of accurate models in a distributed fashion. Many state-of-the-art methods adopt random node selection as a straightforward approach. Such method is particularly ineffective when dealing with data and access pattern heterogeneity, as it increases the likelihood of selecting nodes with low-quality or irrelevant data for DPA. Consequently, it is only after training models over randomly selected nodes that the most suitable ones can be identified based on the predictive performance. This results in more time and resource consumption, and increased network load. In this work, holistic knowledge of nodes’ data characteristics and access patterns is crucial. Such knowledge enables the successful selection of a subset of suitable nodes for each DPA task (query) before model training. Our method engages the most suitable nodes by predicting their relevant distributed data and learning predictive models per query. We introduce a novel DPA query-centric mechanism for node and relevant data selection. We contribute with (i) predictive selection mechanisms based on the availability and relevance of data per DPA query and (ii) various distributed machine learning mechanisms that engage the most suitable nodes for model training. We evaluate the efficiency of our mechanism and provide a comparative assessment with other methods found in the literature. Our experiments showcase that our mechanism significantly outperforms other approaches being applicable in DPA.

SS-ALDL: Consistency-based semi-supervised label distribution learning for acne severity classification

Acne vulgaris is a common skin disease among adolescents. Accurate classification of acne severity is critical to patient treatment. Most existing acne severity classification models ignore the number of acne lesions and the similarity between samples. Moreover, training a supervised model requires collecting a large amount of labeled data, which is labor-intensive and time-consuming. To solve the above problems, this study presents a consistency-based semi-supervised label distribution learning (SS-ALDL) framework, which is the first acne-specific semi-supervised framework for acne severity classification. It generates three distributions based on acne grading criteria, including acne severity, lesion counts, and grading transformed by counts. These three distributions are integrated by multi-task learning loss and optimized in supervised training for joint acne image grading and counting. Furthermore, a feature similarity consistency learning method is proposed for semi-supervised training. By maintaining the batch-level feature similarity matrix between different samples under different perturbations, the proposed method can effectively explore extra semantic information from the unlabeled data. The performance of the proposed model is evaluated on the ACNE04 dataset, the RetinaMNIST dataset, and a private dataset. It achieves the best classification accuracy and the lowest mean absolute error. These experimental results show that the proposed method outperforms other state-of-the-art semi-supervised methods and can significantly reduce the manual assessment workload.

VCSA: Verifiable and collusion-resistant secure aggregation for federated learning using symmetric homomorphic encryption

As a novel distributed learning framework for protecting personal data privacy, federated learning, (FL) has attained widespread attention through sharing gradients among users without collecting their data. However, an untrusted cloud server may infer users’ individual information from gradients and global model. In addition, it may even forge incorrect aggregated results to save resources. To deal with these issues, despite that the existing works can protect local model privacy and achieve verifiability of aggregated results, they are defective in protecting global model privacy, guaranteeing verifiability if collusion attacks occur, and suffer from high computation cost. To further tackle the above challenges, a verifiable and collusion-resistant secure aggregation scheme for FL is proposed, named VCSA. Concretely, we combine symmetric homomorphic encryption with single masking to protect model privacy. Meanwhile, we adopt verifiable multi-secret sharing and generalized Pedersen commitment to achieve verifiability and prevent users from uploading incorrect shares. Furthermore, high model accuracy can be ensured even if some users go offline. Security analysis illustrates that our VCSA enhances the security of FL, realizes verifiability despite collusion attacks and robustness to dropout. Performance evaluation displays that our VCSA can reduce at least 28.27% and 79.15% regarding computation cost compared to existing schemes.

Color distribution learning modulates saccade endpoints: a study of the global effect

Color distribution learning modulates saccade endpoints: a study of the global effect

Mobility-aware federated self-supervised learning in vehicular network

The development of the Internet of Things has led to a significant increase in the number of devices, consequently generating a vast amount of data and resulting in an influx of unlabeled data. Collecting these data enables the training of robust models to support a broader range of applications. However, labeling these data can be costly, and the models dependent on labeled data are often unsuitable for rapidly evolving fields like vehicular networks and mobile Internet of Things, where new data continuously emerge. To address this challenge, Self-Supervised Learning (SSL) offers a way to train models without the need for labels. Nevertheless, the data stored locally in vehicles are considered private, and vehicles are reluctant to share data with others. Federated Learning (FL) is an advanced distributed machine learning approach that protects each vehicle’s privacy by allowing models to be trained locally and the model parameters to be exchanged across multiple devices simultaneously. Additionally, vehicles capture images while driving through cameras mounted on their rooftops. If a vehicle’s velocity is too high, the captured images, donated as local data, may be blurred. Simple aggregation of such data can negatively impact the accuracy of the aggregated model and slow down the convergence speed of FL. This paper proposes a FL algorithm for aggregation based on image blur levels, which is called FLSimCo. This algorithm does not require labels and serves as a pre-training stage for SSL in vehicular networks. Simulation results demonstrate that the proposed algorithm achieves fast and stable convergence.

Residual [formula omitted]-Nearest Neighbors Label Distribution Learning

Label Distribution Learning (LDL) is a novel learning paradigm that assigns label distribution to each instance. It aims to learn the label distribution of training instances and predict unknown ones. Algorithm Adaptation (AA)-kNN is one of the most representative LDL baselines that adapts the kNN algorithm to LDL. Its generalization risk has been proven to approach zero if given infinite training data. Despite its theoretical advantage, AA-kNN generally performs poorly because real-world LDL problems only have finite or small training data. In this paper, we improve AA-kNN and propose a novel method called Residual k-Nearest Neighbors Label Distribution Learning (RkNN-LDL). First, RkNN-LDL introduces residual label distribution learning. Second, RkNN-LDL exploits the neighborhood structure of label distribution. In theoretical analysis, we prove that RkNN-LDL has a tighter generalization bound than AA-kNN. Besides, extensive experiments validate that RkNN-LDL beats several state-of-the-art LDL methods and statistically outperforms AA-kNN.