Distributed Expectation Maximization Research Articles

Adaptive Federated Learning for Non-Convex Optimization Problems in Edge Computing Environment

Federated learning (FL), as one of novel machine learning frameworks, has been widely applied in edge computing (EC) fields. Most of the existing FL frameworks are implemented based on gradient descent (GD), and the training resources are assumed to be sufficient. However, in practical training, not all optimization problem can be well addressed by GD under FL framework in edge computing environment, and especially the edge resources are also limited. In this paper, based on the limited training resource and the FL framework, we study a kind of non-convex optimization problems, namely the gaussian mixture model (GMM), and propose an adaptive distributed expectation maximization (DEM) algorithm of GMM, which are not well addressed by using gradient descent. In particular, we analyze and give the convergence bound of the adaptive DEM. With the bound of adaptive DEM, we propose a local update control algorithm, which can achieve the best compromise between local update and global parameter aggregation of FL to achieve the maximization utilization of given resource budget. The experimental results demonstrate that adaptive DEM can well work out a kind of non-convex optimization problem under the FL framework and edge resource constrained environment.

Visual Analysis of Sports Actions Based on Machine Learning and Distributed Expectation Maximization Algorithm.

In order to improve the scientificity of sports action analysis, this paper constructs a sports action analysis model based on machine learning based on the greedy algorithm and the bat algorithm. According to the structural characteristics of the model, the structure of the model is reflected in the form of face order, that is, the face neighborhood structure. Moreover, this paper judges the degree of similarity between model faces through the pros and cons of the order and applies it to the structural similarity matrix between models. In addition, this paper establishes corresponding mathematical models for the shape and structure of the model and constructs the shape similarity matrix, the surface neighborhood structure similarity matrix, and the structure similarity matrix between the source model and the target model. Finally, this paper designs and implements CAD model retrieval methods based on greedy algorithm and bat algorithm and designs experiments to compare the performance of the algorithm proposed in this paper with traditional algorithms. The result of the experiment shows that the algorithm proposed in this paper has obvious advantages in sports action analysis compared with the traditional algorithm.

Fully Homomorphic based Privacy-Preserving Distributed Expectation Maximization on Cloud

Expectation maximization (EM) is a clustering-based machine learning algorithm that is widely used in many areas of science (e.g., bioinformatics and computer vision) to find maximum likelihood and maximum a posteriori estimates for models with latent variables. To deploy such an algorithm in cloud environments, security and privacy issues need be considered to avoid data breaches or abuses by external malicious parties or even by cloud service providers. However, the processing performance of the EM algorithm poses a challenge in terms of building a secure environment. This article describes an innovative and practical privacy-preserving EM algorithm for cloud systems that addresses this challenge, and estimates the EM parameters in an accurate and secure manner. Fully homomorphic encryption (FHE) is used to ensure the privacy of both the EM algorithm computations and the users' sensitive data in the cloud. A distributed-based approach is also proposed to overcome the overheads of FHE computations and ensure a fast convergence of the EM algorithm. The conducted experiments demonstrate a significant improvement in the convergence time of the distributed EM algorithm, while achieving a high level of accuracy and reducing the associated computational FHE overheads.

An Asynchronous Distributed Expectation Maximization Algorithm for Massive Data: The DEM Algorithm

ABSTRACTThe family of expectation--maximization (EM) algorithms provides a general approach to fitting flexible models for large and complex data. The expectation (E) step of EM-type algorithms is time-consuming in massive data applications because it requires multiple passes through the full data. We address this problem by proposing an asynchronous and distributed generalization of the EM called the distributed EM (DEM). Using DEM, existing EM-type algorithms are easily extended to massive data settings by exploiting the divide-and-conquer technique and widely available computing power, such as grid computing. The DEM algorithm reserves two groups of computing processes called workers and managers for performing the E step and the maximization step (M step), respectively. The samples are randomly partitioned into a large number of disjoint subsets and are stored on the worker processes. The E step of DEM algorithm is performed in parallel on all the workers, and every worker communicates its results to the managers at the end of local E step. The managers perform the M step after they have received results from a γ-fraction of the workers, where γ is a fixed constant in (0, 1]. The sequence of parameter estimates generated by the DEM algorithm retains the attractive properties of EM: convergence of the sequence of parameter estimates to a local mode and linear global rate of convergence. Across diverse simulations focused on linear mixed-effects models, the DEM algorithm is significantly faster than competing EM-type algorithms while having a similar accuracy. The DEM algorithm maintains its superior empirical performance on a movie ratings database consisting of 10 million ratings. Supplementary material for this article is available online.

Distributed Expectation-Maximization Algorithm for Speaker Localization in Reverberant Environments

Localization of acoustic sources has attracted a considerable amount of research attention in recent years. A major obstacle to achieving high localization accuracy is the presence of reverberation, the influence of which obviously increases with the number of active speakers in the room. Human hearing is capable of localizing acoustic sources even in extreme conditions. In this study, we propose to combine a method based on human hearing mechanisms and a modified incremental distributed expectation-maximization (IDEM) algorithm. Rather than using phase difference measurements that are modeled by a mixture of complex-valued Gaussians, as proposed in the original IDEM framework, we propose to use time difference of arrival measurements in multiple subbands and model them by a mixture of real-valued truncated Gaussians. Moreover, we propose to first filter the measurements in order to reduce the effect of the multipath conditions. The proposed method is evaluated using both simulated data and real-life recordings.

Accelerating distributed Expectation–Maximization algorithms with frequent updates

Expectation–Maximization (EM) is a popular approach for parameter estimation in many applications, such as image understanding, document classification, and genome data analysis. Despite the popularity of EM algorithms, it is challenging to efficiently implement these algorithms in a distributed environment for handling massive data sets. In particular, many EM algorithms that frequently update the parameters have been shown to be much more efficient than their concurrent counterparts. Accordingly, we propose two approaches to parallelize such EM algorithms in a distributed environment so as to scale to massive data sets. We prove that both approaches maintain the convergence properties of the EM algorithms. Based on the approaches, we design and implement a distributed framework, FreEM, to support the implementation of frequent updates for the EM algorithms. We show its efficiency through two categories of EM applications, clustering and topic modeling. These applications include k-means clustering, fuzzy c-means clustering, parameter estimation for the Gaussian Mixture Model, and variational inference for Latent Dirichlet Allocation. We extensively evaluate our framework on both a cluster of local machines and the Amazon EC2 cloud. Our evaluation shows that the EM algorithms with frequent updates implemented on FreEM can converge much faster than those implementations with traditional concurrent updates.

Distributed Classification of Localization Attacks in Sensor Networks Using Exchange-Based Feature Extraction and Classifier

Secure localization under different forms of attack has become an essential task in wireless sensor networks. Despite the significant research efforts in detecting the malicious nodes, the problem of localization attack type recognition has not yet been well addressed. Motivated by this concern, we propose a novel exchange-based attack classification algorithm. This is achieved by a distributed expectation maximization extractor integrated with the PECPR-MKSVM classifier. First, the mixed distribution features based on the probabilistic modeling are extracted using a distributed expectation maximization algorithm. After feature extraction, by introducing the theory from support vector machine, an extensive contractive Peaceman-Rachford splitting method is derived to build the distributed classifier that diffuses the iteration calculation among neighbor sensors. To verify the efficiency of the distributed recognition scheme, four groups of experiments were carried out under various conditions. The average success rate of the proposed classification algorithm obtained in the presented experiments for external attacks is excellent and has achieved about 93.9% in some cases. These testing results demonstrate that the proposed algorithm can produce much greater recognition rate, and it can be also more robust and efficient even in the presence of excessive malicious scenario.

Distributed Localization of a RF Target in NLOS Environments

We propose a novel distributed expectation maximization (EM) method for non-cooperative RF target localization using a wireless sensor network. We consider the scenario where few or no sensors receive line-of-sight signals from the target. In the case of non-line-of-sight signals, the signal path consists of a single reflection between the transmitter and receiver. Each sensor is able to measure the time difference of arrival of the target's signal with respect to a reference sensor, as well as the angle of arrival of the target's signal. We derive a distributed EM algorithm where each node makes use of its local information to compute summary statistics, and then shares these statistics with its neighbors to improve its estimate of the target localization. We show that our distributed algorithm converges, and simulation results suggest that our method achieves an accuracy close to the centralized EM algorithm. We apply the distributed EM algorithm to a set of experimental measurements with a network of four nodes, which confirm that the algorithm is able to localize a RF target in a realistic non-line-of-sight scenario.

Distributed data mining patterns and services: an architecture and experiments

SUMMARYDistributed data mining implements techniques for analyzing data on distributed computing systems by exploiting data distribution and parallel algorithms. The grid is a computing infrastructure for implementing distributed high‐performance applications and solving complex problems, offering effective support to the implementation and use of data mining and knowledge discovery systems. The Web Services Resource Framework has become the standard for the implementation of grid services and applications, and it can be exploited for developing high‐level services for distributed data mining applications. This paper describes how distributed data mining patterns, such as collective learning, ensemble learning, and meta‐learning models, can be implemented as Web Services Resource Framework mining services by exploiting the grid infrastructure. The goal of this work was to design a distributed architectural model that can be exploited for different distributed mining patterns deployed as grid services for the analysis of dispersed data sources. In order to validate such an approach, we presented also the implementation of two clustering algorithms on the developed architecture. In particular, the distributed k‐means and distributed expectation maximization were exploited as pilot examples to show the suitability of the implemented service‐oriented framework. An extensive evaluation of its performance was provided. Copyright © 2011 John Wiley & Sons, Ltd.

Distributed multi-target tracking using joint probabilistic data association and average consensus filter

In this paper, a distributed multi-target tracking (MTT) algorithm suitable for implementation in wireless sensor networks is proposed. For this purpose, the Monte Carlo (MC) implementation of joint probabilistic data-association filter (JPDAF) is applied to the well-known problem of multi-target tracking in a cluttered area. Also, to make the tracking algorithm scalable and usable for sensor networks of many nodes, the distributed expectation maximization algorithm is exploited via the average consensus filter, in order to diffuse the nodes’ information over the whole network. The proposed tracking system is robust and capable of modeling any state space with nonlinear and non-Gaussian models for target dynamics and measurement likelihood, since it uses the particle-filtering methods to extract samples from the desired distributions. To encounter the data-association problem that arises due to the unlabeled measurements in the presence of clutter, the well-known JPDAF algorithm is used. Furthermore, some simplifications and modifications are made to MC–JPDAF algorithm in order to reduce the computation complexity of the tracking system and make it suitable for low-energy sensor networks. Finally, the simulations of tracking tasks for a sample network are given.

Distributed Unsupervised Gaussian Mixture Learning for Density Estimation in Sensor Networks

This paper considers the problem of density estimation and clustering in sensor networks. It is assumed that measurements of the sensors can statistically be modeled by a common Gaussian mixture model (GMM). In this paper, a distributed expectation maximization (DEM) algorithm is developed to estimate the model order and the parameters of this model. Scalability and fault tolerance are two important advantages of this method. In the E-step of this algorithm, each node calculates local sufficient statistics using its local observations. A distributed averaging approach is then used to diffuse local sufficient statistics to neighboring nodes and estimate global sufficient statistics in each node. In the M-step, each node updates parameters of the GMM using the estimated global sufficient statistics. Diffusion speed and convergence of the proposed algorithm are also studied. The proposed method is then used for environmental monitoring and also distributed target classification. Simulation results approve the promising performance of this algorithm.

A distributed EM algorithm to estimate the parameters of a finite mixture of components

In this paper, a distributed expectation maximization (DEM) algorithm is first introduced in a general form for estimating the parameters of a finite mixture of components. This algorithm is used for density estimation and clustering of data distributed over nodes of a network. Then, a distributed incremental EM algorithm (DIEM) with a higher convergence rate is proposed. After a full derivation of distributed EM algorithms, convergence of these algorithms is analyzed based on the negative free energy concept used in statistical physics. An analytical approach is also developed for evaluating the convergence rate of both incremental and distributed incremental EM algorithms. It is analytically shown that the convergence rate of DIEM is much faster than that of the DEM algorithm. Finally, simulation results approve that DIEM remarkably outperforms DEM for both synthetic and real data sets.