Local Likelihood Function Research Articles

A modified EM-type algorithm to estimate semi-parametric mixtures of non-parametric regressions

Semi-parametric Gaussian mixtures of non-parametric regressions (SPGMNRs) are a flexible extension of Gaussian mixtures of linear regressions (GMLRs). The model assumes that the component regression functions (CRFs) are non-parametric functions of the covariate(s) whereas the component mixing proportions and variances are constants. Unfortunately, the model cannot be reliably estimated using traditional methods. A local-likelihood approach for estimating the CRFs requires that we maximize a set of local-likelihood functions. Using the Expectation-Maximization (EM) algorithm to separately maximize each local-likelihood function may lead to label-switching. This is because the posterior probabilities calculated at the local E-step are not guaranteed to be aligned. The consequence of this label-switching is wiggly and non-smooth estimates of the CRFs. In this paper, we propose a unified approach to address label-switching and obtain sensible estimates. The proposed approach has two stages. In the first stage, we propose a model-based approach to address the label-switching problem. We first note that each local-likelihood function is a likelihood function of a Gaussian mixture model (GMM). Next, we reformulate the SPGMNRs model as a mixture of these GMMs. Lastly, using a modified version of the Expectation Conditional Maximization (ECM) algorithm, we estimate the mixture of GMMs. In addition, using the mixing weights of the local GMMs, we can automatically choose the local points where local-likelihood estimation takes place. In the second stage, we propose one-step backfitting estimates of the parametric and non-parametric terms. The effectiveness of the proposed approach is demonstrated on simulated data and real data analysis.

Robust federated learning under statistical heterogeneity via Hessian spectral decomposition

Federated Learning (FL) is a collaborative machine learning paradigm in which a global model is learned via aggregating local ones. Although statistical heterogeneity of the local training data is necessary for the generalisability of the global model, it also introduces local model “drift” that slows down the convergence. Thus, how to optimally aggregate local models in FL remains an open problem. Recognising that training data lends varying evidential credence to different parts of a local model, we propose a novel approach to exploit such evidential asymmetry in FL aggregation in not independent and identically distributed (non-IID) data by applying a unique weight coefficient to each of the local parameter updates. To this end, we measure the parameter-level evidential credence making use of the eigenvalues of the Hessian of the local likelihood function, which are theoretically connected to the observed Fisher information. We employ these eigenpairs to propose a novel aggregation method, which we name FedHess. Our experiments show FedHess achieves smoother and faster convergence to a more accurate global model when compared with popular baselines such as Federated Average (FedAvg), FedProx, SCAFFOLD, Federated Curvature (FedCurv) and FedDF across different types of heterogeneous training data drawn from a number of benchmark datasets.

A Novel EM-Type Algorithm to Estimate Semi-Parametric Mixtures of Partially Linear Models

Semi- and non-parametric mixture of normal regression models are a flexible class of mixture of regression models. These models assume that the component mixing proportions, regression functions and/or variances are non-parametric functions of the covariates. Among this class of models, the semi-parametric mixture of partially linear models (SPMPLMs) combine the desirable interpretability of a parametric model and the flexibility of a non-parametric model. However, local-likelihood estimation of the non-parametric term poses a computational challenge. Traditional EM optimisation of the local-likelihood functions is not appropriate due to the label-switching problem. Separately applying the EM algorithm on each local-likelihood function will likely result in non-smooth function estimates. This is because the local responsibilities calculated at the E-step of each local EM are not guaranteed to be aligned. To prevent this, the EM algorithm must be modified so that the same (global) responsibilities are used at each local M-step. In this paper, we propose a one-step backfitting EM-type algorithm to estimate the SPMPLMs and effectively address the label-switching problem. The proposed algorithm estimates the non-parametric term using each set of local responsibilities in turn and then incorporates a smoothing step to obtain the smoothest estimate. In addition, to reduce the computational burden imposed by the use of the partial-residuals estimator of the parametric term, we propose a plug-in estimator. The performance and practical usefulness of the proposed methods was tested using a simulated dataset and two real datasets, respectively. Our finite sample analysis revealed that the proposed methods are effective at solving the label-switching problem and producing reasonable and interpretable results in a reasonable amount of time.

Local likelihood of quantile difference under left-truncated, right-censored and dependent assumptions

We, in this paper, focus on the inference of conditional quantile difference (CQD) for left-truncated and right-censored model. Based on local conditional likelihood function of the observed data, local likelihood ratio function and smoothed local log-likelihood ratio (log-SLL) of the CQD are constructed, and the maximum local likelihood estimator of the CQD is further defined from the log-SLL. When the observations are assumed to be a sequence of stationary α-mixing random variables, we establish asymptotic normality of the defined estimator, and prove the Wilks' theorem of adjusted log-SLL. Besides, we define another estimator of the CQD based on product-limit estimator of conditional distribution function and give its asymptotic normality. Also, simulation study and real data analysis are conducted to investigate the finite sample behaviour of the proposed methods.

A novel Bayesian federated learning framework to address multi-dimensional heterogeneity problem

<abstract><p>Federated learning (FL) has attracted a lot of interests as a promising machine learning approach to protect user privacy and data security. It requires the clients to send model parameters to the server rather than private datasets, thus protecting privacy to a significant extent. However, there are several types of heterogeneities (data, model, objective and systems) in FL scenario, posing distinct challenges to the canonical FL algorithm (FedAvg). In this work, we propose a novel FL framework that integrates knowledge distillation and Bayesian inference to address this multi-dimensional heterogeneity problem. On the client side, we approximate the local likelihood function using a scaled multi-dimensional Gaussian probability density function (PDF). Moreover, each client is allowed to design customized model according to the requirement through knowledge distillation. On the server side, a multi-Gaussian product mechanism is employed to construct and maximize the global likelihood function, greatly enhancing the accuracy of the aggregated model in the case of data heterogeneity. Finally, we show in extensive empirical experiments on various datasets and settings that global model and local model can achieve better performance and require fewer communication rounds to converge compared with other FL techniques.</p></abstract>

Fitting Non-Parametric Mixture of Regressions: Introducing an EM-Type Algorithm to Address the Label-Switching Problem

The non-parametric Gaussian mixture of regressions (NPGMRs) model serves as a flexible approach for the determination of latent heterogeneous regression relationships. This model assumes that the component means, variances and mixing proportions are smooth unknown functions of the covariates where the error distribution of each component is assumed to be Gaussian and hence symmetric. These functions are estimated over a set of grid points using the Expectation-Maximization (EM) algorithm to maximise the local-likelihood functions. However, maximizing each local-likelihood function separately does not guarantee that the local responsibilities and corresponding labels, obtained at the E-step of the EM algorithm, align at each grid point leading to a label-switching problem. This results in non-smooth estimated component regression functions. In this paper, we propose an estimation procedure to account for label switching by tracking the roughness of the estimated component regression functions. We use the local responsibilities to obtain a global estimate of the responsibilities which are then used to maximize each local-likelihood function. The performance of the proposed procedure is demonstrated using a simulation study and through an application using real world data. In the case of well-separated mixture regression components, the procedure gives similar results to competitive methods. However, in the case of poorly separated mixture regression components, the procedure outperforms competitive methods.

Distributed Fault Diagnosis of Nonlinear Stochastic Systems With Monitoring Networks

The design of a distributed fault detection and isolation filter is addressed for stochastic nonlinear systems subjected to multiple failure modes. The monolithic process is monitored by a network of communicating detection nodes, in lieu of a singleton, supervising diagnostic unit. Each detection node is a fault-sensitive estimator unit with processing capabilities and limited observation access to the process. The detection nodes exchange post-processed information over a communication network. An average consensus protocol is employed to fuse the detection nodes' outputs to obtain an agreement over the local likelihood functions. The proposed approach conducts online hypothesis testing without a bank of estimators, reducing the computational complexity of the algorithm significantly. Numerical simulations validate the efficiency of the proposed algorithm.

Speaker Tracking Based on Distributed Particle Filter and Iterative Covariance Intersection in Distributed Microphone Networks

Speaker tracking in distributed microphone networks is a challenging task due to the adverse effects of reverberation and noise. In this paper, a speaker tracking method based on distributed particle filter (DPF) and modified iterative covariance intersection (MICI) algorithm is proposed in distributed microphone networks. Specifically, the time difference of arrival (TDOA) of speech signals received by a pair of microphones at each node is first estimated using the generalized cross-correlation function. Next, multiple TDOAs are considered as the local measurement via a choice strategy and the multiple-hypothesis model is used as the local likelihood function of the DPF. Finally, the MICI algorithm is proposed to fuse local estimates to implement a global consistent speaker tracking where the local estimates at individual nodes are allowed to be unknown cross-correlations. The proposed method can successfully track the speaker in reverberant and noisy environments, and it is robust to the node faults and suitable to track speaker in networks with the unknown number of nodes. Simulation results demonstrate that the proposed method has better performance over the existing methods when SNR <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$&lt; $</tex-math></inline-formula> 10 dB. Real-world experiments reveal validity of the proposed method.

Estimation of Time-Varying Single Particle Tracking Models using Local Likelihood

Single particle tracking (SPT) is a class of experimental techniques and mathematical algorithms for following the motion of very small particles moving inside living cells, including viruses, proteins, and strands of RNA, to mention a few. There are several models in the literature to describe the motion of such small particles. Amongst the most common ones is the so-called simple diffusion model, which corresponds to a random walk with noisy observations. Maximum Likelihood (ML) is an estimation method to determine the value of parameters in a model. It has been utilised in previous works in the context of SPT to estimate key parameters in motion models. However, these works pose the limitation that the parameters are considered time-invariant. An early attempt to include time-varying parameters (TVP) is SPT considers a probability of change in a jump Markov model, however, it does not allow to continuously track TVP, requiring the inclusion of probabilities that might have no physical meaning. Here, the estimation problem of TVP is posed as a local time-invariant likelihood function, which is optimised (locally) to obtain parameters within a window of nominated span. This local likelihood function is centred at time t. To develop the idea, we introduce a weight K((u(i) - t)/h), where u(i) is the data inside the window, h is the window size. and t is a chosen point inside the window. The estimation algorithm continues when the time points are increased by one unit and terminates when the last data point is included in the window. Results of the estimation considering time-varying diffusion have been carried out with different window lengths. Our estimation algorithm is capable of producing a TVP for this simple motion model, having, as expected, less variance when considering longer windows.

A multivariate test against spurious long memory

This paper provides a multivariate score-type test against spurious long memory. We prove the consistency of the test against the alternatives of random level shifts and smooth trends. The test statistic is based on the weighted sum of the partial derivatives of the multivariate local Whittle likelihood function. To apply the test to fractionally cointegrated series, the test statistic is calculated for the linearly transformed system after estimating the cointegrating matrix. We derive the limiting distribution and show consistency of this procedure. The test is applied to log-absolute returns and log-realized volatilities of the S&P 500, DAX, FTSE, and NIKKEI.

Local Whittle likelihood estimators and tests for spatial lattice data

Likelihood methods for large spatial datasets on a d-dimensional lattice are often difficult, if not infeasible, to implement due to the computational limitation. When the number of observations gets increasingly large, it is challenging to evaluate the exact likelihood even if normality is assumed, since the covariance matrix is huge. To alleviate the computational difficulties, one can approximate the exact parametric likelihood through the Whittle likelihood, which is based on periodogram ordinates and spectral density functions. By applying fast Fourier transform, it requires less computational operations to calculate these magnitudes. In this paper, we define a local Whittle likelihood function, which can be used to estimate spectral density functions of general spatial lattice processes and a local Whittle likelihood ratio statistic for testing particular parametric spatial classes. When the bandwidth used is large, this method amounts to ordinary Whittle likelihood, while for moderate or small bandwidth, the method is essentially nonparametric estimation of spectral density functions. In addition, the asymptotic properties of the proposed estimator and associated test statistic are derived. It is shown that the asymptotic distribution of the test statistic is not affected by the underlying spatial asymptotic framework or the shape of the sampling regions. Simulation results and real data examples are also reported to illustrate the finite sample performance of the methodology.

Distributed particle filter based speaker tracking in distributed microphone networks under non-Gaussian noise environments

Non-Gaussian noise distorts the speech signals and leads to the degradation of speaker tracking performance. In this paper, a distributed particle filter (DPF) based speaker tracking method in distributed microphone networks under non-Gaussian noise and reverberant environments is proposed. A generalized correntropy function is first presented to estimate the time differences of arrival (TDOA) for speech signals at each node in distributed microphone networks. Next, to address spurious TDOA estimations caused by noise and reverberation, a multiple-hypothesis likelihood model is introduced to calculate the local likelihood functions of the DPF. Finally, a DPF fusing local likelihood functions with an average consensus algorithm is employed to estimate a moving speaker's positions. The proposed method can accurately track the speaker under non-Gaussian noise and reverberant environments, and it is scalable and robust against the nodes failure in distributed networks. Simulation experiments show the validation of the proposed speaker tracking method.

Distributed Marginalized Auxiliary Particle Filter for Speaker Tracking in Distributed Microphone Networks

In this paper, a distributed marginalized auxiliary particle filter DMAPF is proposed for speaker tracking in distributed microphone networks. After marginalizing the state-space model, the speaker's velocity and position are estimated using the distributed Kalman filter and the distributed auxiliary particle filter APF, respectively. To overcome the adverse effects of noise and reverberation, a time difference of arrival selection scheme is presented to construct the local observation vector, based on the generalized cross-correlation function of the microphone pair signals at each node. Next, the multiple-hypothesis model is used as the local likelihood function of the DMAPF. Finally, the DMAPF is employed to estimate the time-varying positions of a moving speaker. The proposed method combines the strengths of the marginalized particle filter, APF, and distributed estimation. It can track the speaker successfully in noisy and reverberant environments. Moreover, it requires only local communication among neighboring nodes, and is scalable for speaker tracking. Experimental results reveal the validity of the proposed speaker tracking method.

Distributed Particle Filter for Target Tracking: With Reduced Sensor Communications

For efficient and accurate estimation of the location of objects, a network of sensors can be used to detect and track targets in a distributed manner. In nonlinear and/or non-Gaussian dynamic models, distributed particle filtering methods are commonly applied to develop target tracking algorithms. An important consideration in developing a distributed particle filtering algorithm in wireless sensor networks is reducing the size of data exchanged among the sensors because of power and bandwidth constraints. In this paper, we propose a distributed particle filtering algorithm with the objective of reducing the overhead data that is communicated among the sensors. In our algorithm, the sensors exchange information to collaboratively compute the global likelihood function that encompasses the contribution of the measurements towards building the global posterior density of the unknown location parameters. Each sensor, using its own measurement, computes its local likelihood function and approximates it using a Gaussian function. The sensors then propagate only the mean and the covariance of their approximated likelihood functions to other sensors, reducing the communication overhead. The global likelihood function is computed collaboratively from the parameters of the local likelihood functions using an average consensus filter or a forward-backward propagation information exchange strategy.

Estimation of a change point in the variance function based on the χ2-distribution

ABSTRACTLet us consider that the variance function or its νth derivative in a regression model has a change/discontinuity point at an unknown location. To use the local polynomial fits, the log-variance function which break the positivity is targeted. The location and the jump size of the change point are estimated based on a one-sided kernel-weighted local-likelihood function which is provided by the χ2-distribution. The whole structure of the log-variance function is then estimated using the data sets split by the estimated location. Asymptotic results of the proposed estimators are described. Numerical works demonstrate the performances of the methods with simulated and real examples.

Identical inferences about correlated evolution arise from ancestral state reconstruction and independent contrasts.

Inferences about the evolution of continuous traits based on reconstruction of ancestral states have often been considered more error-prone than analysis of independent contrasts. Here we show that both methods in fact yield identical estimators for the correlation coefficient and regression gradient of correlated traits, indicating that reconstructed ancestral states are a valid source of information about correlated evolution. We show that the independent contrast associated with a pair of sibling nodes on a phylogenetic tree can be expressed in terms of the maximum likelihood ancestral state function at those nodes and their common parent. This expression gives rise to novel formulae for independent contrasts for any model of evolution admitting of a local likelihood function. We thus derive new formulae for independent contrasts applicable to traits evolving under directional drift, and use simulated data to show that these directional contrasts provide better estimates of evolutionary model parameters than standard independent contrasts, when traits in fact evolve with a directional tendency.

Consensus-based Distributed Particle Filtering With Distributed Proposal Adaptation

We develop a distributed particle filter for sequential estimation of a global state in a decentralized wireless sensor network. A global state estimate that takes into account the measurements of all sensors is computed in a distributed manner, using only local calculations at the individual sensors and local communication between neighboring sensors. The paper presents two main contributions. First, the likelihood consensus scheme for distributed calculation of the joint likelihood function (used by the local particle filters) is generalized to arbitrary local likelihood functions. This generalization overcomes the restriction to exponential-family likelihood functions that limited the applicability of the original likelihood consensus (Hlinka , “Likelihood consensus and its application to distributed particle filtering,” IEEE Trans. Signal Process., vol. 60, pp. 4334–4349, Aug. 2012). The second contribution is a consensus-based distributed method for adapting the proposal densities used by the local particle filters. This adaptation takes into account the measurements of all sensors, and it can yield a significant performance improvement or, alternatively, a significant reduction of the number of particles required for a given level of accuracy. The performance of the proposed distributed particle filter is demonstrated for a target tracking problem.

Belief consensus algorithms for fast distributed target tracking in wireless sensor networks

In distributed target tracking for wireless sensor networks, agreement on the target state can be achieved by the construction and maintenance of a communication path, in order to exchange information regarding local likelihood functions. Such an approach lacks robustness to failures and is not easily applicable to ad-hoc networks. To address this, several methods have been proposed that allow agreement on the global likelihood through fully distributed belief consensus (BC) algorithms, operating on local likelihoods in distributed particle filtering (DPF). However, a unified comparison of the convergence speed and communication cost has not been performed. In this paper, we provide such a comparison and propose a novel BC algorithm based on belief propagation (BP). According to our study, DPF based on metropolis belief consensus (MBC) is the fastest in loopy graphs, while DPF based on BP consensus is the fastest in tree graphs. Moreover, we found that BC-based DPF methods have lower communication overhead than data flooding when the network is sufficiently sparse.

Likelihood Consensus and Its Application to Distributed Particle Filtering

We consider distributed state estimation in a wireless sensor network without a fusion center. Each sensor performs a global estimation task---based on the past and current measurements of all sensors---using only local processing and local communications with its neighbors. In this estimation task, the joint (all-sensors) likelihood function (JLF) plays a central role as it epitomizes the measurements of all sensors. We propose a distributed method for computing, at each sensor, an approximation of the JLF by means of consensus algorithms. This "likelihood consensus" method is applicable if the local likelihood functions of the various sensors (viewed as conditional probability density functions of the local measurements) belong to the exponential family of distributions. We then use the likelihood consensus method to implement a distributed particle filter and a distributed Gaussian particle filter. Each sensor runs a local particle filter, or a local Gaussian particle filter, that computes a global state estimate. The weight update in each local (Gaussian) particle filter employs the JLF, which is obtained through the likelihood consensus scheme. For the distributed Gaussian particle filter, the number of particles can be significantly reduced by means of an additional consensus scheme. Simulation results are presented to assess the performance of the proposed distributed particle filters for a multiple target tracking problem.

Nonparametric estimation of the regression function having a change point in generalized linear models

In this paper, the local polynomial fit based on the kernel weighted local-likelihood function and the location of the change point is considered as an estimator for the regression function or its νth derivative. Using the data sets split by the location, we estimate the left and right parts of the regression function or its νth derivative. The global L2 rate of convergence of the estimated function is derived. The finite-sample performances of the proposed estimators are illustrated using simulated and real examples.