Expectation Conditional Maximization Research Articles

Robust Clustering in Regression Analysis via the Contaminated Gaussian Cluster-Weighted Model

The Gaussian cluster-weighted model (CWM) is a mixture of regression models with random covariates that allows for flexible clustering of a random vector composed of response variables and covariates. In each mixture component, a Gaussian distribution is adopted for both the covariates and the responses given the covariates. To make the approach robust with respect to the presence of mildly atypical observations, the contaminated Gaussian CWM is introduced. In addition to the parameters of the Gaussian CWM, each mixture component has a parameter controlling the proportion of outliers, one controlling the proportion of leverage points, one specifying the degree of contamination with respect to the response variables, and another specifying the degree of contamination with respect to the covariates. Crucially, these parameters do not have to be specified a priori, adding flexibility to the approach. Furthermore, once the model is estimated and the observations are assigned to the components, a finer intra-group classification in typical points, (mild) outliers, good leverage points, and bad leverage points—concepts of primary importance in robust regression analysis—can be directly obtained. Relations with other mixture-based contaminated models are analyzed, identifiability conditions are provided, an expectation-conditional maximization algorithm is outlined for parameter estimation, and various implementation and operational issues are discussed. Properties of the estimators of the regression coefficients are evaluated through Monte Carlo experiments and compared with other procedures. A sensitivity study is also conducted based on a real data set.

Lifetime prognostics for deteriorating systems with time-varying random jumps

In this paper, we propose a jump diffusion process with non-homogeneous compound Poisson process to model the degradation process with randomly occurring jumps, which combines two stochastic processes, i.e., traditional diffusion process to describe the continuous degradation and non-homogeneous compound Poisson process to depict random jumps with a time-varying intensity. The approximated analytical lifetime under the concept of the first passage time (FPT) is obtained by a time–space transformation technique. To identify the model parameters, we first present a general method based on Maximum Likelihood Estimation (MLE) for the proposed model, and then specifically provide a two-step approach for linear jump diffusion process via combining MLE and Expectation Conditional Maximization (ECM) algorithm. Finally, a numerical example and a study on the furnace wall are provided to illustrate the effectiveness of the proposed method.

A Penalized Likelihood Method for Structural Equation Modeling.

A penalized likelihood (PL) method for structural equation modeling (SEM) was proposed as a methodology for exploring the underlying relations among both observed and latent variables. Compared to the usual likelihood method, PL includes a penalty term to control the complexity of the hypothesized model. When the penalty level is appropriately chosen, the PL can yield an SEM model that balances the model goodness-of-fit and model complexity. In addition, the PL results in a sparse estimate that enhances the interpretability of the final model. The proposed method is especially useful when limited substantive knowledge is available for model specifications. The PL method can be also understood as a methodology that links the traditional SEM to the exploratory SEM (Asparouhov & Muthén in Struct Equ Model Multidiscipl J 16:397-438, 2009). An expectation-conditional maximization algorithm was developed to maximize the PL criterion. The asymptotic properties of the proposed PL were also derived. The performance of PL was evaluated through a numerical experiment, and two real data illustrations were presented to demonstrate its utility in psychological research.

Mixture Of Probabilistic Factor Analyzers For Market Risk Measurement: Empirical Evidence From The Tunisian Foreign Exchange Market

In this paper, we propose a new approach for Basel-Compliant Value-at-Risk (VaR) estimation in financial portfolio risk management, which combines Gaussian Mixture Models with probabilistic factor analysis models. This new mixed specification provides an alternative, compact, model to handle co-movements, heterogeneity and intra-frame correlations in financial data. This results in a model which concurrently performs clustering and dimensionality reduction, and can be considered as a reduced dimension mixture of probabilistic factor analyzers. For maximum likelihood estimation we have used an iterative approach based on the Alternating Expectation Conditional Maximization (AECM) algorithm. Using a set of historical data in a rolling time window, from the Tunisian foreign exchange market, the model structure as well as its parameters are determined and estimated. Then, the fitted model combined with a modified Monte-Carlo simulation algorithm was used to predict the VaR. Through a Backtesting analysis, we found that this new specification exhibits a good fit to the data compared to other competing approaches, improves the accuracy of VaR prediction, possesses more flexibility, and can avoid serious violations when a financial crisis occurs.

Multivariate-$t$ nonlinear mixed models with application to censored multi-outcome AIDS studies.

In multivariate longitudinal HIV/AIDS studies, multi-outcome repeated measures on each patient over time may contain outliers, and the viral loads are often subject to a upper or lower limit of detection depending on the quantification assays. In this article, we consider an extension of the multivariate nonlinear mixed-effects model by adopting a joint multivariate-$t$ distribution for random effects and within-subject errors and taking the censoring information of multiple responses into account. The proposed model is called the multivariate-$t$ nonlinear mixed-effects model with censored responses (MtNLMMC), allowing for analyzing multi-outcome longitudinal data exhibiting nonlinear growth patterns with censorship and fat-tailed behavior. Utilizing the Taylor-series linearization method, a pseudo-data version of expectation conditional maximization either (ECME) algorithm is developed for iteratively carrying out maximum likelihood estimation. We illustrate our techniques with two data examples from HIV/AIDS studies. Experimental results signify that the MtNLMMC performs favorably compared to its Gaussian analogue and some existing approaches.

Multi-Relay Communications in the Presence of Phase Noise and Carrier Frequency Offsets

Impairments such as time varying phase noise (PHN) and carrier frequency offset (CFO) result in loss of synchronization and poor performance of multi-relay communication systems. Joint estimation of these impairments is necessary in order to correctly decode the received signal at the destination. In this paper, we address spectrally efficient multi-relay transmission scenarios where all the relays simultaneously communicate with the destination. We propose an iterative pilot-aided algorithm based on the expectation conditional maximization for joint estimation of multipath channels, Wiener PHNs, and CFOs in decode-and-forward-based multi-relay orthogonal frequency division multiplexing systems. Next, a new expression of the hybrid Cramer–Rao lower bound (HCRB) for the multi-parameter estimation problem is derived. Finally, an iterative receiver based on an extended Kalman filter for joint data detection and PHN tracking is employed. Numerical results show that the proposed estimator outperforms existing algorithms and its mean square error performance is close to the derived HCRB at different signal-to-noise ratios for different PHN variances. In addition, the combined estimation algorithm and the iterative receiver can significantly improve average bit-error rate (BER) performance compared with existing algorithms. In addition, the BER performance of the proposed system is close to the ideal case of perfect channel impulse responses, PHNs, and CFOs estimation.

Frequency-Asynchronous Multiuser Joint Channel-Parameter Estimation, CFO Compensation, and Channel Decoding

This paper investigates a channel-coded multiuser system operated with orthogonal frequency-division multiplexing (OFDM) and interleaved division multiple-access (IDMA). To realize the potential advantage of OFDM-IDMA, two challenges must be addressed. The first challenge is the estimation of multiple channel parameters. An issue is how to contain the estimation errors of the channel parameters of the multiple users, considering that the overall estimation errors may increase with the number of users because the estimations of their channel parameters are intertwined with each other. The second challenge is that the transmitters of the multiple users may be driven by different RF oscillators. The associated frequency asynchrony may cause multiple CFOs at the receiver. Compared with a single-user receiver where the single CFO can be compensated away, a particular difficulty for a multiuser receiver is that it is not possible to compensate for all the multiple CFOs simultaneously. To tackle the two challenges, we put forth a framework to solve the problems of multiuser channel-parameter estimation, CFO compensation, and channel decoding jointly and iteratively. The framework employs the space alternating generalized expectation-maximization (SAGE) algortihm to decompose the multisuser problem into multiple single-user problems, and the expectation-conditional maximization (ECM) algorithm to tackle each of the single-user subproblems. Iterative executions of SAGE and ECM in the framework allow the two aforementioned challenges to be tackled in an optimal manner. Simulations and real experiments based on software-defined radio indicate that, compared with other approaches, our approach can achieve significant performance gains.

Auto Regulated Data Provisioning Scheme with Adaptive Buffer Resilience Control on Federated Clouds

We investigate the channel state information (CSI) in multi-input multi-output (MIMO) cooperative networks that employ the amplify-and-forward transmission scheme. Least squares and expectation conditional maximization have been proposed in the system. However, neither of these two approaches takes advantage of channel sparsity, and they cause estimation performance loss. Unlike linear channel estimation methods, several compressed channel estimation methods are proposed in this study to exploit the sparsity of the MIMO cooperative channels based on the theory of compressed sensing. First, the channel estimation problem is formulated as a compressed sensing problem by using sparse decomposition theory. Second, the lower bound is derived for the estimation, and the MIMO relay channel is reconstructed via compressive sampling matching pursuit algorithms. Finally, based on this model, we propose a novel algorithm so called sparsity adaptive expectation maximization (SAEM) by using Kalman filter and expectation maximization algorithm so that it can exploit channel sparsity alternatively and also track the true support set of time-varying channel. Kalman filter is used to provide soft information of transmitted signals to the EM-based algorithm. Various numerical simulation results indicate that the proposed sparse channel estimation technique outperforms the previous estimation schemes.

Penalized spline joint models for longitudinal and time-to-event data

ABSTRACTThe joint models for longitudinal data and time-to-event data have recently received numerous attention in clinical and epidemiologic studies. Our interest is in modeling the relationship between event time outcomes and internal time-dependent covariates. In practice, the longitudinal responses often show non linear and fluctuated curves. Therefore, the main aim of this paper is to use penalized splines with a truncated polynomial basis to parameterize the non linear longitudinal process. Then, the linear mixed-effects model is applied to subject-specific curves and to control the smoothing. The association between the dropout process and longitudinal outcomes is modeled through a proportional hazard model. Two types of baseline risk functions are considered, namely a Gompertz distribution and a piecewise constant model. The resulting models are referred to as penalized spline joint models; an extension of the standard joint models. The expectation conditional maximization (ECM) algorithm is applied to estimate the parameters in the proposed models. To validate the proposed algorithm, extensive simulation studies were implemented followed by a case study. In summary, the penalized spline joint models provide a new approach for joint models that have improved the existing standard joint models.

Clustering Multivariate Longitudinal Observations: The Contaminated Gaussian Hidden Markov Model

The Gaussian hidden Markov model (HMM) is widely considered for the analysis of heterogenous continuous multivariate longitudinal data. To robustify this approach with respect to possible elliptical heavy-tailed departures from normality, due to the presence of outliers, spurious points, or noise (collectively referred to as bad points herein), the contaminated Gaussian HMM is here introduced. The contaminated Gaussian distribution represents an elliptical generalization of the Gaussian distribution and allows for automatic detection of bad points in the same natural way as observations are typically assigned to the latent states in the HMM context. Once the model is fitted, each observation has a posterior probability of belonging to a particular state and, inside each state, of being a bad point or not. In addition to the parameters of the classical Gaussian HMM, for each state we have two more parameters, both with a specific and useful interpretation: one controls the proportion of bad points and one specifies their degree of atypicality. A sufficient condition for the identifiability of the model is given, an expectation-conditional maximization algorithm is outlined for parameter estimation and various operational issues are discussed. Using a large-scale simulation study, but also an illustrative artificial dataset, we demonstrate the effectiveness of the proposed model in comparison with HMMs of different elliptical distributions, and we also evaluate the performance of some well-known information criteria in selecting the true number of latent states. The model is finally used to fit data on criminal activities in Italian provinces. Supplementary materials for this article are available online

Generalized linear latent models for multivariate longitudinal measurements mixed with hidden Markov models

This article presents a generalized linear latent variable model for analyzing multivariate longitudinal data within the hidden Markov model framework. The relationships among multiple items are captured by several common latent factors. The linear coregionalization method is adopted to model the temporal processes of latent variables. The merit of this modeling strategy lies in the fact that the processes among latent variables are nonseparate and codependent from each other. To account for possible heterogeneity and interrelationship among the longitudinal data, a hidden Markov model is introduced to model the transition probabilities across different latent states over time. The Monte Carlo expectation conditional maximization (MCECM) algorithm is developed to estimate unknown parameters in the proposed model. The Wald- and score-type statistics are proposed to test the related dependence of processes. A simulation study is conducted to investigate the performance of the proposed methodology. An example from a longitudinal study of cocaine use is taken to illustrate the proposed methodology.

Maximum-Likelihood Estimation of Parameters of NHPP Software Reliability Models Using Expectation Conditional Maximization Algorithm

Since its introduction in 1977, the expectation maximization (EM) algorithm has been one of the most important and widely used estimation method in estimating parameters of distributions in the presence of incomplete information. In this paper, a variant of the EM algorithm, the expectation conditional maximization (ECM) algorithm, is introduced for the first time and it provides a promising alternative in estimating the parameters of nonhomogeneous poisson (NHPP) software reliability growth models (SRGM). This algorithm circumvents the difficult M-step of the EM algorithm by replacing it by a series of conditional maximization steps. The utility of the ECM approach is demonstrated in the estimation of parameters of several well-known models for both time domain and time interval software failure data. Numerical examples with real-data indicate that the ECM algorithm performs well in estimating parameters of NHPP SRGM with complex mean value functions and can produce a faster rate of convergence.

Joint Frequency Offset and Channel Estimation in Distributed MIMO-OFDM Systems

In recent times, cooperative communications or distributed multi-input multi-output (DMIMO) systems are being considered as key enabler for increasing spectral efficiency and coverage area in 4G and beyond wireless communication system. DMIMO system employs multiple relays each with single or multiple antennas in between the source and destination to form a virtual antenna array. However, a few of the fundamental difficulties in deployment of DMIMO networks with coherent communications are synchronization, tracking of independent oscillators, and channel estimation. The signal received at the destination is characterized by multiple carrier frequency offsets (MCFOs) and MIMO channel effects due to sparsely distributed relay nodes. In this paper, we address the issue of joint estimation of frequency offset and channel gains in a DMIMO orthogonal frequency division multiplexing (OFDM) systems deploying multiple antennas at the relays. We consider estimate-and-forward (EF) protocol for relaying where the coarse estimation process at the low cost relays generates inter carrier interference (ICI) in a multi-carrier system resulting in severe performance degradation and decrease in reliability of the network. The two iterative estimators, expectation conditional maximization (ECM) and space-alternating generalized expectation maximization (SAGE), are proposed to jointly estimate MCFOs and channel in the presence of ICI. The robustness of the estimators are evaluated through mathematical analysis and validated using simulations. The performance of the estimators is measured in terms of mean square error (MSE) and bit error rate (BER). Simulation results show that the proposed estimators provide a significant performance gain in DMIMO-OFDM systems with MIMO configuration at the relays compared to single-input single-output (SISO) system.

Latent variable selection in structural equation models

Structural equation models (SEMs) are often formulated using a prespecified parametric structural equation. In many applications, however, the formulation of the structural equation is unknown, and its misspecification may lead to unreliable statistical inference. This paper develops a general SEM in which latent variables are linearly regressed on themselves, thereby avoiding the need to specify outcome/explanatory latent variables. A penalized likelihood method with a proper penalty function is proposed to simultaneously select latent variables and estimate the coefficient matrix in formulating the structural equation. Under some regularity conditions, we show the consistency and the oracle property of the proposed estimators. We also develop an expectation/conditional maximization (ECM) algorithm involving a minorization–maximization algorithm that facilitates the second M-step. Simulation studies are performed and a real data set is analyzed to illustrate the proposed methods. • Propose a PL method to identify the structure of coefficient matrix. • Show the consistency and the oracle property of the proposed estimators. • Propose a minorization–maximization algorithm to facilitate the second M-step.

Parsimonious mixtures of multivariate contaminated normal distributions.

A mixture of multivariate contaminated normal distributions is developed for model-based clustering. In addition to the parameters of the classical normal mixture, our contaminated mixture has, for each cluster, a parameter controlling the proportion of mild outliers and one specifying the degree of contamination. Crucially, these parameters do not have to be specified apriori, adding a flexibility to our approach. Parsimony is introduced via eigen-decomposition of the component covariance matrices, and sufficient conditions for the identifiability of all the members of the resulting family are provided. An expectation-conditional maximization algorithm is outlined for parameter estimation and various implementation issues are discussed. Using a large-scale simulation study, the behavior of the proposed approach is investigated and comparison with well-established finite mixtures is provided. The performance of this novel family of models is also illustrated on artificial and real data.

Model-based time-varying clustering of multivariate longitudinal data with covariates and outliers

A class of multivariate linear models under the longitudinal setting, in which unobserved heterogeneity may evolve over time, is introduced. A latent structure is considered to model heterogeneity, having a discrete support and following a first-order Markov chain. Heavy-tailed multivariate distributions are introduced to deal with outliers. Maximum likelihood estimation is performed to estimate parameters by using expectation–maximization and expectation–conditional-maximization algorithms. Notes on model identifiability and robustness are provided, along with all computational details needed to implement the proposal. Three applications on artificial and real data are illustrated. These focus on the potential effects of outliers on clustering and their identification.

Hidden Markov latent variable models with multivariate longitudinal data.

Cocaine addiction is chronic and persistent, and has become a major social and health problem in many countries. Existing studies have shown that cocaine addicts often undergo episodic periods of addiction to, moderate dependence on, or swearing off cocaine. Given its reversible feature, cocaine use can be formulated as a stochastic process that transits from one state to another, while the impacts of various factors, such as treatment received and individuals' psychological problems on cocaine use, may vary across states. This article develops a hidden Markov latent variable model to study multivariate longitudinal data concerning cocaine use from a California Civil Addict Program. The proposed model generalizes conventional latent variable models to allow bidirectional transition between cocaine-addiction states and conventional hidden Markov models to allow latent variables and their dynamic interrelationship. We develop a maximum-likelihood approach, along with a Monte Carlo expectation conditional maximization (MCECM) algorithm, to conduct parameter estimation. The asymptotic properties of the parameter estimates and statistics for testing the heterogeneity of model parameters are investigated. The finite sample performance of the proposed methodology is demonstrated by simulation studies. The application to cocaine use study provides insights into the prevention of cocaine use.

ECM and SAGE based joint estimation of timing and frequency offset for DMIMO–OFDM system

Distributed multiple-input multiple-output (DMIMO) technology is a key enabler of coverage extension and enhancement of link reliability in wireless networks through distributed spatial diversity. DMIMO employs classic relay channels in between the source and the destination to opportunistically form a virtual antenna array (VAA) for emulating cooperative diversity. Use of multiple antennas at the relays further increases capacity and reliability of the relay–destination channel through multiplexing and diversity of MIMO antennas respectively. In such network, the signal received at the destination is characterized by multiple timing offsets (MTO) due to different propagation delay and multiple carrier frequency offsets (MCFO) due to independent oscillators of the relays. Hence, synchronization becomes a crucial issue in DMIMO in order to realize the distributed coherence. In this paper, we address joint estimation of MCFO and MTO in DMIMO orthogonal frequency division multiplexing (OFDM) with MIMO configuration at the relays for estimate-and-forward (EF) relaying protocol. Two iterative algorithms, based on expectation conditional maximization (ECM) and space alternating generalized expectation–maximization (SAGE) are proposed for joint estimation in presence of inter carrier interference (ICI). The robustness of both the estimators to ICI is evaluated by mathematical analysis and supported by extensive simulations. The performance of the proposed estimators is assessed in terms of mean square error (MSE) and bit error rate (BER). The theoretical Cramer–Rao lower bound (CRLB) of estimator error variance is also derived.

Extending multivariate- t linear mixed models for multiple longitudinal data with censored responses and heavy tails.

The analysis of complex longitudinal data is challenging due to several inherent features: (i) more than one series of responses are repeatedly collected on each subject at irregularly occasions over a period of time; (ii) censorship due to limits of quantification of responses arises left- and/or right- censoring effects; (iii) outliers or heavy-tailed noises are possibly embodied within multiple response variables. This article formulates the multivariate- t linear mixed model with censored responses (MtLMMC), which allows the analysts to model such data in the presence of the above described features simultaneously. An efficient expectation conditional maximization either (ECME) algorithm is developed to carry out maximum likelihood estimation of model parameters. The implementation of the E-step relies on the mean and covariance matrix of truncated multivariate- t distributions. To enhance the computational efficiency, two auxiliary permutation matrices are incorporated into the procedure to determine the observed and censored parts of each subject. The proposed methodology is demonstrated via a simulation study and a real application on HIV/AIDS data.

Constrained fuzzy evidential multivariate model identified by EM algorithm: a soft sensor to monitoring imprecise and uncertain process parameters

This paper focuses on the problem of monitoring/estimating process parameters in the insufficient case when only imprecise and uncertain information can be obtained, possibly due to limited precision and reliability of sensors in industries. To solve this problem, a constrained fuzzy evidential multivariate model is proposed as a soft sensor to monitor imprecise and uncertain process parameters. The most challenging task involved in the modeling is how to identify structure parameters of the monitor model, especially under sets of constraints. To tackle this challenge, we represent the imprecise and uncertain information as fuzzy belief functions in the evidence theory framework, and then propose a restricted fuzzy evidential Expectation-Conditional Maximization algorithm (RFE2CM) for maximum likelihood estimation from fuzzy belief functions under linear inequality constraints. Also, the convergence property of the restricted fuzzy evidential EM algorithm is discussed. In order to validate the performance of the proposed model and algorithm, some numerical simulations are conducted as well as an experimental simulation on a real ball mill in a power plant. The numerical and experimental simulation results show that the proposed model and algorithm can not only be feasibly applicable to monitor the process parameters in insufficient informatics cases, but also have high prediction accuracy with small mean square errors.