Generalized Expectation Maximization Research Articles

A Nonlinear Quantile Regression for Accelerated Destructive Degradation Testing Data

Traditional regression approaches to accelerated destructive degradation test (ADDT) data have modeled the mean curve as representative. However, maximum likelihood estimates (MLEs) of the mean model are likely to be biased when the data are non-Gaussian or highly skewed. The median model can be an alternative for skewed degradation data. In this work, we introduce a nonlinear quantile regression (QR) approach for estimating quantile curves of ADDT data. We propose an iterative QR algorithm that uses the generalized expectation-maximization (GEM) framework to estimate the parameters of the nonlinear QR ADDT model, based on the asymmetric Laplace distribution to accommodate non-Gaussian and skewed errors. Using the asymptotic properties of the QR parameter estimates, we estimate variance-covariance matrix for the τ th QR parameters using order statistics and bootstrap methods. We propose a new prediction method of the quantile of the failure-time distribution in the normal use condition. Confidence intervals for the quantiles of the failure-time distribution are constructed using the parametric bootstrap method. The proposed model is illustrated using an industrial application and compared with the existing model. Various quantile curve estimates derived using the QR ADDT model provide a more flexible modeling framework than the traditional mean ADDT modeling approach.

Rethinking Peculiar Images by Diffusion Models: Revealing Local Minima’s Role

Recent significant advancements in diffusion models have revolutionized image generation, enabling the synthesis of highly realistic images with text-based guidance. These breakthroughs have paved the way for constructing datasets via generative artificial intelligence (AI), offering immense potential for various applications. However, two critical challenges hinder the widespread adoption of synthesized data: computational cost and the generation of peculiar images. While computational costs have improved through various approaches, the issue of peculiar image generation remains relatively unexplored. Existing solutions rely on heuristics, extra training, or AI-based post-processing to mitigate this problem. In this paper, we present a novel approach to address both issues simultaneously. We establish that both gradient descent and diffusion sampling are specific cases of the generalized expectation maximization algorithm. We hypothesize and empirically demonstrate that peculiar image generation is akin to the local minima problem in optimization. Inspired by optimization techniques, we apply naive momentum and positive-negative momentum to diffusion sampling. Last, we propose new metrics to evaluate the peculiarity. Experimental results show momentum effectively prevents peculiar image generation without extra computation.

Multipath-assisted positioning enhancement via DC-SLAM with MAP-PF: An in-depth analysis and performance evaluation

In indoor environments, traditional 5G positioning methods, such as trilateration and triangulation, encounter Non-Line-Of-Sight (NLOS) interference, significantly reducing the positioning accuracy. Notably, the quasi-static indoor channel offers valuable environmental features conducive to positioning. We present the newly developed Deep Coupling-Simultaneous Localization And Mapping (DC-SLAM) method that utilizes specular reflection multipath. DC-SLAM jointly estimates the delay and angle of multipath components, as well as the positions of mirror Virtual Anchors (VAs) and User Equipment (UE). Using raw sensor data, specifically channel state information sequences as observations, DC-SLAM integrates the Space Alternating Generalized Expectation Maximization (SAGE) with Bayesian filtering into a single algorithm employing the Maximum A Posteriori Probability-Penalty Function (MAP-PF) approach. In multipath signal processing, the penalty function serves as a prior distribution, guiding the algorithm to find a likelihood function peak closely aligned with the tracker-estimated delay and angle. In Bayesian filtering, it functions as the observational likelihood, updating information on predicted positions of UE and VAs. We carried out a numerical simulation experiment in an office setting. Results indicate that the accuracy of multipath delay and angle determined by MAP-PF generally surpasses that of pure SAGE. Furthermore, the precision of positions for VAs and UE derived from MAP-PF markedly exceeds those from belief propagation-SLAM, a non-joint estimation approach. Optimally, DC-SLAM achieves an average positioning accuracy of 0.11 m in a 2D plane for UE.

Semi-Supervised Mixture Learning for Graph Neural Networks With Neighbor Dependence.

A graph neural network (GNN) is a powerful architecture for semi-supervised learning (SSL). However, the data-driven mode of GNNs raises some challenging problems. In particular, these models suffer from the limitations of incomplete attribute learning, insufficient structure capture, and the inability to distinguish between node attribute and graph structure, especially on label-scarce or attribute-missing data. In this article, we propose a novel framework, called graph coneighbor neural network (GCoNN), for node classification. It is composed of two modules: GCoNN Γ and GCoNN Γ° . GCoNN Γ is trained to establish the fundamental prototype for attribute learning on labeled data, while GCoNN [Formula: see text] learns neighbor dependence on transductive data through pseudolabels generated by GCoNN Γ . Next, GCoNN Γ is retrained to improve integration of node attribute and neighbor structure through feedback from GCoNN [Formula: see text] . GCoNN tends to convergence iteratively using such an approach. From a theoretical perspective, we analyze this iteration process from a generalized expectation-maximization (GEM) framework perspective which optimizes an evidence lower bound (ELBO) by amortized variational inference. Empirical evidence demonstrates that the state-of-the-art performance of the proposed approach outperforms other methods. We also apply GCoNN to brain functional networks, the results of which reveal response features across the brain which are physiologically plausible with respect to known language and visual functions.

Low complexity joint angle and delay estimation for underwater multipath acoustic channel based on sparse Bayesian learning

In the field of underwater acoustic communication, multipath signals arrive at the receiving end with different angles and path delays, which can cause degradation in communication quality. When performing angle-delay joint estimation, sparse Bayesian learning (SBL) has many advantages over traditional methods, such as high accuracy, resolution and global minimum results, but at the cost of high computational complexity. Regarding this, in this study, we propose a low-complexity SBL based framework to jointly estimate the angle and delay of the received signal. We first propose a low-complexity approximation SBL algorithm applied to joint estimation, which can keep the super-resolution and estimation accuracy but significantly reduce the computation time. To further tackle the problem of performance degradation of the approximate algorithm in underwater low signal-to-noise ratio environments, we utilize above result as a starting point for the space-alternating generalized Expectation Maximization (SAGE) algorithm to achieve a more precise estimation. Simulation and experimental results demonstrate that the proposed algorithm has an excellent performance in various underwater multipath scenarios.

Model-based clustering of multipath propagation in powerline communication channels

Powerline communication (PLC) channels are known to exhibit multipath propagation behaviour. The authors present a model-based framework to address the challenge of clustering multipath propagation components (MPCs) in PLC channels for indoor low-voltage (LV) environments. The framework employs a range of finite-mixture models (FMMs), including the gamma mixture model, the inverse gamma mixture model, the Gaussian mixture model, the inverse Gaussian mixture model, the Nakagami mixture model, the inverse Nakagami mixture model (INMM) and the Rayleigh mixture model, to identify clusters of MPCs. A measurement campaign of an unknown indoor LV PLC channel is conducted to obtain a channel response. From the channel response, the delay and magnitude parameters of the MPCs are extracted using the space-alternating generalised expectation maximisation algorithm adopted only for these parameters. A maximum likelihood approach and the expectation–maximisation algorithm are employed to fit the FMMs to the MPC delay-magnitude dataset to cluster MPCs in the delay domain. The results of the model-fitting process are then evaluated using the corrected Akaike information criterion (AICc), which enables a fair comparison of the candidate models over the feasible and finite range of clusters. A novel algorithm is introduced for estimating the feasible and finite range of clusters using the extracted delay and magnitude MPC parameters. The AICc’s ranking results show that the INMM model provides the best fit. Davies–Bouldin (DB) and Calinski–Harabasz (CH) indexes are used to compare the model-based clustering approach to the conventional distance-based clustering methods. Validation results show that CH and DB indexes closely agree in the optimal number of MPC clusters for model-based clustering, which corresponds to the most within-cluster compactness of MPCs and to the most between-cluster separation in the delay domain.

Improving Next Location Recommendation Services With Spatial-Temporal Multi-Group Contrastive Learning

Next location recommendation services play a pivotal role in Location-Based Social Networks (LBSNs) due to their ability to provide personalized recommendations of attractive destinations, resulting in substantial benefits for both users and service providers. Recent research indicates that these services are influenced by both sequential and geographical factors. However, we argue that most of these services fail to fully exploit the latent multi-group knowledge of location semantics and user preferences, resulting in suboptimal performance. Therefore, we propose STMGCL, a novel spatial-temporal multi-group contrastive learning-based method to discover intrinsic multi-group information for improving next location recommendation services. Specifically, STMGCL designs Spatial Group Contrastive Learning (SGCL) to extract multiple group knowledge regarding location semantics. Additionally, it develops Temporal Group Contrastive Learning (TGCL) to explore multiple user preference group information through a self-attention based encoder. Finally, we leverage a multi-task learning strategy and a generalized Expectation Maximization (EM) algorithm to ensure that STMGCL is optimized end-to-end with guaranteed convergence. Extensive experiments conducted on four real-world datasets demonstrate the superior performance of STMGCL over baselines.

Expectation-maximization type algorithms for direction of arrival estimation in unknown nonuniform noise

The expectation–maximization (EM) algorithm updates all of the parameter estimates simultaneously, which is not applicable to direction of arrival (DOA) estimation in unknown nonuniform noise. In this work, we present several computationally efficient EM-type algorithms, which update the parameter estimates sequentially, for solving both the deterministic and stochastic maximum–likelihood (ML) direction finding problems in unknown nonuniform noise. Specifically, we design a generalized EM (GEM) algorithm and a space-alternating generalized EM (SAGE) algorithm for computing the deterministic ML estimator. Simulation results show that the SAGE algorithm outperforms the GEM algorithm. Moreover, we design two SAGE algorithms for computing the stochastic ML estimator, in which the first updates the DOA estimates simultaneously while the second updates the DOA estimates sequentially. Simulation results show that the second SAGE algorithm outperforms the first one.

Noise-boosted recurrent backpropagation

A statistical formulation of recurrent backpropagation (RBP) allows direct noise boosting for time-varying classification and regression. The noise boost reduces training iterations and improves accuracy. The injected noise is just that noise that makes the current signal more probable. This noise-boost result extends the two recent results that backpropagation is a special case of the generalized expectation maximization (EM) algorithm and that careful noise injection can always speed the average convergence of the EM algorithm to a local maximum of the log-likelihood surface. The noise-benefit conditions differ for additive and multiplicative noise in RBP. We tested noise-boosted RBP classifiers on 11 classes of sports video clips and tested RBP regressors on predicting the dollar-rupee exchange rate. Injecting noisy-EM (NEM) noise outperformed injecting blind noise or injecting no noise at all. Additive NEM noise usually outperformed multiplicative noise. The best case of NEM noise injection with RBP training of a recurrent neural classification model speeded up its training by 60% and improved its classification accuracy by 9.51% compared with noiseless RBP training and accuracy. The best performance of the NEM noise with the RBP training of a recurrent neural regression model yielded a 38% speed-up in training and also reduced the squared error by 49.3%. The injection of the additive NEM noise in the output and hidden neurons performed best.

Reliability analysis of constant partially accelerated life tests under progressive first failure type-II censored data from Lomax model: EM and MCMC algorithms

<abstract><p>Examining life-testing experiments on a product or material usually requires a long time of monitoring. To reduce the testing period, units can be tested under more severe than normal conditions, which are called accelerated life tests (ALTs). The objective of this study is to investigate the problem of point and interval estimations of the Lomax distribution under constant stress partially ALTs based on progressive first failure type-II censored samples. The point estimates of unknown parameters and the acceleration factor are obtained by using maximum likelihood and Bayesian approaches. Since reliability data are censored, the maximum likelihood estimates (MLEs) are derived utilizing the general expectation-maximization (EM) algorithm. In the process of Bayesian inference, the Bayes point estimates as well as the highest posterior density credible intervals of the model parameters and acceleration factor, are reported. This is done by using the Markov Chain Monte Carlo (MCMC) technique concerning both symmetric (squared error) and asymmetric (linear-exponential and general entropy) loss functions. Monte Carlo simulation studies are performed under different sizes of samples for comparison purposes. Finally, the proposed methods are applied to oil breakdown times of insulating fluid under two high-test voltage stress level data.</p></abstract>

A novel exponential degradation approach for predicting the remaining useful life of roadheader bearings

Based on the optimized exponential-degradation model (OEDM), a novel approach for predicting the remaining useful life(RUL) of roadheader bearings under different working conditions is proposed in this study. Specifically, the exponential process is used to construct the degradation process from a single performance characteristic under variable operating conditions, the generalized expectation maximization is employed to estimate model parameters, and the proposed degradation model is updated after new data is available. In the traditional exponential degradation method, the hyperparameters are only optimized, which leads to low calculation accuracy under severe working conditions. In the proposed method, the Bayesian algorithm and the Drift Brownian motion algorithm were respectively employed to optimize hyperparameters and stochastic parameters to ensure the high accuracy of the prediction results. In addition, degradation characteristics combined with sensory data acquired through condition monitoring were used to continuously update the RUL in the proposed degradation model. Finally, the effectiveness of the proposed model is verified by a simulation case and a case study. The results show that compared with the linear degradation model and the general exponential degradation model, the proposed OEDM performs well in practical applications and has a higher prediction accuracy. This study provides a reference for predictive maintenance of critical parts of tunneling machinery and cost reduction of tunneling.

Multichannel blind deconvolution for full Stokes image recovery

Random, time-varying blur due to atmospheric turbulence is known to significantly limit image quality. In division-of-time polarimeters, turbulence also randomly distorts the images that are used to infer the polarization content of the scene, leading to substantial Stokes image estimation errors. This research proposes a method to jointly estimate the full polarization properties of a scene along with the point spread functions (PSFs) for each imaging channel in the presence of isoplanatic turbulence. In particular, this research significantly expands on an existing algorithm, to include circular polarization in the framework of a generalized expectation maximization approach. The effectiveness of the approach is demonstrated on laboratory data using a surrogate phase screen placed near the entrance pupil of an imaging system.

Statistical image watermark decoder using NSM-HMT in NSCT-FGPCET magnitude domain

The image watermarking technology is an effective technology for protecting image copyright at present. Image watermarking techniques have a constraint relationship between imperceptibility, robustness, and watermark capacity. Balancing imperceptibility, robustness, and watermark capacity becomes a tricky problem. This paper proposes a statistical image watermarking scheme using nonsubsampled Contourlet transform (NSCT) domain fast generic polar complex exponential transform (FGPCET) magnitudes and nonsymmetric mixtures (NSM) based hidden Markov tree (HMT). In the embedding part, to enhance robustness and imperceptibility, we insert the watermark signal into the robust local NSCT domain FGPCET magnitudes through the multiplicative method. In the decoding part, a statistical image watermarking decoder is designed using the NSM-HMT model and maximum likelihood criterion. Here, marginal characteristics and strong correlations of local NSCT domain FGPCET magnitudes are described by NSM-HMT. In addition, the generalized expectation-maximization (GEM)-clusterized method of moments (CMM) approach is used for accurate parameters. Extensive Monte Carlo experiments validate the better performance of the proposed image watermarking method compared to other well-known methods.

Maximum weighted likelihood estimator for robust heavy-tail modelling of finite mixture models

Insurance claim severity data are characterized by complex distributional phenomenons, where flexible density estimation tools such as the finite mixture models (FMM) are necessary. However, maximum likelihood estimations (MLE) often produce unstable tail estimates for the FMM. Motivated by this challenge, this article presents a maximum weighted likelihood estimator (MWLE) for robust estimations of heavy-tailed FMM. Under some regularity conditions, the proposed MWLE is consistent and asymptotically normal. Since the MWLE has a probabilistic interpretation, we are able to develop two distinctive versions of the Generalized Expectation-Maximization (GEM) algorithm to estimate the MWLE parameters more efficiently and reliably than the standard gradient-based algorithms. We apply the proposed MWLE to two simulation studies and a real motor insurance dataset to demonstrate that it better extrapolates the extreme losses than the MLE, without sacrificing the flexibility of the FMM in capturing the small attritional claims.

Correlation analysis of traction load sequences based on optimal delay method

The correlation between traction load sequences is an important basic information for traction load modeling, but the correlation and time difference between traction load sequences restrict each other. The analysis method for the correlation between traction load sequences based on the optimal delay method is proposed in this paper. This method appropriately delays the sequences between different traction loads and taps the indirect correlation between traction load sequences. Firstly, the maximum correlation coefficient between the two traction load sequences after delaying is taken as the objective function to establish the optimal delay model. The delay time range and the monotonicity of sequence occurrence time constitute the constraints. Secondly, combined with the normalized root mean square error (NRMSE) index, the generalized expectation maximization (EM) algorithm is proposed to solve the optimal delay model. Finally, the traction load sequence with one and two trains at any period of time from the four traction substations of high-speed railway are taken as case examples. Considering the effects of no-load removal processing, inverse transformation, and different test intervals, the maximum correlation coefficients of Pearson, Spearman and Kendall, as well as the corresponding optimal delay amount are calculated. The analysis results show that the proposed method in this paper can dig out the potential delay correlation between traction load sequences, as well as provide basic reference information for traction load modeling and its application.

Multisample Online Learning for Probabilistic Spiking Neural Networks.

Spiking neural networks (SNNs) capture some of the efficiency of biological brains for inference and learning via the dynamic, online, and event-driven processing of binary time series. Most existing learning algorithms for SNNs are based on deterministic neuronal models, such as leaky integrate-and-fire, and rely on heuristic approximations of backpropagation through time that enforces constraints such as locality. In contrast, probabilistic SNN models can be trained directly via principled online, local, and update rules that have proven to be particularly effective for resource-constrained systems. This article investigates another advantage of probabilistic SNNs, namely, their capacity to generate independent outputs when queried over the same input. It is shown that the multiple generated output samples can be used during inference to robustify decisions and to quantify uncertainty-a feature that deterministic SNN models cannot provide. Furthermore, they can be leveraged for training in order to obtain more accurate statistical estimates of the log-loss training criterion and its gradient. Specifically, this article introduces an online learning rule based on generalized expectation-maximization (GEM) that follows a three-factor form with global learning signals and is referred to as GEM-SNN. Experimental results on structured output memorization and classification on a standard neuromorphic dataset demonstrate significant improvements in terms of log-likelihood, accuracy, and calibration when increasing the number of samples used for inference and training.

Automatic particle detection, localization, and physical parameter estimation via combined expectation maximization and deep learning

Single particle tracking is a popular tool that is widely used for studying the dynamic mechanism of nanometer-scale biomolecules. Previously, we proposed a general Expectation Maximization - based framework for simultaneous localization refinement and parameter estimation. Our approach significantly outperforms other methods in terms of localization and parameter estimation, particularly at low signal intensities. The input for this framework is a sequence of images pre-processed from the raw image so in which an individual particle has been isolated and the data linked to form the desired image sequence.

Deep Expectation-Maximization for Joint MIMO Channel Estimation and Signal Detection

To overcome the influence of channel estimation error on signal detection, this paper presents a model-driven deep learning method for joint channel estimation and signal detection in multiple-input multiple-output (MIMO) wireless communication systems. To improve the robustness of the method, we use the Student's <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">t</i> -distribution to model the environment noise, and model the joint problem probabilistically by taking the channel state information (CSI) as a latent variable, then derive a generalized expectation maximization (GEM) algorithm to fit the model. In GEM, the expectation step is used for robust CSI estimation while the maximization step detects the transmitted signal with the estimated CSI. To reduce the computational complexity of GEM, we unfold it into a deep neural network. The network contains only a few trainable parameters, which is easy to train and has low space complexity. Based on our experimental results, we find that the proposed GEM algorithm outperforms a variety of signal detection algorithms which take the CSI estimation and signal detection as independent modules. Further, we find empirically that the proposed GEM network outperforms state-of-the-art deep learning based joint channel estimation and signal detection methods and has a good generalization ability against the number of antennas, the modulation mode, the level of signal-to-noise ratio and channel correlation.

Robust LPV models identification approach based on shifted asymmetric Laplace distribution

This paper focuses on the robust parameters estimation algorithm of linear parameters varying (LPV) models. The classical robust identification techniques deal with the polluted training data, for example, outliers in white noise. The paper extends this robustness to both symmetric and asymmetric noise with outliers to achieve stronger robustness. Without the assumption of Gaussian white noise pollution, the paper employs asymmetric Laplace distribution to model broader noise, especially the asymmetrically distributed noise, since it is an asymmetric heavy-tailed distribution. Furthermore, the asymmetric Laplace (AL) distribution is represented as the product of Gaussian distribution and exponential distribution to decompose this complex AL distribution. Then, a shifted parameter is introduced as the regression term to connect the probabilistic models of the noise and the predict output that obeys shifted AL distribution. In this way, the posterior probability distribution of the unobserved variables could be deduced and the robust parameters estimation problem is solved in the general Expectation Maximization algorithm framework. To demonstrate the advantage of the proposed algorithm, a numerical simulation example is employed to identify the parameters of LPV models and to illustrate the convergence.

Analysis of a generalised expectation–maximisation algorithm for Gaussian mixture models: a control systems perspective

The expectation–maximisation (EM) algorithm is one of the most popular methods used to solve the problem of parametric distribution-based clustering in unsupervised learning. In this paper, we propose to analyse a generalised EM (GEM) algorithm in the context of Gaussian mixture models, where the maximisation step in the EM is replaced by an increasing step. We show that this GEM algorithm can be understood as a linear time-invariant (LTI) system with a feedback nonlinearity. Therefore, we explore some of its convergence properties by leveraging tools from robust control theory. Lastly, we explain how the proposed GEM can be designed and present a pedagogical example to understand the advantages of the proposed approach.