Generalized Expectation-maximization Algorithm Research Articles

Rethinking Peculiar Images by Diffusion Models: Revealing Local Minima’s Role

Recent significant advancements in diffusion models have revolutionized image generation, enabling the synthesis of highly realistic images with text-based guidance. These breakthroughs have paved the way for constructing datasets via generative artificial intelligence (AI), offering immense potential for various applications. However, two critical challenges hinder the widespread adoption of synthesized data: computational cost and the generation of peculiar images. While computational costs have improved through various approaches, the issue of peculiar image generation remains relatively unexplored. Existing solutions rely on heuristics, extra training, or AI-based post-processing to mitigate this problem. In this paper, we present a novel approach to address both issues simultaneously. We establish that both gradient descent and diffusion sampling are specific cases of the generalized expectation maximization algorithm. We hypothesize and empirically demonstrate that peculiar image generation is akin to the local minima problem in optimization. Inspired by optimization techniques, we apply naive momentum and positive-negative momentum to diffusion sampling. Last, we propose new metrics to evaluate the peculiarity. Experimental results show momentum effectively prevents peculiar image generation without extra computation.

Neutron source reconstruction using a generalized expectation-maximization algorithm on one-dimensional neutron images from the Z facility.

Magnetized Liner Inertial Fusion experiments have been performed at the Z facility at Sandia National Laboratories. These experiments use deuterium fuel, which produces 2.45MeV neutrons on reaching thermonuclear conditions. To study the spatial structure of neutron production, the one-dimensional imager of neutrons diagnostic was fielded to record axial resolved neutron images. In this diagnostic, neutrons passing through a rolled edge aperture form an image on a CR-39-based solid state nuclear track detector. Here, we present a modified generalized expectation-maximization algorithm to reconstruct an axial neutron emission profile of the stagnated fusion plasma. We validate the approach by comparing the reconstructed neutron emission profile to an x-ray emission profile provided by a time-integrated pinhole camera.

Model-based clustering of multipath propagation in powerline communication channels

Powerline communication (PLC) channels are known to exhibit multipath propagation behaviour. The authors present a model-based framework to address the challenge of clustering multipath propagation components (MPCs) in PLC channels for indoor low-voltage (LV) environments. The framework employs a range of finite-mixture models (FMMs), including the gamma mixture model, the inverse gamma mixture model, the Gaussian mixture model, the inverse Gaussian mixture model, the Nakagami mixture model, the inverse Nakagami mixture model (INMM) and the Rayleigh mixture model, to identify clusters of MPCs. A measurement campaign of an unknown indoor LV PLC channel is conducted to obtain a channel response. From the channel response, the delay and magnitude parameters of the MPCs are extracted using the space-alternating generalised expectation maximisation algorithm adopted only for these parameters. A maximum likelihood approach and the expectation–maximisation algorithm are employed to fit the FMMs to the MPC delay-magnitude dataset to cluster MPCs in the delay domain. The results of the model-fitting process are then evaluated using the corrected Akaike information criterion (AICc), which enables a fair comparison of the candidate models over the feasible and finite range of clusters. A novel algorithm is introduced for estimating the feasible and finite range of clusters using the extracted delay and magnitude MPC parameters. The AICc’s ranking results show that the INMM model provides the best fit. Davies–Bouldin (DB) and Calinski–Harabasz (CH) indexes are used to compare the model-based clustering approach to the conventional distance-based clustering methods. Validation results show that CH and DB indexes closely agree in the optimal number of MPC clusters for model-based clustering, which corresponds to the most within-cluster compactness of MPCs and to the most between-cluster separation in the delay domain.

Expectation-maximization type algorithms for direction of arrival estimation in unknown nonuniform noise

The expectation–maximization (EM) algorithm updates all of the parameter estimates simultaneously, which is not applicable to direction of arrival (DOA) estimation in unknown nonuniform noise. In this work, we present several computationally efficient EM-type algorithms, which update the parameter estimates sequentially, for solving both the deterministic and stochastic maximum–likelihood (ML) direction finding problems in unknown nonuniform noise. Specifically, we design a generalized EM (GEM) algorithm and a space-alternating generalized EM (SAGE) algorithm for computing the deterministic ML estimator. Simulation results show that the SAGE algorithm outperforms the GEM algorithm. Moreover, we design two SAGE algorithms for computing the stochastic ML estimator, in which the first updates the DOA estimates simultaneously while the second updates the DOA estimates sequentially. Simulation results show that the second SAGE algorithm outperforms the first one.

Switching state-space modeling of neural signal dynamics.

Linear parametric state-space models are a ubiquitous tool for analyzing neural time series data, providing a way to characterize the underlying brain dynamics with much greater statistical efficiency than non-parametric data analysis approaches. However, neural time series data are frequently time-varying, exhibiting rapid changes in dynamics, with transient activity that is often the key feature of interest in the data. Stationary methods can be adapted to time-varying scenarios by employing fixed-duration windows under an assumption of quasi-stationarity. But time-varying dynamics can be explicitly modeled by switching state-space models, i.e., by using a pool of state-space models with different dynamics selected by a probabilistic switching process. Unfortunately, exact solutions for state inference and parameter learning with switching state-space models are intractable. Here we revisit a switching state-space model inference approach first proposed by Ghahramani and Hinton. We provide explicit derivations for solving the inference problem iteratively after applying a variational approximation on the joint posterior of the hidden states and the switching process. We introduce a novel initialization procedure using an efficient leave-one-out strategy to compare among candidate models, which significantly improves performance compared to the existing method that relies on deterministic annealing. We then utilize this state inference solution within a generalized expectation-maximization algorithm to estimate model parameters of the switching process and the linear state-space models with dynamics potentially shared among candidate models. We perform extensive simulations under different settings to benchmark performance against existing switching inference methods and further validate the robustness of our switching inference solution outside the generative switching model class. Finally, we demonstrate the utility of our method for sleep spindle detection in real recordings, showing how switching state-space models can be used to detect and extract transient spindles from human sleep electroencephalograms in an unsupervised manner.

Mixture Composite Regression Models with Multi-type Feature Selection

The aim of this article is to present a mixture composite regression model for claim severity modeling. Claim severity modeling poses several challenges such as multimodality, tail-heaviness, and systematic effects in data. We tackle this modeling problem by studying a mixture composite regression model for simultaneous modeling of attritional and large claims and for considering systematic effects in both the mixture components as well as the mixing probabilities. For model fitting, we present a group-fused regularization approach that allows us to select the explanatory variables that significantly impact the mixing probabilities and the different mixture components, respectively. We develop an asymptotic theory for this regularized estimation approach, and fitting is performed using a novel generalized expectation-maximization algorithm. We exemplify our approach on a real motor insurance dataset.

Deep Expectation-Maximization for Joint MIMO Channel Estimation and Signal Detection

To overcome the influence of channel estimation error on signal detection, this paper presents a model-driven deep learning method for joint channel estimation and signal detection in multiple-input multiple-output (MIMO) wireless communication systems. To improve the robustness of the method, we use the Student's <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">t</i> -distribution to model the environment noise, and model the joint problem probabilistically by taking the channel state information (CSI) as a latent variable, then derive a generalized expectation maximization (GEM) algorithm to fit the model. In GEM, the expectation step is used for robust CSI estimation while the maximization step detects the transmitted signal with the estimated CSI. To reduce the computational complexity of GEM, we unfold it into a deep neural network. The network contains only a few trainable parameters, which is easy to train and has low space complexity. Based on our experimental results, we find that the proposed GEM algorithm outperforms a variety of signal detection algorithms which take the CSI estimation and signal detection as independent modules. Further, we find empirically that the proposed GEM network outperforms state-of-the-art deep learning based joint channel estimation and signal detection methods and has a good generalization ability against the number of antennas, the modulation mode, the level of signal-to-noise ratio and channel correlation.

Analysis of a generalised expectation–maximisation algorithm for Gaussian mixture models: a control systems perspective

The expectation–maximisation (EM) algorithm is one of the most popular methods used to solve the problem of parametric distribution-based clustering in unsupervised learning. In this paper, we propose to analyse a generalised EM (GEM) algorithm in the context of Gaussian mixture models, where the maximisation step in the EM is replaced by an increasing step. We show that this GEM algorithm can be understood as a linear time-invariant (LTI) system with a feedback nonlinearity. Therefore, we explore some of its convergence properties by leveraging tools from robust control theory. Lastly, we explain how the proposed GEM can be designed and present a pedagogical example to understand the advantages of the proposed approach.

A study of uplink and downlink channel spatial characteristics in an urban micro scenario at 28 GHz

This paper presents an empirical study of the uplink and downlink azimuth angle of arrival (AoA) in an urban micro (UMi) scenario at 28 GHz. At present, most UMi measurements are conducted in the downlink and then the uplink situation is inferred assuming channel reciprocity. Although the channel correlation coefficient of the uplink and downlink can be as high as 0.8, this does not mean that they are the same. Only a real uplink measurement can accurately describe its channel conditions, and this is what this study does. A receiver equipped with a rotatable horn antenna is mounted at the base station and the user terminal, respectively, in simulating the uplink and downlink. To improve the angular resolution, we extract the multipath components (MPCs) using the space-alternating generalized expectation-maximization algorithm. Also, a spatial lobe approach is used to cluster the MPCs in the power angular spectrum. By matching MPCs with objects in the environment, we find that direct propagation and first-order reflections are dominant in line-of-sight and non-line-of-sight cases. By comparing our measurement with those in standard channel models, we verify that the AoA of clusters follows a Gaussian distribution in the uplink and downlink. In addition, a two-dimensional Gaussian distribution for ray AoA and power is established to reflect their correlation.

Joint angle and delay estimation for GNSS multipath signals based on multiple sparse Bayesian Learning

Multipath signals formed by signal reflection coming from objects in the vicinity of Global Navigation Satellite System (GNSS) receivers result in a degradation of the tracking performance and an increase in the positioning error. By estimating the parameters of both line-of-sight signal and the multipath signals, superior multipath mitigation, spoofing suppression, and localization can be attained. We propose using the multiple sparse Bayesian learning method together with the joint angle and delay estimation technique in GNSS multipath environment to fully exploit the sparsity present in both the spatial and the temporal domains. We also extend the techniques to the estimation of fractional Doppler frequency besides the angle and delay. To counteract the intrinsic drawbacks of sparse representations, two different algorithms based on on-grid and off-grid estimators are proposed to either reduce the complexity or enhance the resolution such that the proposed multipath mitigation approach can be adapted to various GNSS practical situations. Subsequently, a third algorithm with improved resolution is obtained by applying the Space Alternating Generalized Expectation–Maximization algorithm to refine the MSBL-based joint angle and delay estimates. Simulation results indicate that the three proposed algorithms can effectively resolve the GNSS multipath signals and have better performance than existing methods even in severe situations, like the cases of signals with low carrier-to-noise-power-density ratio and spatially and temporally correlated multipath.

Proportional likelihood ratio mixed model for discrete longitudinal data.

Rathouz and Gao [2] and Luo and Tsai [3] proposed valuable extensions to the generalized linear model for modeling a nonlinear monotonic relationship between the mean response and a set of covariates. In their extensions for discrete data the baseline response distribution is unspecified and is estimated from the data. We propose to extend this model for the analysis of longitudinal data by incorporating random effects into the linear predictor, and using maximum likelihood for estimation and inference. Motivated in particular by longitudinal studies of clinical scale outcomes, we developed an estimation procedure for a finite-support response using a generalized expectation-maximization algorithm where Gauss-Hermite quadrature is employed to approximate the integrals in the E step of the algorithm. Upon convergence, the observed information matrix is estimated through second-order numerical differentiation of the log-likelihood function. Asymptotic properties of the maximum likelihood estimates follow under the usual regularity conditions. Simulation studies are conducted to assess its finite-sample properties and compare the proposed model to the generalized linear mixed model. The proposed method is illustrated in an analysis of data from a longitudinal study of Huntington's disease.

Cluster-Based Millimeter-Wave Outdoor-to-Indoor Propagation Characteristics Based on 32 GHz Measurement Analysis

Although the fifth-generation (5G) networks are deployed at millimeter-wave (mmWave) frequencies, little information about the multipath clustering statistics for mmWave outdoor-to-indoor (O2I) propagation has been reported. This letter attempts to overcome this deficiency somewhat by investigating cluster-based propagation statistics for mmWave O2I that are based upon measurements. The measurements were conducted at 32 GHz at two office buildings. The multipath components (MPCs) were resolved by the space-alternating generalized expectation-maximization algorithm and the MPC clustering was performed by the K -power-means algorithm. We observed that numerous reflections bouncing back-and-forth between the windows cause large numbers of clusters. Other statistics such as the number of rays per clusters, cluster power and shadowing, cluster delay, and angular spread were also calculated.

A validated robust and automatic procedure for vibration analysis of bridge structures using MEMS accelerometers

Abstract Today, short- and long-term structural health monitoring (SHM) of bridge infrastructures and their safe, reliable and cost-effective maintenance has received considerable attention. From a surveying or civil engineer’s point of view, vibration-based SHM can be conducted by inspecting the changes in the global dynamic behaviour of a structure, such as natural frequencies (i. e. eigenfrequencies), mode shapes (i. e. eigenforms) and modal damping, which are known as modal parameters. This research work aims to propose a robust and automatic vibration analysis procedure that is so-called robust time domain modal parameter identification (RT-MPI) technique. It is novel in the sense of automatic and reliable identification of initial eigenfrequencies even closely spaced ones as well as robustly and accurately estimating the modal parameters of a bridge structure using low numbers of cost-effective micro-electro-mechanical systems (MEMS) accelerometers. To estimate amplitude, frequency, phase shift and damping ratio coefficients, an observation model consisting of: (1) a damped harmonic oscillation model, (2) an autoregressive model of coloured measurement noise and (3) a stochastic model in the form of the heavy-tailed family of scaled t-distributions is employed and jointly adjusted by means of a generalised expectation maximisation algorithm. Multiple MEMS as part of a geo-sensor network were mounted at different positions of a bridge structure which is precalculated by means of a finite element model (FEM) analysis. At the end, the estimated eigenfrequencies and eigenforms are compared and validated by the estimated parameters obtained from acceleration measurements of high-end accelerometers of type PCB ICP quartz, velocity measurements from a geophone and the FEM analysis. Additionally, the estimated eigenfrequencies and modal damping are compared with a well-known covariance driven stochastic subspace identification approach, which reveals the superiority of our proposed approach. We performed an experiment in two case studies with simulated data and real applications of a footbridge structure and a synthetic bridge. The results show that MEMS accelerometers are suitable for detecting all occurring eigenfrequencies depending on a sampling frequency specified. Moreover, the vibration analysis procedure demonstrates that amplitudes can be estimated in submillimetre range accuracy, frequencies with an accuracy better than 0.1 Hz and damping ratio coefficients with an accuracy better than 0.1 and 0.2 % for modal and system damping, respectively.

Self-tuning robust adjustment within multivariate regression time series models with vector-autoregressive random errors

The iteratively reweighted least-squares approach to self-tuning robust adjustment of parameters in linear regression models with autoregressive (AR) and t-distributed random errors, previously established in Kargoll et al. (in J Geod 92(3):271–297, 2018. https://doi.org/10.1007/s00190-017-1062-6), is extended to multivariate approaches. Multivariate models are used to describe the behavior of multiple observables measured contemporaneously. The proposed approaches allow for the modeling of both auto- and cross-correlations through a vector-autoregressive (VAR) process, where the components of the white-noise input vector are modeled at every time instance either as stochastically independent t-distributed (herein called “stochastic model A”) or as multivariate t-distributed random variables (herein called “stochastic model B”). Both stochastic models are complementary in the sense that the former allows for group-specific degrees of freedom (df) of the t-distributions (thus, sensor-component-specific tail or outlier characteristics) but not for correlations within each white-noise vector, whereas the latter allows for such correlations but not for different dfs. Within the observation equations, nonlinear (differentiable) regression models are generally allowed for. Two different generalized expectation maximization (GEM) algorithms are derived to estimate the regression model parameters jointly with the VAR coefficients, the variance components (in case of stochastic model A) or the cofactor matrix (for stochastic model B), and the df(s). To enable the validation of the fitted VAR model and the selection of the best model order, the multivariate portmanteau test and Akaike’s information criterion are applied. The performance of the algorithms and of the white noise test is evaluated by means of Monte Carlo simulations. Furthermore, the suitability of one of the proposed models and the corresponding GEM algorithm is investigated within a case study involving the multivariate modeling and adjustment of time-series data at four GPS stations in the EUREF Permanent Network (EPN).

Deep Active Shape Model for Robust Object Fitting.

Object recognition and localization is still a very challenging problem, despite recent advances in deep learning (DL) approaches, especially for objects with varying shapes and appearances. Statistical models, such as an Active Shape Model (ASM), rely on a parametric model of the object, allowing an easy incorporation of prior knowledge about shape and appearance in a principled way. To take advantage of these benefits, this paper proposes a new ASM framework that addresses two tasks: (i) comparing the performance of several image features used to extract observations from an input image; and (ii) improving the performance of the model fitting by relying on a probabilistic framework that allows the use of multiple observations and is robust to the presence of outliers. The goal in (i) is to maximize the quality of the observations by exploring a wide set of handcrafted features (HOG, SIFT, and texture templates) and more recent DL-based features. Regarding (ii), we use the Generalized Expectation-Maximization algorithm to deal with outliers and to extend the fitting process to multiple observations. The proposed framework is evaluated in the context of facial landmark fitting and the segmentation of the endocardium of the left ventricle in cardiac magnetic resonance volumes. We experimentally observe that the proposed approach is robust not only to outliers, but also to adverse initialization conditions and to large search regions (from where the observations are extracted from the image). Furthermore, the results of the proposed combination of the ASM with DL-based features are competitive with more recent DL approaches (e.g. FCN [1], U-Net [2] and CNN Cascade [3]), showing that it is possible to combine the benefits of statistical models and DL into a new deep ASM probabilistic framework.

Image reconstructions with active illumination in strong-turbulence scenarios with single-frame blind deconvolution approaches.

Image formation over long horizontal or slant paths is of interest in surveillance and remote sensing. Image reconstructions of isolated objects are presented using active illumination in long-path, strong-turbulence conditions using a wave-optics simulation to produce the images. Fast-running reconstruction algorithms are used, including a novel single-frame blind iterative deconvolution algorithm and a generalized expectation maximization algorithm. Significant improvements in image quality and image recognizability can be found for spherical-wave Rytov variances up to 0.4 and for up to 10 atmospheric coherence lengths across the aperture in uniform-turbulence scenarios over a 30km range. For these conditions, the isoplanatic patch angle is comparable to the diffraction angle, and there are 20 or more isoplanatic patches across the objects considered. The results are compared with idealized atmospheric phase correction with an incoherent beacon. Several image quality metrics are considered. Results for strongest turbulence are explained in terms of a local average of the point spread function and the central limit theorem for cases in which there are many isoplanatic patches across the object.

A Cluster-Based Three-Dimensional Channel Model for Vehicle-to-Vehicle Communications

Vehicle-to-vehicle (V2V) communications have received a lot of attention as it can significantly improve efficiency and safety of road traffic. In general, V2V scenarios have some special features such as high mobility of transceivers, low antenna height, and small communication ranges, which significantly affect the propagation characteristics. Therefore, an accurate channel model is required to characterize V2V propagation channel. However, some important features of V2V channels have not been well characterized, for example, most existing V2V channel models tend to only consider the distribution of multipath components (MPCs) in a horizontal dimension, whereas ignoring vertical dimension, which is inconsistent with the distribution of MPCs in the actual channel. Moreover, the dynamic clusters of MPCs have not been well modeled in the existing V2V models. Therefore, in order to better model the V2V channel, a cluster-based three-dimensional (3D) channel model is proposed in this paper, which is based on the measurements conducted at 5.9 GHz in urban and suburban scenarios. In the proposed model, the distribution of MPCs clusters in both horizontal and vertical dimensions is considered. The space-alternating generalized expectation-maximization algorithm is used to extract MPCs, and the clustering and tracking algorithms are used to identify and track the clusters of dynamic MPCs. In the proposed model, all MPC clusters are divided into two categories: global-clusters and scatterer-clusters, and the distribution of the two clusters are characterized by a series of inter- and intra-cluster parameters. It is found that both the azimuth spread and the elevation spread follow the lognormal distribution. In addition, the power of MPCs within a cluster has the truncated-Gaussian distribution, whereas the angle of MPCs within a cluster has the Laplacian distribution. Finally, the accuracy of the model is verified by comparing the measurements and simulations.

Deformable multimodal registration for navigation in beating-heart cardiac surgery.

Minimally invasive beating-heart surgery is currently performed using endoscopes and without navigation. Registration of intraoperative ultrasound to a preoperative cardiac CT scan is a valuable step toward image-guided navigation. The registration was achieved by first extracting a representative point set from each ultrasound image in the sequence using a deformable registration. A template shape representing the cardiac chambers was deformed through a hierarchy of affine transformations to match each ultrasound image using a generalized expectation maximization algorithm. These extracted point sets were matched to the CT by exhaustively searching over a large number of precomputed slices of 3D geometry. The result is a similarity transformation mapping the intraoperative ultrasound to preoperative CT. Complete data sets were acquired for four patients. Transesophageal echocardiography ultrasound sequences were deformably registered to a model of oriented points with a mean error of 2.3mm. Ultrasound and CT scans were registered to a mean of 3mm, which is comparable to the error of 2.8mm expected by merging ultrasound registration with uncertainty of cardiac CT. The proposed algorithm registered 3D CT with dynamic 2D intraoperative imaging. The algorithm aligned the images in both space and time, needing neither dynamic CT imaging nor intraoperative electrocardiograms. The accuracy was sufficient for navigation in thoracoscopically guided beating-heart surgery.

Timing and Frequency Synchronization and Doubly Selective Channel Estimation for OFDMA Uplink

In this brief, a novel algorithm is proposed for timing and frequency synchronization and doubly selective channel (DSC) estimation in the uplink of orthogonal frequency division multiple access systems. By using a basis expansion model (BEM) to represent a DSC, a maximum likelihood approach is proposed to jointly estimate the timing and carrier frequency offsets of different users as well as the BEM coefficients of the time-varying channels. A space-alternating generalized expectation-maximization algorithm is then employed to transform the maximization problem for all users into several simpler maximization problems for a single user. Implementation issues are investigated and, the computational complexity and mean square error of the proposed scheme are evaluated. Performance assessment of the new scheme shows a significant improvement as compared with previous methods.

Complex ISNMF: A Phase-Aware Model for Monaural Audio Source Separation

This paper introduces a phase-aware probabilistic model for audio source separation. Classical source models in the short-time Fourier transform domain use circularly-symmetric Gaussian or Poisson random variables. This is equivalent to assuming that the phase of each source is uniformly distributed, which is not suitable for exploiting the underlying structure of the phase. Drawing on preliminary works, we introduce here a Bayesian anisotropic Gaussian source model in which the phase is no longer uniform. Such a model permits us to favor a phase value that originates from a signal model through a Markov chain prior structure. The variance of the latent variables are structured with nonnegative matrix factorization (NMF). The resulting model is called complex Itakura–Saito NMF (ISNMF) since it generalizes the ISNMF model to the case of nonisotropic variables. It combines the advantages of ISNMF, which uses a distortion measure adapted to audio and yields a set of estimates which preserve the overall energy of the mixture, and of complex NMF, which enables one to account for some phase constraints. We derive a generalized expectation-maximization algorithm to estimate the model parameters. Experiments conducted on a musical source separation task in a semiinformed setting show that the proposed approach outperforms state-of-the-art phase-aware separation techniques.