Expectation-maximization Procedure Research Articles

Nonlinear Identification With Local Model Networks Using GTLS Techniques and Equality Constraints

Local model networks approximate a nonlinear system through multiple local models fitted within a partition space. The main advantage of this approach is that the identification of complex nonlinear processes is alleviated by the integration of structured knowledge about the process. This paper extends these concepts by the integration of quantitative process knowledge into the identification procedure. Quantitative knowledge describes explicit dependences between inputs and outputs and is integrated in the parameter estimation process by means of equality constraints. For this purpose, a constrained generalized total least squares algorithm for local parameter estimation is presented. Furthermore, the problem of proper integration of constraints in the partitioning process is treated where an expectation-maximization procedure is combined with constrained parameter estimation. The benefits and the applicability of the proposed concepts are demonstrated by means of two illustrative examples and a practical application using real measurement data.

Optimality and stability of the K-hyperline clustering algorithm

K-hyperline clustering is an iterative algorithm based on singular value decomposition and it has been successfully used in sparse component analysis. In this paper, we prove that the algorithm converges to a locally optimal solution for a given set of training data, based on Lloyd’s optimality conditions. Furthermore, the local optimality is shown by developing an Expectation-Maximization procedure for learning dictionaries to be used in sparse representations and by deriving the clustering algorithm as its special case. The cluster centroids obtained from the algorithm are proved to tessellate the space into convex Voronoi regions. The stability of clustering is shown by posing the problem as an empirical risk minimization procedure over a function class. It is proved that, under certain conditions, the cluster centroids learned from two sets of i.i.d. training samples drawn from the same probability space become arbitrarily close to each other, as the number of training samples increase asymptotically.

Segmentation of fault networks determined from spatial clustering of earthquakes

[1] We present a new method of data clustering applied to earthquake catalogs, with the goal of reconstructing the seismically active part of fault networks. We first use an original method to separate clustered events from uncorrelated seismicity using the distribution of volumes of tetrahedra defined by closest neighbor events in the original and randomized seismic catalogs. The spatial disorder of the complex geometry of fault networks is then taken into account by defining faults as probabilistic anisotropic kernels. The structure of those kernels is motivated by properties of discontinuous tectonic deformation and by previous empirical observations of the geometry of faults and of earthquake clusters at many spatial and temporal scales. Combining this a priori knowledge with information theoretical arguments, we propose the Gaussian mixture approach implemented in an expectation maximization (EM) procedure. A cross-validation scheme is then used that allows the determination of the number of kernels which provides an optimal data clustering of the catalog. This three-step approach is applied to a high-quality catalog of relocated seismicity following the 1986 Mount Lewis (Ml = 5.7) event in California. It reveals that events cluster along planar patches of about 2 km2, i.e., comparable to the size of the main event. The finite thickness of those clusters (about 290 m) suggests that events do not occur on well-defined and smooth Euclidean fault core surfaces but rather that there exist a deforming area and a damage zone surrounding faults which may be seismically active at depth. Finally, we propose a connection between our methodology and multiscale spatial analysis, based on the derivation of a spatial fractal dimension of about 1.8 for the set of hypocenters in the Mount Lewis area, consistent with recent observations on relocated catalogs.

The construction of semantic memory: grammar-based representations learned from relational episodic information.

After acquisition, memories underlie a process of consolidation, making them more resistant to interference and brain injury. Memory consolidation involves systems-level interactions, most importantly between the hippocampus and associated structures, which takes part in the initial encoding of memory, and the neocortex, which supports long-term storage. This dichotomy parallels the contrast between episodic memory (tied to the hippocampal formation), collecting an autobiographical stream of experiences, and semantic memory, a repertoire of facts and statistical regularities about the world, involving the neocortex at large. Experimental evidence points to a gradual transformation of memories, following encoding, from an episodic to a semantic character. This may require an exchange of information between different memory modules during inactive periods. We propose a theory for such interactions and for the formation of semantic memory, in which episodic memory is encoded as relational data. Semantic memory is modeled as a modified stochastic grammar, which learns to parse episodic configurations expressed as an association matrix. The grammar produces tree-like representations of episodes, describing the relationships between its main constituents at multiple levels of categorization, based on its current knowledge of world regularities. These regularities are learned by the grammar from episodic memory information, through an expectation-maximization procedure, analogous to the inside–outside algorithm for stochastic context-free grammars. We propose that a Monte-Carlo sampling version of this algorithm can be mapped on the dynamics of “sleep replay” of previously acquired information in the hippocampus and neocortex. We propose that the model can reproduce several properties of semantic memory such as decontextualization, top-down processing, and creation of schemata.

Point target probabilistic multiple hypothesis tracking

Probabilistic Multiple Hypothesis Tracking (PMHT) is an algorithm for multi-target tracking in clutter with computational requirements, which are linear in the number of targets and the number of measurements. In order to achieve this, the PMHT removes the point target constraint, and uses the expectation maximisation procedure to optimise both data association probabilities and the target trajectory state estimates. However, PMHT is known to have high track-loss percentage in comparison with Probabilistic Data Association, at least in point target tracking. The authors propose a new PMHT-like algorithm to solve some problems of PMHT. In this study they revert the point target constraint. The authors call the new algorithm Point target PMHT. Simulation results show the efficiency of this method.

Bayesian estimation of optimal craniofacial reconstructions

Forensic craniofacial reconstruction (CFR) aims at estimating the facial outlook associated to an unknown skull for victim identification. Computerized CFR techniques are essentially a virtual mimicking of manual CFR techniques and all share the same conceptual model-based framework. We propose a fully automated Bayesian based statistical framework estimating the most probable face, according to a known craniofacial model ( CFM) , given the, possibly inaccurate, skull data. A multivariate Gaussian distribution is assumed for the shape parameters of the CFM, only allowing face-like solutions. The CFM is improved by encoding tissue depth differently as an extra value for 52 landmarks on the face and by incorporating gray-valued texture information. A fully automated and consistent technique is obtained by the use of an implicit target skull representation (TSR). The most plausible face associated to the target skull is calculated using an expectation-maximization procedure that is robust to small (noise) and/or gross errors (outliers). A clinical database of 12 individuals is used for simulating realistic reconstruction scenarios. Validation is performed in terms of reconstruction accuracy and recognition success. Within the same EM reconstruction framework, the proposed procedure is compared to alternative reconstructions using different target skull representations and different CFMs incorporating various amounts of covariance. The results indicate that the proposed CFM performs better than the other models. Furthermore, the use of the implicit TSR generates more consistent and better results compared to a realistic landmark based skull representation. Finally, these results also confirm that the Bayesian framework formulation is indeed robust against noise and outliers in the skull data.

Comparison of the adsorption mechanisms of pyridine in hydrophilic interaction chromatography and in reversed-phase aqueous liquid chromatography

The adsorption isotherms of pyridine were measured by frontal analysis (FA) on a column packed with shell particles of neat porous silica (Halo), using water–acetonitrile mixtures as the mobile phase at 295 K. The isotherm data were measured for pyridine concentrations covering a dynamic range of four millions. The degree of heterogeneity of the surface was characterized by the adsorption energy distribution (AED) function calculated from the raw adsorption data, using the expectation-maximization (EM) procedure. The results showed that two different retention mechanisms dominate in Per aqueous liquid chromatography (PALC) at low acetonitrile concentrations and in hydrophilic interaction chromatography (HILIC) at high acetonitrile concentrations. In the PALC mode, the adsorption mechanism of pyridine on the silica surface is controlled by hydrophobic interactions that take place on very few and ultra-active adsorption sites, which might be pores on the irregular and rugose surface of the porous silica particles. The surface is seriously heterogeneous, with up to five distinct adsorption sites and five different energy peaks on the AED of the packing material. In contrast, in the HILIC mode, the adsorption behavior is quasi-homogeneous and pyridine retention is governed by its adsorption onto free silanol groups. For intermediate mobile phase compositions, the siloxane and the silanol groups are both significantly saturated with acetonitrile and water, respectively, causing a minimum of the retention factor of pyridine on the Halo column.

Using Hidden Markov Models in Vehicular Crash Detection

This paper presents a system for automotive crash detection based on hidden Markov models (HMMs). The crash pulse library used for training comprises a number of head-on and oblique angular crash events involving rigid and offset deformable barriers. Stochastic distribution characteristics of crash signals are validated to ensure conformity with the modeling assumptions. This step is achieved by analyzing the quantile-quantile (Q-Q) plot of actual pulses against the assumed bivariate Gaussian distribution. HMM parameters are next induced by utilizing the expectation-maximization (EM) procedure. The search for an optimal crash pulse model proceeds using the ldquoleave-one-outrdquo technique with the exploration encompassing both fully connected and left-right HMM topologies. The optimal crash pulse architecture is identified as a seven-state left-right HMM with its parameters computed using real and computer-aided engineering (CAE)-generated data. The system described in the paper has the following advantages. First, it is fast and can accurately detect crashes within 6 ms. Second, its implementation is simple and uses only two sensors, which makes it less vulnerable to failures, considering the overall simplicity of interconnects. Finally, it represents a general and modularized algorithm that can be adapted to any vehicle line and readily extended to use additional sensors.

Combined Handwriting and Speech Modalities for User Authentication

In this paper, we report on the development of an efficient user authentication system based on a combined acquisition of online pen and speech signals. The novelty of our approach is in the simultaneous recording of these two modalities, simply asking the user to utter what she/he is writing. The main benefit of this multimodal approach is a better accuracy at no extra costs in terms of access time or inconvenience. Another benefit comes from an increased difficulty for forgers willing to perform imitation attacks as two signals need to be reproduced. We are comparing here two potential scenarios of use. The first one is called <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">spoken</i> <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">signatures</i> where the user signs and says the content of the signature. The second scenario is based on <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">spoken</i> <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">handwriting</i> where the user is prompted to write and read the content of sentences randomly extracted from a text. Data according to these two scenarios have been recorded from a set of 70 users. In the first part of this paper, we describe the acquisition procedure, and we comment on the viability and usability of such simultaneous recordings. Our conclusions are supported by a short survey performed with the users. In the second part, we present the authentication systems that we have developed for both scenarios. More specifically, our strategy was to model independently both streams of data and to perform a fusion at the score level. Starting from a state-of-the-art-modeling algorithm based on Gaussian Mixture Models trained with an Expectation-Maximization procedure, we report on several significant improvements that are brought. As a general observation, the use of both modalities outperforms significantly the modalities used alone.

Direct Importance Estimation with Gaussian Mixture Models

The ratio of two probability densities is called the importance and its estimation has gathered a great deal of attention these days since the importance can be used for various data processing purposes. In this paper, we propose a new importance estimation method using Gaussian mixture models (GMMs). Our method is an extention of the Kullback-Leibler importance estimation procedure (KLIEP), an importance estimation method using linear or kernel models. An advantage of GMMs is that covariance matrices can also be learned through an expectation-maximization procedure, so the proposed method—which we call the Gaussian mixture KLIEP (GM—KLIEP)—is expected to work well when the true importance function has high correlation. Through experiments, we show the validity of the proposed approach.

Agentification of Markov model-based segmentation: Application to magnetic resonance brain scans

Markov random field (MRF) models have been traditionally applied to the task of robust-to-noise image segmentation. Most approaches estimate MRF parameters on the whole image via a global expectation-maximization (EM) procedure. The resulting estimated parameters are likely to be uncharacteristic of local image features. Instead, we propose to distribute a set of local MRF models within a multiagent framework. Local segmentation agents estimate local MRF models via local EM procedures and cooperate to ensure a global consistency of local models. We demonstrate different types of cooperations between agents that lead to additional levels of regularization compared to the standard label regularization provided by MRF. Embedding Markovian EM procedures into a multiagent paradigm shows interesting properties that are illustrated on magnetic resonance (MR) brain scan segmentation. A cooperative tissue and subcortical structure segmentation approach is designed with such a framework, where both models mutually improve. Several experiments are reported and illustrate the working of Markovian EM agents. The evaluation of MR brain scan segmentation was performed using both phantoms and real 3T brain scans. It showed a robustness to intensity non-uniformity and noise, together with a low computational time. Based on these experiments MRF agent-based approach appears to be a very promising new tool for complex image segmentation.

Maximum Likelihood Estimation of Transition Probabilities of Jump Markov Linear Systems

This paper describes an online maximum likelihood estimator for the transition probabilities associated with a jump Markov linear system (JMLS). The maximum likelihood estimator is derived using the reference probability method, which exploits an hypothetical probability measure to find recursions for complex expectations. Expectation maximization (EM) procedure is utilized for maximizing the likelihood function. In order to avoid the exponential increase in the number of statistics of the optimal EM algorithm, we make interacting multiple model (IMM)-type approximations. The resulting method needs the mode weights of an IMM filter with N <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</sup> components, where N is the number of models in the JMLS. The algorithm can also supply base-state estimates and covariances as a by-product. The performance of the estimator is illustrated on two simulated examples and compared to a recently proposed alternative.

Maximum Confidence Hidden Markov Modeling for Face Recognition

This paper presents a hybrid framework of feature extraction and hidden Markov modeling(HMM) for two-dimensional pattern recognition. Importantly, we explore a new discriminative training criterion to assure model compactness and discriminability. This criterion is derived from hypothesis test theory via maximizing the confidence of accepting the hypothesis that observations are from target HMM states rather than competing HMM states. Accordingly, we develop the maximum confidence hidden Markov modeling (MC-HMM) for face recognition. Under this framework, we merge a transformation matrix to extract discriminative facial features. The closed-form solutions to continuous-density HMM parameters are formulated. Attractively, the hybrid MC-HMM parameters are estimated under the same criterion and converged through the expectation-maximization procedure. From the experiments on FERET and GTFD facial databases, we find that the proposed method obtains robust segmentation in presence of different facial expressions, orientations, etc. In comparison with maximum likelihood and minimum classification error HMMs, the proposed MC-HMM achieves higher recognition accuracies with lower feature dimensions.

Cluster analysis of tooth size in subjects with normal occlusion

Tooth-size studies are important in orthodontics. Our aim in this study was to establish normative data on tooth size with a clustering method. Dental casts of 307 subjects with normal occlusion were examined. In these subjects, the tooth-size data sets in the maxilla and the mandible were clustered for men and women by using multivariate normal mixture models. This analysis clustered the data sets into groups of observations that were cohesive and separate from the other groups. The parameters were estimated with expectation-maximization procedures, and the number of clusters was selected via the Bayesian information criterion. Seven heterogeneous clusters in men and 4 clusters in women were identified. This study indicated that, in normal occlusion, tooth sizes are variable. These heterogeneities in the normal occlusion group seem to have contributed to the variability found in normative data. The method used in this study seems to provide a more substantive design for artificial teeth and add an additional dimension in the process of diagnosis of patients. Further applications seem possible in dental anthropometry by simultaneously dealing with the full dentition as a data set.

Comparison between the loading capacities of columns packed with partially and totally porous fine particles: What is the effective surface area available for adsorption?

The adsorption isotherms of phenol, caffeine, insulin, and lysozyme were measured on two C 18-bonded silica columns. The first one was packed with classical totally porous particles (3 μ m Luna(2)-C 18from Phenomenex, Torrance, CA, USA), the second one with shell particles (2.7 μ m Halo-C 18 from Advanced Materials Technology, Wilmington, DE, USA). The measurements were made at room temperature ( T = 295 ± 1 K), using mainly frontal analysis (FA) and also elution by characteristic points (FACP) when necessary. The adsorption energy distributions (AEDs) were estimated by the iterative numerical expectation-maximization (EM) procedure and served to justify the choice of the best adsorption isotherm model for each compound. The best isotherm parameters were derived from either the best fit of the experimental data to a multi-Langmuir isotherm model (MLRA) or from the AED results (equilibrium constants and saturation capacities), when the convergence of the EM program was achieved. The experiments show than the loading capacity of the Luna column is more than twice that of the Halo column for low-molecular-weight compounds. This result was expected; it is in good agreement with the values of the accessible surface area of these two materials, which were calculated from the pore size volume distributions. The pore size volume distributions are validated by the excellent agreement between the calculated and measured exclusion volumes of polystyrene standards by inverse size exclusion chromatography (ISEC). In contrast, the loading capacity ratio of the two columns is 1.5 or less with insulin and lysozyme. This is due to a significant exclusion of these two proteins from the internal pore volumes of the two packing materials. This result raises the problem of the determination of the effective surface area of the packing material, particularly in the case of proteins. This area is about 40 and 30% of the total surface area for insulin and for lysozyme, respectively, based on the pore size volume distribution validated by the ISEC method. The ISEC experiments showed that the largest and the smallest mesopores have rather a cylindrical and a spherical shape, respectively, for both packing materials.

Environmental Independent ASR Model Adaptation/Compensation by Bayesian Parametric Representation

The mismatch between system training and operating conditions can seriously deteriorate the performance of automatic speech recognition (ASR) systems. Various techniques have been proposed to solve this problem in a specified speech environment. Employment of these techniques often involves modification on the ASR system structure. In this paper, we propose an environment-independent (EI) ASR model parameter adaptation approach based on Bayesian parametric representation (BPR), which is able to adapt ASR models to new environments without changing the structure of an ASR system. The parameter set of BPR is optimized by a maximum joint likelihood criterion which is consistent with that of the hidden Markov model (HMM)-based ASR model through an independent expectation-maximization (EM) procedure. Variations of the proposed approach are investigated in the experiments designed in two different speech environments: one is the noisy environment provided by the AURORA 2 database, and the other is the network environment provided by the NTIMIT database. Performances of the proposed EI ASR model compensation approach are compared to those of the cepstral mean normalization (CMN) approach, which is one of the standard techniques for additive noise compensation. The experimental results show that performances of ASR models in different speech environments are significantly improved after being adapted by the proposed BPR model compensation approach

A multi-level thresholding approach using a hybrid optimal estimation algorithm

This paper presented a hybrid optimal estimation algorithm for solving multi-level thresholding problems in image segmentation. The distribution of image intensity is modeled as a random variable, which is approximated by a mixture Gaussian model. The Gaussian’s parameter estimates are iteratively computed by using the proposed PSO + EM algorithm, which consists of two main components: (i) global search by using particle swarm optimization (PSO); (ii) the best particle is updated through expectation maximization (EM) which leads the remaining particles to seek optimal solution in search space. In the PSO + EM algorithm, the parameter estimates fed into EM procedure are obtained from global search performed by PSO, expecting to provide a suitable starting point for EM while fitting the mixture Gaussians model. The preliminary experimental results show that the hybrid PSO + EM algorithm could solve the multi-level thresholding problem quite swiftly, and also provide quality thresholding outputs for complex images.

Brain SPECT with short focal-length cone-beam collimation.

Single-photon emission-computed tomography (SPECT) imaging of deep brain structures is compromised by loss of photons due to attenuation. We have previously shown that a centrally peaked collimator sensitivity function can compensate for this phenomenon, increasing sensitivity over most of the brain. For dual-head instruments, parallel-hole collimators cannot provide variable sensitivity without simultaneously degrading spatial resolution near the center of the brain; this suggests the use of converging collimators. We have designed collimator pairs for dual-head SPECT systems to increase sensitivity, particularly in the center of the brain, and compared the new collimation approach to existing approaches on the basis of performance in estimating activity concentration of small structures at various locations in the brain. The collimator pairs we evaluated included a cone-beam collimator, for increased sensitivity, and a fan-beam collimator, for data sufficiency. We calculated projections of an ellipsoidal uniform background, with 0.9-cm-radius spherical lesions at several locations in the background. From these, we determined ideal signal-to-noise ratios (SNRCRB) for estimation of activity concentration within the spheres, based on the Cramer-Rao lower bound on variance. We also reconstructed, by an ordered-subset expectation-maximization (OS-EM) procedure, images of this phantom, as well as of the Zubal brain phantom, to allow visual assessment and to ensure that they were free of artifacts. The best of the collimator pairs evaluated comprised a cone-beam collimator with 20 cm focal length, for which the focal point is inside the brain, and a fan-beam collimator with 40 cm focal length. This pair yielded increased SNRCRB, compared to the parallel-parallel pair, throughout the imaging volume. The factor by which SNRCRB increased ranged from 1.1 at the most axially extreme location to 3.5 at the center. The gains in SNRCRB were relatively robust to mismatches between the center of the brain and the center of the imaging volume. Artifact-free reconstructions of simulated data acquired using this pair were obtained. Combining fan-beam and short-focusing cone-beam collimation should greatly improve dual-head brain SPECT imaging, especially for centrally located structures.

The Impact of Missing Data on Sample Reliability Estimates: Implications for Reliability Reporting Practices

A method for incorporating maximum likelihood (ML) estimation into reliability analyses with item-level missing data is outlined. An ML estimate of the covariance matrix is first obtained using the expectation maximization (EM) algorithm, and coefficient alpha is subsequently computed using standard formulae. A simulation study demonstrated that the EMapproach yields (a) less bias in reliability estimates, (b) dramatically reduces cross-sample fluctuation of estimates, and (c) yields more accurate confidence intervals. Implications for reliability reporting practices are discussed, and the EM procedure is demonstrated using a heuristic data set.

Physical origin of peak tailing on C 18-bonded silica in reversed-phase liquid chromatography

Single component isotherm data of caffeine and phenol were acquired on two different stationary phases for RPLC, using a methanol/water solution (25%, v/v, methanol) as the mobile phase. The columns were the non-endcapped Waters Resolve-C 18, and the Waters XTerra MS C 18. Both columns exhibit similar C 18-chain densities (2.45 and 2.50 μmol/m 2) and differ essentially by the nature of the underivatized solid support (a conventional, highly polar silica made from water glass, hence containing metal impurities, versus a silica–methylsilane hybrid surface with a lower density of less acidic free silanols). Thirty-two adsorption data points were acquired by FA, for caffeine, between 10 −3 and 24 g/l, a dynamic range of 24,000. Twenty-eigth adsorption data points were acquired for phenol, from 0.025 to 75 g/l, a dynamic range of 3000. The expectation-maximization procedure was used to derive the affinity energy distribution (AED) from the raw FA data points, assuming a local Langmuir isotherm. For caffeine, the AEDs converge to a bimodal and a quadrimodal distribution on XTerra MS-C 18 and Resolve-C 18, respectively. The values of the saturation capacity ( q s,1≃0.80 mol/l and q s,2≃0.10 mol/l) and the adsorption constant ( b 1≃3.1 l/mol and b 2≃29.1 l/mol) measured on the two columns for the lowest two energy modes 1 and 2, are comparable. These data are consistent with those previously measured on an endcapped Kromasil-C 18 in a 30/70 (v/v), methanol/water solution ( q s,1=0.9 mol/l and q s,2=0.10 mol/l, b 1=2.4 l/mol and b 2=16.1 l/mol). The presence of two higher energy modes on the Waters Resolve-C 18 column ( q s,3≃0.013 mol/l and q s,4≃2.6 10 −4 mol/l, b 3≃252 l/mol and b 4=13,200 l/mol) and the strong peak tailing of caffeine are explained by the existence of adsorption sites buried inside the C 18-bonded layer. It is demonstrated that strong interactions between caffeine and the water protected bare silica surface cannot explain these high-energy sites because the retention of caffeine on an underivatized Resolve silica column is almost zero. Possible hydrogen-bond interactions between caffeine and the non-protected isolated silanol groups remaining after synthesis amidst the C 18-chain network cannot explain these high energy interactions because, then, the smaller phenol molecule should exhibit similarly strong interactions with these isolated silanols on the same Resolve-C 18 column and, yet, the consequences of such interactions are not observed. These sites are more consistent with the heterogeneity of the local structure of the C 18-bonded layer. Regarding the adsorption of phenol, no matter whether the column is endcapped or not, its molecular interactions with the bare silica were negligible. For both columns, the best adsorption isotherm was the Bilangmuir model (with q s,1≃2 and q s,2≃0.67 mol/l, b 1≃0.61 and b 2≃10.3 l/mol). These parameters are consistent with those measured previously on an endcapped Kromasil-C 18 column under the same conditions (q s,1=1.5 and q s,2=0.71 mol/l, b 1=1.4 l/mol and b 2=11.3 l/mol). As for caffeine, the high-energy sites are definitely located within the C 18-bonded layer, not on the bare surface of the adsorbent.