Multimodal State Research Articles

Gaussian Mixture Model-Based Ensemble Kalman Filtering for State and Parameter Estimation for a PMMA Process

Polymer processes often contain state variables whose distributions are multimodal; in addition, the models for these processes are often complex and nonlinear with uncertain parameters. This presents a challenge for Kalman-based state estimators such as the ensemble Kalman filter. We develop an estimator based on a Gaussian mixture model (GMM) coupled with the ensemble Kalman filter (EnKF) specifically for estimation with multimodal state distributions. The expectation maximization algorithm is used for clustering in the Gaussian mixture model. The performance of the GMM-based EnKF is compared to that of the EnKF and the particle filter (PF) through simulations of a polymethyl methacrylate process, and it is seen that it clearly outperforms the other estimators both in state and parameter estimation. While the PF is also able to handle nonlinearity and multimodality, its lack of robustness to model-plant mismatch affects its performance significantly.

Multimodal emotional state recognition using sequence-dependent deep hierarchical features

Emotional state recognition has become an important topic for human–robot interaction in the past years. By determining emotion expressions, robots can identify important variables of human behavior and use these to communicate in a more human-like fashion and thereby extend the interaction possibilities. Human emotions are multimodal and spontaneous, which makes them hard to be recognized by robots. Each modality has its own restrictions and constraints which, together with the non-structured behavior of spontaneous expressions, create several difficulties for the approaches present in the literature, which are based on several explicit feature extraction techniques and manual modality fusion. Our model uses a hierarchical feature representation to deal with spontaneous emotions, and learns how to integrate multiple modalities for non-verbal emotion recognition, making it suitable to be used in an HRI scenario. Our experiments show that a significant improvement of recognition accuracy is achieved when we use hierarchical features and multimodal information, and our model improves the accuracy of state-of-the-art approaches from 82.5% reported in the literature to 91.3% for a benchmark dataset on spontaneous emotion expressions.

A Study of Multimodal Addressee Detection in Human-Human-Computer Interaction

The goal of addressee detection is to answer the question , “Are you talking to me?” When a dialogue system interacts with multiple users, it is crucial to detect when a user is speaking to the system as opposed to another person. We study this problem in a multimodal scenario, using lexical, acoustic, visual, dialogue state, and beamforming information. Using data from a multiparty dialogue system, we quantify the benefits of using multiple modalities over using a single modality. We also assess the relative importance of the various modalities, as well as of key individual features, in estimating the addressee. We find that energy-based acoustic features are by far the most important, that information from speech recognition and system state is useful as well, and that visual and beamforming features provide little additional benefit. While we find that head pose is affected by whom the speaker is addressing, it yields little nonredundant information due to the system acting as a situational attractor. Our findings would be relevant to multiparty, open-world dialogue systems in which the agent plays an active, conversational role, such as an interactive assistant deployed in a public, open space. For these scenarios , our study suggests that acoustic, lexical, and system-state information is an effective and practical combination of modalities to use for addressee detection. We also consider how our analyses might be affected by the ongoing development of more realistic, natural dialogue systems.

지능형 전시 서비스 구현을 위한 멀티모달 감정 상태 추정 모형

최근 관람객의 반응에 따라 실시간으로 대응하여 관객의 몰입과 만족도를 증대시키는 인터랙티브 전시 서비스에 대한 학계와 산업계의 관심이 높아지고 있다. 이러한 인터랙티브 전시 서비스를 효과적으로 구현하기 위해서는 관객의 반응을 통해 해당 관객이 느끼는 감정 상태를 추정할 수 있는 지능형 기술의 도입이 요구된다. 인간의 감정 상태를 추정하기 위한 시도들은 많은 연구들에서 이루어져 왔고, 그 중 대부분은 사람의 얼굴 표정이나 소리 반응을 통해 감정 상태를 추정하는 방식을 도입하고 있다. 하지만, 최근 소개되고 있는 연구들에 따르면 단일 반응이 아닌 여러 반응을 종합적으로 고려하는 이른바 멀티 모달(multimodal) 접근을 사용했을 경우, 인간의 감정 상태를 보다 정확하게 추정할 수 있다. 이러한 배경에서 본 연구는 키넥트 센서를 통해 측정되는 관객의 얼굴 표정, 몸짓, 움직임 등을 종합적으로 고려한 새로운 멀티모달 감정 상태 추정 모형을 제안하고 있다. 제안모형의 예측 기법으로는 방대한 양의 데이터를 효과적으로 처리하기 위해, 몬테칼로(Monte Carlo) 방법인 계층화 샘플링(stratified sampling) 방법에 기반한 다중회귀분석을 적용하였다. 제안 모형의 성능을 검증하기 위해, 15명의 피실험자로부터 274개의 독립 및 종속변수들로 구성된 602,599건의 관측 데이터를 수집하여 여기에 제안 모형을 적용해 보았다. 그 결과 10~15% 이내의 평균오차 범위 내에서 피실험자의 쾌/불쾌도(valence) 및 각성도(arousal) 상태를 정확하게 추정할 수 있음을 확인할 수 있었다. 이러한 본 연구의 제안 모형은 비교적 구현이 간단하면서도 안정성이 높아, 향후 지능형 전시 서비스 및 기타 원격학습이나 광고 분야 등에 효과적으로 활용될 수 있을 것으로 기대된다. Both researchers and practitioners are showing an increased interested in interactive exhibition services. Interactive exhibition services are designed to directly respond to visitor responses in real time, so as to fully engage visitors' interest and enhance their satisfaction. In order to install an effective interactive exhibition service, it is essential to adopt intelligent technologies that enable accurate estimation of a visitor's emotional state from responses to exhibited stimulus. Studies undertaken so far have attempted to estimate the human emotional state, most of them doing so by gauging either facial expressions or audio responses. However, the most recent research suggests that, a multimodal approach that uses people's multiple responses simultaneously may lead to better estimation. Given this context, we propose a new multimodal emotional state estimation model that uses various responses including facial expressions, gestures, and movements measured by the Microsoft Kinect Sensor. In order to effectively handle a large amount of sensory data, we propose to use stratified sampling-based MRA (multiple regression analysis) as our estimation method. To validate the usefulness of the proposed model, we collected 602,599 responses and emotional state data with 274 variables from 15 people. When we applied our model to the data set, we found that our model estimated the levels of valence and arousal in the 10~15% error range. Since our proposed model is simple and stable, we expect that it will be applied not only in intelligent exhibition services, but also in other areas such as e-learning and personalized advertising.

Photometric Attitude Estimation for Agile Space Objects with Shape Uncertainty

The problem of estimating attitude for actively maneuvering or passively rotating space objects with unknown mass properties/external torques and uncertain shape models is addressed. To account for agile space object maneuvers, angular rates are simply assumed to be random inputs (e.g., process noise), and model uncertainty is accounted for in a bias state with dynamics derived using first principles. Bayesian estimation approaches are used to estimate the resulting severely non-Gaussian and multimodal state distributions. Simulated results are given, conclusions regarding performance are made, and future work is outlined.

Co-existence of wind seas and swells along the west coast of India during non-monsoon season

Abstract. An attempt has been made to understand the co-existence of wind seas and swells along the west coast of India during non-monsoon season. Wave data were collected in different years during non-monsoon season (off Goa during May 2005, off Ratnagiri during January–February 2008 and off Dwarka during December 2007–January 2008), which is fairly a calm weather season along these regions. Diurnal variation in wave parameters is noticeable along the central west coast of India (off Goa and Ratnagiri), which is due to the interaction of multidirectional waves (both wind seas and swells) of varying magnitudes and frequencies. Swells are predominantly mature (91%) and old (88%) during late pre-monsoon and post-monsoon seasons, respectively. Sea Swell Energy Ratio quantifies wind sea, swell and mixed seas prevailing in these regions during non-monsoon season. Intermodal distance (ID) between the energy peaks is moderately separated during non-monsoon season, whereas, during the shamal events, energy peaks are very close to each other (ID ∼ 0). However, pure wind seas (ID ∼ 1) are weakly present and found to co-exist with the swells almost all the time during non-monsoon season. Wind sea growth has been found while the swell propagates opposite to the direction of the wind and wind sea. Wind seas have minimum angular spreads in multimodal state. Under low winds, the interaction between wind sea and swell dominates and thereby the multimodal state reduces to unimodal state. The fetch available for the evolution of the wind sea spectrum has been estimated, and it is found to be less than 150 km. For the fetch limited condition, a non-dimensional empirical relation has been derived relating the significant wind sea height in terms of wind speed and peak wind sea period, and this relation fits for the west coast of India.

Induction of protein conformational change inside the charged electrospray droplet

The behavior of the analyte molecules inside the neutral core of the charged electrospray (ES) droplet is not unambiguously known to date. The possibility of protein conformational change inside the charged ES droplet has been investigated. The ES droplets encapsulating the protein molecules were exposed to the acetic acid vapor in the ionization chamber to absorb the acetic acid vapor. Because of the faster evaporation of water than that of acetic acid, the droplets became enriched with acetic acid and thus altered the solvent environment (e.g. pH and polarity) of the final charged droplets from where the naked charged analytes (proteins) are formed. Thus, the perturbation of the ES droplet solvent environment resulted in the protein conformational change (unfolding) during the short lifespan of the ES droplet and that is reflected by the multimodal charge state distribution in the corresponding mass spectra. Further, the extent of this conformational change inside the ES droplet was found to be related to the structural flexibility of the protein. Although the protein conformational change inside the ES droplet has been driven by using acetic acid vapor in the present study, the results would help in the near future to understand the spontaneity of the conformational change of the analyte on the millisecond timescale of phase transition in the natural way of ES process.

Modular neuron-based body estimation: maintaining consistency over different limbs, modalities, and frames of reference

This paper addresses the question of how the brain maintains a probabilistic body state estimate over time from a modeling perspective. The neural Modular Modality Frame (nMMF) model simulates such a body state estimation process by continuously integrating redundant, multimodal body state information sources. The body state estimate itself is distributed over separate, but bidirectionally interacting modules. nMMF compares the incoming sensory and present body state information across the interacting modules and fuses the information sources accordingly. At the same time, nMMF enforces body state estimation consistency across the modules. nMMF is able to detect conflicting sensory information and to consequently decrease the influence of implausible sensor sources on the fly. In contrast to the previously published Modular Modality Frame (MMF) model, nMMF offers a biologically plausible neural implementation based on distributed, probabilistic population codes. Besides its neural plausibility, the neural encoding has the advantage of enabling (a) additional probabilistic information flow across the separate body state estimation modules and (b) the representation of arbitrary probability distributions of a body state. The results show that the neural estimates can detect and decrease the impact of false sensory information, can propagate conflicting information across modules, and can improve overall estimation accuracy due to additional module interactions. Even bodily illusions, such as the rubber hand illusion, can be simulated with nMMF. We conclude with an outlook on the potential of modeling human data and of invoking goal-directed behavioral control.

From Sticky to Slippery Droplets: Dynamics of Contact Line Depinning on Superhydrophobic Surfaces

This study explores how surface morphology affects the dynamics of contact line depinning of an evaporating sessile droplet on micropillared superhydrophobic surfaces. The result shows that neither a liquid-solid contact area nor an apparent contact line is a critical physical parameter to determine the depinning force. The configuration of a contact line on a superhydrophobic surface is multimodal, composed of both two phases (liquid and air) and three phases (liquid, solid, and air). The multimodal state is dynamically altered when a droplet recedes. The maximal three-phase contact line attainable along the actual droplet boundary is found to be a direct and linear parameter that decides the depinning force on the superhydrophobic surface.

Eggshell membrane as a multimodal solid state platform for generating fluorescent metal nanoclusters

Eggshell membrane (ESM) has been employed as a unique and especially efficient synthetic platform capable of generating fluorescent silver and gold nanoclusters via various chemical routes. Potential applications of the metal/ESM adducts include recyclable catalysts, sensing paper, surface enhanced Raman scattering interface, fluorescent surface patterning and anti-counterfeiting.

BM³E : Discriminative Density Propagation for Visual Tracking

We introduce BM3 E, a Conditional Bayesian Mixture of Experts Markov Model, for consistent probabilistic estimates in discriminative visual tracking. The model applies to problems of temporal and uncertain inference and represents the unexplored bottom-up counterpart of pervasive generative models estimated with Kalman filtering or particle filtering. Instead of inverting a non-linear generative observation model at run-time, we learn to cooperatively predict complex state distributions directly from descriptors that encode image observations - typically bag-of-feature global image histograms or descriptors computed over regular spatial grids. These are integrated in a conditional graphical model in order to enforce temporal smoothness constraints and allow a principled management of uncertainty. The algorithms combine sparsity, mixture modeling, and non-linear dimensionality reduction for efficient computation in high-dimensional continuous state spaces. The combined system automatically self-initializes and recovers from failure. The research has three contributions: (1) We establish the density propagation rules for discriminative inference in continuous, temporal chain models; (2) We propose flexible supervised and unsupervised algorithms for learning feedforward, multivalued contextual mappings (multimodal state distributions) based on compact, conditional Bayesian mixture of experts models; (3) We validate the framework empirically for the reconstruction of 3d human motion in monocular video sequences. Our tests on both real and motion capture-based sequences show significant performance gains with respect to competing nearest-neighbor, regression, and structured prediction methods.

Simulation studies of the fidelity of biomolecular structure ensemble recreation

We examine the ability of Bayesian methods to recreate structural ensembles for partially folded molecules from averaged data. Specifically we test the ability of various algorithms to recreate different transition state ensembles for folding proteins using a multiple replica simulation algorithm using input from "gold standard" reference ensembles that were first generated with a Go-like Hamiltonian having nonpairwise additive terms. A set of low resolution data, which function as the "experimental" phi values, were first constructed from this reference ensemble. The resulting phi values were then treated as one would treat laboratory experimental data and were used as input in the replica reconstruction algorithm. The resulting ensembles of structures obtained by the replica algorithm were compared to the gold standard reference ensemble, from which those "data" were, in fact, obtained. It is found that for a unimodal transition state ensemble with a low barrier, the multiple replica algorithm does recreate the reference ensemble fairly successfully when no experimental error is assumed. The Kolmogorov-Smirnov test as well as principal component analysis show that the overlap of the recovered and reference ensembles is significantly enhanced when multiple replicas are used. Reduction of the multiple replica ensembles by clustering successfully yields subensembles with close similarity to the reference ensembles. On the other hand, for a high barrier transition state with two distinct transition state ensembles, the single replica algorithm only samples a few structures of one of the reference ensemble basins. This is due to the fact that the phi values are intrinsically ensemble averaged quantities. The replica algorithm with multiple copies does sample both reference ensemble basins. In contrast to the single replica case, the multiple replicas are constrained to reproduce the average phi values, but allow fluctuations in phi for each individual copy. These fluctuations facilitate a more faithful sampling of the reference ensemble basins. Finally, we test how robustly the reconstruction algorithm can function by introducing errors in phi comparable in magnitude to those suggested by some authors. In this circumstance we observe that the chances of ensemble recovery with the replica algorithm are poor using a single replica, but are improved when multiple copies are used. A multimodal transition state ensemble, however, turns out to be more sensitive to large errors in phi (if appropriately gauged) and attempts at successful recreation of the reference ensemble with simple replica algorithms can fall short.

Model-based visual hand posture tracking for guiding a dexterous robotic hand

Visual hand gesture based interfaces have been widely used for navigation of virtual environments and control of a robot arm in robotic systems. In fact, hand movement has abundant powers of expression with complex finger joint motions. In this paper, the full degree-of-freedom hand motion is tracked in a vision-based mode towards natural and intuitive control of a dexterous robotic hand. The main obstacle to handle human hand posture tracking is the high dimensionality. A novel three-dimensional model-based approach is proposed. It utilizes a genetic algorithm-based global optimization procedure after each particle-filtering step, which deals with the tracking in high-dimensional and multi-modal state space reasonably well. Experimental results show that present approach improves performance of hand posture tracking significantly and provides a feasible solution for guiding a dexterous hand.

Instability of localized buckling modes in a one-dimensional strut model

Stability of localized solutions arising in a fourth–order differential equation modelling struts is investigated. It was shown by Buffoni et al . in 1996 that the model exhibits many multimodal buckling states bifurcating from a primary buckling mode. In this article, using analytical and numerical techniques, the primary mode is shown to be unstable under dead loading for all axial loads, while it is likely to be stable under rigid loading for small axial loads. Furthermore, for general reversible or conservative systems, stability of the multimodal solutions is established assuming stability of the primary state. Since this hypothesis is not satisfied for the buckling mode arising in the strut model, any multimodal buckling state will be unstable under dead and rigid loading.

Propensity of Statewide Multimodel Transportation

This paper presents the model areas of concern to statewide transportation planning, and the state's function in relation to each mode of transportation. The transition from highway departments and departments of transportation is examined on a state level. Rather than keying on one of several state or highway departments fundamental issues facing most states are analyzed. Furthermore, an attempt is made to detail the problems encountered in defining the role of such a department in the overall statewide planning process and concurrent difficulties in formulating alternatives that adequately reflect nontransportation effects of statewide transportation planning. The necessity of vertical feedback and input from local to regional to statewide planning is examined. Finally, the paper notes the basic differences between traditional highway planning and multimodal state transportation planning and problems encountered in the development of a statewide multimodal methodology.

Michigan Transportation in the Seventies

The State of Michigan and the U.S. Department of Transportation have both introduced programs that alter transportation policy. In Michigan, the Governor has proposed a program to divert a portion of the motor fuel tax to a discretionary account. These funds would then be used to meet urban transportation problems without restriction to modes. His proposal also includes a $1.7 billion highway program, and a move toward a system of regional airports. President Nixon's transportation revenue sharing proposal is analyzed as it might affect these Michigan programs. Multimodal state planning efforts are questioned, based on the measurable impacts found in Michigan. The ability of the civil engineer to maintain a prominent role in establishing transportation policy is questioned. Revised thinking and training are recommended to prepare the profession for the new challenges.