Multimodal Uncertainty Research Articles

Multimodal uncertainty propagation analysis for the morphing wings of cross-domain variant aircraft

Multimodal uncertainty propagation analysis for the morphing wings of cross-domain variant aircraft

Two-stage image colorization via color codebook

Image colorization has a wide range of applications, but it remains a challenging task due to it is an inherently ill-posed problem with multi-modal uncertainty. The advancement of deep learning techniques has provided extensive avenues for addressing image colorization. However, current works mainly suffer from two problems: inaccurate colorization leading to biased color tones (e.g., cool or warm bias) and undersaturation of images. Existing Transformer-based methods can produce impressive results, but they often come with high training costs and may result in color overflow effects. In this paper, we propose a two-stage image colorization strategy based on a color codebook. Clustering methods in the three-dimensional CIE Lab color space is proposed to integrate brightness information so that the colors in the codebook can be lifelike. In the first stage, we treat the colorization task as a classification problem based on a color codebook, and a high-quality codebook is advantageous for enhancing color classification accuracy. In the second stage, different from the traditional Transformer-based method, a pyramid-type Transformer structure is used to extract rich image features to refine the colors, which can solve potential color bands, color errors and color overflow. In addition, the parameters and FLOPs are significantly smaller than other traditional Transformer-based methods. Extensive experiments demonstrate that our method outperforms state-of-the-art approaches. On the ImageNet validation set, the achieved values are 4.60, 25.23, 0.19, and 39.82 in terms of FID, PSNR, LPIPS, and CF, respectively. On the COCO-Stuff validation set, the achieved values are 5.62, 25.15, 0.19, and 36.25 in terms of FID, PSNR, LPIPS, and CF, respectively. The codes are available at https://github.com/Tanghui2000/Two-stage_Image_Colorization_via_Color_Codebook.

Unifying Multi-Modal Uncertainty Modeling and Semantic Alignment for Text-to-Image Person Re-identification

Text-to-Image person re-identification (TI-ReID) aims to retrieve the images of target identity according to the given textual description. The existing methods in TI-ReID focus on aligning the visual and textual modalities through contrastive feature alignment or reconstructive masked language modeling (MLM). However, these methods parameterize the image/text instances as deterministic embeddings and do not explicitly consider the inherent uncertainty in pedestrian images and their textual descriptions, leading to limited image-text relationship expression and semantic alignment. To address the above problem, in this paper, we propose a novel method that unifies multi-modal uncertainty modeling and semantic alignment for TI-ReID. Specifically, we model the image and textual feature vectors of pedestrian as Gaussian distributions, where the multi-granularity uncertainty of the distribution is estimated by incorporating batch-level and identity-level feature variances for each modality. The multi-modal uncertainty modeling acts as a feature augmentation and provides richer image-text semantic relationship. Then we present a bi-directional cross-modal circle loss to more effectively align the probabilistic features between image and text in a self-paced manner. To further promote more comprehensive image-text semantic alignment, we design a task that complements the masked language modeling, focusing on the cross-modality semantic recovery of global masked token after cross-modal interaction. Extensive experiments conducted on three TI-ReID datasets highlight the effectiveness and superiority of our method over state-of-the-arts.

Multi-modal uncertainty estimations in topology-preserving joint biomedical image registration and segmentation

Multi-modal uncertainty estimations in topology-preserving joint biomedical image registration and segmentation

Urban localization using robust filtering at multiple linearization points

We propose a robust Bayesian filtering framework for state and multi-modal uncertainty estimation in urban settings by fusing diverse sensor measurements. Our framework addresses multi-modal uncertainty from various error sources by tracking a separate probability distribution for linearization points corresponding to dynamics, measurements, and cost functions. Multiple parallel robust Extended Kalman filters (R-EKF) leverage these linearization points to characterize the state probability distribution. Employing Rao–Blackwellization, we combine the linearization point distribution with the state distribution, resulting in a unified, efficient, and outlier-resistant Bayesian filter that captures multi-modal uncertainty. Furthermore, we introduce a gradient descent-based optimization method to refine the filter parameters using available data. Evaluating our filter on real-world data from a multi-sensor setup comprising camera, Global Navigation Satellite System (GNSS), and Attitude and Heading Reference System (AHRS) demonstrates improved performance in bounding position errors based on uncertainty, while maintaining competitive accuracy and comparable computation to existing methods. Our results suggest that our framework is a promising direction for safe and reliable localization in urban environments.

L-CoDe:Language-Based Colorization Using Color-Object Decoupled Conditions

Colorizing a grayscale image is inherently an ill-posed problem with multi-modal uncertainty. Language-based colorization offers a natural way of interaction to reduce such uncertainty via a user-provided caption. However, the color-object coupling and mismatch issues make the mapping from word to color difficult. In this paper, we propose L-CoDe, a Language-based Colorization network using color-object Decoupled conditions. A predictor for object-color corresponding matrix (OCCM) and a novel attention transfer module (ATM) are introduced to solve the color-object coupling problem. To deal with color-object mismatch that results in incorrect color-object correspondence, we adopt a soft-gated injection module (SIM). We further present a new dataset containing annotated color-object pairs to provide supervisory signals for resolving the coupling problem. Experimental results show that our approach outperforms state-of-the-art methods conditioned on captions.

An uncertainty propagation analysis method considering multimodal random distribution

<p indent=0mm>In practical engineering applications, random variables may follow a multimodal random distribution, such as damping characteristics of train shock absorber after long-term use and fatigue life of safety cut specimens after repair. Traditional uncertainty propagation analysis methods are mainly used for random variables following a unimodal distribution. Therefore, large errors will be produced when random variables follow a multimodal distribution. In this study, an uncertainty propagation analysis method considering a multimodal random distribution is proposed. First, the probability density function of input multimodal random variables is defined by the Gaussian mixture model; then, the high-order moments of response function are calculated using the efficient trivariate dimension reduction method, which effectively improves the computational efficiency of uncertainty propagation. Once again, the maximum entropy method is used to compute the probability density function of response, whereas an adaptive method based on entropy is used to determine the optimal order of statistical moments. Finally, four numerical examples are demonstrated to analyze the accuracy and efficiency of the proposed method in handling multimodal uncertainty propagation analysis.

How optimal is word recognition under multimodal uncertainty?

Identifying a spoken word in a referential context requires both the ability to integrate multimodal input and the ability to reason under uncertainty. How do these tasks interact with one another? We study how adults identify novel words under joint uncertainty in the auditory and visual modalities, and we propose an ideal observer model of how cues in these modalities are combined optimally. Model predictions are tested in four experiments where recognition is made under various sources of uncertainty. We found that participants use both auditory and visual cues to recognize novel words. When the signal is not distorted with environmental noise, participants weight the auditory and visual cues optimally, that is, according to the relative reliability of each modality. In contrast, when one modality has noise added to it, human perceivers systematically prefer the unperturbed modality to a greater extent than the optimal model does. This work extends the literature on perceptual cue combination to the case of word recognition in a referential context. In addition, this context offers a link to the study of multimodal information in word meaning learning.

Children’s Signaling of Their Uncertain Knowledge State: Prosody, Face, and Body Cues Come First

ABSTRACTChildren achieve their first language milestones initially in gesture and prosody before they do so in speech. However, little is known about the potential precursor role of those features later in development when children start using more complex linguistic skills. In this study, we explore how children’s ability to reflect on their degree of uncertainty develops. A total of 40 Catalan preschool children (and 10 adults) took part in a game in which they had to guess the identity of ten objects which they could touch but not see and then state how certain they were about each guess. An analysis of the children’s reflections on their own uncertainty showed that (a) when preschoolers take a stance, they start to use multimodal uncertainty signals accurately before they are able to self-report their own level of (un)certainty; (b) they use epistemic prosodic and gestural markers before they start using lexical cues; and (c) multimodal cues signaling knowledge state become increasingly more complex over time. These findings suggest that preschoolers’ expression of knowledge states through gestures and prosody reflects an early epistemic sensitivity which develops as they get older.

From qualification to consensus: The role of multimodal uncertainty in collective argumentation regarding complex integration

From qualification to consensus: The role of multimodal uncertainty in collective argumentation regarding complex integration

Multimode processes monitoring based on hierarchical mode division and subspace decomposition

AbstractTo handle multimodal uncertainty and dynamics which are common in actual industrial processes, this paper proposes a multimode process monitoring method based on hierarchical mode division and subspace decomposition. First, hierarchical mode division is presented to achieve hierarchical mode information for multimode process modelling, which can improve the adaptability of industrial processes with multimodal uncertainty. Then, hierarchical subspace decomposition is presented to decompose multimode processes into multiple global subspaces and multiple local subspaces for monitoring. Lastly, the most similar mode with various hierarchical modes at the current sampling point is dynamically determined for monitoring, which considers multimodal dynamics. The simulation tests of the penicillin fermentation processes are developed to show feasibility and superiority of the proposed method.

Differential-drive mobile robot control using a cloud of particles approach

Common control systems for mobile robots include the use of some deterministic control law coupled with some pose estimation method, such as the extended Kalman filter, by considering the certainty equivalence principle. Recent approaches consider the use of partially observable Markov decision process strategies together with Bayesian estimators. These methods are well suited to handle the uncertainty in pose estimation but demand significant processing power. In order to reduce the required processing power and still allow for multimodal or non-Gaussian uncertain distributions, we propose a scheme based on a particle filter and a corresponding cloud of control signals. The approach avoids the use of the certainty equivalence principle by postponing the decision on the optimal estimate to the control stage. As the mapping between the pose space and the control action space is nonlinear and the best estimation of robot pose is uncertain, postponing the decision to the control space makes it possible to select a better control action in the presence of multimodal and non-Gaussian uncertainty models. Simulation results are presented.

A new form of symplectically invariant multimode uncertainty relations

We derive symplectically invariant uncertainty relations for a set of canonically conjugated variables. The uncertainty relations obtained are multimode analogs of the Robertson–Schrodinger inequalities. Our uncertainty relations are equivalent to the necessary and sufficient conditions for a matrix to be a correlation matrix of some quantum state, obtained by R. Simon and coauthors. The advantage of our inequalities, compared to that suggested by Simon, consists in its simplicity and more obvious symplectic invariance. We derive our inequalities for the case of a two-mode system in explicit form. Particular cases of small and large degrees of correlation between the first and second modes are analyzed in detail.

E. C. G. Sudarshan and symmetry in classical dynamics, optics and quantum mechanics

We review work initiated and inspired by Sudarshan in relativistic dynamics, beam optics, partial coherence theory, Wigner distribution methods, multimode quantum optical squeezing, and geometric phases. The 1963 No Interaction Theorem using Dirac's instant form and particle World Line Conditions is recalled. Later attempts to overcome this result exploiting constrained Hamiltonian theory, reformulation of the World Line Conditions and extending Dirac's formalism, are reviewed.Dirac's front form leads to a formulation of Fourier Optics for the Maxwell field, determining the actions of First Order Systems (corresponding to matrices of Sp(2,R) and Sp(4,R)) on polarization in a consistent manner. These groups also help characterize properties and propagation of partially coherent Gaussian Schell Model beams, leading to invariant quality parameters and the new Twist phase.The higher dimensional groups Sp(2n,R) appear in the theory of Wigner distributions and in quantum optics. Elegant criteria for a Gaussian phase space function to be a Wigner distribution, expressions for multimode uncertainty principles and squeezing are described.In geometric phase theory we highlight the use of invariance properties that lead to a kinematical formulation and the important role of Bargmann invariants. Special features of these phases arising from unitary Lie group representations, and a new formulation based on the idea of Null Phase Curves, are presented.

Adaptive Multimodal Fusion by Uncertainty Compensation With Application to Audiovisual Speech Recognition

While the accuracy of feature measurements heavily depends on changing environmental conditions, studying the consequences of this fact in pattern recognition tasks has received relatively little attention to date. In this paper, we explicitly take feature measurement uncertainty into account and show how multimodal classification and learning rules should be adjusted to compensate for its effects. Our approach is particularly fruitful in multimodal fusion scenarios, such as audiovisual speech recognition, where multiple streams of complementary time-evolving features are integrated. For such applications, provided that the measurement noise uncertainty for each feature stream can be estimated, the proposed framework leads to highly adaptive multimodal fusion rules which are easy and efficient to implement. Our technique is widely applicable and can be transparently integrated with either synchronous or asynchronous multimodal sequence integration architectures. We further show that multimodal fusion methods relying on stream weights can naturally emerge from our scheme under certain assumptions; this connection provides valuable insights into the adaptivity properties of our multimodal uncertainty compensation approach. We show how these ideas can be practically applied for audiovisual speech recognition. In this context, we propose improved techniques for person-independent visual feature extraction and uncertainty estimation with active appearance models, and also discuss how enhanced audio features along with their uncertainty estimates can be effectively computed. We demonstrate the efficacy of our approach in audiovisual speech recognition experiments on the CUAVE database using either synchronous or asynchronous multimodal integration models.

Detecting entanglement by symplectic uncertainty relations

A hierarchy of multimode uncertainty relations on the second moments of n pairs of canonical operators is derived in terms of quantities invariant under linear canonical (i.e.,

Multimode Uncertainty Relations and Separability of Continuous Variable States

A multimode uncertainty relation (generalizing the Robertson-Schrödinger relation) is derived as a necessary constraint on the second moments of n pairs of canonical operators. In turn, necessary conditions for the separability of multimode continuous variable states under (m+n)-mode bipartitions are derived from the uncertainty relation. These conditions are proven to be necessary and sufficient for (1+n)-mode Gaussian states and for (m+n)-mode bisymmetric Gaussian states.

Nonparametric belief propagation for self-localization of sensor networks

Automatic self-localization is a critical need for the effective use of ad hoc sensor networks in military or civilian applications. In general, self-localization involves the combination of absolute location information (e.g., from a global positioning system) with relative calibration information (e.g., distance measurements between sensors) over regions of the network. Furthermore, it is generally desirable to distribute the computational burden across the network and minimize the amount of intersensor communication. We demonstrate that the information used for sensor localization is fundamentally local with regard to the network topology and use this observation to reformulate the problem within a graphical model framework. We then present and demonstrate the utility of nonparametric belief propagation (NBP), a recent generalization of particle filtering, for both estimating sensor locations and representing location uncertainties. NBP has the advantage that it is easily implemented in a distributed fashion, admits a wide variety of statistical models, and can represent multimodal uncertainty. Using simulations of small to moderately sized sensor networks, we show that NBP may be made robust to outlier measurement errors by a simple model augmentation, and that judicious message construction can result in better estimates. Furthermore, we provide an analysis of NBP's communications requirements, showing that typically only a few messages per sensor are required, and that even low bit-rate approximations of these messages can be used with little or no performance impact.

Two-mode squeezing in Raman couplers

We investigate two-mode squeezing of compound Stokes-anti-Stokes modes propagating in a Raman coupler with a broad reservoir phonon system. Three types of squeezing: principal squeezing, generalized squeezing and sum squeezing are examined and their relations are identified. The definition of generalized squeezing is extended to an arbitrary number of modes, and a multimode uncertainty product is introduced. The quantum statistical properties of the optical fields propagating in the coupler are treated within the framework of the generalized superposition of the coherent signal and quantum noise. According to this model, generalized squeezing appears whenever the Glauber-Sudarshan quasidistribution does not exist as an ordinary function.