Cross-modal Coherence Research Articles

Perceive, Reason, and Align: Context-guided cross-modal correlation learning for image–text retrieval

Due to the inconsistency in feature representations between different modalities, namely “Heterogeneous gap”, it remains a persistent challenge to correlate images and texts. Existing studies on image–text retrieval (ITR) mainly emphasize on inter-modal correlation learning through aligning instances or their patches from different modalities. However, it is hard to break through performance bottlenecks of ITR without powerfully supporting from intra-modal correlation. Unfortunately, few studies have sufficiently considered two critical tasks in intra-modal correlation learning: (1) intricate contextual information perceiving, and (2) intrinsic semantic relationships reasoning. Therefore, in this paper, we propose the Context-guided Cross-modal Correlation Learning (CCCL) framework for ITR under a novel paradigm: “Perceive, Reason, and Align”. Firstly, in the stage of “Perceive”, the context-guided mechanism based on the self-attention and gate mechanism is proposed to fully discover contextual information within modalities, eliminating unnecessary interactions between local-level patches. Secondly, in the stage of “Reason”, graph convolutional network with the residual structure is used to uncover relationships among patches within each modality to make reasonable inferences. Thirdly, in the stage of “Align”, to achieve precise inter-modal alignment, the complementarity between different modalities from both global-level and local-level is effectively mined and fused. Finally, to optimize our proposed CCCL framework, the hybrid loss is constructed by combining the cross-modal coherence term with the cross-modal alignment term. Our approach yields highly competitive results on two publicly available ITR datasets, that is, Flickr30K and MS-COCO.

From Data to Diagnosis: Enhancing Radiology Reporting With Clinical Features Encoding and Cross-Modal Coherence

From Data to Diagnosis: Enhancing Radiology Reporting With Clinical Features Encoding and Cross-Modal Coherence

Cross-modal coherence and incoherence of early infant interactive behavior: links to attachment in infants born very preterm or full-term

ABSTRACT Infants exhibit flexibly organized configurations of facial, vocal, affective, and motor behavior during caregiver-infant interactions that convey convergent messages about their internal states and desires. Prior work documents that greater cross-modal discrepancy at 4 months predicts disorganized attachment. Here, we evaluated whether: very preterm (VPT) or full-term (FT) status predicts cross-modal coherence or incoherence in infants’ behavior with the caregiver at 3 months; and, regardless of prematurity, whether cross-modal interactive coherence or incoherence predicts 12-month attachment. Participants included 155 infants (85 FT; 70 VPT), and their mothers followed from birth to 12 months (corrected age). Infants’ cross-modal coherent and incoherent responses were scored microanalytically from videotaped en-face interactions. Infants’ attachment security was evaluated during Ainsworth’s Strange Situation. Infants born VPT exhibited more incoherent cross-modal responses and insecure attachment than infants born FT. Regardless of prematurity, infants’ coherent and incoherent cross-modal interactive behaviors at 3 months predicted different attachment patterns at 12 months.

Cross-Modal Coherence for Text-to-Image Retrieval

Common image-text joint understanding techniques presume that images and the associated text can universally be characterized by a single implicit model. However, co-occurring images and text can be related in qualitatively different ways, and explicitly modeling it could improve the performance of current joint understanding models. In this paper, we train a Cross-Modal Coherence Model for text-to-image retrieval task. Our analysis shows that models trained with image–text coherence relations can retrieve images originally paired with target text more often than coherence-agnostic models. We also show via human evaluation that images retrieved by the proposed coherence-aware model are preferred over a coherence-agnostic baseline by a huge margin. Our findings provide insights into the ways that different modalities communicate and the role of coherence relations in capturing commonsense inferences in text and imagery.

High-frequency normal-mode statistics in shallow water: the combined effect of random surface and internal waves.

In an earlier article, the statistical properties of mode propagation were studied at a frequency of 1 kHz in a shallow water environment with random sound-speed perturbations from linear internal waves, using a hybrid transport theory and Monte Carlo numerical simulations. Here, the analysis is extended to include the effects of random linear surface waves, in isolation and in combination with internal waves. Mode coupling rates for both surface and internal waves are found to be significant, but strongly dependent on mode number. Mode phase randomization by surface waves is found to be dominated by coupling effects, and therefore a full transport theory treatment of the range evolution of the cross mode coherence matrix is needed. The second-moment of mode amplitudes is calculated using transport theory, thereby providing the mean intensity while the fourth-moment is calculated using Monte Carlo simulations, which provides the scintillation index. The transport theory results for second-moment statistics are shown to closely reproduce Monte Carlo simulations. Both surface waves and internal waves strongly influence the acoustic field fluctuations.

Physics of backscattering in terms of mode coupling applied to measured seabed roughness spectra in shallow water

Energy conserving coupled integral equations for the forward and backward propagating modal components have been previously developed [J. Acoust. Soc. Am. 130, 2673–2680 (2011)]. A rough seabed surface leads to a backscattered field and modifies the interference structure of the forward propagating field. Perturbation theory applied to the basic coupled integral equations allows for physical insight into the correlation of the roughness spectrum to the forward and backward modal intensities and cross mode coherence. This study applies the Nx2D integral equation coupled-mode approach to 3-D roughness measurements and examines the physics of the coupling of the forward and backward field components and computes the modal intensities as a function of azimuth. The roughness measurements were made in about 20 m of water off Panama City, Florida. [Work supported by ONR Code 322 OA.]

Physical modelling for interactive installations and the performing arts

This article explores new artistic possibilities through four different examples of digital environments where live performance and virtual physical models come together. Physical models (specifically mass-spring systems) are used to enhance mappings between human movement and digital rendering. In the case of multi-modal rendering, physical models improve the cross-modal coherence. Given a physical model's capacity to simulate physical behaviours, they can also be used to visualise or to simulate human movement qualities, and even to simulate a dancer's virtual partner. More applications certainly remain to be tested, but the present study opens new avenues for the exploration of the complex and physically plausible behaviours of mass-spring systems in the context of both artistic practice and art pedagogy.

FORWARD AND BACKWARD MODAL STATISTICS FOR ROUGH SEABED SURFACE SCATTERING IN SHALLOW WATER

Addressed is the physics of sound propagation through a shallow water waveguide whose seabed has a random rough surface. The basic coupled integral equations for the modal amplitudes can be split into forward and backward going coupled integral equations. Numerical solutions of the coupled equations are obtained for multiple roughness realizations from which ensemble averages can be obtained for the modal intensities and cross-mode coherence as a function of range. Sample calculations illustrate some of the basic characteristics of the average cross-mode coherence and intensity with and without seabed attenuation and for different source depths on range scales of 1500 acoustic wavelengths.

Observations and transport theory analysis of low frequency, acoustic mode propagation in the Eastern North Pacific Ocean

Second order mode statistics as a function of range and source depth are presented from the Long Range Ocean Acoustic Propagation EXperiment (LOAPEX). During LOAPEX, low frequency broadband signals were transmitted from a ship-suspended source to a mode-resolving vertical line array. Over a one-month period, the ship occupied seven stations from 50 km to 3200 km distance from the receiver. At each station broadband transmissions were performed at a near-axial depth of 800 m and an off-axial depth of 350 m. Center frequencies at these two depths were 75 Hz and 68 Hz, respectively. Estimates of observed mean mode energy, cross mode coherence, and temporal coherence are compared with predictions from modal transport theory, utilizing the Garrett-Munk internal wave spectrum. In estimating the acoustic observables, there were challenges including low signal to noise ratio, corrections for source motion, and small sample sizes. The experimental observations agree with theoretical predictions within experimental uncertainty.

When sounds look right and images sound correct: Cross-modal coherence enhances claims of pattern presence

How do people decide whether a stimulus contains a pattern? One possibility is that they rely on a global, non-specific signal of coherence. Interestingly, this signal might reflect a combination of different stimulus sources. Consequently, the coherence of one stimulus might influence decisions about coherence of a second, unrelated stimulus. We explored this possibility in three experiments in which participants judged the presence of a pattern in targets from one sensory modality, while being exposed in the background to incidental coherent and incoherent stimuli in a different modality (visual→auditory, auditory→visual). Across all three experiments, using a variety of judgments, coherence of incidental background cross-modal patterns enhanced claims of pattern presence. These findings advance our understanding of how people judge order in the structured as well as in the unstructured world.

Influence of rough seabed surface on statistics of modal energy flux

When sound propagates through a random media, the wave properties of the acoustic field may be viewed as stochastic variables. It is thus natural to investigate the relationship between the statistical properties of the acoustic field and the random fluctuations of the waveguide. The interest here is a 2-D ocean waveguide with a rough seabed. Numerical solutions to the 2-way integral coupled mode equations (ICME), for random realizations of the roughness from a wavenumber power spectrum, provide the statistics of the modal intensity and cross-mode coherence with range. The roughness induces mode coupling within the trapped spectrum, between the trapped and the continuum spectrum, and to the back propagating modal spectrum. Instead of a master equation for the modal intensities to study the connection between the statistics of the acoustics and the fluctuations in the waveguide as has been advocated in previous studies, the conservation law for acoustic energy flux is used to develop an expression for the individual modal Poynting vectors. In addition to exact numerical computations of the range derivatives of the modal energy flux vectors for both forward and backward propagation, a Poynting vector master equation is derived for the case where the Born approximation is valid.

Statistics of low-frequency normal-mode amplitudes in an ocean with random sound-speed perturbations: Shallow-water environments

Second- and fourth-moment mode-amplitude statistics for low-frequency ocean sound propagation through random sound-speed perturbations in a shallow-water environment are investigated using Monte Carlo simulations and a transport theory for the cross-mode coherence matrix. The acoustic observables of mean and mean square intensity are presented and the importance of adiabatic effects and cross-mode coherence decay are emphasized. Using frequencies of 200 and 400 Hz, transport theory is compared with Monte Carlo simulations in a canonical shallow-water environment representative of the summer Mid-Atlantic Bight. Except for ranges less than a horizontal coherence length of the sound structure, the intensity moments from the two calculations are in good agreement. Corrections for the short range behavior are presented. For these frequencies the computed mode coupling rates are extremely small, and the propagation is strongly adiabatic with a rapid decay of cross-mode coherence. Coupling effects are predicted to be important at kilohertz frequencies. Decay of cross-mode coherence has important implications for acoustic interactions with nonlinear internal waves: For the case in which the acoustic path is not at glancing incidence with a nonlinear internal-wave front, adiabatic phase randomizing effects lead to a significantly reduced influence of the nonlinear waves on both mean and mean square intensity.

Transport equations for cross-frequency transport equations for acoustic intensity moments

The second and fourth moment mode-amplitude statistics for ocean sound propagation through random sound-speed perturbations are investigated using exact transport theory for the cross-frequency cross-mode coherence matrix. These exact equations are derived using the method of successive approximations, originally developed by Klyatskin and Tartarskii. These equations allow the determination of the validity of the usual transport equation (involving the Markov approximation), which is the first order approximation (in a infinite sequence of approximations to the exact equations). The range scales for the approach to the asymptotic behavior of the intensity moments, and the decay of the cross-modal coherence is easily determined at all frequencies.

Transport equation approach for second order mode statistics in an ocean with random sound speed perturbations: Coherence

A transport equation has been derived to describe the range evolution of the single frequency cross mode coherence matrix so that acoustic field coherence functions with temporal as well as depth and transverse separations can be easily computed. The theory assumes 2-D propagation in the depth range plane, small angle weak multiple forward scattering, and the Markov approximation, and it has been previously shown to accurately predict the observable of mean intensity for both deep and shallow water environments. This talk will address the issues of: the accuracy of the approximations, relative contributions from coupling and adiabatic effects, scaling with range and frequency, and the functional form of the coherence with regards to lag.

Learning across senses: cross-modal effects in multisensory statistical learning.

It is currently unknown whether statistical learning is supported by modality-general or modality-specific mechanisms. One issue within this debate concerns the independence of learning in one modality from learning in other modalities. In the present study, the authors examined the extent to which statistical learning across modalities is independent by simultaneously presenting learners with auditory and visual streams. After establishing baseline rates of learning for each stream independently, they systematically varied the amount of audiovisual correspondence across 3 experiments. They found that learners were able to segment both streams successfully only when the boundaries of the audio and visual triplets were in alignment. This pattern of results suggests that learners are able to extract multiple statistical regularities across modalities provided that there is some degree of cross-modal coherence. They discuss the implications of their results in light of recent claims that multisensory statistical learning is guided by modality-independent mechanisms.

Statistics of mode amplitudes in an ocean with random sound speed perturbations: Temporal coherence.

A transport equation has been derived to describe the range evolution of the single frequency cross mode coherence matrix including time separations. The theory assumes 2‐D propagation in the depth range plane, small angle, weak multiple forward scattering, and the Markov approximation, and it has been shown to accurately predict the observable of mean intensity for both deep and shallow water (i.e., strongly lossy) situations. This paper will present results for the acoustic observable time coherence and will address the issues of accuracy of the temporal coherence approximations, relative contributions from coupling and adiabatic effects, scaling with range and frequency, and the functional form of the coherence with regards to lag.

Statistics of normal mode amplitudes in an ocean with random sound-speed perturbations: Cross-mode coherence and mean intensity

In this paper Creamer's [(1996). J. Acoust. Soc. Am. 99, 2825-2838] transport equation for the mode amplitude coherence matrix resulting from coupled mode propagation through random fields of internal waves is examined in more detail. It is shown that the mode energy equations are approximately independent of the cross mode coherences, and that cross mode coherences and mode energy can evolve over very similar range scales. The decay of cross mode coherence depends on the relative mode phase randomization caused by coupling and adiabatic effects, each of which can be quantified by the theory. This behavior has a dramatic effect on the acoustic field second moments like mean intensity. Comparing estimates of the coherence matrix and mean intensity from Monte Carlo simulation, and the transport equations, good agreement is demonstrated for a 100-Hz deep-water example.

Statistics of mode amplitudes in shallow water environments: Effects of random linear internal waves and nonlinear internal wave packets.

In shallow water environments there is presently an incomplete understanding of the relative acoustical importance of the random linear internal-wave field compared to nonlinear internal waves packets. Using the random coupled mode theory of [Creamer (1996)], the nonlinear internal wave mode coupling theory of [Colosi (2007)], and Monte Carlo numerical simulation we are able to demonstrate the important propagation physics when the two internal wave fields act alone, and simultaneously. Importantly, we find that for acoustic frequencies less than roughly 500 Hz and for propagation ranges of order 10s of kms mode propagation though the random internal wave field is very nearly adiabatic, and that cross mode coherences decay rapidly within the first 10 km. A surprising consequence of the decay of cross mode coherence is that the mean intensity will be to first order insensitive to nonlinear internal wave packets if they are located past the de-coherence range. It will also be shown that uncorrelated modes have important implications for the approach to saturation. The adiabatic nature of mode propagation though the random internal wave field may have important implications for shallow water acoustic remote sensing.

Volume scattering in a shallow channel

Using a generalization of the modal coherence equations previously developed by Sutton and McCoy [J. Math. Phys. 18, 1052 (1977)], the effect of volume scattering in a shallow channel is treated. The difference between the range behavior of the cross-modal coherence functions and the previously studied self-modal coherence functions is shown. In particular, calculations are made to evaluate the characteristic scales that govern the range decay of the cross-modal coherence functions. A particular example is given to illustrate the effect of the choice of different parameters.