Multimodal Correlations Research Articles

Multimodal Hateful Meme Classification Based on Transfer Learning and a Cross-Mask Mechanism

Hateful memes are malicious and biased sentiment information widely spread on the internet. Detecting hateful memes differs from traditional multimodal tasks because, in conventional tasks, visual and textual information align semantically. However, the challenge in detecting hateful memes lies in their unique multimodal nature, where images and text in memes may be weak or unrelated, requiring models to understand the content and perform multimodal reasoning. To address this issue, we introduce a multimodal fine-grained hateful memes detection model named “TCAM”. The model leverages advanced encoding techniques from TweetEval and CLIP and introduces enhanced Cross-Attention and Cross-Mask Mechanisms (CAM) in the feature fusion stage to improve multimodal correlations. It effectively embeds fine-grained features of data and image descriptions into the model through transfer learning. This paper uses the Area Under the Receiver Operating Characteristic Curve (AUROC) as the primary metric to evaluate the model’s discriminatory ability. This approach achieved an AUROC score of 0.8362 and an accuracy score of 0.764 on the Facebook Hateful Memes Challenge (FHMC) dataset, confirming its high discriminatory capability. The TCAM model demonstrates relatively superior performance compared to ensemble machine learning methods.

Fusion recognition of palmprint and palm vein based on modal correlation.

Biometric authentication prevents losses from identity misuse in the artificial intelligence (AI) era. The fusion method integrates palmprint and palm vein features, leveraging their stability and security and enhances counterfeiting prevention and overall system efficiency through multimodal correlations. However, most of the existing multi-modal palmprint and palm vein feature extraction methods extract only feature information independently from different modalities, ignoring the importance of the correlation between different modal samples in the class to the improvement of recognition performance. In this study, we addressed the aforementioned issues by proposing a feature-level joint learning fusion approach for palmprint and palm vein recognition based on modal correlations. The method employs a sparse unsupervised projection algorithm with a "purification matrix" constraint to enhance consistency in intra-modal features. This minimizes data reconstruction errors, eliminating noise and extracting compact, and discriminative representations. Subsequently, the partial least squares algorithm extracts high grayscale variance and category correlation subspaces from each modality. A weighted sum is then utilized to dynamically optimize the contribution of each modality for effective classification recognition. Experimental evaluations conducted for five multimodal databases, composed of six unimodal databases including the Chinese Academy of Sciences multispectral palmprint and palm vein databases, yielded equal error rates (EER) of 0.0173%, 0.0192%, 0.0059%, 0.0010%, and 0.0008%. Compared to some classical methods for palmprint and palm vein fusion recognition, the algorithm significantly improves recognition performance. The algorithm is suitable for identity recognition in scenarios with high security requirements and holds practical value.

VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Dataset.

In this paper, we propose the Vision-Audio-Language Omni-peRception pretraining model (VALOR) for multimodal understanding and generation. Unlike widely-studied vision-language pretraining models, VALOR jointly models the relationships among vision, audio, and language in an end-to-end manner. It consists of three separate encoders for single modality representations and a decoder for multimodal conditional text generation. We design two pretext tasks to pretrain the VALOR model: Multimodal Grouping Alignment (MGA) and Multimodal Grouping Captioning (MGC). MGA projects vision, language, and audio into the same common space, simultaneously building vision-language, audio-language, and audiovisual-language alignment. MGC learns to generate text tokens under conditions of vision, audio, or both. To promote vision-audio-language pretraining research, we construct a large-scale, high-quality tri-modality dataset named VALOR-1M, containing 1 million audible videos with human-annotated audiovisual captions. Extensive experiments show that VALOR can learn strong multimodal correlations and generalize to various downstream tasks (e.g., retrieval, captioning, and question answering) with different input modalities (e.g., vision-language, audio-language, and audiovisual-language). VALOR achieves new state-of-the-art performance on a series of public cross-modality benchmarks. Code and data are available on the project page at https://casia-iva-group.github.io/projects/VALOR.

Work Together: Correlation-Identity Reconstruction Hashing for Unsupervised Cross-Modal Retrieval

Unsupervised cross-modal hashing has attracted considerable attention to support large-scale cross-modal retrieval. Although promising progresses have been made so far, existing methods still suffer from limited capability on excavating and preserving the intrinsic multi-modal semantics. In this paper, we propose a Correlation-Identity Reconstruction Hashing (CIRH) method to alleviate this challenging problem. We develop a new unsupervised deep cross-modal hash learning framework to model and preserve the heterogeneous multi-modal correlation semantics into both hash codes and functions, and simultaneously, we involve both the hash codes and functions with the descriptive identity semantics. Specifically, we construct a multi-modal collaborated graph to model the heterogeneous multi-modal correlations, and jointly perform the intra-modal and cross-modal semantic aggregation on homogeneous and heterogeneous graph networks to generate a multi-modal complementary representation with correlation reconstruction. Furthermore, an identity semantic reconstruction process is designed to involve the generated representation with identity semantics by reconstructing the input modality representations. Finally, we propose a correlation-identity consistent hash function learning strategy to transfer the modelled multi-modal semantics into the neural networks of modality-specific deep hash functions. Experiments demonstrate the superior performance of the proposed method on both retrieval accuracy and efficiency. We provide our source codes and experimental datasets at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/XizeWu/CIRH</uri> .

Multi-Source Knowledge Reasoning Graph Network for Multi-Modal Commonsense Inference

As a crucial part of natural language processing, event-centered commonsense inference task has attracted increasing attention. With a given observed event, the intention and reaction of the people involved in the event are required to be inferred with artificial intelligent algorithms. To solve this problem, sequence-to-sequence methods are widely studied, where the event is first encoded into a specific representation and then decoded to generate the results. However, all the existing methods learn the event representation only with the textual information, while the visual information is ignored, which is actually helpful for the commonsense reference. In this article, we first define a new task of multi-modal commonsense reference with both textual and visual information. A new event-centered multi-modal dataset is also provided. Then we propose a multi-source knowledge reasoning graph network to solve this task, where three kinds of relational knowledge are considered. Multi-modal correlations are learned to get the event’s multi-modal representation from a global perspective. Intra-event object relations are explored to capture the fine-grained event feature with an object graph. Inter-event semantic relations are also explored through the external knowledge to understand the semantic associations among events with an event graph. We conduct extensive experiments on the new dataset, and the results show the effectiveness of our method.

Multimodal Correlations of GABAergic Interneurons in Layer 5 of Mouse Neocortex

Cortical GABAergic interneurons can be categorized into distinct subtypes based on multiple modalities, however, it remains unclear to what extent these modalities are correlated. Here, we utilized patch-clamp RT-PCR (Patch-PCR) to simultaneously obtain the morpho-electric properties and mRNA profiles of over 600 interneurons (374 were morphologically reconstructed) in layer 5 of the mouse somatosensory cortex, based on which we identified 11 morpho-electric types (M/E-types), 9 neurochemical types (NC-types) and 20 transcriptomic types (T-types). We found that most NC-types comprised multiple M/E-types. While some M/E-types preferentially expressed specific transcriptomic signatures, a clear one-to-one correspondence between M/E-types and T-types was rarely observed. Moreover, we found that most M/E-types were distributed in a specific sub-lamina of cortical layer 5 and derived from dedicated embryonic origins. These results provide a systematic framework linking distinctive gene expression and phenotypic characterization of interneurons in a specific cortical layer.

Visual question answering with attention transfer and a cross-modal gating mechanism

Visual question answering (VQA) is challenging since it requires to understand both language information and corresponding visual contents. A lot of efforts have been made to capture single-step language and visual interactions. However, answering complex questions requires multiple steps of reasoning which gradually adjusts the region of interest to the most relevant part of the given image, which has not been well investigated. To integrate question related object relations into attention mechanism, we propose a multi-step attention architecture to facilitate the modeling of multi-modal correlations. Firstly, an attention transfer mechanism is integrated to gradually adjust the region of interest considering reasoning representation of questions. Secondly, we propose a cross-modal gating strategy to filter out irrelevant information based on multi-modal correlations. Finally, we achieve the state-of-the-art performance on the VQA 1.0 dataset and favorable results on the VQA 2.0 dataset, which verifies the effectiveness of our proposed method.

Correlation Net: Spatiotemporal multimodal deep learning for action recognition

This paper describes a network that captures multimodal correlations over arbitrary timestamps. The proposed scheme operates as a complementary, extended network over a multimodal convolutional neural network (CNN). Spatial and temporal streams are required for action recognition by a deep CNN, but overfitting reduction and fusing these two streams remain open problems. The existing fusion approach averages the two streams. Here we propose a correlation network with a Shannon fusion for learning a pre-trained CNN. A Long-range video may consist of spatiotemporal correlations over arbitrary times, which can be captured by forming the correlation network from simple fully connected layers. This approach was found to complement the existing network fusion methods. The importance of multimodal correlation is validated in comparison experiments on the UCF-101 and HMDB-51 datasets. The multimodal correlation enhanced the accuracy of the video recognition results.

A Study of the Dynamic Relation between Physiological Changes and Spontaneous Expressions

Recent progress in Affective Computing (AC) has enabled integration of physiological cues and spontaneous expressions to reveal a subject’s emotional state. Due to the lack of an effective technique for evaluating multimodal correlations, experience and intuition play a main role in present AC studies when fusing affective cues or modalities, resulting in unexpected outcomes. This study seeks to demonstrate a dynamic correlation between two such affective cues, physiological changes and spontaneous expressions, which were obtained by a combination of stereo vision based tracking and imaging photoplethysmography (iPPG), with a designed protocol involving 20 healthy subjects. The two cues obtained were sampled into a Statistical Association Space (SAS) to evaluate their dynamic correlation. It is found that the probability densities in the SAS increase as the peaks in two cues are approached. Also the complex form of the high probability density region in the SAS suggests a nonlinear correlation between two cues. Finally the cumulative distribution on the zero time-difference surface is found to be small (<0.047) demonstrating a lack of simultaneity. These results show that the two cues have a close interrelation, that is both asynchronous and nonlinear, in which a peak of one cue heralds a peak in the other.

Personalized Subject Learning Based on Topic Detection and Canonical Correlation Analysis

To keep pace with the time, learning from printed medium alone is no longer a comprehensive approach. Fresh digital contents can definitely be the complement of printed education medium. Although timely access to fresh contents is becoming increasingly important for education and gaining such access is no longer a problem, the capacity for human teachers to assimilate such huge amounts of contents is limited. Topic Detection (TD) is then a promising research area that addresses speedy access of desired contents based on topic or subject. On the other hand, personalized education is getting more attention because it facilitates the improvement of creativity and subject learning of the students. This paper reveals a patented Personalized Subject Learning (PSL) system that caters for the need of personalized education and efficiently provides subject based contents. An efficient topic detection algorithm for providing subject content is presented. Moreover, since education contents are multimedia based ones with multimodal, PSL introduces Canonical Correlation Analysis (CCA) method to detect multimodal correlations across different types of media. Due to its novelty, PSL has been used as the key engine in a real world application of personalized education system as the smart education module sponsored by a Smart City project.

Deep unsupervised network for multimodal perception, representation and classification

In this paper, we tackle the problem of multimodal learning for autonomous robots. Autonomous robots interacting with humans in an evolving environment need the ability to acquire knowledge from their multiple perceptual channels in an unsupervised way. Most of the approaches in the literature exploit engineered methods to process each perceptual modality. In contrast, robots should be able to acquire their own features from the raw sensors, leveraging the information elicited by interaction with their environment: learning from their sensorimotor experience would result in a more efficient strategy in a life-long perspective. To this end, we propose an architecture based on deep networks, which is used by the humanoid robot iCub to learn a task from multiple perceptual modalities (proprioception, vision, audition). By structuring high-dimensional, multimodal information into a set of distinct sub-manifolds in a fully unsupervised way, it performs a substantial dimensionality reduction by providing both a symbolic representation of data and a fine discrimination between two similar stimuli. Moreover, the proposed network is able to exploit multimodal correlations to improve the representation of each modality alone.

Sliding perspectives: dissociating ownership from self-location during full body illusions in virtual reality.

Bodily illusions have been used to study bodily self-consciousness and disentangle its various components, among other the sense of ownership and self-location. Congruent multimodal correlations between the real body and a fake humanoid body can in fact trigger the illusion that the fake body is one's own and/or disrupt the unity between the perceived self-location and the position of the physical body. However, the extent to which changes in self-location entail changes in ownership is still matter of debate. Here we address this problem with the support of immersive virtual reality. Congruent visuotactile stimulation was delivered on healthy participants to trigger full body illusions from different visual perspectives, each resulting in a different degree of overlap between real and virtual body. Changes in ownership and self-location were measured with novel self-posture assessment tasks and with an adapted version of the cross-modal congruency task. We found that, despite their strong coupling, self-location and ownership can be selectively altered: self-location was affected when having a third person perspective over the virtual body, while ownership toward the virtual body was experienced only in the conditions with total or partial overlap. Thus, when the virtual body is seen in the far extra-personal space, changes in self-location were not coupled with changes in ownership. If a partial spatial overlap is present, ownership was instead typically experienced with a boosted change in the perceived self-location. We discussed results in the context of the current knowledge of the multisensory integration mechanisms contributing to self-body perception. We argue that changes in the perceived self-location are associated to the dynamical representation of peripersonal space encoded by visuotactile neurons. On the other hand, our results speak in favor of visuo-proprioceptive neuronal populations being a driving trigger in full body ownership illusions.

Content-oriented multimedia document understanding through cross-media correlation

This paper presents a novel method for multimedia document content analysis through modeling multimodal data correlations. We hypothesize that the correlation of different modalities from the same data source can help achieve better multimedia content understanding results than one which explores a single modality. We turn this task into two parts: multimedia data fusion and multimodal correlation propagation. During the first stage, we re-organize the training multimedia data into Modality semAntic Documents (MADs) after extracting quantized multimodal features, and then use multivariate Gaussian distributions to characterize the continuous quantity by latent topic modeling. Model parameters are asymmetrically learned to initialize multimodal correlations in the latent topic space. Accordingly, during the second stage, we construct a Multimodal Correlation Network (MCN) based on the initialized multimodal correlations, and a new mechanism of propagating inter-modality correlations and intra-modality similarities in MCN is further proposed to take the complementary from cross-modalities to facilitate multimedia content analysis. The experimental results of image-audio data retrieval on a 10-categories dataset and content-oriented web page recommendation on the USTODAY dataset show the effectiveness of our method.

Sonographic diagnosis of abscess following breast-conserving surgery with insertion of nonabsorbable mesh

Immediate mesh insertion has been recently used for breast reconstruction after breast-conserving surgery. We report a case of abscess formation following immediate nonabsorbable mesh insertion with breast-conserving surgery. In this article, we demonstrate multimodal breast imaging features and pathologic correlations of the case. In addition, we illustrate characteristic sonographic findings of nonabsorbable mesh fibers to differentiate them from a gossypiboma caused by a retained surgical sponge or tumor recurrence.

Variability of (functional) hemodynamics as measured with simultaneous fNIRS and fMRI during intertemporal choice

Neural processing inferred from hemodynamic responses measured with functional near infrared spectroscopy (fNIRS) may be confounded with individual anatomical or systemic physiological sources of variance. This may hamper the validity of fNIRS signal interpretations and associations between individual traits and brain activation, such as the link between impulsivity-related personality traits and decreased prefrontal cognitive control during reward-based decision making. Hemodynamic responses elicited by an intertemporal choice reward task in 20 healthy subjects were investigated for multimodal correlations of simultaneous fNIRS-fMRI and for an impact of anatomy and scalp fMRI signal fluctuations on fNIRS signals. Moreover, correlations of prefrontal activation with trait "sensitivity to reward" (SR) were investigated for differences between methods. While showing substantial individual variability, temporal fNIRS-fMRI correlations increased with the activation, which both methods consistently detected within right inferior/middle frontal gyrus. Here, up to 41% of fNIRS channel activation variance was explained by individual gray matter volume simulated to be reached by near-infrared light, and up to 20% by scalp-cortex distance. Extracranial fMRI and fNIRS time series showed significant temporal correlations in the temple region. SR was negatively correlated with fMRI but not fNIRS activation elicited by immediate rewards of choice within right inferior/middle frontal gyrus. Higher SR increased the correlation between extracranial fMRI and fNIRS signals and decreased fNIRS-fMRI correlations. Task-related fNIRS signals might be impacted by regionally and individually weighted sources of anatomical and systemic physiological error variance. Trait-activation correlations might be affected or biased by systemic physiological responses, which should be accounted for in future fNIRS studies of interindividual differences.

Self-organizing neural maps for multi-modal associations

To move in a complex and dynamic environment, according to the active perception theory, an agent needs to learn the multi modal correlations between its actions and the changes they induce in the environment [1]. Human beings interact with their environment through several distant organs, whose sensory flow processing are influencing each others as, for example, in the Mc Gurk effect [2]. In the functional view of the cortex, each sensory area processes a specific sensory flow and associative areas integrate these flows in a consistent representation of the world that influences in return uni-modal perceptions as for the ventriloquist effect [3]. At a mesoscopic level, the cortex shows a generic structure composed of multi-layer cortical columns. We propose a bio inspired model of perceptive map which, using a continuous and unsupervised learning, self-organizes to map a sensory data flow. Thus, using a spatial competition mechanism, a perceptive map provides an activity bump representing the current perception. In a multi modal architecture, by connecting multiple perceptive maps to an associative map with reciprocal spatially constrained connectivity, the spatial organization of all perceptions have to relax the multi modal constraints (see figure 1A). Thus, this multimodal architecture provides a way to learn multi modal correlations with generalization. Figure 1 A Example of use of our perceptive map model in a multi modal architecture with two sensors and one actuator. B Generic structure of one cortical column in a perceptive map. More precisely, each perceptive map consists of generic cortical columns with computing and learning which are local and decentralized (see figure1 B). Sensitive layer receives the feed-forward flow coming from a sensor and uses the BCM learning rule [4]. This synaptic learning rule is based on the hebbian one and is able to autonomously raise a selectivity to one stimulus of the upcoming flow. From a computational point of view, the BCM rule uses a sliding threshold between long term potentiation and long term depression, which has been confirmed by biological evidences [5]. The cortical layer receives feedback influence coming from the associative map, whose activity represents the multi modal perception. The perceptive layer is based on the neural field theory [6] and it receives the sensitive activity modulated by the cortical one. Thanks to the competitive mechanism introduced by the lateral connectivity with a difference of Gaussian shape, an activity bump emerges at the map level, where the activity is the most spatially consistent, representing a consensus between the local sensation and the multi modal constraints. To obtain a self-organization of the sensitive layer at the map level, the perceptive layer modulates the BCM activity, so that the spatial consistency of the activity bump is propagated to the selectivity organization. We have also introduced an unlearning term in the BCM learning rule in order to forget the current selectivity if it is not consistent with the received modulation. This unlearning mechanism provides plasticity to the self-organization in order to adapt to the multi modal constraints.

Imaging Characteristics of Dry Age-Related Macular Degeneration

Purpose: To review the current literature regarding the imaging characteristics of dry age-related macular degeneration (AMD) lesions, with a special focus on drusen and geographic atrophy imaging. We also explore the role of novel approach of hyperspectral imaging in AMD.Methods: Review of current literature as well as findings in a small group of patients imaged with hyperspectral imaging.Results: The use of optical coherence tomography, and especially fourier-domain devices, has enhanced our ability to classify various lesions of dry AMD. The increasing role of autofluorescence in characterization and prognostication in geographic atrophy is reviewed. The advances made in automated detection and multimodal imaging are highlighted, with their potential to revolutionize this area of research.Conclusions: Recent advances in retinal imaging have improved our understanding of the characteristics and prognostication of dry AMD, with an increasing role for multimodal imaging and image correlations. The potential future role of hyperspectral imaging in dry AMD is also presented herein.

Multiscale Hybrid Linear Models for Lossy Image Representation

In this paper, we introduce a simple and efficient representation for natural images. We view an image (in either the spatial domain or the wavelet domain) as a collection of vectors in a high-dimensional space. We then fit a piece-wise linear model (i.e., a union of affine subspaces) to the vectors at each downsampling scale. We call this a multiscale hybrid linear model for the image. The model can be effectively estimated via a new algebraic method known as generalized principal component analysis (GPCA). The hybrid and hierarchical structure of this model allows us to effectively extract and exploit multimodal correlations among the imagery data at different scales. It conceptually and computationally remedies limitations of many existing image representation methods that are based on either a fixed linear transformation (e.g., DCT, wavelets), or an adaptive uni-modal linear transformation (e.g., PCA), or a multimodal model that uses only cluster means (e.g., VQ). We will justify both quantitatively and experimentally why and how such a simple multiscale hybrid model is able to reduce simultaneously the model complexity and computational cost. Despite a small overhead of the model, our careful and extensive experimental results show that this new model gives more compact representations for a wide variety of natural images under a wide range of signal-to-noise ratios than many existing methods, including wavelets. We also briefly address how the same (hybrid linear) modeling paradigm can be extended to be potentially useful for other applications, such as image segmentation.