Multimodal Representation Research Articles

EMPDTA: An End-to-End Multimodal Representation Learning Framework with Pocket Online Detection for Drug-Target Affinity Prediction.

Accurately predicting drug-target interactions is a critical yet challenging task in drug discovery. Traditionally, pocket detection and drug-target affinity prediction have been treated as separate aspects of drug-target interaction, with few methods combining these tasks within a unified deep learning system to accelerate drug development. In this study, we propose EMPDTA, an end-to-end framework that integrates protein pocket prediction and drug-target affinity prediction to provide a comprehensive understanding of drug-target interactions. The EMPDTA framework consists of three main modules: pocket online detection, multimodal representation learning for affinity prediction, and multi-task joint training. The performance and potential of the proposed framework have been validated across diverse benchmark datasets, achieving robust results in both tasks. Furthermore, the visualization results of the predicted pockets demonstrate accurate pocket detection, confirming the effectiveness of our framework.

Towards zero-shot object counting via deep spatial prior cross-modality fusion

Existing counting models predominantly operate on a specific category of objects, such as crowds and vehicles. The recent emergence of multi-modal foundational models, e.g., Contrastive Language-Image Pre-training (CLIP), has facilitated class-agnostic counting. This involves counting objects of any given class from a single image based on textual instructions. However, CLIP-based class-agnostic counting models face two primary challenges. Firstly, the CLIP model lacks sensitivity to location information. It generally considers global content rather than the fine-grain location of objects. Therefore, adapting the CLIP model directly is suboptimal. Secondly, these models generally freeze pre-trained vision and language encoders, while neglecting the potential misalignment in the constructed hypothesis space. In this paper, we address these two issues in a unified framework termed Deep Spatial Prior Interaction (DSPI) network. The DSPI leverages the spatial-awareness ability of large-scale pre-trained object grounding models, i.e., Grounding DINO, to incorporate spatial location as an additional prior for a specific query class. This enables the network to be more specifically focused on the precise location of the objects. Additionally, to align the feature space across different modalities, we tailor a meta adapter that extracts textual information into an object query. This serves as an instruction for cross-modality matching. These two modules collaboratively ensure the alignment of multi-modal representations while preserving their discriminative nature. Comprehensive experiments conducted on a diverse set of benchmarks verify the superiority of the proposed model. The code is available at https://github.com/jinyongch/DSPI.

From Delays to Densities: Exploring Data Uncertainty through Speech, Text, and Visualization

AbstractUnderstanding and communicating data uncertainty is crucial for making informed decisions in sectors like finance and healthcare. Previous work has explored how to express uncertainty in various modes. For example, uncertainty can be expressed visually with quantile dot plots or linguistically with hedge words and prosody. Our research aims to systematically explore how variations within each mode contribute to communicating uncertainty to the user; this allows us to better understand each mode's affordances and limitations. We completed an exploration of the uncertainty design space based on pilot studies and ran two crowdsourced experiments examining how speech, text, and visualization modes and variants within them impact decision‐making with uncertain data. Visualization and text were most effective for rational decision‐making, though text resulted in lower confidence. Speech garnered the highest trust despite sometimes leading to risky decisions. Results from these studies indicate meaningful trade‐offs among modes of information and encourage exploration of multimodal data representations.

Unified multimodal fusion transformer for few shot object detection for remote sensing images

Object detection is a fundamental computer vision task with wide applications in remote sensing, but traditional methods strongly rely on large annotated datasets which are difficult to obtain, especially for novel object classes. Few-shot object detection (FSOD) aims to address this by using detectors to learn from very limited labeled data. Recent work fuse multi-modalities like image–text pairs to tackle data scarcity but require external region proposal network (RPN) to align cross-modal pairs which leads to a bias towards base classes and insufficient cross-modal contextual learning. To address these problems, we propose a unified multi-modal fusion transformer (UMFT), which extracts visual features from ViT and textual encodings from BERT to align multi-modal representations in an end-to-end manner. Specifically, affinity-guided fusion (AFM) captures semantically related image–text pairs by modeling their affinity relationships to selectively combine most informative pairs. In addition, cross-modal correlation module (CCM) captures discriminative inter-modal patterns between image and text representations and filters out unrelated features to enhance cross-modal alignment. By leveraging AFM to focus on semantic relationships and CCM to refine inter-modal features, the model better aligns multimodal data without RPN. These representations are passed to detection decoder that predicts bounding boxes, probabilities of class and cross-modal attributes. Evaluation of UMFT on benchmark datasets NWPU VHR-10 and DIOR demonstrated its ability to leverage limited image–text training data via dynamic fusion, achieving new state-of-the-art mean average precision (mAP) for few-shot object detection. Our code will be publicly available at https://github.com/abdullah-azeem/umft.

Call for contributions to a special issue on multimodal representations of authority in discourse

Call for contributions to a special issue on multimodal representations of authority in discourse

Language experts on a sinking island: critiquing language endangerment rhetorics through participation in and around a tabletop game

ABSTRACT The participatory study of language addresses the unequal relationships between the ‘researcher’ and the ‘researched’. As part of this undertaking, university-based researchers seek to develop more accessible ways of knowledge production to involve other participants in the creation of novel forms of representation at the levels relevant to them. The university-based participants of our research addressed this challenge by co-designing a language-centred tabletop game together with participants from a Romanian multilingual village. Bringing their language expertise into action, all participants collaboratively represented, analysed, and reflected on marginalised local language practices. The game was the culmination of several years of participatory research and aimed to explore interpretations of practices that are widely discussed as threatened, allowing for a critical approach to language endangerment rhetorics. Drawing on the ethnographic analysis of eight sessions in which players engaged with each other and the tabletop game’s creators, this article argues that developing innovative multimodal representations of language on the margins can make new knowledge more accessible, but in itself does not guarantee the subversion of language endangerment rhetorics. To have an impact on well-established discourses, the language experts’ participation throughout the research process is needed.

Vision-language constraint graph representation learning for unsupervised vehicle re-identification

Existing vehicle re-identification methods rely on visual features to extract vehicle identity information. However, while individual visual features enable the model to learn limited semantic information, multimodal representations are difficult to extract. A vision-language constraint graph representation learning method guided by textual descriptions is proposed to exploit the cross-modal robustness capacity. Initially, the training set creates the unique conditional prompts for textual feature extraction. These prompts are employed to improve the understanding of visual modalities. We subsequently designed a vision-language constraint graph topology, where each training sample is considered a node in the graph. Under the dual constraints of visual and textual features, the relationship between graph nodes is further explored to construct more reliable positive and negative sample pairs for graph representation learning. Then, neighboring node label smoothing is introduced to mitigate label noise generated by visual feature clustering and achieved by combining pseudo-label assignment results from neighboring node pairs in the graph topology. Extensive experiments have confirmed that the proposed method achieves state-of-the-art performance by combining salient information from visual and textual modalities.

Attention-Like Multimodality Fusion With Data Augmentation for Diagnosis of Mental Disorders Using MRI.

The globally rising prevalence of mental disorders leads to shortfalls in timely diagnosis and therapy to reduce patients' suffering. Facing such an urgent public health problem, professional efforts based on symptom criteria are seriously overstretched. Recently, the successful applications of computer-aided diagnosis approaches have provided timely opportunities to relieve the tension in healthcare services. Particularly, multimodal representation learning gains increasing attention thanks to the high temporal and spatial resolution information extracted from neuroimaging fusion. In this work, we propose an efficient multimodality fusion framework to identify multiple mental disorders based on the combination of functional and structural magnetic resonance imaging. A multioutput conditional generative adversarial network (GAN) is developed to address the scarcity of multimodal data for augmentation. Based on the augmented training data, the multiheaded gating fusion model is proposed for classification by extracting the complementary features across different modalities. The experiments demonstrate that the proposed model can achieve robust accuracies of 75.1 ± 1.5 %, 72.9 ± 1.1 %, and 87.2 ± 1.5 % for autism spectrum disorder (ASD), attention deficit/hyperactivity disorder, and schizophrenia, respectively. In addition, the interpretability of our model is expected to enable the identification of remarkable neuropathology diagnostic biomarkers, leading to well-informed therapeutic decisions.

Prior-Guided Adversarial Learning With Hypergraph for Predicting Abnormal Connections in Alzheimer's Disease.

Alzheimer's disease (AD) is characterized by alterations of the brain's structural and functional connectivity during its progressive degenerative processes. Existing auxiliary diagnostic methods have accomplished the classification task, but few of them can accurately evaluate the changing characteristics of brain connectivity. In this work, a prior-guided adversarial learning with hypergraph (PALH) model is proposed to predict abnormal brain connections using triple-modality medical images. Concretely, a prior distribution from anatomical knowledge is estimated to guide multimodal representation learning using an adversarial strategy. Also, the pairwise collaborative discriminator structure is further utilized to narrow the difference in representation distribution. Moreover, the hypergraph perceptual network is developed to effectively fuse the learned representations while establishing high-order relations within and between multimodal images. Experimental results demonstrate that the proposed model outperforms other related methods in analyzing and predicting AD progression. More importantly, the identified abnormal connections are partly consistent with previous neuroscience discoveries. The proposed model can evaluate the characteristics of abnormal brain connections at different stages of AD, which is helpful for cognitive disease study and early treatment.

Preoperative Grading of Brain Gliomas Using 3D-ResNet18 Based on Multimodal MRI and Attention Mechanism

In response to the prevalent challenge of imprecise preoperative glioma grading prediction, a multimodal Magnetic Resonance Imaging (MRI) image with attention mechanism prediction model based on Residual Network (ResNet) is proposed for accurate prediction. To achieve this, publicly accessible datasets from BraTS 2019 and BraTS 2020 were employed. MRI images across three modalities underwent preprocessing, cropping, and subsequent stacking to generate comprehensive multimodal representations. Meanwhile, An Efficient Channel Attention (ECA) mechanism based on 3D data (3D-ECA) is proposed and introduced to reduce the problem of slow network convergence and overfitting. Notably, experiments demonstrate that the 3D-ECA attention mechanism added to the model improves the network training speed and the classification accuracy of the model, and the classification accuracy reached 88.1%, with an impressive area under the subject working characteristic curve of 0.918. Therefore, the 3D-ResNet18 deep learning model, incorporating multimodal MRI and 3D-ECA attention mechanisms, demonstrates high accuracy and robustness in distinguishing between high-grade and low-grade gliomas. Thus, showcasing significant clinical potential for glioma classification and diagnosis.

MRFTrans: Multimodal Representation Fusion Transformer for monocular 3D semantic scene completion

The complete understanding of 3D scenes is crucial in robotic visual perception, impacting tasks such as motion planning and map localization. However, due to the limited field of view and scene occlusion constraints of sensors, inferring complete scene geometry and semantic information from restricted observations is challenging. In this work, we propose a novel Multimodal Representation Fusion Transformer framework (MRFTrans) that robustly fuses semantic, geometric occupancy, and depth representations for monocular-image-based scene completion. MRFTrans centers on an affinity representation fusion transformer, integrating geometric occupancy and semantic relationships within a transformer architecture. This integration enables the modeling of long-range dependencies within scenes for inferring missing information. Additionally, we present a depth representation fusion method, efficiently extracting reliable depth knowledge from biased monocular estimates. Extensive experiments demonstrate MRFTrans’s superiority, setting a new benchmark on SemanticKITTI and NYUv2 datasets. It significantly enhances completeness and accuracy, particularly in large structures, movable objects, and scene components with major occlusions. The results underscore the benefits of the affinity-aware transformer and robust depth fusion in monocular-image-based completion.

Multi-modal domain adaptation for revealing spatial functional landscape from spatially resolved transcriptomics.

Spatially resolved transcriptomics (SRT) has emerged as a powerful tool for investigating gene expression in spatial contexts, providing insights into the molecular mechanisms underlying organ development and disease pathology. However, the expression sparsity poses a computational challenge to integrate other modalities (e.g. histological images and spatial locations) that are simultaneously captured in SRT datasets for spatial clustering and variation analyses. In this study, to meet such a challenge, we propose multi-modal domain adaption for spatial transcriptomics (stMDA), a novel multi-modal unsupervised domain adaptation method, which integrates gene expression and other modalities to reveal the spatial functional landscape. Specifically, stMDA first learns the modality-specific representations from spatial multi-modal data using multiple neural network architectures and then aligns the spatial distributions across modal representations to integrate these multi-modal representations, thus facilitating the integration of global and spatially local information and improving the consistency of clustering assignments. Our results demonstrate that stMDA outperforms existing methods in identifying spatial domains across diverse platforms and species. Furthermore, stMDA excels in identifying spatially variable genes with high prognostic potential in cancer tissues. In conclusion, stMDA as a new tool of multi-modal data integration provides a powerful and flexible framework for analyzing SRT datasets, thereby advancing our understanding of intricate biological systems.

An autoencoder-based self-supervised learning for multimodal sentiment analysis

Representation learning is a crucial and challenging task within multimodal sentiment analysis. Effective multimodal sentiment representations contain two key aspects: consistency and difference. However, the state-of-the-art multimodal sentiment analysis approaches failed to capture the difference and consistency of sentiment information across diverse modalities. To address the multimodal sentiment representation problem, we propose an autoencoder-based self-supervised learning framework. In the pre-training stage, an autoencoder is designed for each modality, leveraging unlabeled data to learn richer sentiment representations for each modality through sample reconstruction and modality consistency detection tasks. In the fine-tuning stage, the pre-trained autoencoder is injected into MulT (AE-MT) and enhance the model's ability to extract deep sentiment information by incorporating a contrastive learning auxiliary task. Our experiments on the popular Chinese sentiment analysis benchmark (CH-SIMS v2.0) and English sentiment analysis benchmark (MOSEI) demonstrate significant gains over baseline models.

Logit prototype learning with active multimodal representation for robust open-set recognition

Logit prototype learning with active multimodal representation for robust open-set recognition

Fast retrieval of multi-modal embeddings for e-commerce applications

In this paper, we introduce a retrieval framework designed for e-commerce applications, which employs a multi-modal approach to represent items of interest. This approach incorporates both textual descriptions and images of products, alongside a locality-sensitive hashing (LSH) indexing scheme for rapid retrieval of potentially relevant products. Our focus is on a data-independent methodology, where the indexing mechanism remains unaffected by the specific dataset, while the multi-modal representation is learned beforehand. Specifically, we utilize a multi-modal architecture, CLIP, to learn a latent representation of items by combining text and images in a contrastive manner. The resulting item embeddings encapsulate both the visual and textual information of the products, which are then subjected to various types of LSH for balancing between result quality and retrieval speed. We present the findings of our experiments conducted on two real-world datasets sourced from e-commerce platforms, comprising both product images and textual descriptions. Promising results have been achieved, demonstrating favorable retrieval time and average precision. These results were obtained through testing the approach with a specifically selected set of queries and with synthetic queries generated using a Large Language Model.

High frequency oscillations in human memory and cognition: a neurophysiological substrate of engrams?

Despite advances in understanding the cellular and molecular processes underlying memory and cognition, and recent successful modulation of cognitive performance in brain disorders, the neurophysiological mechanisms remain underexplored. High frequency oscillations beyond the classic electroencephalogram spectrum have emerged as a potential neural correlate of fundamental cognitive processes. High frequency oscillations are detected in the human mesial temporal lobe and neocortical intracranial recordings spanning gamma/epsilon (60-150 Hz), ripple (80-250 Hz) and higher frequency ranges. Separate from other non-oscillatory activities, these brief electrophysiological oscillations of distinct duration, frequency and amplitude are thought to be generated by coordinated spiking of neuronal ensembles within volumes as small as a single cortical column. Although the exact origins, mechanisms and physiological roles in health and disease remain elusive, they have been associated with human memory consolidation and cognitive processing. Recent studies suggest their involvement in encoding and recall of episodic memory with a possible role in the formation and reactivation of memory traces. High frequency oscillations are detected during encoding, throughout maintenance, and right before recall of remembered items, meeting a basic definition for an engram activity. The temporal coordination of high frequency oscillations reactivated across cortical and subcortical neural networks is ideally suited for integrating multimodal memory representations, which can be replayed and consolidated during states of wakefulness and sleep. High frequency oscillations have been shown to reflect coordinated bursts of neuronal assembly firing and offer a promising substrate for tracking and modulation of the hypothetical electrophysiological engram.

Shifted window-based Transformer with multimodal representation for the systematic staging of rectal cancer

Shifted window-based Transformer with multimodal representation for the systematic staging of rectal cancer

SCA-PVNet: Self-and-cross attention based aggregation of point cloud and multi-view for 3D object retrieval

To address 3D object retrieval, substantial efforts have been made to generate highly discriminative descriptors for 3D objects represented by a single modality, such as voxels, point clouds, or multiview images. It is promising to leverage complementary information from multimodal representations of 3D objects to further improve retrieval performance. However, multimodal 3D object retrieval has rarely been developed or analyzed for large-scale datasets. In this paper, we propose a self-and-cross-attention-based aggregation of point clouds and multiview images (SCA-PVNet) for 3D object retrieval. With deep features extracted from point clouds and multi-view images, we design two types of feature aggregation modules, namely the in-modality aggregation module (IMAM) and the cross-modality aggregation module (CMAM ), for effective feature fusion. IMAM leverages a self-attention mechanism to aggregate multiview features, whereas CMAM exploits a cross-attention mechanism to interact with point-cloud and multiview features. The final descriptor of a 3D object for object retrieval can be obtained by concatenating the aggregated feature outputs of both modules. Extensive experiments and analyses were conducted on four datasets ranging from small to large scales to demonstrate the superiority of the proposed SCA-PVNet over state-of-the-art methods. In addition to achieving state-of-the-art retrieval performance, our method is more robust in challenging scenarios when views or points are missing during inference.

Fuser: An enhanced multimodal fusion framework with congruent reinforced perceptron for hateful memes detection

As a multimodal form of hate speech on social media, hateful memes are more aggressive and cryptic threats to the real life of humans. Automatic detection of hateful memes is crucial, but the images and texts in most memes are only weakly consistent or even irrelevant. Although existing works have achieved the initial goal of detecting hateful memes with pre-trained models, they are limited to monolithic inference methods while ignoring the semantic differences between multimodal representations. To strengthen the comprehension and reasoning of the hidden meaning behind the memes by combining real-world knowledge, we propose an enhanced multimodal fusion framework with congruent reinforced perceptron for hateful memes detection. Inspired by the human cognitive mechanism, we first divide the extracted multisource representations into main semantics and auxiliary contexts based on their strength and relevance, and then precode them into lightly correlated embeddings with unified spatial dimensions via a novel prefix uniform layer, respectively. To jointly learn the intrinsic correlation between primary and secondary semantics, a congruent reinforced perceptron with brain-like perceptual integration is designed to seamlessly fuse multimodal representations in a shared latent space while maintaining the feature integrity in the sub-fusion space, thereby implicitly reasoning about the subtle metaphors behind the memes. Extensive experiments on four benchmark datasets fully demonstrate the effectiveness and superiority of our architecture compared with previous state-of-the-art methods.

MVACLNet: A Multimodal Virtual Augmentation Contrastive Learning Network for Rumor Detection

In today’s digital era, rumors spreading on social media threaten societal stability and individuals’ daily lives, especially multimodal rumors. Hence, there is an urgent need for effective multimodal rumor detection methods. However, existing approaches often overlook the insufficient diversity of multimodal samples in feature space and hidden similarities and differences among multimodal samples. To address such challenges, we propose MVACLNet, a Multimodal Virtual Augmentation Contrastive Learning Network. In MVACLNet, we first design a Hierarchical Textual Feature Extraction (HTFE) module to extract comprehensive textual features from multiple perspectives. Then, we fuse the textual and visual features using a modified cross-attention mechanism, which operates from different perspectives at the feature value level, to obtain authentic multimodal feature representations. Following this, we devise a Virtual Augmentation Contrastive Learning (VACL) module as an auxiliary training module. It leverages ground-truth labels and extra-generated virtual multimodal feature representations to enhance contrastive learning, thus helping capture more crucial similarities and differences among multimodal samples. Meanwhile, it performs a Kullback–Leibler (KL) divergence constraint between predicted probability distributions of the virtual multimodal feature representations and their corresponding virtual labels to help extract more content-invariant multimodal features. Finally, the authentic multimodal feature representations are input into a rumor classifier for detection. Experiments on two real-world datasets demonstrate the effectiveness and superiority of MVACLNet on multimodal rumor detection.