Multimodal Dataset Research Articles

A global multimodal flood event dataset with heterogeneous text and multi-source remote sensing images

ABSTRACT With the increasing frequency of floods, in-depth flood event analyses are essential for effective disaster relief and prevention. Satellite-based flood event datasets have become the primary data source for flood event analyses instead of limited disaster maps due to their enhanced availability. Nevertheless, despite the vast amount of available remote sensing images, existing flood event datasets continue to pose significant challenges in flood event analyses due to the uneven geographical distribution of data, the scarcity of time series data, and the limited availability of flood-related semantic information. There has been a surge in acceptance of deep learning models for flood event analyses, but some existing flood datasets do not align well with model training, and distinguishing flooded areas has proven difficult with limited data modalities and semantic information. Moreover, efficient retrieval and pre-screening of flood-related imagery from vast satellite data impose notable obstacles, particularly within large-scale analyses. To address these issues, we propose a Multimodal Flood Event Dataset (MFED) for deep-learning-based flood event analyses and data retrieval. It consists of 18 years of multi-source remote sensing imagery and heterogeneous textual information covering flood-prone areas worldwide. Incorporating optical and radar imagery can exploit the correlation and complementarity between distinct image modalities to capture the pixel features in flood imagery. It is worth noting that text modality data, including auxiliary hydrological information extracted from the Global Flood Database and text information refined from online news records, can also offer a semantic supplement to the images for flood event retrieval and analysis. To verify the applicability of the MFED in deep learning models, we carried out experiments with different models using a single modality and different combinations of modalities, which fully verified the effectiveness of the dataset. Furthermore, we also verify the efficiency of the MFED in comparative experiments with existing multimodal datasets and diverse neural network structures.

Cross-modal interaction and multi-source visual fusion for video generation in fetal cardiac screening

To address the limitation of preserving data for dynamic visualization in fetal ultrasound screening, a novel framework is proposed to facilitate the generation of fetal four-chamber echocardiogram videos, incorporating multi-source visual fusion and understanding. The framework utilizes an effective spectrogram-ultrasound synchronizer to align the ultrasound images with time, ensuring the generated video matches the actual heartbeat rhythm. It further employs effective frame interpolation techniques to synthesize a video by incorporating a nonlinear bidirectional motion prediction. By integrating a Transformer model for the autoregressive generation of visual semantic sequence, the proposed framework demonstrates its capability to generate high-resolution frames. Experimental outcomes show the Clip-Similarity of 96.23% and DINOv2-Similarity of 99.77%. Furthermore, a multimodal dataset of fetal echocardiogram examinations has been constructed.

Integrated analysis of -omic landscapes in breast cancer subtypes

The subtypes of breast cancer exhibit diverse histology, molecular features, therapeutic response, aggressiveness, and patient outcomes. Multi-omics high-throughput technologies, which are widely used in cancer research, generated waste amounts of multimodal omic datasets calling for new approaches of integrated analyses to uncover patterns of transcriptomic, genomic, and epigenetic changes in breast cancer subtypes and connect them to disease clinical characteristics. Here, we applied multi-layer self-organizing map (ml-SOM) algorithms to PAM50-classified TCGA breast cancer samples to disentangle the diversity of the effects of gene expression, methylation, copy number, and somatic single nucleotide variation in the disease subtypes. Furthermore, we studied the association of perturbed gene modules with survival, prognosis, and other clinical characteristics. Our findings highlight the power of multi-omic analyses to offer a better understanding of the molecular diversity of breast cancer subtypes compared to single-omic analyses. Moreover, they highlight the complex subtype-characteristic associations between gene expression and epigenetic/genomic factors and their implications for survival and clinical outcomes.

A Survey on Cyberbullying Predictive Model using Deep Learning Techniques

Cyberbullying, a pervasive issue in the current digital age, has prompted the need for advanced predictive models to identify and mitigate online harassment. This survey paper explores the landscape of cyberbullying severity level prediction using an ensemble-based deep learning approach for multimodal datasets. Delving into the realm of deep learning techniques and investigating their efficacy in discerning subtle patterns indicative of cyberbullying behaviour, the survey encompasses a comprehensive review of existing ensemble methodologies, highlighting their strengths and weaknesses in cyberbullying prediction. Diverse datasets, model architectures, and evaluation metrics employed in relevant studies are analysed, aiming to provide a thorough understanding of the current technological status. Additionally, difficulties and potential avenues for upcoming studies are discussed, fostering advancements in the development of robust predictive models to combat cyberbullying effectively. Researchers, practitioners, and policymakers looking for insights on the changing field of cyberbullying prevention using ensemble-based deep-learning methodologies will find this survey to be a valuable resource.

MosaicMPI: a framework for modular data integration across cohorts and -omics modalities.

Advances in molecular profiling have facilitated generation of large multi-modal datasets that can potentially reveal critical axes of biological variation underlying complex diseases. Distilling biological meaning, however, requires computational strategies that can perform mosaic integration across diverse cohorts and datatypes. Here, we present mosaicMPI, a framework for discovery of low to high-resolution molecular programs representing both cell types and states, and integration within and across datasets into a network representing biological themes. Using existing datasets in glioblastoma, we demonstrate that this approach robustly integrates single cell and bulk programs across multiple platforms. Clinical and molecular annotations from cohorts are statistically propagated onto this network of programs, yielding a richly characterized landscape of biological themes. This enables deep understanding of individual tumor samples, systematic exploration of relationships between modalities, and generation of a reference map onto which new datasets can rapidly be mapped. mosaicMPI is available at https://github.com/MorrissyLab/mosaicMPI.

A robust and accurate feature matching method for multi-modal geographic images spatial registration

ABSTRACT While the current research has achieved satisfactory results for the registration of single mode data, there has always been a significant challenge in the registration of multi-modal images due to the obvious nonlinear radiation differences caused by different imaging mechanisms and imaging time. For example, multi-temporal visible, visible-synthetic aperture radar, visible-near infrared, near infrared-short wave infrared, visible-MAP, etc. To address this problem, we propose a Robust and Accurate Feature Matching Method for Multi-modal Geographic Images Spatial Registration (RAMMR) to fully extract common key points between images, weaken the radiation difference between data, and finally accurately match more inliers to realize multi-modal image registration. Considering the influence of noise and edge information on key point extraction, RAMMR first constructs a new scale space by introducing the Side Window Filter (SWF); Then, we improve Harris algorithm to extract key points based on the SWF scale space; After that, we propose an enhanced log-polar descriptor based on the gradient angles and gradient amplitudes of the scale space, which effectively improves the quality of the descriptor and avoids the mismatch of key points; Based on the standard Euclidean distance, we design a re-match strategy to obtain the initial matching results, and Random Sample Consensus (RANSAC) is used to eliminate outliers. Finally, the affine transformation parameters are calculated based on inliers, and multi-modal image registration is realized. RAMMR is evaluated on different multi-modal datasets and compared with some state-of-art methods. The experimental results show that RAMMR accurately registers multi-modal geographic images and obtains comparative results compared with benchmark methods. Our source datasets are publicly available at https://github.com/RSmfmr/multimodal-dataset.

ACORN: Performant and Predicate-Agnostic Search Over Vector Embeddings and Structured Data

Applications increasingly leverage mixed-modality data, and must jointly search over vector data, such as embedded images, text and video, as well as structured data, such as attributes and keywords. Proposed methods for this hybrid search setting either suffer from poor performance or support a severely restricted set of search predicates (e.g., only small sets of equality predicates), making them impractical for many applications. To address this, we present ACORN, an approach for performant and predicate-agnostic hybrid search. ACORN builds on Hierarchical Navigable Small Worlds (HNSW), a state-of-the-art graph-based approximate nearest neighbor index, and can be implemented efficiently by extending existing HNSW libraries. ACORN introduces the idea of predicate subgraph traversal to emulate a theoretically ideal, but impractical, hybrid search strategy. ACORN's predicate-agnostic construction algorithm is designed to enable this effective search strategy, while supporting a wide array of predicate sets and query semantics. We systematically evaluate ACORN on both prior benchmark datasets, with simple, low-cardinality predicate sets, and complex multi-modal datasets not supported by prior methods. We show that ACORN achieves state-of-the-art performance on all datasets, outperforming prior methods with 2--1,000× higher throughput at a fixed recall. Our code is available at: https://github.com/stanford-futuredata/ACORN.

AiGen-FoodReview: A Multimodal Dataset of Machine-Generated Restaurant Reviews and Images on Social Media

AiGen-FoodReview: A Multimodal Dataset of Machine-Generated Restaurant Reviews and Images on Social Media

LidPose: Real-Time 3D Human Pose Estimation in Sparse Lidar Point Clouds with Non-Repetitive Circular Scanning Pattern.

In this paper, we propose a novel, vision-transformer-based end-to-end pose estimation method, LidPose, for real-time human skeleton estimation in non-repetitive circular scanning (NRCS) lidar point clouds. Building on the ViTPose architecture, we introduce novel adaptations to address the unique properties of NRCS lidars, namely, the sparsity and unusual rosetta-like scanning pattern. The proposed method addresses a common issue of NRCS lidar-based perception, namely, the sparsity of the measurement, which needs balancing between the spatial and temporal resolution of the recorded data for efficient analysis of various phenomena. LidPose utilizes foreground and background segmentation techniques for the NRCS lidar sensor to select a region of interest (RoI), making LidPose a complete end-to-end approach to moving pedestrian detection and skeleton fitting from raw NRCS lidar measurement sequences captured by a static sensor for surveillance scenarios. To evaluate the method, we have created a novel, real-world, multi-modal dataset, containing camera images and lidar point clouds from a Livox Avia sensor, with annotated 2D and 3D human skeleton ground truth.

Video multimodal sentiment analysis using cross-modal feature translation and dynamical propagation

Multimodal sentiment analysis on social platforms is crucial for comprehending public opinions and attitudes, thus garnering substantial interest in knowledge engineering. Existing methods like implicit interaction, explicit interaction, and cross-modal translation can effectively integrate sentiment information, but they encounter challenges in establishing efficient emotional correlations across modalities due to data heterogeneity and concealed emotional relationships. To tackle this issue, we propose a video multimodal sentiment analysis model called PEST, which leverages cross-modal feature translation and a dynamic propagation model. Specifically, cross-modal feature translation translates textual, visual, and acoustic features into a common feature space, eliminating heterogeneity and enabling initial modal interaction. Additionally, the dynamic propagation model facilitates in-depth interaction and aids in establishing stable and reliable emotional correlations across modalities. Extensive experiments on the three multimodal sentiment datasets, CMU-MOSI, CMU-MOSEI, and CH-SIMS, demonstrate that PEST exhibits superior performance in both word-aligned and unaligned settings.

Stochastic reconstruction of heterogeneous microstructure combining sliced Wasserstein distance and gradient optimization

Computational reconstruction methods play an important role in integrated computational materials engineering, providing an efficient and inexpensive way for multi-modal and multi-length datasets construction, material design, and property prediction research. However, existing methods are either not computationally inefficient (traditional methods), or data-dependent and difficult to handle various reconstruction tasks (learning-based methods). Hybrid methods show advantages in addressing these problems, but they are still necessary to advance in low dataset dependence, generalization, and controllability. Herein, we propose a novel descriptor-based gradient optimization method, which utilizes multipoint statistical information and sliced Wasserstein metric for accurate microstructure characterization and reconstruction. Our method aims to respect and reproduce all the local patterns from training images, realizing similar higher-order statistics while better reproducing local features. To this end, sliced Wasserstein metric is introduced to measure the distance between higher-order distributions, and gradient optimization is adopted for fast and efficient reconstructions. Hierarchical and multitemplate strategies are developed to further strengthen characterization and reconstruction ability of heterogeneous structures. Additionally, our method can complete 2D-to-2D, 2D-to-3D, and 3D-to-3D reconstruction tasks even in the case of a single sample. Multiple sets of experiments are conducted under different reconstruction tasks for verification, and comparisons of visualization results, statistical parameters, and physical properties show the superiority of our method.

FedCMD: A Federated Cross-modal Knowledge Distillation for Drivers’ Emotion Recognition

Emotion recognition has attracted a lot of interest in recent years in various application areas such as healthcare and autonomous driving. Existing approaches to emotion recognition are based on visual, speech, or psychophysiological signals. However, recent studies are looking at multimodal techniques that combine different modalities for emotion recognition. In this work, we address the problem of recognizing the user’s emotion as a driver from unlabeled videos using multimodal techniques. We propose a collaborative training method based on cross-modal distillation, i.e., “FedCMD” (Federated Cross-Modal Distillation). Federated Learning (FL) is an emerging collaborative decentralized learning technique that allows each participant to train their model locally to build a better generalized global model without sharing their data. The main advantage of FL is that only local data is used for training, thus maintaining privacy and providing a secure and efficient emotion recognition system. The local model in FL is trained for each vehicle device with unlabeled video data by using sensor data as a proxy. Specifically, for each local model, we show how driver emotional annotations can be transferred from the sensor domain to the visual domain by using cross-modal distillation. The key idea is based on the observation that a driver’s emotional state indicated by a sensor correlates with facial expressions shown in videos. The proposed “FedCMD” approach is tested on the multimodal dataset “BioVid Emo DB” and achieves state-of-the-art performance. Experimental results show that our approach is robust to non-identically distributed data, achieving 96.67% and 90.83% accuracy in classifying five different emotions with IID (independently and identically distributed) and non-IID data, respectively. Moreover, our model is much more robust to overfitting, resulting in better generalization than the other existing methods.

End-to-end multi-perspective multimodal posts relevance score reasoning prediction

Driven by the surge in students sharing insights on online platforms, evaluating discussion forum posts has emerged as a significant research focus. This assessment is crucial for both students and educators, as it enables teachers to efficiently gauge student comprehension and helps students tailor their learning approaches to suit their individual needs, thereby fostering personalized growth. However, existing research on post-quality assessment faces challenges: 1) They overlook multimodal information interaction between post and topic, leading to inaccurate evaluations. 2) They are usually classification tasks that fail to provide feedback on the minor and relative differences between posts on the same topic. 3) They are usually evaluated using a single perspective, which is insufficient for capturing the complexity of the relationship between post and topic. Based on the above challenges, we propose a new assessment task, Multimodal Topic-Post Relevance Score Prediction (MTRSP), which analyzes whether a student's post comprehensively answers the question of the discussion topic by combining text and images to predict topic-post relevance scores, i.e., the degree of correctness of the answer. We develop an end-to-end Multi-perspective Topic-Post Relevance Score Reasoning (MTRSR) Method to solve the MTRSP Task, which leverages images and text from both the post and topic to infer topic-post relevance scores based on semantic similarity and logical coherence. Specifically, the topic-post content relevance reasoning module uses multimodal fusion to learn the semantic similarity of posts and topics. The logical coherence inference module examines the logical connections between posts and topics. Finally, we use three newly collected multimodal topic-post datasets and the public dataset Lazada-Home as an evaluation benchmark for the MTRSP task. Experimental results show that our MTRSR method can bring up to 9.02% in the NDCG@3 (Normalized Discounted Cumulative Gain) metric compared to the best-performing text-only model. The source code and dataset will be made public.

Image-text crisis tweet categorization:a caption-based approach

The growth of social media usage this last decade has made available a massive and valuable volume of multimedia data. However, the lack of large multimodal annotated datasets, along with the inherent noise and the diversity of multimodal relations in this type of data presents challenges for machine learning methods. Unlike classic multimodal data, social media data comes with a large diversity of relations between image and text making the interaction between the two modalities more difficult.Previous research concentrated on fusion strategies with separate encoders for each modality. This paper introduces CMB (Caption-based Multimodal BERT), a method of classifying crisis-related social media posts by utilizing information from both images and texts. CMB translates the image modality into a text-compatible space, facilitating intermodal interaction. Furthermore, CMB presents training opportunities to enhance the model's robustness to missing modalities. Experimental results show that CMB is competitive with well-established, costly, and manually crafted multimodal models.

M-CATNAT: A Multimodal dataset to analyze French tweets during natural disasters

The proliferation of social media, especially platforms like X (formerly Twitter), has made available a large volume of real-time data valuable across diverse fields. During natural disasters, such data aids humanitarian efforts by providing crucial insights. However, processing this vast amount of data necessitates automated systems, often relying on annotated datasets for training. While supervised learning dominates this area, multilingual and multimodal annotated datasets are scarce. The present study addresses this gap by introducing M-CATNAT, a multimodal dataset of French tweets about natural disasters. Unlike previous datasets, M-CATNAT integrates annotations for texts, images, and their multimodal combination. Leveraging CrisisMMD guidelines, this work in progress aims to annotate 1,430 tweets, generating over 4,500 labels. The M-CATNAT dataset not only expands resources to non-English languages but also enhances multimodal analysis by furnishing three levels of annotation for each tweet (one per modality plus one for the whole tweet).

Feature Level Fusion of Fingerprints and IRIS for the Enhancement of Biometric System

Background:: With the growing need for information safety and security rules in every corner of the globe. The biometric innovation has been used widely all over the world in daily life. In this view, the Multimodal-based biometric technique has gained popularity and interest due to its capability to resolve various problems associated with single-model biometric identification systems Methods:: In this research, an advanced multimodal-based biometric recognition system is introduced, which depends on multilayer perceptron (MLP), and it identifies humans using biometric features of Fingerprints and IRIS. To build an efficient model, the popular model called Rest- Net50 was used, the gradient method was used and the hinge technique was used as a loss function. For the technique of fusion, different techniques are used to show the effect of fusion in our model. The recognition rate of introduced systems was assessed by experimenting with various techniques on SDUMLA-HMT datasets based on the multimodal biometric datasets. The recognition rate showed that merging two biometric features in biometric recognition systems gives the best results compared to a single biometric feature. Results:: The results obtained showed the superiority of our model with other existing techniques by obtaining a recognition rate of 99.79% with a feature-based fusion method. Conclusion:: The introduced system used the MLP deep learning technique. The feature level fusion was applied to recognize the user using IRIS and fingerprints. To the best of our belief, this is the first research to incorporate MLP and multimodal-based biometric systems using IRIS and fingerprints.

Global structure graph mapping for multimodal change detection

ABSTRACT Multimodal change detection (MCD) combines multiple remote sensing data sources to realize surface change monitoring, which is essential for disaster evaluation and environmental monitoring. However, due to the ‘incomparable’ features in multimodal data, traditional change detection methods for unimodal (homogenous) data no longer apply. To address this issue, this paper proposes a novel MCD method with global structure graph mapping (GSGM) which extracts the ‘comparable’ structural features between multimodal datasets and constructs a global structure graph (GSG) to express the structure information for each of the multi-temporal images, which are then cross-mapped to the other data domain. The change intensity (CI) is determined by measuring the change of GSGs after mapping and the differences between GSGs and mapped GSGs. The forward and backward CI maps (CIMs) are then fused with the latent low-rank representation method (LLRR), and the change map (CM) is obtained by threshold segmentation. Experiments on five multimodal and four unimodal datasets demonstrate the effectiveness and robustness of the proposed method (source code is made available at https://github.com/rshante0426/GSGM).

Image-Aided LiDAR Extraction, Classification, and Characterization of Lane Markings from Mobile Mapping Data

The documentation of roadway factors (such as roadway geometry, lane marking retroreflectivity/classification, and lane width) through the inventory of lane markings can reduce accidents and facilitate road safety analyses. Typically, lane marking inventory is established using either imagery or Light Detection and Ranging (LiDAR) data collected by mobile mapping systems (MMS). However, it is important to consider the strengths and weaknesses of both camera and LiDAR units when establishing lane marking inventory. Images may be susceptible to weather and lighting conditions, and lane marking might be obstructed by neighboring traffic. They also lack 3D and intensity information, although color information is available. On the other hand, LiDAR data are not affected by adverse weather and lighting conditions, and they have minimal occlusions. Moreover, LiDAR data provide 3D and intensity information. Considering the complementary characteristics of camera and LiDAR units, an image-aided LiDAR framework would be highly advantageous for lane marking inventory. In this context, an image-aided LiDAR framework means that the lane markings generated from one modality (i.e., either an image or LiDAR) are enhanced by those derived from the other one (i.e., either imagery or LiDAR). In addition, a reporting mechanism that can handle multi-modal datasets from different MMS sensors is necessary for the visualization of inventory results. This study proposes an image-aided LiDAR lane marking inventory framework that can handle up to five lanes per driving direction, as well as multiple imaging and LiDAR sensors onboard an MMS. The framework utilizes lane markings extracted from images to improve LiDAR-based extraction. Thereafter, intensity profiles and lane width estimates can be derived using the image-aided LiDAR lane markings. Finally, imagery/LiDAR data, intensity profiles, and lane width estimates can be visualized through a web portal that has been developed in this study. For the performance evaluation of the proposed framework, lane markings obtained through LiDAR-based, image-based, and image-aided LiDAR approaches are compared against manually established ones. The evaluation demonstrates that the proposed framework effectively compensates for the omission errors in the LiDAR-based extraction, as evidenced by an increase in the recall from 87.6% to 91.6%.

Similar modality completion-based multimodal sentiment analysis under uncertain missing modalities

Recently, uncertain missing modalities in multimodal sentiment analysis (MSA) brings a new challenge for sentiment analysis. However, existing research cannot accurately complete the missing modalities, and fail to explore the advantages of the text modality in MSA. For the above problems, this work develops a Similar Modality Completion based-MSA model under uncertain missing modalities (termed as SMCMSA). Firstly, we construct the full modalities samples database (FMSD) by screening out the full modality samples from the whole multimodal dataset, and then predicting and marking the sentiment labels of each modality of the samples with three pre-trained unimodal sentiment analysis model (PTUSA). Next, for completing the uncertain missing modalities, we propose a set of missing modalities completion strategies based on the similar modalities selected from FMSD. For the completed multimodal data, we first encode the text, video and audio modality using the encoder of transformer, then we fuse the representation of text into the representations of video and audio under the guidance of a pre-trained model, thereby improving the quality of video and audio. Finally, we conduct sentiment classification based on the representations of text, video and audio with the softmax function respectively, and get the final decision with the decision-level fusion method. Based on benchmark datasets CMU-MOSI and IEMOCAP, extensive experiments have been conducted to verify that our proposed model SMCMSA has better performance than that of the state-of-the-art baseline models.The codes of our model are available at https://github.com/Astro2Sun/SMCMSA.

Non-contact multimodal indoor human monitoring systems: A survey

Indoor human monitoring systems are integral in various applications. They leverage a wide range of sensors, including cameras, radio devices, and inertial measurement units, to collect extensive data from users and the environment. These sensors contribute distinct data modalities, encompassing video feeds from cameras, received signal strength indicators and channel state information from WiFi devices, and three-axis acceleration data from accelerometers. In this context, we present a comprehensive survey of multimodal approaches applied to indoor human monitoring systems, with a specific focus on their relevance in elderly care. Our survey primarily highlights non-contact technologies, particularly cameras and radio devices, as key components in the development of indoor human monitoring systems. Throughout this article, we explore well-established techniques for extracting features from multimodal data sources. Our exploration extends to methodologies for fusing these features and harnessing multiple modalities to improve the accuracy and robustness of machine learning models. Furthermore, we conduct comparative analysis across different data modalities in diverse human monitoring tasks and undertake a comprehensive examination of existing multimodal datasets. This extensive survey not only highlights the significance of indoor human monitoring systems but also emphasizes their versatile applications. In particular, we emphasize their critical role in enhancing the quality of elderly care, offering valuable insights into the development of non-contact monitoring solutions tailored to the needs of aging populations.