Multimodal Data Research Articles

Deep learning with plasma plume image sequences for anomaly detection and prediction of growth kinetics during pulsed laser deposition

Materials synthesis platforms that are designed for autonomous experimentation are capable of collecting multimodal diagnostic data that can be utilized for feedback to optimize material properties. Pulsed laser deposition (PLD) is emerging as a viable autonomous synthesis tool, and so the need arises to develop machine learning (ML) techniques that are capable of extracting information from in situ diagnostics. Here, we demonstrate that intensified-CCD image sequences of the plasma plume generated during PLD can be used for anomaly detection and the prediction of thin film growth kinetics. We develop multi-output (2 + 1)D convolutional neural network regression models that extract deep features from plume dynamics that not only correlate with the measured chamber pressure and incident laser energy, but more importantly, predict parameters of an auto-catalytic film growth model derived from in situ laser reflectivity experiments. Our results demonstrate how ML with in situ plume diagnostics data in PLD can be utilized to maintain deposition conditions in an optimal regime. Further, the predictive capabilities of plume dynamics on the kinetics of film growth or other film properties prior to deposition provides a means for rapid pre-screening of growth conditions for the non-expert, which promises to accelerate materials optimization with PLD.

"Insights into ALD & AUD diagnosis and prognosis: Exploring AI and multi-modal data streams".

The rapid evolution of artificial intelligence (AI) and the widespread embrace of digital technologies have ushered in a new era of clinical research and practice in hepatology. Although its potential is far from realization, these significant strides have generated new opportunities to address existing gaps in the delivery of care for patients with liver disease. In this review, we discuss how AI and opportunities for multimodal data integration can improve the diagnosis, prognosis and management of alcohol-associated liver disease (ALD). An emphasis is made on how these approaches will also benefit the detection and management of alcohol use disorder (AUD). Our discussion encompasses challenges and limitations, concluding with a glimpse into the promising future of these advancements.

Fine-resolution estimation for urban surface water pollution susceptibility with multi-modal earth observation data

Abstract The sustainability and suitability of water resources are of great importance for maintaining urban populations. The landscapes and environment around urban waters have always been the main focus of maintaining water quality for sustainable water supplies. Early-stage field investigations recognized the influence of land use/land cover (LULC) on water quality. To extend the research scope in spatial and temporal dimensions, remote sensing techniques have been utilized to discover the relationships between LULC and water quality. However, these remote sensing datasets generally had a medium spatial resolution, making them unable to support the fine-detailed land classifications that are critical to explore the water quality in an urban area. Moreover, although more details regarding the land surface are available from the currently-generated high-resolution and very-high-resolution remote sensing images, this land surface information is too complex for the state-of-the-art deep learning approaches and benchmark datasets. This manuscript reports our efforts on developing a framework to explore the fine-resolution relationship between surface water pollution and LULC. To address the cost of computing time and limitations of well-labelled datasets, we employ a foundation model-enhanced approach for water extraction and water-surrounded LULC classification. We propose an estimator of surface water pollution susceptibility to main pollutants based on the surrounding LULCs. Selecting the Future City of Beijing as the study area, based on very-high-resolution remote sensing images, the experiment proved that our proposed approach could effectively map the susceptibility of surface water pollution caused by its surrounding land use and land cover. To our knowledge, the relationship of LULCs and water quality have not been investigated using 0.5m spatial resolution data. We hope our work can provide a prospective fine-detailed water quality analysis in the community of water environment of remote sensing.

Multimodal MRI segmentation of key structures for microvascular decompression via knowledge-driven mutual distillation and topological constraints.

Microvascular decompression (MVD) is a widely used neurosurgical intervention for the treatment of cranial nerves compression. Segmentation of MVD-related structures, including the brainstem, nerves, arteries, and veins, is critical for preoperative planning and intraoperative decision-making. Automatically segmenting structures related to MVD is still challenging for current methods due to the limited information from a single modality and the complex topology of vessels and nerves. Considering that it is hard to distinguish MVD-related structures, especially for nerve and vessels with similar topology, we design a multimodal segmentation network with a shared encoder-dual decoder structure and propose a clinical knowledge-driven distillation scheme, allowing reliable knowledge transferred from each decoder to the other. Besides, we introduce a class-wise contrastive module to learn the discriminative representations by maximizing the distance among classes across modalities. Then, a projected topological loss based on persistent homology is proposed to constrain topological continuity. We evaluate the performance of our method on in-house dataset consisting of 100 paired HR-T2WI and 3D TOF-MRA volumes. Experiments indicate that our model outperforms the SOTA in DSC by 1.9% for artery, 3.3% for vein and 0.5% for nerve. Visualization results show our method attains improved continuity and less breakage, which is also consistent with intraoperative images. Our method can comprehensively extract the distinct features from multimodal data to segment the MVD-related key structures and preserve the topological continuity, allowing surgeons precisely perceiving the patient-specific target anatomy and substantially reducing the workload of surgeons in the preoperative planning stage. Our resources will be publicly available at https://github.com/JaronTu/Multimodal_MVD_Seg .

From CNNs to Transformers in Multimodal Human Action Recognition: A Survey

Due to its widespread applications, human action recognition is one of the most widely studied research problems in Computer Vision. Recent studies have shown that addressing it using multimodal data leads to superior performance as compared to relying on a single data modality. During the adoption of deep learning for visual modelling in the last decade, action recognition approaches have mainly relied on Convolutional Neural Networks (CNNs). However, the recent rise of Transformers in visual modelling is now also causing a paradigm shift for the action recognition task. This survey captures this transition while focusing on Multimodal Human Action Recognition (MHAR). Unique to the induction of multimodal computational models is the process of ‘fusing’ the features of the individual data modalities. Hence, we specifically focus on the fusion design aspects of the MHAR approaches. We analyze the classic and emerging techniques in this regard, while also highlighting the popular trends in the adaption of CNN and Transformer building blocks for the overall problem. In particular, we emphasize on recent design choices that have led to more efficient MHAR models. Unlike existing reviews, which discuss Human Action Recognition from a broad perspective, this survey is specifically aimed at pushing the boundaries of MHAR research by identifying promising architectural and fusion design choices to train practicable models. We also provide an outlook of the multimodal datasets from their scale and evaluation viewpoint. Finally, building on the reviewed literature, we discuss the challenges and future avenues for MHAR.

HKA: A Hierarchical Knowledge Alignment Framework for Multimodal Knowledge Graph Completion

Recent years have witnessed the successful application of knowledge graph techniques in structured data processing, while how to incorporate knowledge from visual and textual modalities into knowledge graphs has been given less attention. To better organize them, Multimodal Knowledge Graphs (MKGs), comprising the structural triplets of traditional Knowledge Graphs (KGs) together with entity-related multimodal data (e.g., images and texts), have been introduced consecutively. However, it is still a great challenge to explore MKGs due to their inherent incompleteness. Although most existing Multimodal Knowledge Graph Completion (MKGC) approaches can infer missing triplets based on available factual triplets and multimodal information, they almost ignore the modal conflicts and supervisory effect, failing to achieve a more comprehensive understanding of entities. To address these issues, we propose a novel H ierarchical K nowledge A lignment ( HKA ) framework for MKGC. Specifically, a macro-knowledge alignment module is proposed to capture global semantic relevance between modalities for dealing with modal conflicts in MKG. Furthermore, a micro-knowledge alignment module is also developed to reveal the local consistency information through inter- and intra-modality supervisory effect more effectively. By integrating different modal predictions, a final decision can be made. Experimental results on three benchmark MKGC tasks have demonstrated the effectiveness of the proposed HKA framework.

HGR Correlation Pooling Fusion Framework for Recognition and Classification in Multimodal Remote Sensing Data

This paper investigates remote sensing data recognition and classification with multimodal data fusion. Aiming at the problems of low recognition and classification accuracy and the difficulty in integrating multimodal features in existing methods, a multimodal remote sensing data recognition and classification model based on a heatmap and Hirschfeld–Gebelein–Rényi (HGR) correlation pooling fusion operation is proposed. A novel HGR correlation pooling fusion algorithm is developed by combining a feature fusion method and an HGR maximum correlation algorithm. This method enables the restoration of the original signal without changing the value of transmitted information by performing reverse operations on the sample data. This enhances feature learning for images and improves performance in specific tasks of interpretation by efficiently using multi-modal information with varying degrees of relevance. Ship recognition experiments conducted on the QXS-SROPT dataset demonstrate that the proposed method surpasses existing remote sensing data recognition methods. Furthermore, land cover classification experiments conducted on the Houston 2013 and MUUFL datasets confirm the generalizability of the proposed method. The experimental results fully validate the effectiveness and significant superiority of the proposed method in the recognition and classification of multimodal remote sensing data.

A Comparison of Personalized and Generalized Approaches to Emotion Recognition Using Consumer Wearable Devices: Machine Learning Study

Background There are a wide range of potential adverse health effects, ranging from headaches to cardiovascular disease, associated with long-term negative emotions and chronic stress. Because many indicators of stress are imperceptible to observers, the early detection of stress remains a pressing medical need, as it can enable early intervention. Physiological signals offer a noninvasive method for monitoring affective states and are recorded by a growing number of commercially available wearables. Objective We aim to study the differences between personalized and generalized machine learning models for 3-class emotion classification (neutral, stress, and amusement) using wearable biosignal data. Methods We developed a neural network for the 3-class emotion classification problem using data from the Wearable Stress and Affect Detection (WESAD) data set, a multimodal data set with physiological signals from 15 participants. We compared the results between a participant-exclusive generalized, a participant-inclusive generalized, and a personalized deep learning model. Results For the 3-class classification problem, our personalized model achieved an average accuracy of 95.06% and an F1-score of 91.71%; our participant-inclusive generalized model achieved an average accuracy of 66.95% and an F1-score of 42.50%; and our participant-exclusive generalized model achieved an average accuracy of 67.65% and an F1-score of 43.05%. Conclusions Our results emphasize the need for increased research in personalized emotion recognition models given that they outperform generalized models in certain contexts. We also demonstrate that personalized machine learning models for emotion classification are viable and can achieve high performance.

Overview and Benchmark on Multi-Modal Lidar Point Cloud Registration for Forest Applications

Abstract. Light Detection and Ranging (LIDAR) is widely acknowledged as a robust tool for monitoring forest structure, dynamics, and changes. To achieve a high-complete forest structural model, LiDAR data acquisition from both aerial (above-canopy) and terrestrial (below-canopy) platforms is commonplace. Consequently, in such multi-modal LiDAR cases, robust data registration is required for accurate forest analysis, such as biomass and canopy growth. Yet, multi-modal LiDAR registration remains a significant challenge due to differences in observation perspectives, deficient data overlap, and often inhomogeneity in point distributions and densities. The challenge increases in complex forest environments due to the abundance of unstable features (e.g., leaves) and occlusions. Thus, the dynamic nature of forest scenes needs to be considered when applying registration methods on forest point clouds. In this paper, we overview the latest advancements in registering forest point clouds from multi-modal data acquisitions, aiming to discuss the strengths and weaknesses of the most used LiDAR registration methods for forest applications. To support our investigations, we benchmark two multi-modal registration methods especially designed for forest mapping against traditional global and feature-based approaches. Experiment assessments were conducted using two point clouds acquired from a permanent laser scanning and airborne laser scanning systems at a boreal forest plot.

Global structure graph mapping for multimodal change detection

ABSTRACT Multimodal change detection (MCD) combines multiple remote sensing data sources to realize surface change monitoring, which is essential for disaster evaluation and environmental monitoring. However, due to the ‘incomparable’ features in multimodal data, traditional change detection methods for unimodal (homogenous) data no longer apply. To address this issue, this paper proposes a novel MCD method with global structure graph mapping (GSGM) which extracts the ‘comparable’ structural features between multimodal datasets and constructs a global structure graph (GSG) to express the structure information for each of the multi-temporal images, which are then cross-mapped to the other data domain. The change intensity (CI) is determined by measuring the change of GSGs after mapping and the differences between GSGs and mapped GSGs. The forward and backward CI maps (CIMs) are then fused with the latent low-rank representation method (LLRR), and the change map (CM) is obtained by threshold segmentation. Experiments on five multimodal and four unimodal datasets demonstrate the effectiveness and robustness of the proposed method (source code is made available at https://github.com/rshante0426/GSGM).

Deep kernelized dimensionality reducer for multi-modality heterogeneous data

Deep kernelized dimensionality reducer for multi-modality heterogeneous data

Multi-modal topic modeling from social media data using deep transfer learning

As social media platforms grow rapidly, multi-modal data is becoming more and more prevalent. A user can better understand events by analyzing multimodal data for topics. Automatic topic detection from multimodal data can potentially have tremendous value for advertising and government agencies for whom public opinion matters for strategic decisions and policy making. However , multimodal topic detection is complicated for two reasons: (1) The nature of the multimodal data varies from one medium to another, and (2) The noisy nature of webdata. Conventional topic models are ineffective in dealing with these two problems. This paper proposes, a framework for multimodal topic modeling for social media data that uses topics extracted using Latent Dirichlet Allocation (LDA) and patterns found from images using transfer learning. The proposed framework makes use of textual as well as visual data for topic detection. The experiments are conducted on the benchmark datasets: Flickr8k, Flickr30k, and MCG WEBV. The proposed work outperformed other techniques in terms of accuracy (0.63), precision (0.75), recall (0.97), F-Measure (0.85), Bleu-1(0.68),METEOR (0.17) , ROUGE-L (0.49), and CIDEr (0.573). The proposed work is compared to state-of-the-art methods to demonstrate its accuracy.

Comparison of different group-level templates in gradient-based multimodal connectivity analysis

Abstract The study of large-scale brain connectivity is increasingly adopting unsupervised approaches that derive low-dimensional spatial representations from high-dimensional connectomes, referred to as gradient analysis. When translating this approach to study interindividual variations in connectivity, one technical issue pertains to the selection of an appropriate group-level template to which individual gradients are aligned. Here, we compared different group-level template construction strategies using functional and structural connectome data from neurotypical controls and individuals with autism spectrum disorder (ASD) to identify between-group differences. We studied multimodal magnetic resonance imaging data obtained from the Autism Brain Imaging Data Exchange (ABIDE) Initiative II and the Human Connectome Project (HCP). We designed six template construction strategies that varied in whether: (1) they included typical controls in addition to ASD; or (2) they mapped from one dataset onto another. We found that aligning a combined subject template of the ASD and control subjects from the ABIDE Initiative onto the HCP template exhibited the most pronounced effect size. This strategy showed robust identification of ASD-related brain regions for both functional and structural gradients across different study settings. Replicating the findings on focal epilepsy demonstrated the generalizability of our approach. Our findings will contribute to improving gradient-based connectivity research.

Artificial intelligence in epilepsy - applications and pathways to the clinic.

Artificial intelligence (AI) is rapidly transforming health care, and its applications in epilepsy have increased exponentially over the past decade. Integration of AI into epilepsy management promises to revolutionize the diagnosis and treatment of this complex disorder. However, translation of AI into neurology clinical practice has not yet been successful, emphasizing the need to consider progress to date and assess challenges and limitations of AI. In this Review, we provide an overview of AI applications that have been developed in epilepsy using a variety of data modalities: neuroimaging, electroencephalography, electronic health records, medical devices and multimodal data integration. For each, we consider potential applications, including seizure detection and prediction, seizure lateralization, localization of the seizure-onset zone and assessment for surgical or neurostimulation interventions, and review the performance of AI tools developed to date. We also discuss methodological considerations and challenges that must be addressed to successfully integrate AI into clinical practice. Our goal is to provide an overview of the current state of the field and provide guidance for leveraging AI in future to improve management of epilepsy.

Multimodal Sentiment Analysis for Movie Scenes Based on a Few-Shot Learning Approach

Multimodal sentiment analysis involves discerning the emotions of speakers through diverse features such as sound, text, and images. However, current research in this field heavily relies on extensive supervised datasets, demanding substantial labor and computational resources. This study introduces a meta-learning-based approach for multimodal sentiment analysis, aiming to delve into emotional information within movie scenes. Leveraging meta-learning techniques, this approach seeks to accurately capture emotions in movie scenes using a limited number of annotated samples, achieving model generalization under constrained labeled data. Specifically, we introduce an optimization-based meta-learning approach for the multimodal sentiment analysis tasks in text and vision, enhancing the model’s ability to generalize to new tasks with limited annotations. Additionally, an innovative strategy based on Meta-Prompting is proposed to handle multimodal data. The merits of the proposed method are validated across different datasets.

Multimodal data fusion using sparse canonical correlation analysis and cooperative learning: a COVID-19 cohort study

Through technological innovations, patient cohorts can be examined from multiple views with high-dimensional, multiscale biomedical data to classify clinical phenotypes and predict outcomes. Here, we aim to present our approach for analyzing multimodal data using unsupervised and supervised sparse linear methods in a COVID-19 patient cohort. This prospective cohort study of 149 adult patients was conducted in a tertiary care academic center. First, we used sparse canonical correlation analysis (CCA) to identify and quantify relationships across different data modalities, including viral genome sequencing, imaging, clinical data, and laboratory results. Then, we used cooperative learning to predict the clinical outcome of COVID-19 patients: Intensive care unit admission. We show that serum biomarkers representing severe disease and acute phase response correlate with original and wavelet radiomics features in the LLL frequency channel (cor(Xu1, Zv1) = 0.596, p value < 0.001). Among radiomics features, histogram-based first-order features reporting the skewness, kurtosis, and uniformity have the lowest negative, whereas entropy-related features have the highest positive coefficients. Moreover, unsupervised analysis of clinical data and laboratory results gives insights into distinct clinical phenotypes. Leveraging the availability of global viral genome databases, we demonstrate that the Word2Vec natural language processing model can be used for viral genome encoding. It not only separates major SARS-CoV-2 variants but also allows the preservation of phylogenetic relationships among them. Our quadruple model using Word2Vec encoding achieves better prediction results in the supervised task. The model yields area under the curve (AUC) and accuracy values of 0.87 and 0.77, respectively. Our study illustrates that sparse CCA analysis and cooperative learning are powerful techniques for handling high-dimensional, multimodal data to investigate multivariate associations in unsupervised and supervised tasks.

RGB‐D road segmentation based on cross‐modality feature maintenance and encouragement

AbstractDeep images can provide rich spatial structure information, which can effectively exclude the interference of illumination and road texture in road scene segmentation and make better use of the prior knowledge of road area. This paper first proposes a new cross‐modal feature maintenance and encouragement network. It includes a quantization statistics module as well as a maintenance and encouragement module for effective fusion between multimodal data. Meanwhile, for the problem that if the road segmentation is performed directly using a segmentation network, there will be a lack of supervised guidance with clear physical meaningful information and poor interpretability of learning features, this paper proposes two road segmentation models based on prior knowledge of deep image: disparity information and surface normal vector information. Then, a two‐branch neural network is used to process the colour image and the processed depth image separately, to achieve the full utilization of the complementary features of the two modalities. The experimental results on the KITTI road dataset and Cityscapes dataset show that the method in this paper has good road segmentation performance and high computational efficiency.

Research on application of multimodal data fusion in intelligent building environment perception

With the rapid development of the building industry, intelligent buildings benefit from its safety, energy saving, environmental protection and integration and other advantages have been widely loved by people, most operators also realize the importance of intelligent buildings to bring people humanized and customized services, and in order to realize the personalized service of the building, multi-modal data fusion is an effective method. On the other hand, in today’s Internet of Things society, many practical applications need to deploy a large number of sensing equipment for data collection and processing, so as to carry out high-quality monitoring of the physical world, but due to the inherent limitations of these hardware equipment and the influence of factors such as the environment, single mode data often cannot be completely and comprehensively monitored to the physical world’s changing characteristics. In this development context, multi-modal data fusion has become a research hotspot in the field of machine learning. Based on this, this paper proposes a one-stage fast object detection model with multi-level fusion of multi-modal features and end-to-end characteristics for building indoor environment perception, and conducts experimental analysis on the performance of the model. The verification results show that the accuracy of the proposed method is 50.7% and the running speed is 0.107 s, which has better performance than the existing detection methods.

A road surface reconstruction dataset for autonomous driving

Recent developments in intelligent robot systems, especially autonomous vehicles, put forward higher requirements for safety and comfort. Road conditions are crucial factors affecting the comprehensive performance of ground vehicles. Nonetheless, existing environment perception datasets for autonomous driving lack attention to road surface areas. In this paper, we introduce the road surface reconstruction dataset, providing multi-modal, high-resolution, and high-precision data collected by real-vehicle platform in diverse driving conditions. It covers common road types containing approximately 16,000 pairs of stereo images, point clouds, and ground-truth depth/disparity maps, with accurate data processing pipelines to ensure its quality. Preliminary evaluations reveal the effectiveness of our dataset and the challenge of the task, underscoring substantial opportunities of it as a valuable resource for advancing computer vision techniques. The reconstructed road structure and texture contribute to the analysis and prediction of vehicle responses for motion planning and control systems.

DeepGCN based on variable multi‐graph and multimodal data for ASD diagnosis

AbstractDiagnosing individuals with autism spectrum disorder (ASD) accurately faces great challenges in clinical practice, primarily due to the data's high heterogeneity and limited sample size. To tackle this issue, the authors constructed a deep graph convolutional network (GCN) based on variable multi‐graph and multimodal data (VMM‐DGCN) for ASD diagnosis. Firstly, the functional connectivity matrix was constructed to extract primary features. Then, the authors constructed a variable multi‐graph construction strategy to capture the multi‐scale feature representations of each subject by utilising convolutional filters with varying kernel sizes. Furthermore, the authors brought the non‐imaging information into the feature representation at each scale and constructed multiple population graphs based on multimodal data by fully considering the correlation between subjects. After extracting the deeper features of population graphs using the deep GCN(DeepGCN), the authors fused the node features of multiple subgraphs to perform node classification tasks for typical control and ASD patients. The proposed algorithm was evaluated on the Autism Brain Imaging Data Exchange I (ABIDE I) dataset, achieving an accuracy of 91.62% and an area under the curve value of 95.74%. These results demonstrated its outstanding performance compared to other ASD diagnostic algorithms.