Multimodal Sequences Research Articles

Target and source modality co-reinforcement for emotion understanding from asynchronous multimodal sequences

Perceiving human emotions from a multimodal perspective has received significant attention in knowledge engineering communities. Due to the variable receiving frequency for sequences from various modalities, multimodal streams usually have an inherent asynchronous challenge. Most previous methods performed manual sequence alignment before multimodal fusion, which ignored long-range dependencies among modalities and failed to learn reliable crossmodal element correlations. Inspired by the human perception paradigm, we propose a target and source Modality Co-Reinforcement (MCR) approach to achieve sufficient crossmodal interaction and fusion at different granularities. Specifically, MCR introduces two types of target modality reinforcement units to reinforce the multimodal representations jointly. These target units effectively enhance emotion-related knowledge exchange in fine-grained interactions and capture the crossmodal elements that are emotionally expressive in mixed-grained interactions. Moreover, a source modality update module is presented to provide meaningful features for the crossmodal fusion of target modalities. Eventually, the multimodal representations are progressively reinforced and improved via the above components. Comprehensive experiments are conducted on three multimodal emotion understanding benchmarks. Quantitative results show that MCR significantly outperforms the previous state-of-the-art methods in both word-aligned and unaligned settings. Additionally, qualitative analysis and visualization fully demonstrate the superiority of the proposed modules.

Multimodal Graph for Unaligned Multimodal Sequence Analysis via Graph Convolution and Graph Pooling

Multimodal sequence analysis aims to draw inferences from visual, language, and acoustic sequences. A majority of existing works focus on the aligned fusion of three modalities to explore inter-modal interactions, which is impractical in real-world scenarios. To overcome this issue, we seek to focus on analyzing unaligned sequences, which is still relatively underexplored and also more challenging. We propose Multimodal Graph, whose novelty mainly lies in transforming the sequential learning problem into graph learning problem. The graph-based structure enables parallel computation in time dimension (as opposed to recurrent neural network) and can effectively learn longer intra- and inter-modal temporal dependency in unaligned sequences. First, we propose multiple ways to construct the adjacency matrix for sequence to perform sequence to graph transformation. To learn intra-modal dynamics, a graph convolution network is employed for each modality based on the defined adjacency matrix. To learn inter-modal dynamics, given that the unimodal sequences are unaligned, the commonly considered word-level fusion does not pertain. To this end, we innovatively devise graph pooling algorithms to automatically explore the associations between various time slices from different modalities and learn high-level graph representation hierarchically. Multimodal Graph outperforms state-of-the-art models on three datasets under the same experimental setting.

Multimodal Sentiment Recognition With Multi-Task Learning

Sentiment recognition in social network aims at recognizing the underlying affective states of user-generated content. The research center of sentiment recognition is moving from pure texts to multimodal contents, with the explosive growth of social platforms. Different from the image-text contents on blog and review platforms, multimodal sequences play a dominant role in streaming media, e.g. YouTube and TikTok. Sentiment recognition for multimodal sequences needs to extract the common and specific information of modalities. But current studies only focus on learning the cross-modal fusion representation to explore the inter-modal interaction, while neglecting the interactions and characteristics within each modality. We propose a novel cascade and specific scoring model, which aims at learning better cross-modal and unimodal representations to capture both the inter- and intra-modal interactions for sentiment recognition. Qualitative and quantitative experiments on two benchmarks have demonstrated the competitive performances of the proposed methods.

Modeling and analyzing single-cell multimodal data with deep parametric inference.

The proliferation of single-cell multimodal sequencing technologies has enabled us to understand cellular heterogeneity with multiple views, providing novel and actionable biological insights into the disease-driving mechanisms. Here, we propose a comprehensive end-to-end single-cell multimodal analysis framework named Deep Parametric Inference (DPI). DPI transforms single-cell multimodal data into a multimodal parameter space by inferring individual modal parameters. Analysis of cord blood mononuclear cells (CBMC) reveals that the multimodal parameter space can characterize the heterogeneity of cells more comprehensively than individual modalities. Furthermore, comparisons with the state-of-the-art methods on multiple datasets show that DPI has superior performance. Additionally, DPI can reference and query cell types without batch effects. As a result, DPI can successfully analyze the progression of COVID-19 disease in peripheral blood mononuclear cells (PBMC). Notably, we further propose a cell state vector field and analyze the transformation pattern of bone marrow cells (BMC) states. In conclusion, DPI is a powerful single-cell multimodal analysis framework that can provide new biological insights into biomedical researchers. The python packages, datasets and user-friendly manuals of DPI are freely available at https://github.com/studentiz/dpi.

Deep Multimodal Sequence Fusion by Regularized Expressive Representation Distillation

Multimodal sequence learning aims to utilize information from different modalities to enhance overall performance. Mainstream works often follow an intermediate-fusion pipeline, which explores both modality-specific and modality-supplementary information for fusion. However, the unaligned and heterogeneously distributed multimodal sequences pose significant challenges to the fusion task: 1) to extract both effective unimodal and crossmodal representations and 2) to overcome the overfitting issue in joint multimodal sequence optimization. In this work, we propose regularized expressive representation distillation (RERD) that aims to seek effective multimodal representations and to enhance the generalization of fusion. First, to improve unimodal representation learning, unimodal representations are assigned to multi-head distillation encoders, where the unimodal representations are iteratively updated through distillation attention layers. Second, to alleviate the overfitting issue in joint crossmodal optimization, a multimodal sinkhorn distance regularizer is proposed to reinforce the expressive representation extraction and to reduce the modality gap before fusion adaptively. These representations produce a comprehensive view of the multimodal sequences, which are utilized for downstream fusion tasks. Experimental results on several popular benchmarks demonstrate that the proposed method achieves state-of-the-art performance, compared with widely used baselines for deep multimodal sequence fusion. Source code will be shared publicly.

UniMF: A Unified Multimodal Framework for Multimodal Sentiment Analysis in Missing Modalities and Unaligned Multimodal Sequences

UniMF: A Unified Multimodal Framework for Multimodal Sentiment Analysis in Missing Modalities and Unaligned Multimodal Sequences

Clustering of single-cell multi-omics data with a multimodal deep learning method

Single-cell multimodal sequencing technologies are developed to simultaneously profile different modalities of data in the same cell. It provides a unique opportunity to jointly analyze multimodal data at the single-cell level for the identification of distinct cell types. A correct clustering result is essential for the downstream complex biological functional studies. However, combining different data sources for clustering analysis of single-cell multimodal data remains a statistical and computational challenge. Here, we develop a novel multimodal deep learning method, scMDC, for single-cell multi-omics data clustering analysis. scMDC is an end-to-end deep model that explicitly characterizes different data sources and jointly learns latent features of deep embedding for clustering analysis. Extensive simulation and real-data experiments reveal that scMDC outperforms existing single-cell single-modal and multimodal clustering methods on different single-cell multimodal datasets. The linear scalability of running time makes scMDC a promising method for analyzing large multimodal datasets.

Automatic 1p/19q co-deletion identification of gliomas by MRI using deep learning U-net network

The chromosome 1p/19q co-deletion which is a hallmark of oligodendroglioma plays more crucial role in glioma classification especially in “The 2021 WHO Classification of Tumors of the Central Nervous System”. A more effective non-invasive method to distinguish 1p/19q co-deletion tumor from all gliomas can facilitate the strategy selection of pathologists, physicians, and surgeons. Preoperative MRI, including T1, T2, enhanced T1 and T2-FLAIR, from 61 glioma patients of our facility were reviewed. Data from 89 gliomas subjects from The Cancer Imaging Archive were recruited. Following the preprocessing, we improved the U-net and ResNet152 based on the MRI data of different modalities to determine the 1p/19q codeletion from overall gliomas. The different models were compared. The UMAP result implies that two different data share some similar traits. All the sensitivity, specificity and accuracy of U-net are higher than that of the ResNet152. The test accuracy with four modalities outperforms others significantly, reaching 92.156%. We introduce an efficient pipeline with U-net network for the identification of 1p/19q genotype status. The study implements one step judgement with multi-modal sequence MRI images. It takes a further step to suggest that machine learning can render more possibilities to conventional MRI.

Multi-modal sequence learning for Alzheimer's disease progression prediction with incomplete variable-length longitudinal data.

Alzheimer's disease (AD) is a neurodegenerative disorder with a long prodromal phase. Predicting AD progression will clinically help improve diagnosis and empower sufferers in taking proactive care. However, most existing methods only target individuals with a fixed number of historical visits, and only predict the cognitive scores once at a fixed time horizon in the future, which cannot meet practical requirements. In this study, we consider a flexible yet more challenging scenario in which individuals may suffer from the (arbitrary) modality-missing issue, as well as the number of individuals' historical visits and the length of target score trajectories being not prespecified. To address this problem, a multi-modal sequence learning framework, highlighted by deep latent representation collaborated sequence learning strategy, is proposed to flexibly handle the incomplete variable-length longitudinal multi-modal data. Specifically, the proposed framework first employs a deep multi-modality fusion module that automatically captures complementary information for each individual with incomplete multi-modality data. A comprehensive representation is thus learned and fed into a sequence learning module to model AD progression. In addition, both the multi-modality fusion module and sequence learning module are collaboratively trained to further promote the performance of AD progression prediction. Experimental results on Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset validate the superiority of our method.

Multi‐modal fusion method for human action recognition based on IALC

In occlusion and interaction scenarios, human action recognition (HAR) accuracy is low. To address this issue, this paper proposes a novel multi-modal fusion framework for HAR. In this framework, a module called improved attention long short-term memory (IAL) is proposed, which combines the improved SE-ResNet50 (ISE-ResNet50) with long short-term memory (LSTM). IAL can extract the video sequence features and the skeleton sequence features of human behaviour. To improve the performance of HAR at a high semantic level, the obtained multi-modal sequence features are fed into a couple hidden Markov model (CHMM), and a multi-modal IAL+CHMM method called IALC is developed based on a probability graph model. To test the performance of the proposed method, experiments are conducted on the HMDB51, UCF101, Kinetics 400k, and ActivityNet datasets, and the obtained recognition accuracy are 86.40%, 97.78%, 81.12%, and 69.36% on the four datasets, respectively. The experimental results show that when the environment is complex, the proposed multi-modal fusion method for HAR based on the IALC can achieve more accurate target recognition results.

Cross-modality reinforcement for unaligned sequences sentiment analysis

Human multi-modal emotions analysis includes time series data with different modalities, such as verbal, visual, and auditory. Due to different sampling rates from each modality, the collected data streams are unaligned. The asynchrony cross-modality increases the difficulty of multi-modal fusion. Therefore, we propose a new Cross-Modality Reinforcement model (CMR) based on recent advances in a cross-modality transformer, which performs multi-modal fusion in unaligned multi-modal sequences for emotion prediction. To deal with the long-time dependencies of unaligned sequences, we introduce a time domain aggregation to model the single modal, by aggregating the information in the time dimension, and enhance contextual dependencies. Moreover, a CMR strategy is introduced in our approach.With the main and secondary modalities as inputs to the module, main modal features are strengthened through cross-modality attention and cross-modality gate, and the secondary modality information flows to the main modality potentially, while retaining main modality-specific features and complementing the missing cues. This process gradually learns the common contributing features between the main and secondary modalities and reduces the noise caused by the variability of the modal features. Finally, the enhanced features are used to make predictions about human emotions. We evaluate CMR on two multi-modal sentiment analysis benchmark datasets, and we report the accuracy of 82.7% on the CMU-MOSI and 82.5% and CMU-MOSEI, respectively, which demonstrates our method outperforms current state-of-the-art methods.

Unaligned Multimodal Sequences for Depression Assessment From Speech.

A growing area of mental health research pertains to how an individual's degree of depression might be automatically assessed through analyzing multimodal-based objective markers. However, when combined with machine learning, this research can be challenging due to the existence of unaligned multimodal sequences and the limited amount of annotated training data. In this paper, a novel cross-modal framework for automatic depression severity assessment is proposed. The low-level descriptions (LLDs) from multiple clues (such as text, audio and video) are extracted, after which multimodal fusion via cross-modal attention mechanism is utilized to facilitate the learning of more accurate feature representations. For the features extracted from each modality, the cross-modal attention mechanism is utilized to continuously update the input sequence of the target mode, until the score of the patient's health questionnaire (PHQ-8) can finally be obtained. Moreover, Self-Attention Generative Adversarial Networks (SAGAN) is employed to increase the amount of training data available for depression severity analysis. Experimental results on the depression sub-challenge dataset of the Audio/Visual Emotion Challenge (AVEC 2017 and AVEC 2019) demonstrate the effectiveness of our proposed method.

Neural Marionette: Unsupervised Learning of Motion Skeleton and Latent Dynamics from Volumetric Video

We present Neural Marionette, an unsupervised approach that discovers the skeletal structure from a dynamic sequence and learns to generate diverse motions that are consistent with the observed motion dynamics. Given a video stream of point cloud observation of an articulated body under arbitrary motion, our approach discovers the unknown low-dimensional skeletal relationship that can effectively represent the movement. Then the discovered structure is utilized to encode the motion priors of dynamic sequences in a latent structure, which can be decoded to the relative joint rotations to represent the full skeletal motion. Our approach works without any prior knowledge of the underlying motion or skeletal structure, and we demonstrate that the discovered structure is even comparable to the hand-labeled ground truth skeleton in representing a 4D sequence of motion. The skeletal structure embeds the general semantics of possible motion space that can generate motions for diverse scenarios. We verify that the learned motion prior is generalizable to the multi-modal sequence generation, interpolation of two poses, and motion retargeting to a different skeletal structure.

Behaviour Detection and Recognition of College Basketball Players Based on Multimodal Sequence Matching and Deep Neural Networks.

This study fuses multimodal sequence matching with a deep neural network algorithm for college basketball player behavior detection and recognition to conduct in-depth research and analysis, analyzing the basic components of basketball technical action videos by studying the practical application of technical actions in professional games and teaching videos from self-published authors of short video platforms. The characteristics of the dataset are also analyzed through literature research related to the basketball action dataset. On the established basketball technical action dataset, combined with the SSD target detection algorithm, the video images with cropping of human motion regions to reduce the size of image frames, generating a basketball technical action dataset based on cropped frames, reduces the amount of network training and improves the efficiency of subsequent action recognition training. In this study, by analyzing the characteristics of basic camera motion, a univariate global motion model is proposed to introduce a quadratic term to accurately express the shaking transformation, while the horizontal and vertical motion are independently represented to reduce the model complexity. Comparative experimental results show that the proposed model achieves a good balance between complexity and accuracy of global motion representation. It is suitable for global motion modeling in behavior recognition applications, laying the foundation for global and local motion estimation. On this basis, the visual feature change pattern of the key area of the scene (basketball area) is combined with the group behavior recognition based on motion patterns and the success-failure classification based on key visual information to achieve basketball semantic event recognition. The experimental results at NCAA show that the fusion of global and local motion patterns can effectively improve group behavior recognition performance. The semantic event recognition algorithm combining motion patterns and video key visual information achieves the best performance.

The Weighted Cross-Modal Attention Mechanism With Sentiment Prediction Auxiliary Task for Multimodal Sentiment Analysis

Human brain extracts the spatial and temporal semantic information by processing the multi-modalities, which has contextually meaningful for perceiving and understanding the emotional state of an individual. However, there are two main challenges in modeling multimodal sequences: 1) the different sampling rates from multimodal data make the cross-modal interactions very difficult; 2) how to efficiently fuse unimodal representations and effectively capture relationships among multimodal data. In this paper, we design the weighted cross-modal attention mechanism, which not only captures the temporal correlation information and the spatial dependence information of each modality, but also dynamically adjusts the weight of each modality across different time steps. And the unimodal subtasks are led for assisting the representation learning of specific modality to jointly train the multimodal tasks and unimodal subtasks to explore the complementary relationships of each modality. Our model gets a new state-of-the-art record on the CMU-MOSI dataset and brings noticeable performance improvements on all the metrics. For the CMU-MOSEI dataset, the F1 score of the binary classification, the 7-class task, and the regression task of our model are still the highest among all models and the proposed model is only lower than the multimodal split attention fusion (MSAF) model with aligned data on the accuracy of the binary classification, showing the great performance of the suggested method.

DM2S2: Deep Multimodal Sequence Sets With Hierarchical Modality Attention

There is increasing interest in the use of multimodal data in various web applications, such as digital advertising and e-commerce. Typical methods for extracting important information from multimodal data rely on a mid-fusion architecture that combines the feature representations from multiple encoders. However, as the number of modalities increases, several potential problems with the mid-fusion model structure arise, such as an increase in the dimensionality of the concatenated multimodal features and missing modalities. To address these problems, we propose a new concept that considers multimodal inputs as a set of sequences, namely, deep multimodal sequence sets (DM$^2$S$^2$). Our set-aware concept consists of three components that capture the relationships among multiple modalities: (a) a BERT-based encoder to handle the inter- and intra-order of elements in the sequences, (b) intra-modality residual attention (IntraMRA) to capture the importance of the elements in a modality, and (c) inter-modality residual attention (InterMRA) to enhance the importance of elements with modality-level granularity further. Our concept exhibits performance that is comparable to or better than the previous set-aware models. Furthermore, we demonstrate that the visualization of the learned InterMRA and IntraMRA weights can provide an interpretation of the prediction results.

Graph-Based Multimodal Sequential Embedding for Sign Language Translation

Sign language translation (SLT) is a challenging weakly supervised task without word-level annotations. An effective method of SLT is to leverage multimodal complementarity and to explore implicit temporal cues. In this work, we propose a graph-based multimodal sequential embedding network (MSeqGraph), in which multiple sequential modalities are densely correlated. Specifically, we build a graph structure to realize the intra-modal and inter-modal correlations. First, we design a graph embedding unit (GEU), which embeds a parallel convolution with channel-wise and temporal-wise learning into the graph convolution to learn the temporal cues in each modal sequence and cross-modal complementarity. Then, a hierarchical GEU stacker with a pooling-based skip connection is proposed. Unlike the state-of-the-art methods, to obtain a compact and informative representation of multimodal sequences, the GEU stacker gradually compresses the channel <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$d$</tex-math></inline-formula> with multi-modalities <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$m$</tex-math></inline-formula> rather than the temporal dimension <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$t$</tex-math></inline-formula> . Finally, we adopt the connectionist temporal decoding strategy to explore the entire video’s temporal transition and translate the sentence. Extensive experiments on the USTC-CSL and BOSTON-104 datasets demonstrate the effectiveness of the proposed method.

NIMG-23. MACHINE LEARNING METHODS IN GLIOMA GRADE PREDICTION: A SYSTEMATIC REVIEW

Abstract PURPOSE Machine learning (ML) technologies have demonstrated highly accurate prediction of glioma grade, though it is unclear which methods and algorithms are superior. We have conducted a systematic review of the literature in order to identify the ML applications most promising for future research and clinical implementation. MATERIALS AND METHODS A literature review, in agreement with PRISMA, was conducted by a university librarian in October 2020 and verified by a second librarian in February 2021 using four databases: Cochrane trials (CENTRAL), Ovid Embase, Ovid MEDLINE, and Web of Science core-collection. Keywords and controlled vocabulary included artificial intelligence, machine learning, deep learning, radiomics, magnetic resonance imaging, glioma, and glioblastoma. Screening of publications was done in Covidence, and TRIPOD was used for bias assessment. RESULTS The search identified 11,727 candidate articles with 1,135 articles undergoing full text review. 86 articles published since 1995 met the criteria for our study. 79% of the articles were published between 2018 and 2020. The average glioma prediction accuracy of the highest performing model in each study was 90% (range: 53% to 100%). The most common algorithm used for cML studies was Support Vector Machine (SVM) and for DL studies was Convolutional Neural Network (CNN). BRATS and TCIA datasets were used in 47% of the studies, with the average patient number of study datasets being 186 (range: 23 to 662). The average number of features used in machine learning prediction was 55 (range: 2 to 580). Classical machine learning (cML) was the primary machine learning model in 68% of studies, with deep learning (DL) used in 32%. CONCLUSIONS Using multimodal sequences in ML methods delivers significantly higher grading accuracies than single sequences. Potential areas of improvement for ML glioma grade prediction studies include increasing sample size, incorporating molecular subtypes, and validating on external datasets.

Dancing with the sound in edge computing environments

Conventional motion predictions have achieved promising performance. However, the length of the predicted motion sequences of most literatures are short, and the rhythm of the generated pose sequence has rarely been explored. To pursue high quality, rhythmic, and long-term pose sequence prediction, this paper explores a novel dancing with the sound task, which is appealing and challenging in computer vision field. To tackle this problem, a novel model is proposed, which takes the sound as an indicator input and outputs the dancing pose sequence. Specifically, our model is based on the variational autoencoder (VAE) framework, which encodes the continuity and rhythm of the sound information into the hidden space to generate a coherent, diverse, rhythmic and long-term pose video. Extensive experiments validated the effectiveness of audio cues in the generation of dancing pose sequences. Concurrently, a novel dataset of audiovisual multimodal sequence generation has been released to promote the development of this field.

REGOSTA: A randomized, placebo-controlled, double-blinded, multicenter study evaluating the efficacy and safety of regorafenib (REGO) as maintenance therapy after first-line treatment in patients (pts) with osteosarcoma (OS) and non-osteosarcomas (non-OS) of bone (non-Ewing, non-chondrosarcomas and non-chordomas).

TPS11576 Background: Pts with OS and non-OS of bone are treated with a multimodal sequence therapy of neoadjuvant chemotherapy (CT), surgery and adjuvant CT, followed by a close surveillance until recurrence. At recurrence, the prognosis remains poor with objective response rates of 3-29%, and a median Progression-Free Survival (PFS) of less than 4 months in OS. There is a clinical need to reduce the risk of recurrence after the initial treatment sequence. The REGOBONE study reported a significant clinical benefit of regorafenib compared to placebo in patients with relapsed OS (median PFS: 16.4 versus 4.1 weeks). Methods: This multicenter trial is ongoing to study the efficacy and safety of maintenance REGO in pts &gt; = 16 years, with complete remission after initial treatment sequence of their bone sarcoma. 168 pts will be randomly allocated in a 1:1 ratio to receive either oral REGO or its matching placebo (control arm) at a daily dose of 120mg, continuously and for a maximum of 12 months. Randomization will be stratified according to the following risk factors: metastases (mets) at diagnosis and/or poor response to neoadjuvant CT versus no mets at diagnosis and good response to neoadjuvant CT. The primary objective is to compare the efficacy (Relapse-Free Survival) between the 2 arms. The expected 3-year RFS rates are 55% in the control arm and 74.6% in the REGO arm (HR = 0.5). 66 events will provide 80% power to show significant improvement in RFS, using a 2-sided log-rank test at a 5% level. Secondary endpoints include Time to Treatment Failure, Overall Survival, Quality of Life, safety profile, and compliance to treatment. Radiological endpoints will be evaluated using the RECIST 1.1. Translational objectives will be to identify predictive biomarkers for efficacy of REGO as maintenance therapy using liquid biopsies. As of Feb 1st, 2021, 3 patients have been randomized. 15 sites of the French Sarcoma Group will participate. Clinical trial information: NCT04055220.