Deep Multimodal Network Research Articles

A deep multimodal network for multi-task trajectory prediction

Addressing the complexity of multi-task trajectory prediction, this study introduces a novel Deep Multimodal Network (DMN), which integrates a shared feature extractor and a multi-task prediction module with translational encoders to capture both intra-modal and inter-modal dependencies. Unlike traditional models that focus on single-task forecasting, our DMN efficiently and simultaneously predicts multiple trajectory outputs—locations, travel times, and transportation modes. Compared to baseline models including LSTM and Seq2Seq using a real-world dataset, the DMN demonstrates superior performance, reducing the location prediction error by 67% and the travel time error by 69%, while achieving an accuracy of 91. 44% in travel mode prediction. Statistical tests confirm the significance of these enhancements. Ablation studies further validate the critical role of modeling complex dependencies, highlighting the potential of DMN to advance intelligent and sustainable transportation systems.

Multi-Source and Multi-modal Deep Network Embedding for Cross-Network Node Classification

In recent years, to address the issue of networked data sparsity in node classification tasks, cross-network node classification (CNNC) leverages the richer information from a source network to enhance the performance of node classification in the target network, which typically has sparser information. However, in real-world applications, labeled nodes may be collected from multiple sources with multiple modalities (e.g., text, vision, and video). Naive application of single-source and single-modal CNNC methods may result in sub-optimal solutions. To this end, in this article, we propose a model called Multi-source and Multi-modal Cross-network Deep Network Embedding (M 2 CDNE) for cross-network node classification. In M 2 CDNE, we propose a deep multi-modal network embedding approach that combines the extracted deep multi-modal features to make the node vector representations network invariant. In addition, we apply dynamic adversarial adaptation to assess the significance of marginal and conditional probability distributions between each source and target network to make node vector representations label discriminative. Furthermore, we devise to classify nodes in the target network through the related source classifier and aggregate different predictions utilizing respective network weights, corresponding to the discrepancy between each source and target network. Extensive experiments performed on real-world datasets demonstrate that the proposed M 2 CDNE significantly outperforms the state-of-the-art approaches.

MimicNet: Mimicking manual delineation of human expert for brain tumor segmentation from multimodal MRIs

Existing deep neural networks for brain tumor segmentation from multimodal MRIs rely predominantly on standard segmentation architectures, overlooking the underlying rules in clinical scenarios. To address this gap, we propose a novel deep multimodal network that architecturally mimics these rules and fully replicates the process of manually annotating brain tumors by human experts in machines. Three significant manual annotation rules are mimicked. The three sub-tasks are introduced first to segment three regions sequentially. We incorporate a fine-grained modality attention module in each sub-task to mimic region-aware multimodal fusion. Finally, we propose the deep feature propagation module for multi-scale context exploitation and repurpose deep cascaded attention for location information exploitation. These exploits imitate earlier segmentation knowledge exploitation. We use a new curriculum loss to train the nested network and a stratified training approach to reduce the intramodality domain shift caused by the dataset’s distribution. The proposed approach is evaluated and compared to state-of-the-art methods on three public datasets: BraTS2018, BraTS2019 and BraTS2020. The Dice similarity coefficient on Brats2020 is 0.800,0.930 and 0.857 for enhancing tumor, whole tumor, and tumor core, respectively, ranking sixth top among 285 participating methods. These findings suggest the potential applicability of our approach in brain tumor segmentation from multimodal MRIs.

Deep Multimodal Networks for M-type Star Classification with Paired Spectrum and Photometric Image

Traditional stellar classification methods include spectral and photometric classification separately. Although satisfactory results can be achieved, the accuracy could be improved. In this paper, we pioneer a novel approach to deeply fuse the spectra and photometric images of the sources in an advanced multimodal network to enhance the model’s discriminatory ability. We use Transformer as the fusion module and apply a spectrum–image contrastive loss function to enhance the consistency of the spectrum and photometric image of the same source in two different feature spaces. We perform M-type stellar subtype classification on two data sets with high and low signal-to-noise ratio (S/N) spectra and corresponding photometric images, and the F1-score achieves 95.65% and 90.84%, respectively. In our experiments, we prove that our model effectively utilizes the information from photometric images and is more accurate than advanced spectrum and photometric image classifiers. Our contributions can be summarized as follows: (1) We propose an innovative idea for stellar classification that allows the model to simultaneously consider information from spectra and photometric images. (2) We discover the challenge of fusing low-S/N spectra and photometric images in the Transformer and provide a solution. (3) The effectiveness of Transformer for spectral classification is discussed for the first time and will inspire more Transformer-based spectral classification models.

Gated-Residual Block for Semantic Segmentation Using RGB-D Data

Semantic segmentation is an important technique for scene understanding in the intelligent transportation system. RGB-D data shows great advantages over the unimodal data in this area, and it can be easily obtained from consumer sensors nowadays. How to design effective fusion structures to fuse RGB and depth signals in RGB-D data is a challenging problem. This paper proposes a novel gated-residual block to address this problem. The structure consists of two residual units and one gated fusion unit, the residual unit progressively aggregates modality-specific features from the modality-specific signals and the gate mechanism computes complementary features for them. Based on the gated-residual block, the paper presents the deep multimodal networks, named GRBNet, for RGB-D semantic segmentation. Experiments on ScanNet, Cityscapes and SUN RGB-D datasets verify the effectiveness of the proposed approach and demonstrate that the GRBNet achieved competitive performance.

A deep multimodal network based on bottleneck layer features fusion for action recognition

A deep multimodal network based on bottleneck layer features fusion for action recognition

Sending or not? A multimodal framework for Danmaku comment prediction

Danmaku is an emerging comment design for videos that allows real-time, interactive comments from viewers. Danmaku increases viewers’ interaction with other viewers and streamers, thereby raising viewers’ loyalty and sense of belonging. Sending Danmaku comments demonstrates a higher degree of viewer involvement than traditional static comments below the videos. Therefore, it is necessary and meaningful to learn about viewers’ preferences by observing their behavior, as this may benefit the platform as well as the streamers. However, research on how the multimodal environment affects viewers’ behavior in sending Danmaku comments is quite limited. To fill this gap, we propose a new dataset and a deep neural network integrating multimodal information to predict whether viewers will send Danmaku comments (Deep Multimodal network for Danmaku Forecasting, DMDF) in order to evaluate the impact of the interaction of textual features, audio features and visual features on the behavior of viewers sending Danmaku comments. A series of experimental results based on a real dataset of 249657 samples from Bilibili (a leading Chinese video streaming Website) demonstrate the effectiveness of the proposed DMDF and the helpfulness of all modalities, especially visual and acoustic features, in behavior forecasting. DMDF with the multimodal squeeze-and-excitation (MSE) module we proposed achieves 90.14% on accuracy and 83.60% on F1-score, and it reveals the extent to which a user-generated video can influence viewers to send Danmaku comments, which helps predict viewers’ online viewing behavior. Furthermore, our model contributes to the current work on the video understanding task.

Hierarchical deep multi-modal network for medical visual question answering

Visual Question Answering in Medical domain (VQA-Med) plays an important role in providing medical assistance to the end-users. These users are expected to raise either a straightforward question with a Yes/No answer or a challenging question that requires a detailed and descriptive answer. The existing techniques in VQA-Med fail to distinguish between the different question types sometimes complicates the simpler problems, or over-simplifies the complicated ones. It is certainly true that for different question types, several distinct systems can lead to confusion and discomfort for the end-users. To address this issue, we propose a hierarchical deep multi-modal network that analyzes and classifies end-user questions/queries and then incorporates a query-specific approach for answer prediction. We refer our proposed approach as Hierarchical Question Segregation based Visual Question Answering, in short HQS-VQA. Our contributions are three-fold, viz. firstly, we propose a question segregation (QS) technique for VQA-Med; secondly, we integrate the QS model to the hierarchical deep multi-modal neural network to generate proper answers to the queries related to medical images; and thirdly, we study the impact of QS in Medical-VQA by comparing the performance of the proposed model with QS and a model without QS. We evaluate the performance of our proposed model on two benchmark datasets, viz. RAD and CLEF18. Experimental results show that our proposed HQS-VQA technique outperforms the baseline models with significant margins. We also conduct a detailed quantitative and qualitative analysis of the obtained results and discover potential causes of errors and their solutions.

Deep Top-$k$ Ranking for Image–Sentence Matching

Image-sentence matching is a challenging task for the heterogeneity-gap between different modalities. Ranking-based methods have achieved excellent performance in this task in past decades. Given an image query, these methods typically assume that the correct matched image-sentence pair must rank before all other mismatched ones. However, this assumption may be too strict and prone to the overfitting problem, especially when some sentences in a massive database are similar and confusable with one another. In this paper, we relax the traditional ranking loss and propose a novel deep multi-modal network with a top-k ranking loss to mitigate the data ambiguity problem. With this strategy, query results will not be penalized unless the index of ground truth is outside the range of top-k query results. Considering the non-smoothness and non-convexity of the initial top-k ranking loss, we exploit a tight convex upper bound to approximate the loss and then utilize the traditional back-propagation algorithm to optimize the deep multi-modal network. Finally, we apply the method on three benchmark datasets, namely, Flickr8k, Flickr30k, and MSCOCO. Empirical results on metrics R@K (K = 1, 5, 10) show that our method achieves comparable performance in comparison to state-of-the-art methods.

Recognizing and Presenting the Storytelling Video Structure With Deep Multimodal Networks

In this paper, we propose a novel scene detection algorithm which employs semantic, visual, textual, and audio cues. We also show how the hierarchical decomposition of the storytelling video structure can improve retrieval results presentation with semantically and aesthetically effective thumbnails. Our method is built upon two advancements of the state of the art: first is semantic feature extraction which builds video-specific concept detectors; and second is multimodal feature embedding learning that maps the feature vector of a shot to a space in which the Euclidean distance has task specific semantic properties. The proposed method is able to decompose the video in annotated temporal segments which allow us for a query specific thumbnail extraction. Extensive experiments are performed on different data sets to demonstrate the effectiveness of our algorithm. An in-depth discussion on how to deal with the subjectivity of the task is conducted and a strategy to overcome the problem is suggested.