Features Of Different Modalities Research Articles

Model long-range dependencies for multi-modality and multi-view retinopathy diagnosis through transformers

Early eye examination based on fundus images effectively prevents visual impairment caused by retinopathy. The laborious and error-prone process of interpreting fundus images and the lack of ophthalmologists have driven research toward automated retinopathy diagnosis. However, most previous studies have focused on single-modality fundus images, disregarding the integration of information from multiple views, rendering the results unsatisfactory and inconsistent with clinical practice due to the incomprehensive lesion features and incomplete fundus fields. To address this issue, we introduce multi-modality and multi-view fundus images into the automated retinopathy diagnosis pipeline. In contrast to single fundus images, sequential relationships in multi-modality and multi-view fundus images contain essential long-range dependency information, which is vital for retinopathy diagnosis. Inspired by the recent success of transformers for excavating long-range dependencies in sequence data, in this paper, we propose a transformer-based automated retinopathy diagnosis framework for pathology classification and symptom report generation by integrating multi-modality and multi-view fundus images. Specifically, we present two transformer-based networks to construct long-range dependencies in different fundus images. Moreover, we adopt two novel modules to aggregate features of different modalities and views by modeling long-range dependencies among different fundus image sequences. Experiments are conducted on two in-house datasets, in which each subject provides one color fundus photography image and four-view fundus fluorescein angiography images. The experimental results of retinopathy classification and report generation tasks indicate that our proposed method is superior to other benchmarking methods, achieving a classification accuracy of 85.49% and a report generation BlEU-1 of 0.422.

Hierarchy of Intra- and Cross-modal Redundancy Gains in Visuo-tactile Search: Evidence from the Posterior Contralateral Negativity.

Redundant combination of target features from separable dimensions can expedite visual search. The dimension-weighting account explains these "redundancy gains" by assuming that the attention-guiding priority map integrates the feature-contrast signals generated by targets within the respective dimensions. The present study investigated whether this hierarchical architecture is sufficient to explain the gains accruing from redundant targets defined by features in different modalities, or whether an additional level of modality-specific priority coding is necessary, as postulated by the modality-weighting account (MWA). To address this, we had observers perform a visuo-tactile search task in which targets popped out by a visual feature (color or shape) or a tactile feature (vibro-tactile frequency) as well as any combination of these features. The RT gains turned out larger for visuo-tactile versus visual redundant targets, as predicted by the MWA. In addition, we analyzed two lateralized event-related EEG components: the posterior (PCN) and central (CCN) contralateral negativities, which are associated with visual and tactile attentional selection, respectively. The CCN proved to be a stable somatosensory component, unaffected by cross-modal redundancies. In contrast, the PCN was sensitive to cross-modal redundancies, evidenced by earlier onsets and higher amplitudes, which could not be explained by linear superposition of the earlier CCN onto the later PCN. Moreover, linear mixed-effect modeling of the PCN amplitude and timing parameters accounted for approximately 25% of the behavioral RT variance. Together, these behavioral and PCN effects support the hierarchy of priority-signal computation assumed by the MWA.

Weakly Aligned Multimodal Flame Detection for Fire-Fighting Robots

Flame detection is a key module of fire-fighting robots, especially for autonomous fire suppression. To effectively tackle the fire-fighting tasks, fire-fighting robots are usually equipped with multimodal vision systems. On the one hand, cameras of different modalities can provide complementary visual information. On the other hand, the differences in installation position and resolution between different cameras also result in weakly aligned image pairs, that is, the positions of the same object in different modal images are inconsistent. Directly fusing the image features of different modalities is difficult to meet the accuracy and false alarm requirements of fire-fighting robots. Therefore, we propose a multimodal flame detection model based on projection and attention guidance. First, we use projection to obtain the approximate position of the flame in the thermal image and employ a neighbor sampling module to detect flames around it. Second, we design an attention guidance module based on index matching, which applies the attention map generated by the thermal modality to optimize the regional feature of the color modality. Experiments on multimodal datasets collected by an actual fire-fighting robot validate that the proposed method is effective in both fire and nonfire environments.

Automated multi-modal Transformer network (AMTNet) for 3D medical images segmentation

Objective. Over the past years, convolutional neural networks based methods have dominated the field of medical image segmentation. But the main drawback of these methods is that they have difficulty representing long-range dependencies. Recently, the Transformer has demonstrated super performance in computer vision and has also been successfully applied to medical image segmentation because of the self-attention mechanism and long-range dependencies encoding on images. To the best of our knowledge, only a few works focus on cross-modalities of image segmentation using the Transformer. Hence, the main objective of this study was to design, propose and validate a deep learning method to extend the application of Transformer to multi-modality medical image segmentation. Approach. This paper proposes a novel automated multi-modal Transformer network termed AMTNet for 3D medical image segmentation. Especially, the network is a well-modeled U-shaped network architecture where many effective and significant changes have been made in the feature encoding, fusion, and decoding parts. The encoding part comprises 3D embedding, 3D multi-modal Transformer, and 3D Co-learn down-sampling blocks. Symmetrically, the 3D Transformer block, upsampling block, and 3D-expanding blocks are included in the decoding part. In addition, a Transformer-based adaptive channel interleaved Transformer feature fusion module is designed to fully fuse features of different modalities. Main results. We provide a comprehensive experimental analysis of the Prostate and BraTS2021 datasets. The results show that our method achieves an average DSC of 0.907 and 0.851 (0.734 for ET, 0.895 for TC, and 0.924 for WT) on these two datasets, respectively. These values show that AMTNet yielded significant improvements over the state-of-the-art segmentation networks. Significance. The proposed 3D segmentation network exploits complementary features of different modalities during the feature extraction process at multiple scales to increase the 3D feature representations and improve the segmentation efficiency. This powerful network enriches the research of the Transformer to multi-modal medical image segmentation.

Disentangled Multimodal Representation Learning for Recommendation

Many multimodal recommender systems have been proposed to exploit the rich side information associated with users or items (e.g., user reviews and item images) for learning better user and item representations to improve the recommendation performance. Studies from psychology show that users have individual differences in the utilization of various modalities for organizing information. Therefore, for a certain factor of an item (such as <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">appearance</i> or <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">quality</i> ), the features of different modalities are of varying importance to a user. However, existing methods ignore the fact that different modalities contribute differently towards a user's preference on various factors of an item. In light of this, in this paper, we propose a novel <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Disentangled Multimodal Representation Learning</i> (DMRL) recommendation model, which can capture users' attention to different modalities on each factor in user preference modeling. In particular, we employ a disentangled representation technique to ensure the features of different factors in each modality are independent of each other. A multimodal attention mechanism is then designed to capture users' modality preference for each factor. Based on the estimated weights obtained by the attention mechanism, we make recommendations by combining the preference scores of a user's preferences to each factor of the target item over different modalities. Extensive evaluation on five real-world datasets demonstrate the superiority of our method compared with existing methods.

TSFNet: Triple-Steam Image Captioning

Image captioning is a challenging task that generates a natural language description based on the visual understanding of the given image. Significant region representation is a milestone in image captioning. Despite the great success of existing region-based works, they only focus on the salient objects and encode these objects independently, still plagued by the lack of global contextual information and visual relationships. In fact, the global contextual information and structured visual relationships are exactly the merits of traditional grid features and emerging scene graph features. In this paper, we present a Triple-Steam Feature Fusion Network (TSFNet) to leverage the complementary advantages of the grid, region, and scene graph triple-steam visual representations in image captioning. Concretely, in our TSFNet, a novel Dual-level Attention (DA) mechanism is proposed to simultaneously explore visual intrinsic properties and word-related attributes uniformly of different features. Then attention enhanced features of different modalities are mapped into a joint representation to guide the caption generation. Moreover, we design a new global-aware decoder, which leverages the concatenated representation of triple-steam features and the joint attention representation to obtain global visual guidance information, further refine the complex multimodal reasoning. To verify the effectiveness of our feature fusion model, we perform extensive experiments on the highly competitive MSCOCO dataset to evaluate the model quantitatively and qualitatively. The results illustrate that the proposed framework outperforms many state-of-the-art image captioning approaches in various evaluation metrics, and generates more accurate and abundant captions.

Bridging optical and SAR satellite image time series via contrastive feature extraction for crop classification

Precise crop mapping is crucial for guiding agricultural production, forecasting crop yield, and ensuring food security. Integrating optical and synthetic aperture radar (SAR) satellite image time series (SITS) for crop classification is an essential and challenging task in remote sensing. Previously published studies generally employ a dual-branch network to learn optical and SAR features independently, while ignoring the complementarity and correlation between the two modalities. In this article, we propose a novel method to learn optical and SAR features for crop classification through cross-modal contrastive learning. Specifically, we develop an updated dual-branch network with partial weight-sharing of the two branches to reduce model complexity. Furthermore, we enforce the network to map features of different modalities from the same class to nearby locations in a latent space, while samples from distinct classes are far apart, thereby learning discriminative and modality-invariant features. We conducted a comprehensive evaluation of the proposed method on a large-scale crop classification dataset. Experimental results show that our method consistently outperforms traditional supervised learning approaches, no matter the training samples are adequate or not. Our findings demonstrate that unifying the representations of optical and SAR image time series enables the network to learn more competitive features and suppress inference noise.

Retentive multimodal scale-variable anomaly detection framework with limited data groups for liquid rocket engine

Anomaly detection (AD) plays a significant role in the safe and stable operation of liquid rocket engine (LRE). Facing the fact that LRE operating data are in multiple modalities and extremely scarce, existing methods, which ignore the modal properties, cannot detect anomalies effectively. With a limited number of data groups, they hinder the AD during LRE research and development (R&D) especially in the early stage. A retentive multimodal scale-variable framework is proposed for AD with limited LRE operating data groups. Then the framework utilizes multimodal scale-variable structure to extract and fuse the distinct features of different modalities, which pays more attention to modal properties. To suppress the overfitting and get stronger generalization ability, the framework constructs a memory matrix to fuse it with test data at feature level, which has a retentive memory of samples encountered. To verify the effectiveness of this framework, three experimental cases are designed for research and discussion according to different stages of R&D process. Results show that this framework could recognize the state of LRE with F1-scores over 0.977 in three experimental cases, verifying it has reliable performance and strong potential of AD in the whole R&D cycle of LRE.

Finger Vein and Inner Knuckle Print Recognition Based on Multilevel Feature Fusion Network

Multimodal biometric recognition involves two critical issues: feature representation and multimodal fusion. Traditional feature representation requires complex image preprocessing and different feature-extraction methods for different modalities. Moreover, the multimodal fusion methods used in previous work simply splice the features of different modalities, resulting in an unsatisfactory feature representation. To address these two problems, we propose a Dual-Branch-Net based recognition method with finger vein (FV) and inner knuckle print (IKP). The method combines convolutional neural network (CNN), transfer learning, and triplet loss function to complete feature representation, thereby simplifying and unifying the feature-extraction process of the two modalities. Dual-Branch-Net also achieves deep multilevel fusion of the two modalities’ features. We assess our method on a public FV and IKP homologous multimodal dataset named PolyU-DB. Experimental results show that the proposed method performs best and achieves an equal error rate (EER) of the recognition result of 0.422%.

Local extreme map guided multi-modal brain image fusion.

Multi-modal brain image fusion targets on integrating the salient and complementary features of different modalities of brain images into a comprehensive image. The well-fused brain image will make it convenient for doctors to precisely examine the brain diseases and can be input to intelligent systems to automatically detect the possible diseases. In order to achieve the above purpose, we have proposed a local extreme map guided multi-modal brain image fusion method. First, each source image is iteratively smoothed by the local extreme map guided image filter. Specifically, in each iteration, the guidance image is alternatively set to the local minimum map of the input image and local maximum map of previously filtered image. With the iteratively smoothed images, multiple scales of bright and dark feature maps of each source image can be gradually extracted from the difference image of every two continuously smoothed images. Then, the multiple scales of bright feature maps and base images (i.e., final-scale smoothed images) of the source images are fused by the elementwise-maximum fusion rule, respectively, and the multiple scales of dark feature maps of the source images are fused by the elementwise-minimum fusion rule. Finally, the fused bright feature map, dark feature map, and base image are integrated together to generate a single informative brain image. Extensive experiments verify that the proposed method outperforms eight state-of-the-art (SOTA) image fusion methods from both qualitative and quantitative aspects and demonstrates great application potential to clinical scenarios.

Learning graph-based relationship of dual-modal features towards subject adaptive ASD assessment

Autism Spectrum Disorder (ASD) has been identified as one of the most challenging and intriguing problems in neurodevelopment of children. Recent research suggest that conventional assessment on the basis of explicit behavior observations can be well complemented by evaluation of intrinsic neurophysiological states via analyses of brain imaging data such as electroencephalogram (EEG). However, before any more objective and comprehensive insights for a joint ASD assessment may be obtained, research challenges still remain: (1) how to characterize the interaction relationship of features rooted from recordings in different modalities; and at the same time (2) how to adapt to the individuality of subjects. This study develops a graph-based solution towards individualized assessment of ASD subjects via construction of the relationship of the dual-modal features of eye-tracking recordings and EEG: (1) Relationship Matrix Construction: A shallow encoding module as a variant of Multi-Level Perception (MLP) derives the initial intra- and inter-modal relationship matrix of the features in both modalities; and (2) Graph-based Relationship Learning: A model based on Deep Graph Convolutional Networks (D-GCN) fuses the global information of dual-modal features to learn the final relationship matrix, which is refined in the process of parameter optimization under the regulation of ASD classification. The resulted sample-specific matrices can then be exploited to address the individuality of subjects under examination. Experimental results indicate that: (1) the proposed method is superior to both the single-modal and multi-modal counterparts in ASD classification; (2) it excels in mining hidden connections among features in different modalities in comparison with mainstream methods for correlation measurement; and (3) it holds potentials in mitigating the uncertain variations brought by the individuality of subjects in ASD assessment.

CAR-Net: A Deep Learning-Based Deformation Model for 3D/2D Coronary Artery Registration.

Percutaneous coronary intervention is widely applied for the treatment of coronary artery disease under the guidance of X-ray coronary angiography (XCA) image. However, the projective nature of XCA causes the loss of 3D structural information, which hinders the intervention. This issue can be addressed by the deformable 3D/2D coronary artery registration technique, which fuses the pre-operative computed tomography angiography volume with the intra-operative XCA image. In this study, we propose a deep learning-based neural network for this task. The registration is conducted in a segment-by-segment manner. For each vessel segment pair, the centerlines that preserve topological information are decomposed into an origin tensor and a spherical coordinate shape tensor as network input through independent branches. Features of different modalities are fused and processed for predicting angular deflections, which is a special type of deformation field implying motion and length preservation constraints for vessel segments. The proposed method achieves an average error of 1.13 mm on the clinical dataset, which shows the potential to be applied in clinical practice.

Multi-Level Cross-Modal Semantic Alignment Network for Video–Text Retrieval

This paper strives to improve the performance of video–text retrieval. To date, many algorithms have been proposed to facilitate the similarity measure of video–text retrieval from the single global semantic to multi-level semantics. However, these methods may suffer from the following limitations: (1) largely ignore the relationship semantic which results in semantic levels are insufficient; (2) it is incomplete to constrain the real-valued features of different modalities to be in the same space only through the feature distance measurement; (3) fail to handle the problem that the distributions of attribute labels in different semantic levels are heavily imbalanced. To overcome the above limitations, this paper proposes a novel multi-level cross-modal semantic alignment network (MCSAN) for video–text retrieval by jointly modeling video–text similarity on global, entity, action and relationship semantic levels in a unified deep model. Specifically, both video and text are first decomposed into global, entity, action and relationship semantic levels by carefully designing spatial–temporal semantic learning structures. Then, we utilize KLDivLoss and a cross-modal parameter-share attribute projection layer as statistical constraints to ensure that representations from different modalities in different semantic levels are projected into a common semantic space. In addition, a novel focal binary cross-entropy (FBCE) loss function is presented, which is the first effort to model the unbalanced attribute distribution problem for video–text retrieval. MCSAN is practically effective to take the advantage of the complementary information among four semantic levels. Extensive experiments on two challenging video–text retrieval datasets, namely, MSR-VTT and VATEX, show the viability of our method.

Detecting COVID-19 vaccine hesitancy in India: a multimodal transformer based approach.

COVID-19 has emerged as the greatest threat in recent times, causing extensive mortality and morbidity in the entire world. India is among the highly affected countries suffering severe disruptions due this pandemic. To overcome the adverse effects of COVID-19, vaccination has been identified as the most effective preventive measure globally. However, a growing amount of hesitancy has been observed among the general public regarding the efficacy and possible side-effects of vaccination. Such hesitancy may proved to be the greatest hindrance towards combating this deadly pandemic. This paper introduces a multimodal deep learning method for Indian Twitter user classification, leveraging both content-based and network-based features. To explore the fundamental features of different modalities, improvisations of transformer models, BERT and GraphBERT are utilized to encode the textual and network structure information. The proposed approach thus integrates multiple data representations, utilizing the advances in both transformer based deep learning as well as multimodal learning. Experimental results demonstrates the efficacy of proposed approach over state of the art approaches. Aggregated feature representations from multiple modalities embed additional information that improves the classification results. The findings of the proposed model has been further utilized to perform a study on the dynamics of COVID-19 vaccine hesitancy in India.

BCMF: A bidirectional cross-modal fusion model for fake news detection

In recent years, fake news detection has been a significant task attracting much attention. However, most current approaches utilize the features from a single modality, such as text or image, while the comprehensive fusion between features of different modalities has been ignored. To deal with the above problem, we propose a novel model named Bidirectional Cross-Modal Fusion (BCMF), which comprehensively integrates the textual and visual representations in a bidirectional manner. Specifically, the proposed model is decomposed into four submodules, i.e., the input embedding, the image2text fusion, the text2image fusion, and the prediction module. We conduct intensive experiments on four real-world datasets, i.e., Weibo, Twitter, Politi, and Gossip. The results show 2.2, 2.5, 4.9, and 3.1 percentage points of improvements in classification accuracy compared to the state-of-the-art methods on Weibo, Twitter, Politi, and Gossip, respectively. The experimental results suggest that the proposed model could better capture integrated information of different modalities and has high generalizability among different datasets. Further experiments suggest that the bidirectional fusions, the number of multi-attention heads, and the aggregating function could impact the performance of the cross-modal fake news detection. The research sheds light on the role of bidirectional cross-modal fusion in leveraging multi-modal information to improve the effect of fake news detection.

Progressive multi-scale fusion network for RGB-D salient object detection

Salient object detection (SOD) aims at locating the most significant object within a given image. In recent years, great progress has been made in applying SOD on many vision tasks. The depth map could provide additional spatial prior and boundary cues to boost the performance. Combining the depth information with image data obtained from standard visual cameras has been widely used in recent SOD works, however, introducing depth information in a suboptimal fusion strategy may have negative influence in the performance of SOD. In this paper, we discuss about the advantages of the so-called progressive multi-scale fusion method and propose a mask-guided feature aggregation module (MGFA). The proposed framework can effectively combine the two features of different modalities and, furthermore, alleviate the impact of erroneous depth features, which are inevitably caused by the variation of depth quality. We further introduce a mask-guided refinement module (MGRM) to complement the high-level semantic features and reduce the irrelevant features from multi-scale fusion, leading to an overall refinement of detection. Experiments on five challenging benchmarks demonstrate that the proposed method outperforms 11 state-of-the-art methods under different evaluation metrics.

Enabling modality interactions for RGB-T salient object detection

Most existing RGB and thermal (RGB-T) salient object detection (SOD) techniques focus on investing multi-modality feature fusion strategies for capturing cross-modality complementary information within RGB and thermal images. However, most of these strategies do not allow explicitly extracting the interactions among the features of different modalities, thus leading to insignificant cross-modality complementary information exploitation. In this paper, we propose a novel RGB-T SOD model that alleviates this issue by leveraging a modality-aware and scale-aware feature fusion module. Such a module is capable of capturing the cross-modality complementary information by exploiting the interactions of single-modality features across modalities and the interactions of multi-modality features across scales. A stage-wise feature aggregation module is also proposed to thoroughly exploit the cross-level complementary information and reduce their redundancies for generating accurate saliency maps with sharp boundaries. To this end, a novel multi-level feature aggregation structure with two types of feature aggregation nodes is employed. Experimental results on several benchmark datasets verify the effectiveness and superiorities of our proposed model over some state-of-the-art models.

Semi-supervised cross-modal retrieval with graph-based semantic alignment network

Semi-supervised cross-modal retrieval is an eclectic paradigm which learns common representations via exploiting underlying semantic information from both labeled and unlabeled data. Most existing methods ignore the rich semantic information of text data and are unable to fully utilize the text data in common representation learning. Moreover, they only considered the correlation of the data with the same semantic label, but ignored the correlation between the data with different semantic label. In this paper, we propose a novel semi-supervised cross-modal retrieval method, called Graph-based Semantic Alignment Network (GSAN), which learns common representation by aligning the features of different modalities with semantic embeddings of text data. Firstly, we design a Deep Supervised Semantic Encoding (DSSE) module to train the semantic projector and label predictor which can exploit the semantic embeddings and the predicted labels from unlabeled data of text modality. Then, GAN-based Bidirectional Fusion (GBF) module is designed to learn the mapping networks of two modalities (image and text). In order to make the mapping networks generate semantically discriminative and modality-invariant representations, we utilize the underlying semantic information exploited by DSSE to construct Graph-based Triplet Constraint (GTC) which can enforce feature embeddings from the semantically-matched (image and text) pairs to be more similar and push those mismatched ones away. By the benefit of fully using of semantic information, our approach can only use fewer label data and achieves the performance of state-of-the-art methods. In addition, since we only utilize the mapping networks trained in GBF module to generate common representations in referring stage, our approach is efficient and time saving in real world application. Extensive experiments on four widely-used datasets show the effectiveness of GSAN.

Optical Flow-Aware-Based Multi-Modal Fusion Network for Violence Detection.

Violence detection aims to locate violent content in video frames. Improving the accuracy of violence detection is of great importance for security. However, the current methods do not make full use of the multi-modal vision and audio information, which affects the accuracy of violence detection. We found that the violence detection accuracy of different kinds of videos is related to the change of optical flow. With this in mind, we propose an optical flow-aware-based multi-modal fusion network (OAMFN) for violence detection. Specifically, we use three different fusion strategies to fully integrate multi-modal features. First, the main branch concatenates RGB features and audio features and the optical flow branch concatenates optical flow features with RGB features and audio features, respectively. Then, the cross-modal information fusion module integrates the features of different combinations and applies weights to them to capture cross-modal information in audio and video. After that, the channel attention module extracts valuable information by weighting the integration features. Furthermore, an optical flow-aware-based score fusion strategy is introduced to fuse features of different modalities from two branches. Compared with methods on the XD-Violence dataset, our multi-modal fusion network yields APs that are 83.09% and 1.4% higher than those of the state-of-the-art methods in offline detection, and 78.09% and 4.42% higher than those of the state-of-the-art methods in online detection.

IF-ConvTransformer

Recent advances in sensor based human activity recognition (HAR) have exploited deep hybrid networks to improve the performance. These hybrid models combine Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) to leverage their complementary advantages, and achieve impressive results. However, the roles and associations of different sensors in HAR are not fully considered by these models, leading to insufficient multi-modal fusion. Besides, the commonly used RNNs in HAR suffer from the 'forgetting' defect, which raises difficulties in capturing long-term information. To tackle these problems, an HAR framework composed of an Inertial Measurement Unit (IMU) fusion block and an applied ConvTransformer subnet is proposed in this paper. Inspired by the complementary filter, our IMU fusion block performs multi-modal fusion of commonly used sensors according to their physical relationships. Consequently, the features of different modalities can be aggregated more effectively. Then, the extracted features are fed into the applied ConvTransformer subnet for classification. Thanks to its convolutional subnet and self-attention layers, ConvTransformer can better capture local features and construct long-term dependencies. Extensive experiments on eight benchmark datasets demonstrate the superior performance of our framework. The source code will be published soon.