Features Of Different Modalities Research Articles

Contrastive cross-modal clustering with twin network

Cross-modal clustering (CMC) methods explore the correlation information between multiple modalities to improve clustering performance. However, the obvious differences between heterogeneous modalities make it difficult to obtain the correlation information directly. In this paper, we propose a novel Contrastive Cross-modal Clustering with Twin Network (3CTnet) for CMC, which contrasts the differences of multiple modalities to fully mine the correlation information. The 3CTnet contains two modal-special encoders and an attention-based correlation propagate module (CPM). First, the modal-special encoders are trained by pseudo-labels to learn the clustering structure and feature of single modality. Then we contrast the clustering structures and features of different modalities to explore the inter-cluster and inter-feature correlation information simultaneously. Finally, the CPM is designed to propagate the learned correlation information among modal-special encoders to further optimize the learning of features and clustering structures. The experiments show that 3CTnet outperforms the state-of-the-art CMC methods on six large datasets.

DMFF: Deep multimodel feature fusion for building occupancy detection

The 2022 Global Status Report for Buildings and Construction (Buildings-GSR) indicates that construction activities have returned to pre-pandemic levels in most major economies, alongside more building energy consumption. To achieve the Net Zero emissions target by 2050, particularly in the post-pandemic era, accurate occupancy information is important to enhance building energy efficiency and improve occupancy comfort. While remarkable progress has been made in existing studies, they struggle to make full use of multi-sensor data to achieve high accuracy. Furthermore, there is a high expectation of the multimodel multi-temporary fusion by Transformer. In this study, we present a Transformer-based multimodal, multi-temporal feature fusion method (DMFF) for occupancy detection. To transfer domain knowledge from artificial intelligence into the building area, DMFF includes a pretrain-finetune pipeline and leverages pre-trained visual and sound models. Multiple pre-trained Transformer encoders are employed to extract features of different modalities. Then, we propose a self-attention mechanism for modality fusion to learn relationships among various sensors. Our DMFF method demonstrates superior performance on a real dataset, outperforming machine and deep learning methods (e.g., Convolutional Neural Networks, Random Forest, and Multilayer Perceptrons). Applied to a room setting, DMFF shows promising potential for building energy savings. The code and demo are accessible at https://github.com/kailaisun/multimodel_occupancy.

An object detection algorithm based on infrared-visible dual modal feature fusion

Dual modal image pairs can provide complementary feature information and overcome the limitations of single modal object detection algorithms, thus improving detection performance. In order to fully utilize the features of different modalities, this paper proposes a dual-branch fusion detection network that can simultaneously input both infrared and visible modalities for object detection. The method is based on YOLOv5-s, and two attention modules are proposed according to the characteristics of different modalities to enhance the expression capability of infrared and visible image feature information respectively; At the same time, a dual modal fusion module is designed for cross-modal information complementarity by fusing features from corresponding scales of both modalities; Finally, a feature enhancement module is proposed to improve the multi-scale feature fusion capability. The algorithm is validated on the KAIST, FLIR, and GIR datasets and compared with classic single modal and dual modal detection algorithms. Experimental results show that, compared with the baseline algorithm YOLOv5-s alone detecting visible and infrared images separately, the proposed algorithm improves the detection accuracy by 17.6% and 5.9% on the KAIST dataset respectively, and 19.1% and 13% on the FLIR dataset respectively. The proposed algorithm also has a significant advantage in detection accuracy on the self-built GIR dataset.

Seeking a Hierarchical Prototype for Multimodal Gesture Recognition.

Gesture recognition has drawn considerable attention from many researchers owing to its wide range of applications. Although significant progress has been made in this field, previous works always focus on how to distinguish between different gesture classes, ignoring the influence of inner-class divergence caused by gesture-irrelevant factors. Meanwhile, for multimodal gesture recognition, feature or score fusion in the final stage is a general choice to combine the information of different modalities. Consequently, the gesture-relevant features in different modalities may be redundant, whereas the complementarity of modalities is not exploited sufficiently. To handle these problems, we propose a hierarchical gesture prototype framework to highlight gesture-relevant features such as poses and motions in this article. This framework consists of a sample-level prototype and a modal-level prototype. The sample-level gesture prototype is established with the structure of a memory bank, which avoids the distraction of gesture-irrelevant factors in each sample, such as the illumination, background, and the performers' appearances. Then the modal-level prototype is obtained via a generative adversarial network (GAN)-based subnetwork, in which the modal-invariant features are extracted and pulled together. Meanwhile, the modal-specific attribute features are used to synthesize the feature of other modalities, and the circulation of modality information helps to leverage their complementarity. Extensive experiments on three widely used gesture datasets demonstrate that our method is effective to highlight gesture-relevant features and can outperform the state-of-the-art methods.

A Multi-classification Accessment Framework for Reproducible Evaluation of Multimodal Learning in Alzheimer's Disease.

Multimodal learning is widely used in automated early diagnosis of Alzheimer's disease. However, the current studies are based on an assumption that different modalities can provide more complementary information to help classify the samples from the public dataset Alzheimer's Disease Neuroimaging Initiative (ADNI). In addition, the combination of modalities and different tasks are external factors that affect the performance of multimodal learning. Above all, we summrise three main problems in the early diagnosis of Alzheimer's disease: (i) unimodal vs multimodal; (ii) different combinations of modalities; (iii) classification of different tasks. In this paper, to experimentally verify these three problems, a novel and reproducible multi-classification framework for Alzheimer's disease early automatic diagnosis is proposed to evaluate and verify the above issues. The multi-classification framework contains four layers, two types of feature representation methods, and two types of models to verify these three issues. At the same time, our framework is extensible, that is, it is compatible with new modalities generated by new technologies. Following that, a series of experiments based on the ADNI-1 dataset are conducted and some possible explanations for the early diagnosis of Alzheimer's disease are obtained through multimodal learning. Experimental results show that SNP has the highest accuracy rate of 57.09% in the early diagnosis of Alzheimer's disease. In the modality combination, the addition of Single Nucleotide Polymorphism modality improves the multi-modal machine learning performance by 3% to 7%. Furthermore, we analyse and discuss the most related Region of Interest and Single Nucleotide Polymorphism features of different modalities.

Pattern recognition study of musical rhythm perception based on multimodal deep learning

Abstract Rhythm perception is becoming more and more important in the field of music information processing and music understanding. The study first adopts signal processing methods to extract musical features, then uses feature fusion techniques to integrate features of different modalities into a single feature vector. Based on this model, the study identifies the rhythmic activation function of music and combines it with the hidden Markov model to infer the rhythm of the music. One of the key points of the study is to perform rhythm recognition on music containing drums, to explore the recognition effect. One of the focuses of the study is to recognize the rhythm of music containing drums to explore the recognition effect.In addition, the study also analyzes the Softmax output values of the music and compares the recognition effect of different models. The results show that the rhythm recognition of music using the multimodal deep learning method performs the best in terms of the F-Measure value, the Cemgil value, the Goto value, and the P-score value, with the respective 65.65%, 66.76%, 36.75%, and 36.75%. 66.76%, 36.75%, and 75.68%.Especially in the drum music recognition, the position of each drum music is accurately recognized, proving the model’s effectiveness in this paper. The research provides a new feasible method for the recognition and understanding of music rhythms and a valuable reference for the research in this field.

RFTNet: Region–Attention Fusion Network Combined with Dual-Branch Vision Transformer for Multimodal Brain Tumor Image Segmentation

Brain tumor image segmentation plays a significant auxiliary role in clinical diagnosis. Recently, deep learning has been introduced into multimodal segmentation tasks, which construct various Convolutional Neural Network (CNN) structures to achieve excellent performance. However, most CNN-based segmentation methods have poor capability for global feature extraction. Transformer is good at modeling long-distance dependencies, but it can cause local information loss and usually has a high computational complexity. In addition, it is difficult to fully exploit the brain tumor features of different modalities. To address these issues, in this paper, we propose a region–attention fusion (RAF) network that combines a dual-branch vision Transformer (DVT), called RFTNet. In RFTNet, the DVT is exploited to capture the delicate local information and global semantics separately by two branches. Meanwhile, a novel RAF is employed to effectively fuse the images of the different modalities. Finally, we design a new hybrid loss function, called region-mixed loss function (RML) to calculate the importance of each pixel and solve the problem of class imbalance. The experiments on BrasTS2018 and BraTS2020 datasets show that our method obtains a higher segmentation accuracy than other models. Furthermore, ablation experiments prove the effectiveness of each key component in RFTNet.

DRCM: a disentangled representation network based on coordinate and multimodal attention for medical image fusion.

Recent studies on medical image fusion based on deep learning have made remarkable progress, but the common and exclusive features of different modalities, especially their subsequent feature enhancement, are ignored. Since medical images of different modalities have unique information, special learning of exclusive features should be designed to express the unique information of different modalities so as to obtain a medical fusion image with more information and details. Therefore, we propose an attention mechanism-based disentangled representation network for medical image fusion, which designs coordinate attention and multimodal attention to extract and strengthen common and exclusive features. First, the common and exclusive features of each modality were obtained by the cross mutual information and adversarial objective methods, respectively. Then, coordinate attention is focused on the enhancement of the common and exclusive features of different modalities, and the exclusive features are weighted by multimodal attention. Finally, these two kinds of features are fused. The effectiveness of the three innovation modules is verified by ablation experiments. Furthermore, eight comparison methods are selected for qualitative analysis, and four metrics are used for quantitative comparison. The values of the four metrics demonstrate the effect of the DRCM. Furthermore, the DRCM achieved better results on SCD, Nabf, and MS-SSIM metrics, which indicates that the DRCM achieved the goal of further improving the visual quality of the fused image with more information from source images and less noise. Through the comprehensive comparison and analysis of the experimental results, it was found that the DRCM outperforms the comparison method.

Multi-Sensor Data Fusion Method Based on Self-Attention Mechanism

In 3D reconstruction tasks, single-sensor data fusion based on deep learning is limited by the integrity and accuracy of the data, which reduces the accuracy and reliability of the fusion results. To address this issue, this study proposes a multi-sensor data fusion method based on a self-attention mechanism. A multi-sensor data fusion model for acquiring multi-source and multi-modal data is constructed, with the core component being a convolutional neural network with self-attention (CNN-SA), which employs CNNs to process multi-source and multi-modal data by extracting their features. Additionally, it introduces an SA mechanism to weigh and sum the features of different modalities, adaptively focusing on the importance of different modal data. This enables mutual support, complementarity, and correction among the multi-modal data. Experimental results demonstrate that the accuracy of the CNN-SA network is improved by 72.6%, surpassing the improvements of 29.9% for CNN-CBAM, 23.6% for CNN, and 11.4% for CNN-LSTM, exhibiting enhanced generalization capability, accuracy, and robustness. The proposed approach will contribute to the effectiveness of multi-sensor data fusion processing.

Graph-Based Progressive Fusion Network for Multi-Modality Vehicle Re-Identification

Vehicle re-identification (Re-ID) is a critical task in intelligent transportation, aiming to match vehicle images of the same identity captured by non-overlapping cameras. However, it is difficult to achieve satisfactory results based on RGB images alone in darkness. Therefore, it is of great importance to consider multi-modality vehicle re-identification. Currently, the proposed works deal with different modality features through direct summation and fusion based on heat map, which however ignores the relationship between them. Meanwhile, there is a huge gap between the different modalities, which needs to be reduced. In this paper, to solve the above two problems, we propose a Graph-based Progressive Fusion Network (GPFNet) using a graph convolutional network to adaptively fuse multi-modality features in an end-to-end learning framework. GPFNet consists of a CNN feature extraction module (FEM), a GCN feature fusion module (FFM), and a loss function module (LFM). Firstly, in FEM, we employ a multi-stream network architecture to extract single-modality features and common-modality features and employ a random modality substitution module to extract mixed-modality features. Secondly, in FFM, we design an efficient graph structure to associate the features of different modalities and adopt a progressive two-stage strategy to fuse them. Finally, in LFM, we use GCN-aware multi-modality loss to constrain the features. For reducing modality differences and contributing better initial mixed-modality features to FFM, we propose random modality substitution as a data enhancement method for multi-modality datasets. Extensive experiments on multi-modality vehicle Re-ID datasets RGBN300 and RGBNT100 show that our model achieves state-of-the-art performance.

Co-learning-assisted progressive dense fusion network for cardiovascular disease detection using ECG and PCG signals

Electrocardiograms (ECGs) and phonocardiograms (PCGs) are two modalities to provide complementary diagnostic information for improving the early detection accuracy of cardiovascular diseases (CVDs). Existing multi-modality methods mainly used the early or late feature fusion strategy, which did not simultaneously utilize the complementary information contained in low-level detail features and high-level semantic features of different modalities. Meanwhile, they were specially designed for the multi-modality scenario with both ECGs and PCGs, without considering the missing-modality scenarios with only ECGs or PCGs in clinical practice. To address these challenges, we developed a Co-learning-assisted Progressive Dense fusion network (CPDNet) for end-to-end CVD detection, with a three-branch interweaving architecture consisting of ECG and PCG modality-specific encoders and a progressive dense fusion encoder, which could be used for both multi-modality and missing-modality scenarios. Specifically, we designed a novel progressive dense fusion strategy, which not only progressively fused multi-level complementary information of different modalities from low-level details to high-level semantics, but also employed the dense fusion during feature fusion at each level to further enrich available multi-modality information through mutual guidance of features at different levels. Meanwhile, the strategy integrated cross-modality region-aware and multi-scale feature optimization modules to fully evaluate the contributions of different modalities and signal regions and enhance the feature extraction ability of the network for multi-scale target regions. Moreover, we designed a novel co-learning strategy to guide the learning process of the CPDNet by combining intra-modality and joint losses, which made each encoder well-trained. This strategy could not only assist our fusion strategy by making modality-specific encoders provide sufficiently discriminative features for the fusion encoder, but also enable the CPDNet to robustly handle missing-modality scenarios by independently using the corresponding modality-specific encoder. Experimental results on public and private datasets demonstrated that our method not only outperformed state-of-the-art multi-modality methods by at least 5.05% for average accuracy in the multi-modality scenario, but also achieved better performance than single-modality models in the missing-modality scenarios.

TranSiam: Aggregating multi-modal visual features with locality for medical image segmentation

Automatic segmentation of medical images plays an important role in the diagnosis of diseases. On single-modal data, convolutional neural networks have demonstrated satisfactory performance. However, multi-modal data encompasses a greater amount of information rather than single-modal data. Multi-modal data can be effectively used to improve the segmentation accuracy of regions of interest by analyzing both spatial and temporal information. In this study, we propose a dual-path segmentation model for multi-modal medical images, named TranSiam. Taking into account that there is a significant diversity between the different modalities, TranSiam employs two parallel CNNs to extract the features which are specific to each of the modalities. In our method, two parallel CNNs extract detailed and local information in the low-level layer, and the Transformer layer extracts global information in the high-level layer. Finally, we fuse the features of different modalities via a locality-aware aggregation block (LAA block) to establish the association between different modal features. The LAA block is used to locate the region of interest and suppress the influence of invalid regions on multi-modal feature fusion. TranSiam uses LAA blocks at each layer of the encoder in order to fully fuse multi-modal information at different scales. Extensive experiments on several multi-modal datasets have shown that TranSiam achieves satisfying results.

Understanding public opinions on Chinese short video platform by multimodal sentiment analysis using deep learning-based techniques

PurposeWith the rapid development of short videos in China, the public has become accustomed to using short videos to express their opinions. This paper aims to solve problems such as how to represent the features of different modalities and achieve effective cross-modal feature fusion when analyzing the multi-modal sentiment of Chinese short videos (CSVs).Design/methodology/approachThis paper aims to propose a sentiment analysis model MSCNN-CPL-CAFF using multi-scale convolutional neural network and cross attention fusion mechanism to analyze the CSVs. The audio-visual and textual data of CSVs themed on “COVID-19, catering industry” are collected from CSV platform Douyin first, and then a comparative analysis is conducted with advanced baseline models.FindingsThe sample number of the weak negative and neutral sentiment is the largest, and the sample number of the positive and weak positive sentiment is relatively small, accounting for only about 11% of the total samples. The MSCNN-CPL-CAFF model has achieved the Acc-2, Acc-3 and F1 score of 85.01%, 74.16 and 84.84%, respectively, which outperforms the highest value of baseline methods in accuracy and achieves competitive computation speed.Practical implicationsThis research offers some implications regarding the impact of COVID-19 on catering industry in China by focusing on multi-modal sentiment of CSVs. The methodology can be utilized to analyze the opinions of the general public on social media platform and to categorize them accordingly.Originality/valueThis paper presents a novel deep-learning multimodal sentiment analysis model, which provides a new perspective for public opinion research on the short video platform.

Multiple Graph Affinity Interactive Network and a Variable Illumination Dataset for RGBT Image Salient Object Detection

Salient object detection (SOD) of images refers to simulating the attention mechanism of human vision to capture the most attractive objects in an image. Current SOD mainly relies on RGB images captured by optical cameras. However, existing optical cameras are not comparable to the human visual system, especially in poorly illumination scenes. Our human visual system is able to resolve scenes well in low light conditions, while optical cameras can barely image without enough illumination. To make machine vision closer to the imaging of the human eye, we propose to use thermal infrared (T) images to compensate RGB images and build a variable illumination RGBT dataset named VI-RGBT1500 for SOD. This dataset is collected under three different illumination conditions including sufficient illumination, uneven illumination and insufficient illumination to fully demonstrate the superiority of the RGBT image combination. Furthermore, we propose a multiple graph affinity interactive (MGAI) network to validate the proposed dataset. Our network structure is simple using only the MGAI to fuse the features of different modalities. Meanwhile, the MGAI model highlights valuable information during the interaction, which facilitates feature representation under variable illumination. The proposed VI-RGBT1500 dataset and three publicly available RGBT SOD datasets are used for the comparison experiments, and the results with the state-of-the-art methods prove that our VI-RGBT1500 dataset is valuable and the performance of the MGAI network is competitive. The VI-RGBT1500 dataset and the MGAI network are available at: https://github.com/huanglm-me/VI-RGBT1500.

Exploring the potential of Siamese network for RGBT object tracking

Siamese tracking is one of the most promising object tracking methods today due to its balance of performance and speed. However, it still performs poorly when faced with some challenges such as low light or extreme weather. This is caused by the inherent limitations of visible images, and a common way to cope with it is to introduce infrared data as an aid to improve the robustness of tracking. However, most of the existing RGBT trackers are variants of MDNet (Hyeonseob Nam and Bohyung Han, Learning multi-domain convolutional neural networks for visual tracking, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 4293–4302.), which have significant limitations in terms of operational efficiency. On the contrary, the potential of Siamese tracking in the field of RGBT tracking has not been effectively exploited due to the reliance on large-scale training data. To solve this dilemma, in this paper, we propose an end-to-end Siamese RGBT tracking framework that is based on cross-modal feature enhancement and self-attention (SiamFEA). We draw on the idea of migration learning and employ local fine-tuning to reduce the dependence on large-scale RGBT data and verify the feasibility of this approach, and then we propose a reliable fusion approach to efficiently fuse the features of different modalities. Specifically, we first propose a cross-modal feature enhancement module to exploit the complementary properties of dual-modality, followed by capturing non-local attention in channel and spatial dimensions for adaptive weighted fusion, respectively. Our network was trained end-to-end on the LasHeR (Chenglong Li, Wanlin Xue, Yaqing Jia, Zhichen Qu, Bin Luo, Jin Tang, LasHeR: A Large-scale High-diversity Benchmark for RGBT Tracking, CoRR abs/2104.13202, 2021) training set and reached new SOTAs on GTOT (C. Li, H. Cheng, S. Hu, X. Liu, J. Tang, L. Lin, Learning collaborative sparse representation for grayscale-thermal tracking, IEEE Trans. Image Process, 25 (12) (2016) 5743–5756.), RGBT234 (C. Li, X. Liang, Y. Lu, N. Zhao, and J. Tang, “Rgb-t object tracking: Benchmark and baseline,” Pattern Recognition, vol. 96, p. 106977, 2019.), and LasHeR (Chenglong Li, Wanlin Xue, Yaqing Jia, Zhichen Qu, Bin Luo, Jin Tang, LasHeR: A Large-scale High-diversity Benchmark for RGBT Tracking, CoRR abs/2104.13202, 2021) while running in real-time.

Multi-antenna intelligent spectrum sensing in the presence of non-Gaussian interference

The secondary user (SU) only detects one feature of the primary user (PU) signal in conventional spectrum sensing. However, single feature detection does not fully explore the PU information. Meanwhile, non-Gaussian interference will also affect the spectrum sensing performance. In this paper, we propose a novel spectrum sensing method using multimodal fusion and convolutional neural networks (CNN). First, the received signal at multiple antennas is preprocessed, where the generalized covariance matrix and the generalized Wigner-Ville distribution are used to characterize two different modes of the received signals as the input of CNN. Then, a CNN model using multimodal fusion is constructed. In this model, the attention mechanism is first used to extract features of different modalities, and the multimodal fusion technique fuses the extracted features to obtain the feature vector. Finally, the detection statistics and detection thresholds for spectrum sensing are constructed and compared using the output feature vectors of CNN. Simulation results show that the performance of multi-mode fusion spectrum sensing is better than the single-mode counterpart. This method can effectively realize spectrum sensing with both Gaussian noise and non-Gaussian interference, and it has a high detection probability at low SNR.

Model long-range dependencies for multi-modality and multi-view retinopathy diagnosis through transformers

Early eye examination based on fundus images effectively prevents visual impairment caused by retinopathy. The laborious and error-prone process of interpreting fundus images and the lack of ophthalmologists have driven research toward automated retinopathy diagnosis. However, most previous studies have focused on single-modality fundus images, disregarding the integration of information from multiple views, rendering the results unsatisfactory and inconsistent with clinical practice due to the incomprehensive lesion features and incomplete fundus fields. To address this issue, we introduce multi-modality and multi-view fundus images into the automated retinopathy diagnosis pipeline. In contrast to single fundus images, sequential relationships in multi-modality and multi-view fundus images contain essential long-range dependency information, which is vital for retinopathy diagnosis. Inspired by the recent success of transformers for excavating long-range dependencies in sequence data, in this paper, we propose a transformer-based automated retinopathy diagnosis framework for pathology classification and symptom report generation by integrating multi-modality and multi-view fundus images. Specifically, we present two transformer-based networks to construct long-range dependencies in different fundus images. Moreover, we adopt two novel modules to aggregate features of different modalities and views by modeling long-range dependencies among different fundus image sequences. Experiments are conducted on two in-house datasets, in which each subject provides one color fundus photography image and four-view fundus fluorescein angiography images. The experimental results of retinopathy classification and report generation tasks indicate that our proposed method is superior to other benchmarking methods, achieving a classification accuracy of 85.49% and a report generation BlEU-1 of 0.422.

Hierarchy of Intra- and Cross-modal Redundancy Gains in Visuo-tactile Search: Evidence from the Posterior Contralateral Negativity.

Redundant combination of target features from separable dimensions can expedite visual search. The dimension-weighting account explains these "redundancy gains" by assuming that the attention-guiding priority map integrates the feature-contrast signals generated by targets within the respective dimensions. The present study investigated whether this hierarchical architecture is sufficient to explain the gains accruing from redundant targets defined by features in different modalities, or whether an additional level of modality-specific priority coding is necessary, as postulated by the modality-weighting account (MWA). To address this, we had observers perform a visuo-tactile search task in which targets popped out by a visual feature (color or shape) or a tactile feature (vibro-tactile frequency) as well as any combination of these features. The RT gains turned out larger for visuo-tactile versus visual redundant targets, as predicted by the MWA. In addition, we analyzed two lateralized event-related EEG components: the posterior (PCN) and central (CCN) contralateral negativities, which are associated with visual and tactile attentional selection, respectively. The CCN proved to be a stable somatosensory component, unaffected by cross-modal redundancies. In contrast, the PCN was sensitive to cross-modal redundancies, evidenced by earlier onsets and higher amplitudes, which could not be explained by linear superposition of the earlier CCN onto the later PCN. Moreover, linear mixed-effect modeling of the PCN amplitude and timing parameters accounted for approximately 25% of the behavioral RT variance. Together, these behavioral and PCN effects support the hierarchy of priority-signal computation assumed by the MWA.

Weakly Aligned Multimodal Flame Detection for Fire-Fighting Robots

Flame detection is a key module of fire-fighting robots, especially for autonomous fire suppression. To effectively tackle the fire-fighting tasks, fire-fighting robots are usually equipped with multimodal vision systems. On the one hand, cameras of different modalities can provide complementary visual information. On the other hand, the differences in installation position and resolution between different cameras also result in weakly aligned image pairs, that is, the positions of the same object in different modal images are inconsistent. Directly fusing the image features of different modalities is difficult to meet the accuracy and false alarm requirements of fire-fighting robots. Therefore, we propose a multimodal flame detection model based on projection and attention guidance. First, we use projection to obtain the approximate position of the flame in the thermal image and employ a neighbor sampling module to detect flames around it. Second, we design an attention guidance module based on index matching, which applies the attention map generated by the thermal modality to optimize the regional feature of the color modality. Experiments on multimodal datasets collected by an actual fire-fighting robot validate that the proposed method is effective in both fire and nonfire environments.

Automated multi-modal Transformer network (AMTNet) for 3D medical images segmentation

Objective. Over the past years, convolutional neural networks based methods have dominated the field of medical image segmentation. But the main drawback of these methods is that they have difficulty representing long-range dependencies. Recently, the Transformer has demonstrated super performance in computer vision and has also been successfully applied to medical image segmentation because of the self-attention mechanism and long-range dependencies encoding on images. To the best of our knowledge, only a few works focus on cross-modalities of image segmentation using the Transformer. Hence, the main objective of this study was to design, propose and validate a deep learning method to extend the application of Transformer to multi-modality medical image segmentation. Approach. This paper proposes a novel automated multi-modal Transformer network termed AMTNet for 3D medical image segmentation. Especially, the network is a well-modeled U-shaped network architecture where many effective and significant changes have been made in the feature encoding, fusion, and decoding parts. The encoding part comprises 3D embedding, 3D multi-modal Transformer, and 3D Co-learn down-sampling blocks. Symmetrically, the 3D Transformer block, upsampling block, and 3D-expanding blocks are included in the decoding part. In addition, a Transformer-based adaptive channel interleaved Transformer feature fusion module is designed to fully fuse features of different modalities. Main results. We provide a comprehensive experimental analysis of the Prostate and BraTS2021 datasets. The results show that our method achieves an average DSC of 0.907 and 0.851 (0.734 for ET, 0.895 for TC, and 0.924 for WT) on these two datasets, respectively. These values show that AMTNet yielded significant improvements over the state-of-the-art segmentation networks. Significance. The proposed 3D segmentation network exploits complementary features of different modalities during the feature extraction process at multiple scales to increase the 3D feature representations and improve the segmentation efficiency. This powerful network enriches the research of the Transformer to multi-modal medical image segmentation.