Multimodal Image Features Research Articles

Multi-Modal Object Detection Method Based on Dual-Branch Asymmetric Attention Backbone and Feature Fusion Pyramid Network

With the simultaneous acquisition of the infrared and optical remote sensing images of the same target becoming increasingly easy, using multi-modal data for high-performance object detection has become a research focus. In remote sensing multi-modal data, infrared images lack color information, it is hard to detect difficult targets with low contrast, and optical images are easily affected by illuminance. One of the most effective ways to solve this problem is to integrate multi-modal images for high-performance object detection. The challenge of fusion object detection lies in how to fully integrate multi-modal image features with significant modal differences and avoid introducing interference information while taking advantage of complementary advantages. To solve these problems, a new multi-modal fusion object detection method is proposed. In this paper, the method is improved in terms of two aspects: firstly, a new dual-branch asymmetric attention backbone network (DAAB) is designed, which uses a semantic information supplement module (SISM) and a detail information supplement module (DISM) to supplement and enhance infrared and RGB image information, respectively. Secondly, we propose a feature fusion pyramid network (FFPN), which uses a Transformer-like strategy to carry out multi-modal feature fusion and suppress features that are not conducive to fusion during the fusion process. This method is a state-of-the-art process for both FLIR-aligned and DroneVehicle datasets. Experiments show that this method has strong competitiveness and generalization performance.

CFNet: Automatic multi-modal brain tumor segmentation through hierarchical coarse-to-fine fusion and feature communication

Automatic segmentation of brain tumors employing images from multi-modalities is important for preoperative diagnosis and prognostic assessment. The rich complementary information contained within multi-modal images allows for improved brain tumor segmentation performance when models are trained on multi-modal data. However, accurate segmentation of small lesion regions from medical images remains challenging, due to the irregular shapes and low boundary contrast of brain gliomas. To address these challenges, we propose a coarse-to-fine feature fusion network (CFNet) that effectively incorporates multi-modal image features through modal interaction, semantic perception, and feature fusion. Specifically, the modality cross-attention fusion module is proposed introducing complementary features and mapping them into a unified semantic space to learn complementary representations between modalities. Furthermore, the multi-scale context perception module enables the model to focus on multi-scale lesion fine-grained information while capturing deep semantics. Moreover, the further fusion of fine-grained features with coarse-grained features enhances edge features and the complementary of features. We evaluate the proposed CFNet on two public brain tumor segmentation datasets BraTS2019 and BraTS2020. Experimental results show that the proposed framework outperforms state-of-the-art methods in Dice score, HD95, Sensitivity, demonstrating the effectiveness of CFNet on brain tumor segmentation. Code will be available at https://github.com/YaruC/CFNet.

Multi-grained contrastive representation learning for label-efficient lesion segmentation and onset time classification of acute ischemic stroke

Ischemic lesion segmentation and the time since stroke (TSS) onset classification from paired multi-modal MRI imaging of unwitnessed acute ischemic stroke (AIS) patients is crucial, which supports tissue plasminogen activator (tPA) thrombolysis decision-making. Deep learning methods demonstrate superiority in TSS classification. However, they often overfit task-irrelevant features due to insufficient paired labeled data, resulting in poor generalization. We observed that unpaired data are readily available and inherently carry task-relevant cues, but are less often considered and explored. Based on this, in this paper, we propose to fully excavate the potential of unpaired unlabeled data and use them to facilitate the downstream AIS analysis task. We first analyze the utility of features at the varied grain and propose a multi-grained contrastive learning (MGCL) framework to learn task-related prior representations from both coarse-grained and fine-grained levels. The former can learn global prior representations to enhance the location ability for the ischemic lesions and perceive the healthy surroundings, while the latter can learn local prior representations to enhance the perception ability for semantic relation between the ischemic lesion and other health regions. To better transfer and utilize the learned task-related representation, we designed a novel multi-task framework to simultaneously achieve ischemic lesion segmentation and TSS classification with limited labeled data. In addition, a multi-modal region-related feature fusion module is proposed to enable the feature correlation and synergy between multi-modal deep image features for more accurate TSS decision-making. Extensive experiments on the large-scale multi-center MRI dataset demonstrate the superiority of the proposed framework. Therefore, it is promising that it helps better stroke evaluation and treatment decision-making.

Visible and Infrared Object Tracking Based on Multimodal Hierarchical Relationship Modeling

Visible RGB and Thermal infrared (RGBT) object tracking has emerged as a prominent area of focus within the realm of computer vision. Nevertheless, the majority of existing RGBT tracking methods, which predominantly rely on Transformers, primarily emphasize the enhancement of features extracted by convolutional neural networks. Unfortunately, the latent potential of Transformers in representation learning has been inadequately explored. Furthermore, most studies tend to overlook the significance of distinguishing between the importance of each modality in the context of multimodal tasks. In this paper, we address these two critical issues by introducing a novel RGBT tracking framework centered on multimodal hierarchical relationship modeling. Through the incorporation of multiple Transformer encoders and the deployment of self-attention mechanisms, we progressively aggregate and fuse multimodal image features at various stages of image feature learning. Throughout the process of multimodal interaction within the network, we employ a dynamic component feature fusion module at the patch-level to dynamically assess the relevance of visible information within each region of the tracking scene. Our extensive experimentation, conducted on benchmark datasets such as RGBT234, GTOT, and LasHeR, substantiates the commendable performance of our proposed approach in terms of accuracy, success rate, and tracking speed.

Monitoring drought induced photosynthetic and fluorescent variations of potatoes by visible and thermal imaging analysis

Accurate determination of photosynthetic parameters is critical to evaluate crop physiological and growth processes. This study aimed to estimate photosynthetic and fluorescence variables of potatoes by visible and thermal imaging fusion analysis. Two-years pot experiments were conducted in a climate chamber with different irrigation treatments. Multi-modal image features of crop canopy were extracted from both visible and thermal images by color component extraction, discrete wavelet transformation, gray level co-occurrence matrix, and local binary pattern algorithms. Extracted features were subsequently employed to build partial least squares regression (PLSR) models for estimation of transpiration rate (Tr), net photosynthetic rate (An), stomatal conductance (GSW), electron transport rate (ETR), and maximum photochemical efficiency under Photosystem II (Fv'/Fm'). Results showed that Mask Region-Convolutional Neural Network (Mask R-CNN) performed satisfactorily on canopy segmentation with intersection over union of 87.29 % and 86.93 % in visible and thermal images, respectively. Three different types of models that either using only visible image features (PLSRRGB), or only thermal image features (PLSRT) or both visible and thermal image features (PLSRRGB+T) as inputs were compared. Results showed that PLSRRGB+T had superior estimation performance in terms of R2 and RMSE. It achieved the highest R2 of 0.85 with An and the lowest R2 of 0.66 with GSW for Zhongshu 5, while it had the highest R2 of 0.86 with Fv'/Fm', and the lowest R2 of 0.71 with Tr and An for D681. This implied the potential of visible and thermal image-driven method for quick and accurate estimation of photosynthetic traits of crops grown in controlled environment.

Ultrasound radiomics models based on multimodal imaging feature fusion of papillary thyroid carcinoma for predicting central lymph node metastasis.

This retrospective study aimed to establish ultrasound radiomics models to predict central lymph node metastasis (CLNM) based on preoperative multimodal ultrasound imaging features fusion of primary papillary thyroid carcinoma (PTC). In total, 498 cases of unifocal PTC were randomly divided into two sets which comprised 348 cases (training set) and 150 cases (validition set). In addition, the testing set contained 120 cases of PTC at different times. Post-operative histopathology was the gold standard for CLNM. The following steps were used to build models: the regions of interest were segmented in PTC ultrasound images, multimodal ultrasound image features were then extracted by the deep learning residual neural network with 50-layer network, followed by feature selection and fusion; subsequently, classification was performed using three classical classifiers-adaptive boosting (AB), linear discriminant analysis (LDA), and support vector machine (SVM). The performances of the unimodal models (Unimodal-AB, Unimodal-LDA, and Unimodal-SVM) and the multimodal models (Multimodal-AB, Multimodal-LDA, and Multimodal-SVM) were evaluated and compared. The Multimodal-SVM model achieved the best predictive performance than the other models (P < 0.05). For the Multimodal-SVM model validation and testing sets, the areas under the receiver operating characteristic curves (AUCs) were 0.910 (95% CI, 0.894-0.926) and 0.851 (95% CI, 0.833-0.869), respectively. The AUCs of the Multimodal-SVM model were 0.920 (95% CI, 0.881-0.959) in the cN0 subgroup-1 cases and 0.828 (95% CI, 0.769-0.887) in the cN0 subgroup-2 cases. The ultrasound radiomics model only based on the PTC multimodal ultrasound image have high clinical value in predicting CLNM and can provide a reference for treatment decisions.

Multi-Head multimodal deep interest recommendation network

From machine learning recommendation to deep learning recommendation, reinforcement learning recommendation, and recommendation model compression, the network structure of the recommendation algorithm becomes more complex, more expressive, and more lightweight with higher real-time performance. However, most of these models are dedicated to recommendations by mining ratings and text information, and researchers have optimized the network structures of many recommendation models. However, mining image information and enriching recommendation model features are less studied, and there is still some room for optimization of deep recommendation model structures and real-time performance. To this end, this paper proposes a model called MMDIN by adding multi-head and multi-modal modules to the DIN model. The multi-modal module extracts image features, which enriches the feature set that the model can use and strengthens the cross-combination and fitting expression capabilities of the model. Meanwhile, the multi-head mechanism extracts features from different dimensions, which further enhances the model prediction performance. Experimental results show that the proposed MMDIN model improves the recommendation prediction performance, and it outperforms state-of-the-art methods in comprehensive indicators.

Adaptive zero-learning medical image fusion

Medical fusion image can integrate complementary information from multi-modal medical images, so that more comprehensive and accurate image results could be obtained. This paper proposes a new two-scale zero-learning medical images fusion method combined with pre-trained Res2Net and an adaptive guided filter. Firstly, this method utilizes the guided filter to decompose a medical image into a base layer representing large-scale intensity variations, and a detail layer containing small scale changes. Considering the influence of the parameters of the guided filter on the fusion images and the time-consuming on parameter selection, an adaptive guided filter based on the multi-modal medical image features is proposed. Then, detail layers are fused by an elementwise-sum strategy for retaining more detail information of source images. And the base layers are to fused with deep feature maps extracted from a pre-trained neural network Res2Net. Finally, the fused detail and basic layer will be reconstructed to obtain medical fusion image. The proposed method is demonstrated its superiority by ablation studies, and compared with 7 typical and advanced image fusion methods in visual effect and evaluation index. The experimental results show that the proposed image fusion method outperforms other methods in retaining effective detailed information and image clarity.

Cross-Modal Prostate Cancer Segmentation via Self-Attention Distillation.

The automatic and accurate segmentation of the prostate cancer from the multi-modal magnetic resonance images is of prime importance for the disease assessment and follow-up treatment plan. However, how to use the multi-modal image features more efficiently is still a challenging problem in the field of medical image segmentation. In this paper, we develop a cross-modal self-attention distillation network by fully exploiting the encoded information of the intermediate layers from different modalities, and the generated attention maps of different modalities enable the model to transfer significant and discriminative information that contains more details. Moreover, a novel spatial correlated feature fusion module is further employed for learning more complementary correlation and non-linear information of different modality images. We evaluate our model in five-fold cross-validation on 358 MRI images with biopsy confirmed. Without bells and whistles, our proposed network achieves state-of-the-art performance on extensive experiments.

Colposcopic multimodal fusion for the classification of cervical lesions

Objective: Cervical cancer is one of the two biggest killers of women and early detection of cervical precancerous lesions can effectively improve the survival rate of patients. Manual diagnosis by combining colposcopic images and clinical examination results is the main clinical diagnosis method at present. Developing an intelligent diagnosis algorithm based on artificial intelligence is an inevitable trend to solve the objectification of diagnosis and improve the quality and efficiency of diagnosis. Approach: A colposcopic multimodal fusion convolutional neural network (CMF-CNN) was proposed for the classification of cervical lesions. Mask region convolutional neural network was used to detect the cervical region while the encoding network EfficientNet-B3 was introduced to extract the multimodal image features from the acetic image and iodine image. Finally, Squeeze-and-Excitation, Atrous Spatial Pyramid Pooling, and convolution block were also adopted to encode and fuse the patient’s clinical text information. Main results: The experimental results showed that in 7106 cases of colposcopy, the accuracy, macro F1-score, macro-areas under the curve of the proposed model were 92.70%, 92.74%, 98.56%, respectively. They are superior to the mainstream unimodal image classification models. Significance: CMF-CNN proposed in this paper combines multimodal information, which has high performance in the classification of cervical lesions in colposcopy, so it can provide comprehensive diagnostic aid.

Defect Detection for a Vertical Shaft Surface Based on Multimodal Sensors

Hydroelectricity is a major source of renewable electricity originating from a turbine driven by dammed large-volume water via a penstock or a drop shaft. Shafts suffer from risks of collapse due to the pressure from exterior structures and the erosion from inner water flow. Vertical shafts are an important part of hydroelectric power generation systems, and detecting defects in shafts guarantees the stable operation of hydropower stations. However, shaft defect detection is a great challenge due to the poor conditions, large drop in height, limited entrance size, lack of light and damp air, where suitable technology and equipment are not available. Aiming at defect detection for vertical shafts, we have developed a defect-detecting system based on unmanned airships, integrated panoramic CCD cameras, 3D laser scanners, inertial measurement units, barometric altimeters, illumination sensors, and control modules. Shaft defect detection methods (SDDMs) are proposed by fusing the multi-modal image features to extract typical defects on concrete surfaces. Compared with machine learning methods, the proposed method achieves the highest overall accuracy of 90.90&#x0025; for defect detection. Our system was validated by experiments in the shafts of the Nuozhadu hydropower station to be functional for defect detection, which demonstrates its capability of reducing the risk of collapse and improving safety.

MMI-Fuse: Multimodal Brain Image Fusion With Multiattention Module

Medical imaging plays a pivotal role in the clinical diagnosis of brain disease. There are many imaging methods to detect the state of tissues in the brain. While these imaging methods have advantages, they also have shortcomings. For example, magnetic resonance imaging (MRI) contains structural information but no functional characteristics of tissue, while positron emission tomography (PET) possesses functional characteristics but no structural information. The attention mechanism has been widely used in image fusion tasks, such as fusion of infrared and visible images and medical images. However, those attention models lack a balance mechanism for multimodal image features, affecting the final fusion performance. This paper proposes an end-to-end multimodal brain image fusion framework, MMI-fuse. Specifically, we first apply an autoencoder to extract the features of source images. Then, an information preservation weighted channel spatial attention model (ICS) is proposed to fuse the image features. We set an adaptive weight according to the information preservation degree of features. Finally, we use a decoder model to restructure the fused medical image. The proposed method increased the quality of fused images and decreased the fusion time effectively by the help of the improved attention model and encoder-decoder structure. To validate the performance of the proposed method, we collected 1590 pairs of multimodal brain images from the Harvard dataset and performed extensive experiments. Seven methods and five metrics were selected for the comparison experiments. The results demonstrate that the proposed method achieved notable performance on both the visual quality and objective metric score among these seven approaches. Moreover, the proposed method takes the least time among all compared methods.

Robust registration for ultra-field infrared and visible binocular images.

The ultra-field infrared and visible image registration is a challenging task due to its nonlinear imaging and multi-modal image features. In this paper, a robust registration method is proposed for the ultra-field infrared and visible images. First, control points are extracted utilizing phase congruency and optimized based on the guidance map, which is proposed according to significant structures information. Second, ROI pair matching is accomplished based on epipolar curve. Its effect is equivalent to a search window that is popular in methods with the standard field of view, and it can overcome the content differences in the search window caused by nonlinear imaging and vision disparity. Third, a descriptor, named multiple phase congruency directional pattern (MPCDP), is established and composed of distribution information and main direction. The phase congruency amplitudes are encoded as binary patterns, and then they are represented as weighted histogram for distribution information. Six pairs of ultra-field infrared and visible images are employed for registration experiments, and the results demonstrate that the performance of the proposed is robust and accurate in five types of ultra-field scenes and two different camera relationships.

A Geometric Framework for Feature Mappings in Multimodal Fusion of Brain Image Data.

Fusing multimodal brain image features to empower statistical analysis has attracted considerable research interest. Generally, a feature mapping is learned in the fusion process so the cross-modality relationship in the multimodal data can be more effectively extracted in a common feature space. Most of the prior work achieve this goal by data-driven approaches without considering the geometry properties of the feature spaces where the data are embedded. It results in a huge sacrifice of untapped information. Here, we propose to fuse the multimodal brain images through a novel geometric approach. The key idea is to encode various brain image features with the local metric change on brain shapes, such that the feature mapping can be efficiently solved by some geometric mapping functions, i.e., quasiconformal and harmonic mappings. We approach our multimodal fusion framework (MFRM) in two steps: surface feature mapping and volumetric feature mapping. For each of them, we design an informative Riemannian metric based on distinct brain anatomical features and achieve image fusion via diffeomorphic maps. We evaluate our proposed method on two brain image cohorts. The experimental results reveal the effectiveness of our proposed framework which yields better statistical performances than state-of-the-art data-driven methods.

Multiple Network Fusion with Low-Rank Representation for Image-Based Age Estimation

Image-based age estimation is a challenging task since there are ambiguities between the apparent age of face images and the actual ages of people. Therefore, data-driven methods are popular. To improve data utilization and estimation performance, we propose an image-based age estimation method. Theoretically speaking, the key idea of the proposed method is to integrate multi-modal features of face images. In order to achieve it, we propose a multi-modal learning framework, which is called Multiple Network Fusion with Low-Rank Representation (MNF-LRR). In this process, different deep neural network (DNN) structures, such as autoencoders, Convolutional Neural Networks (CNNs), Recursive Neural Networks (RNNs), and so on, can be used to extract semantic information of facial images. The outputs of these neural networks are then represented in a low-rank feature space. In this way, feature fusion is obtained in this space, and robust multi-modal image features can be computed. An experimental evaluation is conducted on two challenging face datasets for image-based age estimation extracted from the Internet Move Database (IMDB) and Wikipedia (WIKI). The results show the effectiveness of the proposed MNF-LRR.

Neurobiological support to the diagnosis of ADHD in stimulant-naïve adults: pattern recognition analyses of MRI data.

In adulthood, the diagnosis of attention-deficit/hyperactivity disorder (ADHD) has been subject of recent controversy. We searched for a neuroanatomical signature associated with ADHD spectrum symptoms in adults by applying, for the first time, machine learning-based pattern classification methods to structural MRI and diffusion tensor imaging (DTI) data obtained from stimulant-naïve adults with childhood-onset ADHD and healthy controls (HC). Sixty-seven ADHD patients and 66 HC underwent high-resolution T1-weighted and DTI acquisitions. A support vector machine (SVM) classifier with a non-linear kernel was applied on multimodal image features extracted on regions of interest placed across the whole brain. The discrimination between a mixed-gender ADHD subgroup and individually matched HC (n = 58 each) yielded area-under-the-curve (AUC) and diagnostic accuracy (DA) values of up to 0.71% and 66% (P = 0.003) respectively. AUC and DA values increased to 0.74% and 74% (P = 0.0001) when analyses were restricted to males (52 ADHD vs. 44 HC). Although not at the level of clinically definitive DA, the neuroanatomical signature identified herein may provide additional, objective information that could influence treatment decisions in adults with ADHD spectrum symptoms.

Multi-modal multiple kernel learning for accurate identification of Tourette syndrome children

Tourette syndrome (TS) is a childhood-onset neurobehavioral disorder characterized by the presence of multiple motor and vocal tics. To date, TS diagnosis remains somewhat limited and studies using advanced diagnostic methods are of great importance. In this paper, we introduce an automatic classification framework for accurate identification of TS children based on multi-modal and multi-type features, which is robust and easy to implement. We present in detail the feature extraction, feature selection, and classifier training methods. In addition, in order to exploit complementary information revealed by different feature modalities, we integrate multi-modal image features using multiple kernel learning (MKL). The performance of our framework has been validated in classifying 44 TS children and 48 age- and gender-matched healthy children. When combining features using MKL, the classification accuracy reached 94.24% using nested cross-validation. Most discriminative brain regions were mostly located in the cortico-basal ganglia, frontal cortico-cortical circuits, which are thought to be highly related to TS pathology. These results show that our method is reliable for early TS diagnosis, and promising for prognosis and treatment outcome.

SU-D-BRA-03: What Image Features Are Useful for Tumor Segmentation in Multi-Modal Images?

Purpose: We hypothesize that derived image features will improve accuracy of segmenting gross tumor volume (GTV) using (semi-)automatic methods when compared with raw intensities. Features from multi-modal images, however, require more CPU time and memory, for extraction, training and classification. In this study, we examine what image features are useful for segmenting GTV in multi-modal images. Methods: Eight cases of high grade brain tumor from MICCAI 2013 for brain tumor segmentation competition were examined. In each case, images of co-registered four MRI modalities: Flair, T1, T1 contrast and T2, as well as the ground truth are provided in the data set. Three cases were used for probing important features and the other five cases were for evaluation of segmentations with the selected features. Besides original image intensity, additional 7 candidate features were extracted: minimum, maximum, gradient, entropy, spot, ripple and edge, resulting in a 32-D (4 x 8) feature vector for each voxel. For selecting important features, a random forest with 100 decision trees was trained using 32-D features from the 3 probing cases. The top ranked features reported by the random forest were selected and used in a Conditional Random Fields (CRF) based semi-automatic method we previously developed to segment the 5 testing cases. Dice coefficent metric to the ground truth are reported and compared to intensity-only segmentation using the same segmentation method. Results: The feature selection random forest ranked intensity, minimum, maximum and entropy the most important features across all 4 MRI modalities. With the selected features (16-D), the CRF segmentation of 5 test cases yielded Dice coefficent 0.847±0.114 (mean±1SD), significantly better (p =0.016) than the Dice coefficient 0.797±0.135 of intensity-only (4-D) CRF segmentation. Conclusion: Multi-modal image features selected by a machine learning method can improve the accuracy of brain tumor segmentation significantly when compared with intensity-only features. Research supported by Varian Medical System

Classification of ADHD children through multimodal magnetic resonance imaging

Attention deficit/hyperactivity disorder (ADHD) is one of the most common diseases in school-age children. To date, the diagnosis of ADHD is mainly subjective and studies of objective diagnostic method are of great importance. Although many efforts have been made recently to investigate the use of structural and functional brain images for the diagnosis purpose, few of them are related to ADHD. In this paper, we introduce an automatic classification framework based on brain imaging features of ADHD patients and present in detail the feature extraction, feature selection, and classifier training methods. The effects of using different features are compared against each other. In addition, we integrate multimodal image features using multi-kernel learning (MKL). The performance of our framework has been validated in the ADHD-200 Global Competition, which is a world-wide classification contest on the ADHD-200 datasets. In this competition, our classification framework using features of resting-state functional connectivity (FC) was ranked the 6th out of 21 participants under the competition scoring policy and performed the best in terms of sensitivity and J-statistic.

Disparity disambiguation by fusion of signal- and symbolic-level information

We describe a method for resolving ambiguities in low-level disparity calculations in a stereo-vision scheme by using a recurrent mechanism that we call signal-symbol loop. Due to the local nature of low-level processing it is not always possible to estimate the correct disparity values produced at this level. Symbolic abstraction of the signal produces robust, high confidence, multimodal image features which can be used to interpret the scene more accurately and therefore disambiguate low-level interpretations by biasing the correct disparity. The fusion process is capable of producing more accurate dense disparity maps than the low- and symbolic-level algorithms can produce independently. Therefore we describe an efficient fusion scheme that allows symbolic- and low-level cues to complement each other, resulting in a more accurate and dense disparity representation of the scene.