Image Modalities Research Articles

Multimodal hate speech detection via multi-scale visual kernels and knowledge distillation architecture

People increasingly use social media platforms to express themselves by posting visuals and texts. As a result, hate content is on the rise, necessitating practical visual caption analysis. Thus, the relationship between image and caption modalities is crucial in visual caption analysis. Contrarily, most methods combine features from the image and caption modalities using deep learning architectures with millions of parameters already trained without integrating a specialized attention module, resulting in less desirable outcomes. This paper suggests a novel multi-modal architecture for identifying hateful memetic information in response to the above observation. The proposed architecture contains a novel "multi-scale kernel attentive visual" (MSKAV) module that uses an efficient multi-branch structure to extract discriminative visual features. Additionally, MSKAV utilizes an adaptive receptive field using multi-scale kernels. MSKAV also incorporates a multi-directional visual attention module to highlight spatial regions of importance. The proposed model also contains a novel "knowledge distillation-based attentional caption" (KDAC) module. It uses a transformer-based self-attentive block to extract discriminative features from meme captions. Thorough experimentation on multi-modal hate speech benchmarks MultiOff, Hateful Memes, and MMHS150K datasets achieved accuracy scores of 0.6250, 0.8750, and 0.8078, respectively. It also reaches impressive AUC scores of 0.6557, 0.8363, and 0.7665 on the three datasets, respectively, beating SOTA multi-modal hate speech identification models.

Medical Diagnosis with Multimodal Image Fusion Techniques

Image Fusion is an effective approach utilized to draw out all the significant information from the source images, which supports experts in evaluation and quick decision making. Multi modal medical image fusion produces a composite fused image utilizing various sources to improve quality and extract complementary information. It is extremely challenging to gather every piece of information needed using just one imaging method. Therefore, images obtained from different modalities are fused Additional clinical information can be gleaned through the fusion of several types of medical image pairings. This study's main aim is to present a thorough review of medical image fusion techniques which also covers steps in fusion process, levels of fusion, various imaging modalities with their pros and cons, and the major scientific difficulties encountered in the area of medical image fusion. This paper also summarizes the quality assessments fusion metrics. The various approaches used by image fusion algorithms that are presently available in the literature are classified into four broad categories i) Spatial fusion methods ii) Multiscale Decomposition based methods iii) Neural Network based methods and iv) Fuzzy Logic based methods. the benefits and pitfalls of the existing literature are explored and Future insights are suggested. Moreover, this study is anticipated to create a solid platform for the development of better fusion techniques in medical applications.

Estimation of interlayer textural relationships to discriminate the benignancy/malignancy of brain tumors

A computer-aided diagnosis system is a popular tool to predict the risk factors of brain tumors. However, the existing techniques are unable to provide high detection accuracy due to the inability of capturing the hidden features of brain tumors. In this view, the present work decomposes the MR modality images into different layers using non-sub-sampled shearlet transformation (NSST). Following this, the proposed detection pipeline employs adaptive pulse-coupled neural network (A-PCNN) module and a fuzzy c-means (FCM) clustering algorithm for enhancement and segmentation of suspicious regions respectively. Interlayer textural relationships of tumors are estimated in terms of the layer-wise difference of the gray-level cooccurrence matrix (GLCM), similarity indices, and entropy distributions. Thenceforth, those feature coefficients are coded with binary sequences to express the textural homogeneity/randomness of tumors. Finally, these handcrafted multi-scale feature quantifiers are fed to some standard classifiers such as the k-nearest neighbor (kNN) technique, the linear square support vector machine (LS-SVM), and the decision tree (DT) for discriminating the state of benignancy/malignancy of tumors. Experimental results ensure that the proposed detection model shows superior performance compared to existing methods.

Effect of Alumina Nanoparticles on Lentil Seed Germination Using Biospeckle Optical Coherence Tomography

The industrial use of nanoparticles is rapidly increasing in agricultural products, and it causes numerous effects on plant growth and seed germination. There are limited studies about the uptake, accumulation, and effect of nanoparticles on plant growth. Moreover, there are no studies that have been able to measure the biological activity of seeds’ interiors under nanoparticle treatment prior to germination. Hence, in this study, the possible size-defendant effect of alumina nanoparticles on lentil (Lens culinaris) seed germination was monitored using biospeckle Optical Coherence Tomography (bOCT) at an early stage before germination. bOCT is a non-contact, non-destructive, and non-invasive image modality developed by researchers to visualize internal biological activities in vivo. Previous studies by researchers using the technique have demonstrated its potential to observe the effect of different stimuli on seed germination at an early stage before germination occurs. In the present study, the possible size-defendant effect of 200 nm and 2000 nm alumina nanoparticles (AlNPs) on lentil seed germination was examined at a concentration of 100 mg/L. The results imply that the AlNPs could enhance lentil seed germination and seedling growth compared to control treatments, and the effect is more significant as the particle size decreases. This is believed to be due to the enhancement of energy metabolism under exposure to AlNPs.

DSA-Net: Infrared and Visible Image Fusion via Dual-Stream Asymmetric Network.

Infrared and visible image fusion technologies are used to characterize the same scene using diverse modalities. However, most existing deep learning-based fusion methods are designed as symmetric networks, which ignore the differences between modal images and lead to source image information loss during feature extraction. In this paper, we propose a new fusion framework for the different characteristics of infrared and visible images. Specifically, we design a dual-stream asymmetric network with two different feature extraction networks to extract infrared and visible feature maps, respectively. The transformer architecture is introduced in the infrared feature extraction branch, which can force the network to focus on the local features of infrared images while still obtaining their contextual information. The visible feature extraction branch uses residual dense blocks to fully extract the rich background and texture detail information of visible images. In this way, it can provide better infrared targets and visible details for the fused image. Experimental results on multiple datasets indicate that DSA-Net outperforms state-of-the-art methods in both qualitative and quantitative evaluations. In addition, we also apply the fusion results to the target detection task, which indirectly demonstrates the fusion performances of our method.

General cross-modality registration framework for visible and infrared UAV target image registration

In all-day-all-weather tasks, well-aligned multi-modality images pairs can provide extensive complementary information for image-guided UAV target detection. However, multi-modality images in real scenarios are often misaligned, and images registration is extremely difficult due to spatial deformation and the difficulty narrowing cross-modality discrepancy. To better overcome the obstacle, in this paper, we construct a General Cross-Modality Registration (GCMR) Framework, which explores generation registration pattern to simplify the cross-modality image registration into a easier mono-modality image registration with an Image Cross-Modality Translation Network (ICMTN) module and a Multi-level Residual Dense Registration Network (MRDRN). Specifically, ICMTN module is used to generate a pseudo infrared image taking a visible image as input and correct the distortion of structural information during the translation of image modalities. Benefiting from the favorable geometry correct ability of the ICMTN, we further employs MRDRN module which can fully extract and exploit the mutual information of misaligned images to better registered Visible and Infrared image in a mono-modality setting. We evaluate five variants of our approach on the public Anti-UAV datasets. The extensive experimental results demonstrate that the proposed architecture achieves state-of-the-art performance.

Dual-Centre Harmonised Multimodal Positron Emission Tomography/Computed Tomography Image Radiomic Features and Machine Learning Algorithms for Non-small Cell Lung Cancer Histopathological Subtype Phenotype Decoding

AimsWe aimed to build radiomic models for classifying non-small cell lung cancer (NSCLC) histopathological subtypes through a dual-centre dataset and comprehensively evaluate the effect of ComBat harmonisation on the performance of single- and multimodality radiomic models. Materials and methodsA public dataset of NSCLC patients from two independent centres was used. Two image fusion methods, namely guided filtering-based fusion and image fusion based on visual saliency map and weighted least square optimisation, were used. Radiomic features were extracted from each scan, including first-order, texture and moment-invariant features. Subsequently, ComBat harmonisation was applied to the extracted features from computed tomography (CT), positron emission tomography (PET) and fused images to correct the centre effect. For feature selection, least absolute shrinkage and selection operator (Lasso) and recursive feature elimination (RFE) were investigated. For machine learning, logistic regression (LR), support vector machine (SVM) and AdaBoost were evaluated for classifying NSCLC subtypes. Training and evaluation of the models were carried out in a robust framework to offset plausible errors and performance was reported using area under the curve, balanced accuracy, sensitivity and specificity before and after harmonisation. N-way ANOVA was used to assess the effect of different factors on the performance of the models. ResultsSupport vector machine fed with selected features by recursive feature elimination from a harmonised PET feature set achieved the highest performance (area under the curve = 0.82) in classifying NSCLC histopathological subtypes. Although the performance of the models did not significantly improve for CT images after harmonisation, the performance of PET and guided filtering-based fusion feature signatures significantly improved for almost all models. Although the selection of the image modality and feature selection methods was effective on the performance of the model (ANOVA P-values <0.001), machine learning and harmonisation did not change the performance significantly (ANOVA P-values = 0.839 and 0.292, respectively). ConclusionThis study confirmed the potential of radiomic analysis on PET, CT and hybrid images for histopathological classification of NSCLC subtypes.

A deep learning approach for automatic tumor delineation in stereotactic radiotherapy for non-small cell lung cancer using diagnostic PET-CT and planning CT.

Accurate delineation of tumor targets is crucial for stereotactic body radiation therapy (SBRT) for non-small cell lung cancer (NSCLC). This study aims to develop a deep learning-based segmentation approach to accurately and efficiently delineate NSCLC targets using diagnostic PET-CT and SBRT planning CT (pCT). The diagnostic PET was registered to pCT using the transform matrix from registering diagnostic CT to the pCT. We proposed a 3D-UNet-based segmentation method to segment NSCLC tumor targets on dual-modality PET-pCT images. This network contained squeeze-and-excitation and Residual blocks in each convolutional block to perform dynamic channel-wise feature recalibration. Furthermore, up-sampling paths were added to supplement low-resolution features to the model and also to compute the overall loss function. The dice similarity coefficient (DSC), precision, recall, and the average symmetric surface distances were used to assess the performance of the proposed approach on 86 pairs of diagnostic PET and pCT images. The proposed model using dual-modality images was compared with both conventional 3D-UNet architecture and single-modality image input. The average DSC of the proposed model with both PET and pCT images was 0.844, compared to 0.795 and 0.827, when using 3D-UNet and nnUnet. It also outperformed using either pCT or PET alone with the same network, which had DSC of 0.823 and 0.732, respectively. Therefore, our proposed segmentation approach is able to outperform the current 3D-UNet network with diagnostic PET and pCT images. The integration of two image modalities helps improve segmentation accuracy.

A Cross-Scale Iterative Attentional Adversarial Fusion Network for Infrared and Visible Images

Recent existing methods generally adopt a simple concatenation or addition strategy to integrate features at the fusion layer, failing to adequately consider the intrinsic characteristics of different modal images and feature interaction of different scales, which may produce a limited fusion performance. Toward this end, we introduce a cross-scale iterative attentional adversarial fusion network, namely CrossFuse. More specifically, in the generator, we design a cross-modal attention integrated module to merge the intrinsic content of different modal images. The parallel spatial-independent and channel-independent pathways are proposed to calculate the attentional weights, which are assigned to measure the activity levels of source images at the same scale. Moreover, we construct a cross-scale iterative decoder framework to interact with different modality features at different scales, which can constantly optimize their activity levels. By this means, the generator learns to integrate their modality characteristics via attentional weights in an iterative manner, and the generated result characterizes competitive infrared radiant intensity and distinct visible detail description. Extensive experiments on three different benchmarks demonstrate that our CrossFuse outperforms other nine state-of-the-art methods in terms of fusion performance, generalization ability and computational efficiency. Our codes will be released at https://github.com/Zhishe-Wang/CrossFuse.

Feature Fusion Based on Transformer for Cross-modal Retrieval

With the popularity of the Internet and the rapid growth of multimodal data, multimodal retrieval has gradually become a hot area of research. As one of the important branches of multimodal retrieval, image-text retrieval aims to design a model to learn and align two modal data, image and text, in order to build a bridge of semantic association between the two heterogeneous data, so as to achieve unified alignment and retrieval. The current mainstream image-text cross-modal retrieval approaches have made good progress by designing a deep learning-based model to find potential associations between different modal data. In this paper, we design a transformer-based feature fusion network to fuse the information of two modalities in the feature extraction process, which can enrich the semantic connection between the modalities. Meanwhile, we conduct experiments on the benchmark dataset Flickr30k and get competitive results, where recall at 10 achieves 96.2% accuracy in image-to-text retrieval.

Attentive continuous generative self-training for unsupervised domain adaptive medical image translation.

Self-training is an important class of unsupervised domain adaptation (UDA) approaches that are used to mitigate the problem of domain shift, when applying knowledge learned from a labeled source domain to unlabeled and heterogeneous target domains. While self-training-based UDA has shown considerable promise on discriminative tasks, including classification and segmentation, through reliable pseudo-label filtering based on the maximum softmax probability, there is a paucity of prior work on self-training-based UDA for generative tasks, including image modality translation. To fill this gap, in this work, we seek to develop a generative self-training (GST) framework for domain adaptive image translation with continuous value prediction and regression objectives. Specifically, we quantify both aleatoric and epistemic uncertainties within our GST using variational Bayes learning to measure the reliability of synthesized data. We also introduce a self-attention scheme that de-emphasizes the background region to prevent it from dominating the training process. The adaptation is then carried out by an alternating optimization scheme with target domain supervision that focuses attention on the regions with reliable pseudo-labels. We evaluated our framework on two cross-scanner/center, inter-subject translation tasks, including tagged-to-cine magnetic resonance (MR) image translation and T1-weighted MR-to-fractional anisotropy translation. Extensive validations with unpaired target domain data showed that our GST yielded superior synthesis performance in comparison to adversarial training UDA methods.

Cross-modal multi-scale feature fusion-based RGB-T saliency object detection method

To cope with the challenge of significant target detection in complex scenes, this study proposes an RGB-T significant target detection method called CMFF. The method utilizes the complete potential of RGB and thermal infrared modal images and employs a codec structure and cross-modal multiscale feature fusion techniques. In the coding stage, two VGG16 backbone networks are used for multi-level feature extraction and CBAM attention module feature enhancement, and the enhanced features are fused using a stepwise fusion approach. Meanwhile, the weights of the two modalities are assigned using the L1-parametric fusion strategy to enhance the complementarity between them. In the decoding stage, global features are extracted from the high-level fused features by introducing the pyramid pooling module (PPM), and the low-level fused features are fused with multi-scale features in the up-sampling and encoding stages to enrich the global and local information of the feature map. Finally, this study conducted comparison experiments on the publicly available VT5000 dataset, and the method achieved an F-measure value of 0.863 and a mean absolute error (MAE) of 0.062, which significantly improved the overall detection performance relative to the six existing methods.

Exploring latent weight factors and global information for food-oriented cross-modal retrieval

Food-oriented cross-modal retrieval aims to retrieve relevant recipes given food images or vice versa. The modality semantic gap between recipes and food images (text and image modalities) is the main challenge. Though several studies are introduced to bridge this gap, they still suffer from two major limitations: 1) The simple embedding concatenation only can capture the simple interactions rather than complex interactions between different recipe components. 2) The image feature extraction based on convolutional neural networks only considers the local features and ignores the global features of an image, as well as the interactions between different extracted features. This paper proposes a novel method based on Latent Component Weight Factors and Global Information (LCWF-GI) to learn the robust recipe and image representations for food-oriented cross-modal retrieval. This proposed method integrates the textual embeddings of different recipe components into a compact embedding to represent the recipes with the latent component-specific weight factors. A transformer encoder is utilised to capture the intra-modality interactions and the importance of different extracted image features for enhanced image representations. Finally, the bi-directional triplet loss is further used to perform retrieval learning. Experimental results on the Recipe 1M dataset show that our LCWF-GI method achieves competent improvements.

Adaptive Template Reconstruction for Effective Pattern Classification.

A novel instance-based algorithm for pattern classification is presented and evaluated in this paper. This new method is motivated by the challenge of pattern classifications where only limited and/or noisy training data are available. For every classification, the proposed system transforms the query data and the training templates based on their distributions in the feature space. One of the major novelties of the proposed method is the concept of template reconstruction enabling improved performance with limited training data. The technique is compared with similar algorithms and evaluated using both the image and time-series modalities to demonstrate its effectiveness and versatility. Two public image databases, FASHION-MNIST and CIFAR-10, were used to test its effectiveness for the classification of images using small amounts of training samples. An average classification improvement of 2~3% was observed while using a small subset of the training database, compared to the performances achieved by state-of-the-art techniques using the full datasets. To further explore its capability in solving more challenging classification problems such as non-stationary time-series electroencephalography (EEG) signals, a clinical grade 64-electrode EEG database, as well as a low-quality (high-noise level) EEG database, obtained using a low-cost system equipped with a single dry sensor, have also been used to test the algorithm. Adaptive reconstruction of the feature instances has been seen to have substantially improved class separation and matching performance for both still images and time-series signals. In particular, the method is found to be effective for the classification of noisy non-stationary data with limited training data volumes, indicating its potential suitability for a wide range of applications.

Attention-based 3D CNN with Multi-layer Features for Alzheimer's Disease Diagnosis using Brain Images.

Structural MRI and PET imaging play an important role in the diagnosis of Alzheimer's disease (AD), showing the morphological changes and glucose metabolism changes in the brain respectively. The manifestations in the brain image of some cognitive impairment patients are relatively inconspicuous, for example, it still has difficulties in achieving accurate diagnosis through sMRI in clinical practice. With the emergence of deep learning, convolutional neural network (CNN) has become a valuable method in AD-aided diagnosis, but some CNN methods cannot effectively learn the features of brain image, making the diagnosis of AD still presents some challenges. In this work, we propose an end-to-end 3D CNN framework for AD diagnosis based on ResNet, which integrates multi-layer features obtained under the effect of the attention mechanism to better capture subtle differences in brain images. The attention maps showed our model can focus on key brain regions related to the disease diagnosis. Our method was verified in ablation experiments with two modality images on 792 subjects from the ADNI database, where AD diagnostic accuracies of 89.71% and 91.18% were achieved based on sMRI and PET respectively, and also outperformed some state-of-the-art methods.

Self-Enhanced Mixed Attention Network for Three-Modal Images Few-Shot Semantic Segmentation.

As an important computer vision technique, image segmentation has been widely used in various tasks. However, in some extreme cases, the insufficient illumination would result in a great impact on the performance of the model. So more and more fully supervised methods use multi-modal images as their input. The dense annotated large datasets are difficult to obtain, but the few-shot methods still can have satisfactory results with few pixel-annotated samples. Therefore, we propose the Visible-Depth-Thermal (three-modal) images few-shot semantic segmentation method. It utilizes the homogeneous information of three-modal images and the complementary information of different modal images, which can improve the performance of few-shot segmentation tasks. We constructed a novel indoor dataset VDT-2048-5i for the three-modal images few-shot semantic segmentation task. We also proposed a Self-Enhanced Mixed Attention Network (SEMANet), which consists of a Self-Enhanced module (SE) and a Mixed Attention module (MA). The SE module amplifies the difference between the different kinds of features and strengthens the weak connection for the foreground features. The MA module fuses the three-modal feature to obtain a better feature. Compared with the most advanced methods before, our model improves mIoU by 3.8% and 3.3% in 1-shot and 5-shot settings, respectively, which achieves state-of-the-art performance. In the future, we will solve failure cases by obtaining more discriminative and robust feature representations, and explore achieving high performance with fewer parameters and computational costs.

Multiparametric MRI radiomics in prostate cancer for predicting Ki-67 expression and Gleason score: a multicenter retrospective study

PurposeProstate cancer (PCa) with high Ki-67 expression and high Gleason Scores (GS) tends to have aggressive clinicopathological characteristics and a dismal prognosis. In order to predict the Ki-67 expression status and the GS in PCa, we sought to construct and verify MRI-based radiomics signatures.Methods and materialsWe collected T2-weighted imaging (T2WI), diffusion-weighted imaging (DWI), and apparent diffusion coefficient (ADC) images from 170 PCa patients at three institutions and extracted 321 original radiomic features from each image modality. We used support vector machine (SVM) and least absolute shrinkage and selection operator (LASSO) logistic regression to select the most informative radiomic features and built predictive models using up sampling and feature selection techniques. Using receiver operating characteristic (ROC) analysis, the discriminating power of this feature was determined. Subsequent decision curve analysis (DCA) assessed the clinical utility of the radiomic features. The Kaplan–Meier (KM) test revealed that the radiomics-predicted Ki-67 expression status and GS were prognostic factors for PCa survival.ResultThe hypothesized radiomics signature, which included 15 and 9 selected radiomics features, respectively, was significantly correlated with pathological Ki-67 and GS outcomes in both the training and validation datasets. Areas under the curve (AUC) for the developed model were 0.813 (95% CI 0.681,0.930) and 0.793 (95% CI 0.621, 0.929) for the training and validation datasets, respectively, demonstrating discrimination and calibration performance. The model's clinical usefulness was verified using DCA. In both the training and validation sets, high Ki-67 expression and high GS predicted by radiomics using SVM models were substantially linked with poor overall survival (OS).ConclusionsBoth Ki-67 expression status and high GS correlate with PCa patient survival outcomes; therefore, the ability of the SVM classifier-based model to estimate Ki-67 expression status and the Lasso classifier-based model to assess high GS may enhance clinical decision-making.

Multimodal brain image fusion based on error texture elimination and salient feature detection.

As an important clinically oriented information fusion technology, multimodal medical image fusion integrates useful information from different modal images into a comprehensive fused image. Nevertheless, existing methods routinely consider only energy information when fusing low-frequency or base layers, ignoring the fact that useful texture information may exist in pixels with lower energy values. Thus, erroneous textures may be introduced into the fusion results. To resolve this problem, we propose a novel multimodal brain image fusion algorithm based on error texture removal. A two-layer decomposition scheme is first implemented to generate the high- and low-frequency subbands. We propose a salient feature detection operator based on gradient difference and entropy. The proposed operator integrates the gradient difference and amount of information in the high-frequency subbands to effectively identify clearly detailed information. Subsequently, we detect the energy information of the low-frequency subband by utilizing the local phase feature of each pixel as the intensity measurement and using a random walk algorithm to detect the energy information. Finally, we propose a rolling guidance filtering iterative least-squares model to reconstruct the texture information in the low-frequency components. Through extensive experiments, we successfully demonstrate that the proposed algorithm outperforms some state-of-the-art methods. Our source code is publicly available at https://github.com/ixilai/ETEM.

Haar wavelet downsampling: A simple but effective downsampling module for semantic segmentation

Downsampling operations such as max pooling or strided convolution are ubiquitously utilized in Convolutional Neural Networks (CNNs) to aggregate local features, enlarge receptive field, and minimize computational overhead. However, for a semantic segmentation task, pooling features over the local neighbourhood may result in the loss of important spatial information, which is conducive for pixel-wise predictions. To address this issue, we introduce a simple yet effective pooling operation called the Haar Wavelet-based Downsampling (HWD) module. This module can be easily integrated into CNNs to enhance the performance of semantic segmentation models. The core idea of HWD is to apply Haar wavelet transform for reducing the spatial resolution of feature maps while preserving as much information as possible. Furthermore, to investigate the benefits of HWD, we propose a novel metric, named as feature entropy index (FEI), which measures the degree of information uncertainty after downsampling in CNNs. Specifically, the FEI can be used to indicate the ability of downsampling methods to preserve essential information in semantic segmentation. Our comprehensive experiments demonstrate that the proposed HWD module could (1) effectively improve the segmentation performance across different modality image datasets with various CNN architectures, and (2) efficiently reduce information uncertainty compared to the conventional downsampling methods. Our implementation are available at https://github.com/apple1986/HWD.

Identity Feature Disentanglement for Visible-Infrared Person Re-Identification

Visible-infrared person re-identification (VI-ReID) task aims to retrieve persons from different spectrum cameras (i.e., visible and infrared images). The biggest challenge of VI-ReID is the huge cross-modal discrepancy caused by different imaging mechanisms. Many VI-ReID methods have been proposed by embedding different modal person images into a shared feature space to narrow the cross-modal discrepancy. However, these methods ignore the purification of identity features, which results in identity features containing different modal information and failing to align well. In this article, an identity feature disentanglement method is proposed to disentangle the identity features from identity-irrelevant information, such as pose and modality. Specifically, images of different modalities are first processed to extract shared features that reduce the cross-modal discrepancy preliminarily. Then the extracted feature of each image is disentangled into a latent identity variable and an identity-irrelevant variable. In order to enforce the latent identity variable to contain as much identity information as possible and as little identity-irrelevant information, an ID-discriminative loss and an ID-swapping reconstruction process are additionally designed. Extensive quantitative and qualitative experiments on two popular public VI-ReID datasets, RegDB and SYSU-MM01, demonstrate the efficacy and superiority of the proposed method.