Cross-modality Image Research Articles

A light-weight, efficient, and general cross-modal image fusion network

Existing cross-modal image fusion methods pay limited research attention to image fusion efficiency and network architecture. However, the efficiency and accuracy of image fusion have an important impact on practical applications. To solve this problem, we propose a light-weight, efficient, and general cross-modal image fusion network, termed as AE-Netv2. Firstly, we analyze the influence of different network architectures (e.g., group convolution, depth-wise convolution, inceptionNet, squeezeNet, shuffleNet, and multi-scale module) on image fusion quality and efficiency, which provides a reference for the design of image fusion architecture. Secondly, we explore the commonness and characteristics of different image fusion tasks, which provides a research basis for further research on the continuous learning characteristics of the human brain. Finally, positive sample loss is added to the similarity loss to reduce the difference of data distribution of different cross-modal image fusion tasks. Comprehensive experiments demonstrate the superiority of our method compared to state-of-the-art methods in different fusion tasks at a real-time speed of 100+ FPS on GTX 2070. Compared with the fastest image fusion method based on deep learning, the efficiency of AE-Netv2 is improved by 2.14 times. Compared with the image fusion model with the smallest model size, the size of our model is reduced by 11.59 times.

Examining the Feasibility of Quantifying Receptor Availability Using Cross-Modality Paired-Agent Imaging

PurposeThe ability to noninvasively quantify receptor availability (RA) in solid tumors is an aspirational goal of molecular imaging, often challenged by the influence of non-specific accumulation of the contrast agent. Paired-agent imaging (PAI) techniques aim to compensate for this effect by imaging the kinetics of a targeted agent and an untargeted isotype, often simultaneously, and comparing the kinetics of the two agents to estimate RA. This is usually accomplished using two spectrally distinct fluorescent agents, limiting the technique to superficial tissues and/or preclinical applications. Applying the approach in humans using conventional imaging modalities is generally infeasible since most modalities are unable to routinely image multiple agents simultaneously. We examine the ability of PAI to be implemented in a cross-modality paradigm, in which the targeted and untargeted agent kinetics are imaged with different modalities and used to recover receptor availability.ProceduresEighteen mice bearing orthotopic brain tumors were administered a solution containing three contrast agents: (1) a fluorescent agent targeted to epidermal growth factor receptor (EGFR), (2) an untargeted fluorescent isotype, and (3) a gadolinium-based contrast agent (GBCA) for MRI imaging. The kinetics of all three agents were imaged for 1 h after administration using an MRI-coupled fluorescence tomography system. Paired-agent receptor availability was computed using (1) the conventional all-optical approach using the targeted and untargeted optical agent images and (2) the cross-modality approach using the targeted optical and untargeted MRI-GBCA images. Receptor availability estimates between the two methods were compared.ResultsReceptor availability values using the cross-modality approach were highly correlated to the conventional, single-modality approach (r = 0.94; p < 0.00001).ConclusionThese results suggest that cross-modality paired-agent imaging for quantifying receptor availability is feasible. Ultimately, cross-modality paired-agent imaging could facilitate rapid, noninvasive receptor availability quantification in humans using hybrid clinical imaging modalities.

Deep-Learning-Based Virtual Refocusing of Images Using an Engineered Point-Spread Function

We present a virtual image refocusing method over an extended depth of field (DOF) enabled by cascaded neural networks and a double-helix point-spread function (DH-PSF). This network model, referred to as W-Net, is composed of two cascaded generator and discriminator network pairs. The first generator network learns to virtually refocus an input image onto a user-defined plane, while the second generator learns to perform a cross-modality image transformation, improving the lateral resolution of the output image. Using this W-Net model with DH-PSF engineering, we extend the DOF of a fluorescence microscope by ~20-fold. This approach can be applied to develop deep learning-enabled image reconstruction methods for localization microscopy techniques that utilize engineered PSFs to improve their imaging performance, including spatial resolution and volumetric imaging throughput.

Cross-Modality Imaging of Murine Tumor Vasculature\u2014a Feasibility Study

Tumor vasculature and angiogenesis play a crucial role in tumor progression. Their visualization is therefore of utmost importance to the community. In this proof-of-principle study, we have established a novel cross-modality imaging (CMI) pipeline to characterize exactly the same murine tumors across scales and penetration depths, using orthotopic models of melanoma cancer. This allowed the acquisition of a comprehensive set of vascular parameters for a single tumor. The workflow visualizes capillaries at different length scales, puts them into the context of the overall tumor vessel network and allows quantification and comparison of vessel densities and morphologies by different modalities. The workflow adds information about hypoxia and blood flow rates. The CMI approach includes well-established technologies such as magnetic resonance imaging (MRI), positron emission tomography (PET), computed tomography (CT), and ultrasound (US), and modalities that are recent entrants into preclinical discovery such as optical coherence tomography (OCT) and high-resolution episcopic microscopy (HREM). This novel CMI platform establishes the feasibility of combining these technologies using an extensive image processing pipeline. Despite the challenges pertaining to the integration of microscopic and macroscopic data across spatial resolutions, we also established an open-source pipeline for the semi-automated co-registration of the diverse multiscale datasets, which enables truly correlative vascular imaging. Although focused on tumor vasculature, our CMI platform can be used to tackle a multitude of research questions in cancer biology.

Non-linear and selective fusion of cross-modal images

Existing image fusion methods pay little research attention to human visual characteristics. However, human visual characteristics play an important role in visual processing tasks. To solve this problem, we propose a cross-modal image fusion method that combines illuminance factors and attention mechanisms. Human visual characteristics are studied and simulated in cross-modal image fusion task. Firstly, in order to reject high and low-frequency mixing and reduce the halo effect, we perform cross-modal image multi-scale decomposition. Secondly, in order to remove highlights, the visual saliency map and the deep feature map are combined with the illuminance fusion factor to perform high-low frequency non-linear fusion. Thirdly, the feature maps are selected through a channel attention network to obtain the final fusion map. Finally, we validate our image fusion method on public datasets of infrared and visible images. The experimental results demonstrate the superiority of our fusion method under the complex illumination environment. In addition, the experimental results also demonstrate the effectiveness of our simulation of human visual characteristics.

Cross-modality image feature fusion diagnosis in breast cancer

Considering the complementarity of mammography and breast MRI, the research of feature fusion diagnosis based on cross-modality images was explored to improve the accuracy of breast cancer diagnosis. 201 patients with both mammography and breast MRI were collected retrospectively, including 117 cases of benign lesions and 84 cases of malignant ones. Two feature optimization strategies of sequential floating forward selection (SFFS), SFFS-1 and SFFS-2, were defined based on the sequential floating forward selection method. Each strategy was used to analyze the diagnostic performance of single-modality images and then to study the feature fusion diagnosis of cross-modality images. Three feature fusion approaches were compared: optimizing MRI features and then fusing those of mammography; optimizing mammography features and then fusing those of MRI; selecting the effective features from the whole feature set (mammography and MRI). Support vector machine, Naive Bayes, and K-nearest neighbor were employed as the classifiers and were finally integrated to get better performance. The average accuracy and area under the ROC curve (AUC) of MRI (88.56%, 0.9 for SFFS-1, 88.39%, 0.89 for SFFS-2) were better than mammography (84.25%, 0.84 for SFFS-1, 80.43%, 0.80 for SFFS-2). Furthermore, compared with a single modality, the average accuracy and AUC of cross-modality feature fusion can improve from 85.40% and 0.86 to 89.66% and 0.91. Classifier integration improved the accuracy and AUC from 90.49%, 0.92 to 92.37%, and 0.97. Cross-modality image feature fusion can achieve better diagnosis performance than a single modality. Feature selection strategy SFFS-1 has better efficiency than SFFS-2. Classifier integration can further improve diagnostic accuracy.

Disentangle domain features for cross-modality cardiac image segmentation

Unsupervised domain adaptation (UDA) generally learns a mapping to align the distribution of the source domain and target domain. The learned mapping can boost the performance of the model on the target data, of which the labels are unavailable for model training. Previous UDA methods mainly focus on domain-invariant features (DIFs) without considering the domain-specific features (DSFs), which could be used as complementary information to constrain the model. In this work, we propose a new UDA framework for cross-modality image segmentation. The framework first disentangles each domain into the DIFs and DSFs. To enhance the representation of DIFs, self-attention modules are used in the encoder which allows attention-driven, long-range dependency modeling for image generation tasks. Furthermore, a zero loss is minimized to enforce the information of target (source) DSFs, contained in the source (target) images, to be as close to zero as possible. These features are then iteratively decoded and encoded twice to maintain the consistency of the anatomical structure. To improve the quality of the generated images and segmentation results, several discriminators are introduced for adversarial learning. Finally, with the source data and their DIFs, we train a segmentation network, which can be applicable to target images. We validated the proposed framework for cross-modality cardiac segmentation using two public datasets, and the results showed our method delivered promising performance and compared favorably to state-of-the-art approaches in terms of segmentation accuracies. The source code of this work will be released via https://zmiclab.github.io/projects.html, once this manuscript is accepted for publication.

Grayscale Enhancement Colorization Network for Visible-Infrared Person Re-Identification

Visible-infrared person re-identification (VI-ReID) is an emerging and challenging cross-modality image matching problem because of the explosive surveillance data in night-time surveillance applications. To handle the large modality gap, various generative adversarial network models have been developed to eliminate the cross-modality variations based on a cross-modal image generation framework. However, the lack of point-wise cross-modality ground-truths makes it extremely challenging to learn such a cross-modal image generator. To address these problems, we learn the correspondence between single-channel infrared images and three-channel visible images by generating intermediate grayscale images as auxiliary information to colorize the single-modality infrared images. We propose a grayscale enhancement colorization network (GECNet) to bridge the modality gap by retaining the structure of the colored image which contains rich information. To simulate the infrared-to-visible transformation, the point-wise transformed grayscale images greatly enhance the colorization process. Our experiments conducted on two visible-infrared cross-modality person re-identification datasets demonstrate the superiority of the proposed method over the state-of-the-arts.

Recurrent neural network-based volumetric fluorescence microscopy

Volumetric imaging of samples using fluorescence microscopy plays an important role in various fields including physical, medical and life sciences. Here we report a deep learning-based volumetric image inference framework that uses 2D images that are sparsely captured by a standard wide-field fluorescence microscope at arbitrary axial positions within the sample volume. Through a recurrent convolutional neural network, which we term as Recurrent-MZ, 2D fluorescence information from a few axial planes within the sample is explicitly incorporated to digitally reconstruct the sample volume over an extended depth-of-field. Using experiments on C. elegans and nanobead samples, Recurrent-MZ is demonstrated to significantly increase the depth-of-field of a 63×/1.4NA objective lens, also providing a 30-fold reduction in the number of axial scans required to image the same sample volume. We further illustrated the generalization of this recurrent network for 3D imaging by showing its resilience to varying imaging conditions, including e.g., different sequences of input images, covering various axial permutations and unknown axial positioning errors. We also demonstrated wide-field to confocal cross-modality image transformations using Recurrent-MZ framework and performed 3D image reconstruction of a sample using a few wide-field 2D fluorescence images as input, matching confocal microscopy images of the same sample volume. Recurrent-MZ demonstrates the first application of recurrent neural networks in microscopic image reconstruction and provides a flexible and rapid volumetric imaging framework, overcoming the limitations of current 3D scanning microscopy tools.

Self-Attentive Spatial Adaptive Normalization for Cross-Modality Domain Adaptation.

Despite the successes of deep neural networks on many challenging vision tasks, they often fail to generalize to new test domains that are not distributed identically to the training data. The domain adaptation becomes more challenging for cross-modality medical data with a notable domain shift. Given that specific annotated imaging modalities may not be accessible nor complete. Our proposed solution is based on the cross-modality synthesis of medical images to reduce the costly annotation burden by radiologists and bridge the domain gap in radiological images. We present a novel approach for image-to-image translation in medical images, capable of supervised or unsupervised (unpaired image data) setups. Built upon adversarial training, we propose a learnable self-attentive spatial normalization of the deep convolutional generator network's intermediate activations. Unlike previous attention-based image-to-image translation approaches, which are either domain-specific or require distortion of the source domain's structures, we unearth the importance of the auxiliary semantic information to handle the geometric changes and preserve anatomical structures during image translation. We achieve superior results for cross-modality segmentation between unpaired MRI and CT data for multi-modality whole heart and multi-modal brain tumor MRI (T1/T2) datasets compared to the state-of-the-art methods. We also observe encouraging results in cross-modality conversion for paired MRI and CT images on a brain dataset. Furthermore, a detailed analysis of the cross-modality image translation, thorough ablation studies confirm our proposed method's efficacy.

An Attention Enhanced Cross-Modal Image–Sound Mutual Generation Model for Birds

Abstract Cross-modal bird image–audio mutual generation has appealing potential benefits for bird classification. To achieve promising cross-modal bird visual–audio mutual generation, we propose an attention enhanced cross-modal cycle adversarial generation network. Specifically, the attention module endows our model with long-term intra-modality dependency and inter-modality dependency capabilities, which can provide more information during the generation process and further improve the generation performance. Moreover, because there was no dataset concerning bird visual–audio mutual generation, the authors established a novel bird cross-modal generation dataset, called Bird_Crossmodal_Generation (BCG). Based on BCG, our model obtains promising performance and achieves significant improvement under both inception score and Frechet inception distance criteria. The experimental results validate the feasibility of the proposed task and the superiority of our model. Additionally, this investigation provides a basis for more researchers to develop cross-modality methods for bird visual–audio generation.

Cross-modal image sentiment analysis via deep correlation of textual semantic

Social media has become indispensable to people’s lives, where they can share their views and emotion with images and texts. Analyzing social images for sentiment prediction can help understand human social behavior and provide better recommendation results. Most current researches on image sentiment analysis have achieved quite good progress, which ignores the semantic correlation between an image and its corresponding descriptive sentences (caption). To capture the complementary multimodal information for joint sentiment classification, in this paper, we propose a novel cross-modal Semantic Content Correlation (SCC) method based on deep matching and hierarchical networks, which bridges the correlation between images and captions. Specifically, pre-trained convolutional neural networks (CNNs) are leveraged to encode the visual sub-regions contents, and a GloVe is employed to embed the textual semantic. Relying on visual contents and textual semantic, a joint attention network is proposed to learn the content correlation of the image and its caption, which is then exported as an image–text pair. To exploit the dependence of visual contents on textual semantic in caption effectively, the caption is processed by a Class-Aware Sentence Representation (CASR) network with a class dictionary, and a fully connected layer concatenates the outputs of CASR into a class-aware vector. Finally, the class-aware distributed vector is fed into an Inner-class Dependency Long Short-Term Memory network (IDLSTM) with the image–text pair as a query to further capture the cross-modal non-linear correlations for sentiment prediction. The performance of extensive experiments conducted on three datasets verifies the effectiveness of the model SCC.

Infrared and Visible Cross-Modal Image Retrieval Through Shared Features

Image retrieval is one of the key techniques of computer vision, and has been studied for a long time. Nevertheless, little attention is paid to infrared and visible cross-modal retrieval which can be widely used in various applications, e.g., infrared and visible surveillance systems. In this paper, we propose a shared features based infrared-visible cross-modal image retrieval method. The similar visual features are extracted from infrared and visible images as the shared features, and the Euclidean distance is used to measure the similarity between these features. The core of the proposed method comes from three aspects: 1) Feature separation network can separate image features into shared features and exclusive features; 2) Maximum Mean Discrepancy (MMD) loss is employed to constrain the distribution of shared features, which can reduce the retrieval error caused by different imaging angles and similarity of infrared images. 3) The cross-layer fusion encoder compensates for the context loss in the convolution of infrared images. Experimental results on the Infrared-Visible dataset demonstrate the proposed method is effective and outperforms the state-of-the-art approaches.

Attend to the Difference: Cross-Modality Person Re-Identification via Contrastive Correlation.

The problem of cross-modality person re-identification has been receiving increasing attention recently, due to its practical significance. Motivated by the fact that human usually attend to the difference when they compare two similar objects, we propose a dual-path cross-modality feature learning framework which preserves intrinsic spatial structures and attends to the difference of input cross-modality image pairs. Our framework is composed by two main components: a Dual-path Spatial-structure-preserving Common Space Network (DSCSN) and a Contrastive Correlation Network (CCN). The former embeds cross-modality images into a common 3D tensor space without losing spatial structures, while the latter extracts contrastive features by dynamically comparing input image pairs. Note that the representations generated for the input RGB and Infrared images are mutually dependant to each other. We conduct extensive experiments on two public available RGB-IR ReID datasets, SYSU-MM01 and RegDB, and our proposed method outperforms state-of-the-art algorithms by a large margin with both full and simplified evaluation modes.

Synthetic CT Generation of the Pelvis in Patients With Cervical Cancer: A Single Input Approach Using Generative Adversarial Network.

Multi-modality imaging constitutes a foundation of precision medicine, especially in oncology where reliable and rapid imaging techniques are needed in order to insure adequate diagnosis and treatment. In cervical cancer, precision oncology requires the acquisition of 18F-labeled 2-fluoro-2-deoxy-D-glucose (FDG) positron emission tomography (PET), magnetic resonance (MR), and computed tomography (CT) images. Thereafter, images are co-registered to derive electron density attributes required for FDG-PET attenuation correction and radiation therapy planning. Nevertheless, this traditional approach is subject to MR-CT registration defects, expands treatment expenses, and increases the patient’s radiation exposure. To overcome these disadvantages, we propose a new framework for cross-modality image synthesis which we apply on MR-CT image translation for cervical cancer diagnosis and treatment. The framework is based on a conditional generative adversarial network (cGAN) and illustrates a novel tactic that addresses, simplistically but efficiently, the paradigm of vanishing gradient vs. feature extraction in deep learning. Its contributions are summarized as follows: 1) The approach –termed sU-cGAN-uses, for the first time, a shallow U-Net (sU-Net) with an encoder/decoder depth of 2 as generator; 2) sU-cGAN’s input is the same MR sequence that is used for radiological diagnosis, i.e. T2-weighted, Turbo Spin Echo Single Shot (TSE-SSH) MR images; 3) Despite limited training data and a single input channel approach, sU-cGAN outperforms other state of the art deep learning methods and enables accurate synthetic CT (sCT) generation. In conclusion, the suggested framework should be studied further in the clinical settings. Moreover, the sU-Net model is worth exploring in other computer vision tasks.

Cross-modality imaging of bisphosphonate-treated murine jawbones.

In this proof-of-principle study, we established and implemented a cross-modality imaging (CMI) pipeline to characterize and compare bisphosphonate (BIS)-treated jawbones of Sprague-Dawley rats after tooth extraction after physical therapies (photobiomodulation and extracorporeal shockwave therapy (PBMT and ESWT)). We showcase the feasibility of such a CMI approach and its compatibility across imaging modalities to probe the same region of interest (ROI) of the same jawbone. Jawbones were imaged in toto in 3D using micro-Computed Tomography to identify ROIs for subsequent sequential 2D analysis using well-established technologies such as Atomic Force Microscopy and Scanning Electron Microscopy, and recent imaging approaches in biomedical settings, such as micro-X-Ray Fluorescence Spectroscopy. By combining these four modalities, multiscale information on the morphology, topography, mechanical stiffness (Young's modulus), and calcium, zinc and phosphorus concentrations of the bone was collected. Based on the CMI pipeline, we characterized and compared the jawbones of a previously published clinically relevant rat model of BIS-related osteonecrosis of the jawbone (BRONJ) before and after treatment with BISs, PBMT and ESWT. While we did not find that physical therapies altered the mechanical and elemental jawbone parameters with significance (probably due to the small sample size of only up to 5 samples per group), both ESWT and PBMT reduced pore thicknesses and bone-to-enamel distances significantly compared to the controls. Although focused on BIS-treated jawbones, the established CMI platform can be beneficial in the study of bone-related diseases in general (such as osteoarthritis or -porosis) to acquire complementary hallmarks and better characterize disease status and alleviation potentials.

Anatomy-Regularized Representation Learning for Cross-Modality Medical Image Segmentation.

An increasing number of studies are leveraging unsupervised cross-modality synthesis to mitigate the limited label problem in training medical image segmentation models. They typically transfer ground truth annotations from a label-rich imaging modality to a label-lacking imaging modality, under an assumption that different modalities share the same anatomical structure information. However, since these methods commonly use voxel/pixel-wise cycle-consistency to regularize the mappings between modalities, high-level semantic information is not necessarily preserved. In this paper, we propose a novel anatomy-regularized representation learning approach for segmentation-oriented cross-modality image synthesis. It learns a common feature encoding across different modalities to form a shared latent space, where 1) the input and its synthesis present consistent anatomical structure information, and 2) the transformation between two images in one domain is preserved by their syntheses in another domain. We applied our method to the tasks of cross-modality skull segmentation and cardiac substructure segmentation. Experimental results demonstrate the superiority of our method in comparison with state-of-the-art cross-modality medical image segmentation methods.

Robotic cross-modal generative adversarial network based on variational Bayesian Gaussian mixture noise model

Multimodal fusion is essential for robots to fully perceive the external environment. Single modal information limits the ability of robots to recognize and grasp objects. Meanwhile, traditional cross-modal data generation methods produce poor images, resulting in the bad effects of multimodal fusion. To solve the problem of the poor image effects of multimodal generation and the lack of data for multimodal fusion, this study proposes a variational Bayesian Gaussian mixture-conditional generative adversarial network (BGM-CGAN) for generating diverse cross-modal noise data. With the variational Bayesian Gaussian mixture algorithm, a uniform distributed random noise group is generated into a single mixed variable. The generated mixed variable is then generated through a Gaussian mixture model to generate a series of Gaussian mixed noise groups. The generated mixed noise group is randomly selected into a single Gaussian noise and imported into a modal image and fused with the modal image to successfully generate a high-resolution heterogeneous modal image. The method restores the heterogeneous modal information and solves the problem of the insufficient information and poor quality of generated images under a single modal. In the end, we use a variety of evaluation indexes (IS, FID, SSIM, and PSNR) to compare the proposed BGM-CGAN and other algorithms for cross-modal image generation capabilities. Results show the effectiveness and feasibility of the proposed algorithm. In addition, the BGM-CGAN algorithm has wide application prospects and can be extended to cross-modal material retrieval, cross-modal texture recognition, and other fields.

CF Distance: A New Domain Discrepancy Metric and Application to Explicit Domain Adaptation for Cross-Modality Cardiac Image Segmentation

Domain adaptation has great values in unpaired cross-modality image segmentation, where the training images with gold standard segmentation are not available from the target image domain. The aim is to reduce the distribution discrepancy between the source and target domains. Hence, an effective measurement for this discrepancy is critical. In this work, we propose a new metric based on characteristic functions of distributions. This metric, referred to as CF distance, enables explicit domain adaptation, in contrast to the implicit manners minimizing domain discrepancy via adversarial training. Based on this CF distance, we propose an unsupervised domain adaptation framework for cross-modality cardiac segmentation, which consists of image reconstruction and prior distribution matching. We validated the method on two tasks, i.e., the CT-MR cross-modality segmentation and the multi-sequence cardiac MR segmentation. Results showed that the proposed explicit metric was effective in domain adaptation, and the segmentation method delivered promising and superior performance, compared to other state-of-the-art techniques. The data and source code of this work has been released via https://zmiclab.github.io/projects.html.

Variable length deep cross-modal hashing based on Cauchy probability function

With the rapid development of multimedia technology, considerable achievement has been achieved in image retrieval technology. On the one hand, users’ demand for cross-modal retrieval is increasing rapidly. On the other hand, deep hashing algorithms as one of the most prominent high-dimensional reduction methods have received extensive attention. Under such background, the cross-modal retrieval method based on deep hashing came into being. Although the cross-modal hashing learning method has moved a long way in recent years, it still has space of promotion. First, existing cross-modal based hashing algorithms can only map the feature vectors into binary codes with a fixed length. The length of hash codes cannot be modified according to the practical situation. Second, the saturation level of the probability function used in existing methods is too high to concentrate relevant samples very well, which results in low retrieval efficiency. To solve the above problem, a novel variable-length hash code based on adaptive weight was proposed in this paper. The length of hash codes could be adjusted according to the importance of each bit. And a novel probability function based on Cauchy distribution was proposed to generate compact binary codes and make hashing retrieval more efficient. The experiment shows that the proposed cross-modal image retrieval algorithm based on deep hashing outperforms existing related algorithms on the accuracy of retrieval.