Latent Low-rank Representation Research Articles

Global structure graph mapping for multimodal change detection

ABSTRACT Multimodal change detection (MCD) combines multiple remote sensing data sources to realize surface change monitoring, which is essential for disaster evaluation and environmental monitoring. However, due to the ‘incomparable’ features in multimodal data, traditional change detection methods for unimodal (homogenous) data no longer apply. To address this issue, this paper proposes a novel MCD method with global structure graph mapping (GSGM) which extracts the ‘comparable’ structural features between multimodal datasets and constructs a global structure graph (GSG) to express the structure information for each of the multi-temporal images, which are then cross-mapped to the other data domain. The change intensity (CI) is determined by measuring the change of GSGs after mapping and the differences between GSGs and mapped GSGs. The forward and backward CI maps (CIMs) are then fused with the latent low-rank representation method (LLRR), and the change map (CM) is obtained by threshold segmentation. Experiments on five multimodal and four unimodal datasets demonstrate the effectiveness and robustness of the proposed method (source code is made available at https://github.com/rshante0426/GSGM).

Efficient subspace clustering and feature extraction via [formula omitted]-norm and [formula omitted]-norm minimization

The nuclear norm-based Latent Low-Rank Representation (LatLRR) has gained much attention due to its success in subspace clustering and feature extraction. However, it suffers from high computational costs due to the calculation of singular value decomposition for large matrices. To this end, we develop an efficient subspace clustering and feature extraction method (ESCFE) which substitutes the nuclear norm with the ℓ2,1-norm and ℓ1,2-norm respectively. Theoretically proof shows both the ℓ2,1-norm and ℓ1,2-norm can serve as the convex surrogates of the nuclear norm while can derive closed-form solutions. Furthermore, the ℓ2,1-norm (or ℓ1,2-norm) regularization promotes column (or row) structure sparsity due to the discriminative nature inherited from the ℓ1-norm. Thus the proposed ESCFE is robust to outliers in data and can extract features with joint sparsity. Extensive experiments on multiple benchmark datasets demonstrate the superiority of our method in both efficiency and effectiveness.

Using Sparse Parts in Fused Information to Enhance Performance in Latent Low-Rank Representation-Based Fusion of Visible and Infrared Images.

Latent Low-Rank Representation (LatLRR) has emerged as a prominent approach for fusing visible and infrared images. In this approach, images are decomposed into three fundamental components: the base part, salient part, and sparse part. The aim is to blend the base and salient features to reconstruct images accurately. However, existing methods often focus more on combining the base and salient parts, neglecting the importance of the sparse component, whereas we advocate for the comprehensive inclusion of all three parts generated from LatLRR image decomposition into the image fusion process, a novel proposition introduced in this study. Moreover, the effective integration of Convolutional Neural Network (CNN) technology with LatLRR remains challenging, particularly after the inclusion of sparse parts. This study utilizes fusion strategies involving weighted average, summation, VGG19, and ResNet50 in various combinations to analyze the fusion performance following the introduction of sparse parts. The research findings show a significant enhancement in fusion performance achieved through the inclusion of sparse parts in the fusion process. The suggested fusion strategy involves employing deep learning techniques for fusing both base parts and sparse parts while utilizing a summation strategy for the fusion of salient parts. The findings improve the performance of LatLRR-based methods and offer valuable insights for enhancement, leading to advancements in the field of image fusion.

Nighttime Thermal Infrared Image Translation Integrating Visible Images

Nighttime Thermal InfraRed (NTIR) image colorization, also known as the translation of NTIR images into Daytime Color Visible (DCV) images, can facilitate human and intelligent system perception of nighttime scenes under weak lighting conditions. End-to-end neural networks have been used to learn the mapping relationship between temperature and color domains, and translate NTIR images with one channel into DCV images with three channels. However, this mapping relationship is an ill-posed problem with multiple solutions without constraints, resulting in blurred edges, color disorder, and semantic errors. To solve this problem, an NTIR2DCV method that includes two steps is proposed: firstly, fuse Nighttime Color Visible (NCV) images with NTIR images based on an Illumination-Aware, Multilevel Decomposition Latent Low-Rank Representation (IA-MDLatLRR) method, which considers the differences in illumination conditions during image fusion and adjusts the fusion strategy of MDLatLRR accordingly to suppress the adverse effects of nighttime lights; secondly, translate the Nighttime Fused (NF) image to DCV image based on HyperDimensional Computing Generative Adversarial Network (HDC-GAN), which ensures feature-level semantic consistency between the source image (NF image) and the translated image (DCV image) without creating semantic label maps. Extensive comparative experiments and the evaluation metrics values show that the proposed algorithms perform better than other State-Of-The-Art (SOTA) image fusion and translation methods, such as FID and KID, which decreased by 14.1 and 18.9, respectively.

Development and validation of survival prognostic models for head and neck cancer patients using machine learning and dosiomics and CT radiomics features: a multicentric study

BackgroundThis study aimed to investigate the value of clinical, radiomic features extracted from gross tumor volumes (GTVs) delineated on CT images, dose distributions (Dosiomics), and fusion of CT and dose distributions to predict outcomes in head and neck cancer (HNC) patients.MethodsA cohort of 240 HNC patients from five different centers was obtained from The Cancer Imaging Archive. Seven strategies, including four non-fusion (Clinical, CT, Dose, DualCT-Dose), and three fusion algorithms (latent low-rank representation referred (LLRR),Wavelet, weighted least square (WLS)) were applied. The fusion algorithms were used to fuse the pre-treatment CT images and 3-dimensional dose maps. Overall, 215 radiomics and Dosiomics features were extracted from the GTVs, alongside with seven clinical features incorporated. Five feature selection (FS) methods in combination with six machine learning (ML) models were implemented. The performance of the models was quantified using the concordance index (CI) in one-center-leave-out 5-fold cross-validation for overall survival (OS) prediction considering the time-to-event.ResultsThe mean CI and Kaplan-Meier curves were used for further comparisons. The CoxBoost ML model using the Minimal Depth (MD) FS method and the glmnet model using the Variable hunting (VH) FS method showed the best performance with CI = 0.73 ± 0.15 for features extracted from LLRR fused images. In addition, both glmnet-Cindex and Coxph-Cindex classifiers achieved a CI of 0.72 ± 0.14 by employing the dose images (+ incorporated clinical features) only.ConclusionOur results demonstrated that clinical features, Dosiomics and fusion of dose and CT images by specific ML-FS models could predict the overall survival of HNC patients with acceptable accuracy. Besides, the performance of ML methods among the three different strategies was almost comparable.

Reweighted robust and discriminative latent subspace projection for face recognition

Subspace projection has been widely studied to implement the feature extraction for face recognition. However, it scarcely facilitates exploring the label information in the subspace projection procedure. The feature extraction and classification are usually divided into two separate procedures. This fails to explore the discriminative information from the training data, and thus degrades the discrimination of the extracted features and lowers the face recognition accuracy. To address this issue, we present a novel reweighted robust and discriminative latent subspace projection (ReRDLSP) method which organically integrates the reweighted latent low-rank representation model and discriminative ridge regression into a union framework. We employ a reweighted nuclear norm minimization to realize the low-rank regularization, which provides adaptive weighted values assignment, regularizes the representation matrix adaptively, and thus boosts the capability and flexibility of the method. Moreover, a discriminative ridge regression method is presented by employing a nonnegative relaxation, which enlarges the margins of the face samples from different classes as much as possible and boosts the freedom of fitting the label information. The computational complexity and convergence analysis are also discussed in detail. Extensive experiments are demonstrated and very promising results are achieved compared with some state-of-the-art face recognition methods.

Medical image fusion method based on saliency measurement improvement and local structure similarity correction

Medical image fusion is an important tool in medical diagnosis, because the quality of the fused image is closely related to the diagnosis result. However, the approaches of highlighting details through the acquisition of high-contrast lead to information distortion in the fused image and then increase the misdiagnosis rate. In order to solve the problem, a medical image fusion method based on saliency measurement improvement and local structural similarity correction is proposed in this paper. First, multi-level decomposition latent low-rank representation (MDLatLRR) combined with nonsubsampled shearlet transformation (NSST) is presented to decompose source images to obtain high SNR base layers and multi-scale detail layers. Then, the saliency measurement improvement with the source image mask filtering is advanced for base layer fusion rule. Specially, after the base layer is refined by the source image mask for the salient area extraction, the salient area is refreshed by its mean gray to calculate the fusion coefficient. Finally, a correction method based on structural similarity index measure (SSIM) is put forward to deal with the information distortion in the initial fused image, in which the correction weight is acquired through the weighted structural similarity between the initial fused image and the source ones. Experiments show that our method has excellent subjective visual effects, and outperforms the other states-of-the-art methods in objective indexes.

BTSFusion: Fusion of infrared and visible image via a mechanism of balancing texture and salience

In recent years, deep learning research has received significant attention in the field of infrared and visible image fusion. However, the issue of designing loss functions in deep learning-based image fusion methods has not been well-addressed. To tackle this problem, we propose a novel mechanism of utilizing traditional fusion methods as loss functions to guide the training of deep learning models. We incorporate the superior aspects of two traditional methods, namely Guided Filter (GF) and Latent Low-Rank Representation (LatLRR), into the design of the loss function, proposing a fusion method for infrared and visible images that balances both texture and saliency, termed BTSFusion. The proposed network is not only lightweight but also preserves the maximum amount of valuable information in source images. It is worth noting that the complexity of BTSFusion primarily lies in the design of the loss function, which allows it to remain an end-to-end network, as demonstrated by efficiency comparison experiments that highlight the excellent computational efficiency of our algorithm. Furthermore, through subjective observations and objective comparisons, we validated the performance of the proposed method by comparing it with twelve state-of-the-art methods on two public datasets. The source code will be publicly available at https://github.com/YQ-097/BTSFusion.

Latent Low-Rank Representation With Weighted Distance Penalty for Clustering.

Latent low-rank representation (LatLRR) is a critical self-representation technique that improves low-rank representation (LRR) by using observed and unobserved samples. It can simultaneously learn the low-dimensional structure embedded in the data space and capture the salient features. However, LatLRR ignores the local geometry structure and can be affected by the noise and redundancy in the original data space. To solve the above problems, we propose a latent LRR with weighted distance penalty (LLRRWD) for clustering in this article. First, a weighted distance is proposed to enhance the original Euclidean distance by enlarging the distance among the unconnected samples, which can enhance the discriminitation of the distance among the samples. By leveraging on the weighted distance, a weighted distance penalty is introduced to the LatLRR model to enable the method to preserve both the local geometric information and global information, improving discrimination of the learned affinity matrix. Moreover, a weight matrix is imposed on the sparse error norm to reduce the effect of noise and redundancy. Experimental results based on several benchmark databases show the effectiveness of our method in clustering.

Information fusion for fully automated segmentation of head and neck tumors from PET and CT images.

PET/CT images combining anatomic and metabolic data provide complementary information that can improve clinical task performance. PET image segmentation algorithms exploiting the multi-modal information available are still lacking. Our study aimed to assess the performance of PET and CT image fusion for gross tumor volume (GTV) segmentations of head and neck cancers (HNCs) utilizing conventional, deep learning (DL), and output-level voting-based fusions. The current study is based on a total of 328 histologically confirmed HNCs from six different centers. The images were automatically cropped to a 200 × 200 head and neck region box, and CT and PET images were normalized for further processing. Eighteen conventional image-level fusions were implemented. In addition, a modified U2-Net architecture as DL fusion model baseline was used. Three different input, layer, and decision-level information fusions were used. Simultaneous truth and performance level estimation (STAPLE) and majority voting to merge different segmentation outputs (from PET and image-level and network-level fusions), that is, output-level information fusion (voting-based fusions) were employed. Different networks were trained in a 2D manner with a batch size of 64. Twenty percent of the dataset with stratification concerning the centers (20% in each center) were used for final result reporting. Different standard segmentation metrics and conventional PET metrics, such as SUV, were calculated. In single modalities, PET had a reasonable performance with a Dice score of 0.77±0.09, while CT did not perform acceptably and reached a Dice score of only 0.38±0.22. Conventional fusion algorithms obtained a Dice score range of [0.76-0.81] with guided-filter-based context enhancement (GFCE) at the low-end, and anisotropic diffusion and Karhunen-Loeve transform fusion (ADF), multi-resolution singular value decomposition (MSVD), and multi-level image decomposition based on latent low-rank representation (MDLatLRR) at the high-end. All DL fusion models achieved Dice scores of 0.80. Output-level voting-based models outperformed all other models, achieving superior results with a Dice score of 0.84 for Majority_ImgFus, Majority_All, and Majority_Fast. A mean error of almost zero was achieved for all fusions using SUVpeak , SUVmean and SUVmedian . PET/CT information fusion adds significant value to segmentation tasks, considerably outperforming PET-only and CT-only methods. In addition, both conventional image-level and DL fusions achieve competitive results. Meanwhile, output-level voting-based fusion using majority voting of several algorithms results in statistically significant improvements in the segmentation of HNC.

A Novel Saliency-Based Decomposition Strategy for Infrared and Visible Image Fusion

The image decomposition strategy that extracts salient features from the source image is crucial for image fusion. To this end, we proposed a novel saliency-based decomposition strategy for infrared and visible image fusion. In particular, the latent low-rank representation (LatLRR) and rolling guidance filter (RGF) are together employed to process source images, which is called DLatLRR_RGF. In this method, the source images are first decomposed to salient components and base components based on LatLRR, and the salient components are filtered by RGF. Then, the final base components can be calculated by the difference between the source image and the processed salient components. The fusion rule based on the nuclear-norm and modified spatial frequency is used to fuse the salient components. The base components are fused by the l2-energy minimization model. Finally, the fused image can be obtained by the fused base components and saliency detail components. Multiple groups of experiments on different pairs of infrared and visible images demonstrate that, compared with other state-of-the-art fusion algorithms, our proposed method possesses superior fusion performance from subjective and objective perspectives.

LatLRR-CNN: an infrared and visible image fusion method combining latent low-rank representation and CNN

LatLRR-CNN: an infrared and visible image fusion method combining latent low-rank representation and CNN

Nested multi-scale transform fusion model: The response evaluation of chemoradiotherapy for patients with lung tumors

Background and Objective: The response evaluation of chemoradiotherapy is an important method of precision treatment for patients with malignant lung tumors. In view of the existing evaluation criteria for chemoradiotherapy, it is difficult to synthesize the geometric and shape characteristics of lung tumors. In the present, the response evaluation of chemoradiotherapy is limited. Therefore, this paper constructs a response evaluation system of chemoradiotherapy based on PET/CT images. Methods: There are two parts in the system: a nested multi-scale fusion model and an attribute sets for the Response evaluation of chemoradiotherapy (AS-REC). In the first part, a new nested multi-scale transform method, i.e., latent low-rank representation (LATLRR) and non-subsampled contourlet transform (NSCT), is proposed. Then, the average gradient self-adaptive weighting is used for the low-frequency fusion rule, and the regional energy fusion rule is used for the high-frequency fusion rule. Further, the low-rank part fusion image is obtained by the inverse NSCT, and the fusion image is generated by adding the low-rank part fusion image and the significant part fusion image. In the second part, AS-REC is constructed to evaluate the growth direction of the tumor, the degree of tumor metabolic activity, and the tumor growth state. Results: the numerical results clearly show that the performance of our proposed method outperforms in comparison with several existing methods, among them, the value of Qabf increased by up to 69%. Conclusions: Through the experiment of three reexamination patients, the effectiveness of the evaluation system of radiotherapy and chemotherapy are proved.

A novel cell image fusion approach based on the collaboration of multilevel latent Low-Rank representation and the convolutional neural network

Multilevel cell image fusion is an image enhancement technique that improves the quality of images. The procedure of the multilevel approach follows an image decomposition and fusion strategy. The image decomposition phase partitions the medical observations into the basic (luminance and contour information) and detailed (structural information) components. Whereas the image fusion stage reconstructs the fused image from the decomposed data patterns. Recent techniques have paid more attention to the fusion of detailed components. These components contain hidden data patterns and they are fused with a well-adopted strategy. The basic components are considered less important features and they are combined with weighted sum rules. Which results in the generation of an insignificant feature map containing more detailed texture and less luminance information. We have addressed the given problem by performing the efficient fusion of sub-components. The basic components of the image are affected by blur regions and varying illuminations. These distortions are minimized by the CNN-based fusion approach. Which generates a powerful decision map by processing the features through multiple stages, which include focus map generation, binary segmentation, consistency verification, and decision map. Parallel strategies are followed for the fusion of detailed data components. These components are combined with a nuclear-norm-based fusion framework. Which is the efficient fusion policy for the combination of local structural patterns. We have evaluated the model by nine quality metrics. The performance of the proposed model is compared with state-of-art methods. It has overpowered the related techniques in qualitative and quantitative analysis.

Infrared-visible image fusion method based on sparse and prior joint saliency detection and LatLRR-FPDE

In this paper, a new infrared and visible image fusion method based on saliency detection and LatLRR-FPDE is proposed to ensure the saliency of the fused target and improve the clarity of the texture detail information of the latent low-rank representation (LatLRR) fusion method. In the feature representation stage, source images are decomposed into low-rank and sparse parts based on LatLRR. Then Fourth order Partial Differential Equation (FPDE) is used to extract high-frequency detail information from the low-rank part. In the fusion decision stage, according to the characteristic difference of the decomposed sub-bands, a multi-scale fusion scheme is carried out using rules such as EM (expectation-maximization), average (AVG), and the maximum absolute value. To guarantee the integrity of thermal radiation features in the target area in fusion results, an infrared image saliency detection algorithm is proposed, which utilizes the fusion information obtained from the sparse and prior saliency feature. Specifically, the algorithm uses convex hull center prior and contrast prior to obtain the background-suppressed saliency map. Meanwhile, to make full use of the high-dimensional saliency features in the image decomposition stage, the prior saliency map and the sparse saliency map are fused to improve the accuracy of saliency detection without the demand to extract redundant feature, which largely increases time-efficiency of the model. Finally, infrared feature compensation is performed on the target region of the fused image based on the saliency mapping strategy to make the final fused image having a better infrared target feature representation. The experiments are based on registered public datasets and compared with various popular image fusion algorithms. The subjective and objective results show that the proposed algorithm can effectively keep the integrity of salient infrared features and has superior visual quality.

Enhancing Underwater Imagery via Latent Low-Rank Decomposition and Image Fusion

The quality of underwater images is usually degraded due to the absorption and scattering effect of seawater, which leads to image distortion and reduces the accuracy and efficiency of subsequent vision tasks. To solve the above problems, an enhancement method for underwater images based on latent low-rank decomposition and image fusion is proposed in this article. First, a color correction method based on adaptive channel compensation is implemented to remove color distortion. Second, an improved Laplace sharpening method and gamma correction technology are applied to effectively improve the sharpness and contrast of the underwater image. Finally, a dual-image weighted image fusion based on latent low-rank representation is proposed to integrate the enhanced image. The experimental results show that this method can obtain better results than other traditional methods on both color reproduction and detail preservation. Furthermore, a set of application tests is executed to further demonstrate the effectiveness of the proposed method.

Image fusion of dual-discriminator generative adversarial network and latent low-rank representation

Image fusion of dual-discriminator generative adversarial network and latent low-rank representation

Hybrid Multimodal Medical Image Fusion Method Based on LatLRR and ED-D2GAN

In order to better preserve the anatomical structure information of Computed Tomography (CT) source images and highlight the metabolic information of lesion regions in Positron Emission Tomography (PET) source images, a hybrid multimodal medical image fusion method (LatLRR-GAN) based on Latent low-rank representation (LatLRR) and the dual discriminators Generative Adversarial Network (ED-D2GAN) is proposed. Firstly, considering the denoising capability of LatLRR, source images were decomposed by LatLRR. Secondly, the ED-D2GAN model was put forward as the low-rank region fusion method, which can fully extract the information contained by the low-rank region images. Among them, encoder and decoder networks were used in the generator; convolutional neural networks were also used in dual discriminators. Thirdly, a threshold adaptive weighting algorithm based on the region energy ratio is proposed as the salient region fusion rule, which can improve the overall sharpness of the fused image. The experimental results show that compared with the best methods of the other six methods, this paper is effective in multiple objective evaluation metrics, including the average gradient, edge intensity, information entropy, spatial frequency and standard deviation. The results of the two experiments are improved by 35.03%, 42.42%, 4.66%, 8.59% and 11.49% on average.

Enhancement and Noise Suppression of Single Low-Light Grayscale Images

Low-light images have low contrast and high noise, making them not easily readable. Most existing image-enhancement methods focus on color images. In the present study, an enhancement and denoising algorithm for single low-light grayscale images is proposed. The algorithm is based on the multi-exposure fusion framework. First, on the basis of the low-light tone-mapping operators, the optimal virtual exposure image is constructed according to the information entropy criterion. Then, the latent low-rank representation is applied to two images to generate low-ranking parts and saliency parts to reduce noise after fusion. Next, the initial weight map is constructed based on the information contained in the decomposed images, and an adaptive weight refined algorithm is proposed to restore as much structural information as possible and keep the details while avoiding halo artifacts. When solving the weight maps, the decomposition and optimization of the nonlinear problem is converted into a total variation model, and an iterative method is used to reduce the computational complexity. Last, the normalized weight map is used for image fusion to obtain the enhanced image. The experimental results showed that the proposed method performed well both in the subjective and objective evaluation of state-of-the-art enhancement methods for low-light grayscale images.

Latent Low Rank Representation Applied to Pulsed Thermography Data For Carbon Fibre Reinforced Polymer Inspection

ABSTRACT This paper explores the implementation of Latent Low-Rank Representation (LatLRR) on pulsed thermographic data. LatLRR decomposes an image in the form of a linear association of three types of information: observed, unobserved and noise. This information is then used in order to separate the salient and principal features. This study has found that when used as a post-processing method prior to the application of state-of-the-art signal processing techniques, such as principal component thermography (PCT) and pulsed phase thermography (PPT), LatLRR significantly improves defect detection: 18% for PCT and 92% for PPT. Nevertheless, no noticeable improvement was measured when LatLRR was used to reconstruct a noiseless version of each image of a dataset, before processing it with a state-of-the-art algorithm. The investigations conducted on each type of feature returned by the LatLRR have also failed to provide results regarding the detection of defects.