Original Multispectral Image Research Articles

CrossDiff: Exploring Self-SupervisedRepresentation of Pansharpening via Cross-Predictive Diffusion Model.

Fusion of a panchromatic (PAN) image and corresponding multispectral (MS) image is also known as pansharpening, which aims to combine abundant spatial details of PAN and spectral information of MS images. Due to the absence of high-resolution MS images, available deep-learning-based methods usually follow the paradigm of training at reduced resolution and testing at both reduced and full resolution. When taking original MS and PAN images as inputs, they always obtain sub-optimal results due to the scale variation. In this paper, we propose to explore the self-supervised representation for pansharpening by designing a cross-predictive diffusion model, named CrossDiff. It has two-stage training. In the first stage, we introduce a cross-predictive pretext task to pre-train the UNet structure based on conditional Denoising Diffusion Probabilistic Model (DDPM). While in the second stage, the encoders of the UNets are frozen to directly extract spatial and spectral features from PAN and MS images, and only the fusion head is trained to adapt for pansharpening task. Extensive experiments show the effectiveness and superiority of the proposed model compared with state-of-the-art supervised and unsupervised methods. Besides, the cross-sensor experiments also verify the generalization ability of proposed self-supervised representation learners for other satellite datasets. Code is available at https://github.com/codgodtao/CrossDiff.

A self-supervised remote sensing image fusion framework with dual-stage self-learning and spectral super-resolution injection

Pan-sharpening is a very productive technique to enhance the spatial details of multispectral images with the aid of panchromatic images. Nowadays, deep learning-based pan-sharpening has scored tremendous achievements. However, strict requirement for training image pairs and low generalization hamper the development of supervised pan-sharpening with limited samples. Unsupervised image fusion is an effective technique to fuse images without adequate ground truth as training samples. Existing unsupervised pan-sharpening methods are commonly based on the image generator model, suffering from unsatisfactory spatial details. In this study, we proposed a self-supervised pan-sharpening method with the aid of spectral super-resolution named sSRPNet. Coarsening-scale self-learning exploits the internal information in multispectral images at a coarsening scale and trains the initial fusion model without other labels. Spectral super-resolution injection explores the missing spatial details in the initial fused images and recovers it. Degradation self-learning introduces strong spectral constraints with original multispectral images. Both reduced-resolution and full-resolution experiments on three datasets have proved the superiority of sSRPNet. Furthermore, the proposed spectral super-resolution injection can be implemented on any existing pan-sharpening algorithms, improving their performance.

Recognition of Rubber Tree Powdery Mildew Based on UAV Remote Sensing with Different Spatial Resolutions

Rubber tree is one of the essential tropical economic crops, and rubber tree powdery mildew (PM) is the most damaging disease to the growth of rubber trees. Accurate and timely detection of PM is the key to preventing the large-scale spread of PM. Recently, unmanned aerial vehicle (UAV) remote sensing technology has been widely used in the field of agroforestry. The objective of this study was to establish a method for identifying rubber trees infected or uninfected by PM using UAV-based multispectral images. We resampled the original multispectral image with 3.4 cm spatial resolution to multispectral images with different spatial resolutions (7 cm, 14 cm, and 30 cm) using the nearest neighbor method, extracted 22 vegetation index features and 40 texture features to construct the initial feature space, and then used the SPA, ReliefF, and Boruta–SHAP algorithms to optimize the feature space. Finally, a rubber tree PM monitoring model was constructed based on the optimized features as input combined with KNN, RF, and SVM algorithms. The results show that the simulation of images with different spatial resolutions indicates that, with resolutions higher than 7 cm, a promising classification result (>90%) is achieved in all feature sets and three optimized feature subsets, in which the 3.4 cm resolution is the highest and better than 7 cm, 14 cm, and 30 cm. Meanwhile, the best classification accuracy was achieved by combining the Boruta–SHAP optimized feature subset and SVM model, which were 98.16%, 96.32%, 95.71%, and 88.34% at 3.4 cm, 7 cm, 14 cm, and 30 cm resolutions, respectively. Compared with SPA–SVM and ReliefF–SVM, the classification accuracy was improved by 6.14%, 5.52%, 12.89%, and 9.2% and 1.84%, 0.61%, 1.23%, and 6.13%, respectively. This study’s results will guide rubber tree plantation management and PM monitoring.

Multi-Fusion algorithms for Detecting Land Surface Pattern Changes Using Multi-High Spatial Resolution Images and Remote Sensing Analysis

Producing accurate Land-Use and Land-Cover (LU/LC) maps using low-spatial-resolution images is a difficult task. Pan-sharpening is crucial for estimating LU/LC patterns. This study aimed to identify the most precise procedure for estimating LU/LC by adopting two fusion approaches, namely Color Normalized Brovey (BM) and Gram-Schmidt Spectral Sharpening (GS), on high-spatial-resolution Multi-sensor and Multi-spectral images, such as (1) the Unmanned Aerial Vehicle (UAV) system, (2) the WorldView-2 satellite system, and (3) low-spatial-resolution images like the Sentinel-2 satellite, to generate six levels of fused images with the three original multi-spectral images. The Maximum Likelihood method (ML) was used for classifying all nine images. A confusion matrix was used to evaluate the accuracy of each single classified image. The obtained results were statistically compared to determine the most reliable, accurate, and appropriate LU/LC map and procedure. It was found that applying GS to the fused image, which integrated WorldView-2 and Sentinel-2 satellite images and was classified by the ML method, produced the most accurate results. This procedure has an overall accuracy of 88.47% and a kappa coefficient of 0.85. However, the overall accuracies of the three classified multispectral images range between 86.84% to 76.49%. Furthermore, the accuracy assessment of the fused images by the Brovey method and the rest of the GS method and classified by the ML method ranges between 85.75% to 76.68%. This proposed procedure shows a lot of promise in the academic sphere for mapping LU/LC. Previous researchers have mostly used satellite images or datasets with similar spatial and spectral resolution, at least for tropical areas like the study area of this research, to detect land surface patterns. However, no one has previously investigated and examined the use and application of different datasets that have different spectral and spatial resolutions and their accuracy for mapping LU/LC. This study has successfully adopted different datasets provided by different sensors with varying spectral and spatial levels to investigate this. Doi: 10.28991/ESJ-2023-07-04-013 Full Text: PDF

Estimation of Leaf Nitrogen Content in Rice Using Vegetation Indices and Feature Variable Optimization with Information Fusion of Multiple-Sensor Images from UAV

LNC (leaf nitrogen content) in crops is significant for diagnosing the crop growth status and guiding fertilization decisions. Currently, UAV (unmanned aerial vehicles) remote sensing has played an important role in estimating the nitrogen nutrition of crops at the field scale. However, many existing methods of evaluating crop nitrogen based on UAV imaging techniques usually have used a single type of imagery such as RGB or multispectral images, seldom considering the usage of information fusion from different types of UAV imagery for assessing the crop nitrogen status. In this study, GS (Gram–Schmidt Pan Sharpening) was utilized to fuse images from two sensors of digital RGB and multispectral cameras mounted on UAV, and the specific bands of the multispectral cameras are blue, green, red, rededge and NIR. The color space transformation method, HSV (Hue-Saturation-Value), was used to separate soil background noise from crops due to the high spatial resolution of UAV images. Two methods of optimizing feature variables, the Successive Projection Algorithm (SPA) and the Competitive Adaptive Reweighted Sampling method (CARS), combined with two regularization regression algorithms, LASSO and RIDGE, were adopted to estimate the LNC, compared to the commonly used Random Forest algorithm. The results showed that: (1) the accuracy of LNC estimation using the fusion image is improved distinctly by a comparison to the original multispectral image; (2) the denoised images performed better than the original multispectral images in evaluating LNC in rice; (3) the RIDGE-SPA combined method, using SPA to select the MCARI, SAVI and OSAVI, had the best performance for LNC in rice, with an R2 of 0.76 and an RMSE of 10.33%. It can be demonstrated that the information fusion of multiple-sensor imagery from UAV coupling with the methods of optimizing feature variables can estimate the rice LNC more effectively, which can also provide a reference for guiding the decision making of fertilization in rice fields.

Calculation method of wilting index based on fractal dimension of multispectral images for the soybean canopy

Drought stress is an important environmental factor that restricts the quality and high yield of the soybean. Rapid and nondestructive monitoring of canopy traits under drought stress is very critical to the cultivation and quality improvement of for the soybean plants. To solve the problems of traditional artificial chemical tests in determining soybean wilt index, such as cumbersome, time-consuming, and laborious, a method for calculating wilting index was proposed based on the fractal dimension of multispectral images for the soybean canopy. First, the soybean variety of Suinong 26 was chosen as the experimental subject, the multispectral images of five channels (RGB, GRE, NIR, RED and REG) of the soybean canopy were acquired. Second, the soybean canopy image was extracted by K-means clustering algorithm from the original multispectral image, with an effective segmentation rate (ESR) of 0.9729. Third, on the base of the multispectral image enhanced by the gamma transform for the soybean canopy, a method for calculating wilting index (LBC,LDBC, LDB) was constructed for the soybean canopy by using fractal dimension algorithms including box-counting (BC) dimension, differential box-counting (DBC) dimension and double blanket (DB) dimension. Finally, the proposed method was proved by using the average leaf inclination (LI) and the projected leaf area (PLA). The experimental results showed that the determination coefficients R2 were all above 0.85. The LBC got the highest correlation with LI and PLA in the RGB channel with coefficient determination R2 above 0.8729. The LDB got the highest correlation with LI and PLA in the GRE, NIR, and REG channels with determination coefficient R2 above 0.8824. The LDBC got the highest correlation with LI and PLA in the RED channel with determination coefficient R2 above 0.8857. This achievement presented that the proposed wilting index could characterize the change degree for the ecology and morphology when soybeans were affected by drought stress, which can provide the technical support to scientifically monitor the phenotypic traits in the process of variety breeding, field cultivation for the soybean plant.

MS-Pansharpening Algorithm Based on Dual Constraint Guided Filtering

The difference and complementarity of spatial and spectral information between multispectral (MS) image and panchromatic (PAN) image have laid the foundation for the fusion of the two types of images. In recent years, MS and PAN image fusion (also known as MS-Pansharpening) has gained attention as an important research area in remote sensing (RS) image processing. This paper proposes an MS-Pansharpening algorithm based on dual constraint Guided Filtering in the nonsubsampled shearlet transform (NSST) domain. The innovation is threefold. First, the dual constraint guided image filtering (DCGIF) model, based on spatial region average gradient correlation and vector correlation formed by neighborhood elements is proposed. Further, the PAN image detail information extraction scheme, based on the model, is provided, which extracts more complete and accurate detail information, thus avoiding, to some extent, the spectral distortion caused by the injection of non-adaptive information. Second, the weighted information injection model, based on the preservation of the correlation between the band spectra, is proposed. The model determines the information injection weight of each band pixel based on the spectral proportion between bands of the original MS image, which ensures the spectral correlation between bands of the fused MS image. Finally, a new MS-Pansharpening algorithm in NSST domain is proposed. The MS and PAN high frequency sub-bands of NSST are used to extract more effective spatial details. Then the proposed DCGIF model is used to extract the effective spatial detail injection information through the weighted joint method based on the regional energy matrix. Finally, the weighted information injection model is used to inject it into each band of MS to complete information fusion. Experimental results show that the proposed approach has better fusion effect than some conventional MS-Pansharpening algorithms, which can effectively improve the spatial resolution of the fused MS image and maintain the spectral characteristics of MS.

Multispectral Image Enhancement Based on the Dark Channel Prior and Bilateral Fractional Differential Model

Compared with single-band remote sensing images, multispectral images can obtain information on the same target in different bands. By combining the characteristics of each band, we can obtain clearer enhanced images; therefore, we propose a multispectral image enhancement method based on the improved dark channel prior (IDCP) and bilateral fractional differential (BFD) model to make full use of the multiband information. First, the original multispectral image is inverted to meet the prior conditions of dark channel theory. Second, according to the characteristics of multiple bands, the dark channel algorithm is improved. The RGB channels are extended to multiple channels, and the spatial domain fractional differential mask is used to optimize the transmittance estimation to make it more consistent with the dark channel hypothesis. Then, we propose a bilateral fractional differentiation algorithm that enhances the edge details of an image through the fractional differential in the spatial domain and intensity domain. Finally, we implement the inversion operation to obtain the final enhanced image. We apply the proposed IDCP_BFD method to a multispectral dataset and conduct sufficient experiments. The experimental results show the superiority of the proposed method over relative comparison methods.

A Blind Full-Resolution Quality Evaluation Method for Pansharpening

Pansharpening methods have been developed for nearly 40 years; however, how to quantitatively evaluate the quality of pansharpened images at full resolution (FR) is probably the most debated topic in this field due to the inherent unavailable of the real HR MS reference image. In this article, a novel blind FR quality evaluation method for pansharpening is proposed. In the proposed method, spatial and spectral features that are sensitive to spatial and spectral distortions of fused images are comprehensively considered and jointly learned based on online multivariate Gaussian (MVG) to construct the evaluation model. It directly outputs the quality of fused images, rather than the stepwise evaluation of spectral score, spatial score, and final overall quality score by the weighted combination of them, which may introduce contradictory results. First, a pristine benchmark evaluation model is established on the spatial features from the original high-spatial-resolution (HR) panchromatic (PAN) image and the spectral invariant assumption between ideal fused and original multispectral (MS) images. Second, a testing evaluation model for the fused image is founded. Finally, the quality of the fused image is measured based on the distance between the testing and benchmark models. The experimental results demonstrated the superior performance of the proposed method. Furthermore, the proposed method can be generalized to other interesting tasks, such as the nonreference evaluation for pansharpening with missing information and the nonreference evaluation for hyperspectral image fusion. The source code is available on <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/yyxhpkq/MQNR</uri> .

Unsupervised Cycle-Consistent Generative Adversarial Networks for Pan Sharpening

Deep learning based pan-sharpening has received significant research interest in recent years. Most of existing methods fall into the supervised learning framework in which they down-sample the multi-spectral (MS) and panchromatic (PAN) images and regard the original MS images as ground truths to form training samples. Although impressive performance could be achieved, they have difficulties generalizing to the original full-scale images due to the scale gap, which makes them lack of practicability. In this paper, we propose an unsupervised generative adversarial framework that learns from the full-scale images without the ground truths to alleviate this problem. We extract the modality-specific features from the PAN and MS images with a two-stream generator, perform fusion in the feature domain, and then reconstruct the pan-sharpened images. Furthermore, we introduce a novel hybrid loss based on the cycle-consistency and adversarial scheme to improve the performance. Comparison experiments with the state-of-the-art methods are conducted on GaoFen-2 and WorldView-3 satellites. Results demonstrate that the proposed method can greatly improve the pan-sharpening performance on the full-scale images, which clearly show its practical value. Codes are available at https://github.com/zhysora/UCGAN.

Multiresolution Analysis Based on Dual-Scale Regression for Pansharpening

Pansharpening technique is used to merge the original multispectral image (MS) with a high spatial resolution panchromatic image (PAN). Due to its robustness, the multiresolution analysis (MRA) is an important part of pansharpening. The scale regression model is effective for improving MRA. However, the existing MRA based on scale regression results into single-scale regression information, thus affecting the final pansharpening result. To address this problem, in this work, we propose a dual-scale regression-based MRA for pansharpening. First, we establish a scale regression-based model. Then, this model is improved using a high-pass modulation (HPM) injection scheme. Finally, the dual-scale information is added to the scale regression to construct the dual-scale regression for obtaining the final pansharpening result. We perform experiments using five datasets. The results show that the proposed method obtains a better pansharpening result as compared to various state-of-the-art MRA methods. In addition, the quantitative and qualitative analysis of the results shows that the proposed method achieves appropriate spatial and spectral resolution fusion. Therefore, it has a great potential in pansharpening technique.

Developing a new IHS-based fusion methodology for GeoEye-1 satellite imagery with the ELSHORA fusion technique

In this research, a new image fusion technique was developed to merge the spatial details in the panchromatic (PAN) image and the spectral details in the multispectral (MS) image to produce fused images with the highest spatial and spectral qualities. This fusion technique is superior to the other existing fusion techniques in that it can produce sharp images with no color distortion for all types of land cover. This is because of using weighting parameters for all MS bands in the calculation of the I component according to the intersecting areas between the MS bands and the PAN band, and the I component will be an average of the four modified MS bands, which minimizes the gray level differences between PAN and I images. Moreover, using the variable vegetation coefficient β for the NIR band makes this technique produce the highest spectral and spatial qualities for each land cover class. To evaluate the performance of the ELSHORA fusion technique, it was compared to the traditional intensity-hue-saturation (IHS) fusion technique, the fast IHS (FIHS) fusion technique, and four other standard image fusion techniques; principal component analysis, hyper-spherical color space, multiplicative resolution merge and Brovey transform. These fusion techniques were applied to three sets of GeoEye-1 PAN and MS images covering different land covers in Tanta city, Egypt. The fused images are compared statistically and visually to the original PAN and MS images. The results showed that the ELSHORA fusion technique was chosen as the best method for all datasets. This is because it has produced the highest spectral quality in which no color distortion can be detected and has provided significant spatial quality, visually and analytically. To test the spatial and temporal transferability of ELSHORA fusion technique, it was compared to the original FIHS fusion technique to fuse three different datasets covering different classes of land cover (urban, agriculture, and mixed) located in different places and acquired at different times. The results indicate that ELSHORA fusion technique is spectrally and spatially in the lead for all datasets. This study proved the efficiency of ELSHORA fusion technique to produce images of high spectral and spatial resolution at different times and places.

Mathematical description of the processes of synthesis of digital multispectral images

The article discusses the features of the formation of digital multispectral images corresponding to narrow registration zones. The processes of signal transformations are analyzed during the synthesis of such images from the original multispectral images with overlapping spectral flux registration zones. Questions of the mathematical description of the differential and multiplicative method for the synthesis of spectrozonal images are discussed. Analytical expressions are given that correspond to signal transformations of the original digital multispectral images in their differential and multiplicative synthesis.

Panchromatic and multispectral image fusion using the spatial frequency and the à trous wavelet transform

We propose a panchromatic (Pan) and multispectral (MS) image fusion method in order to synthesize an MS image with enhanced spatial resolution and preserved spectral information of the original MS image. The proposed method is based on the spatial frequency and the à trous wavelet transform (SFATWT). The à trous wavelet transform (ATWT) is used to decompose Pan and MS images into low and high frequencies. The spatial frequency is exploited in the image fusion rules to improve the details or the high-frequency information which will be injected into the MS image. The ATWT reconstruction makes it possible to produce MS image with high spatial resolution. To validate the proposed method, two image sets are used. The first one is acquired by the SPOT (Satellite Pour l’Observation de la Terre) satellite with a resolution ratio of 2. The second one is the Algerian satellite AlSat 2A images with a resolution ratio of 4. The fused results are evaluated by means of qualitative and quantitative comparisons. The assessment of fused products has shown that the proposed method is promising.

Improving Urban Impervious Surfaces Mapping through Integrating Statistical Methods and Spectral Mixture Analysis

Impervious surfaces have been widely considered as the key indicator for evaluating urbanization and environmental quality. As one of the most widely applied methods, spectral mixture analysis (SMA) has been commonly used for mapping urban impervious surface fractions. When implementing SMA, the original multispectral remote-sensing reflectance images are served as the foundation and key to successful SMA. However, the limited spectral variances among different land covers from the original reflectance images make it challenging in information extraction and results in unsatisfactory mapping results. To address this issue, a new method has been proposed in this study to improve urban impervious surface mapping through integrating statistical methods and SMA. In particular, two traditional statistical methods, principal component analysis (PCA) and minimum noise fraction rotation (MNF) were applied to highlight the spectral variances among different land covers. Three endmember classes (impervious surface, soil, and vegetation) and corresponding spectra were identified and extracted from the vertices of the 2-D space plots generated by the first three components of each of the statistical analysis methods, PCA and MNF. A new dataset was generated by stacking the first three components of the PCA and MNF (in a total of six components), and a fully constrained linear SMA was implemented to map the fractional impervious surfaces. Results indicate that a promising performance has been achieved by the proposed new method with the systematic error (SE) of −3.45% and mean absolute error (MAE) of 11.52%. Comparative analysis results also show a much better performance achieved by the proposed statistical method-based SMA than the conventional SMA.

Weighted manifold regularized sparse representation of featured injected details for pansharpening

ABSTRACT Sparse representation (SR)-based pansharpening methods which combine the dictionary and estimated sparse coefficients have achieved visually and quantitatively great results in pansharpening problem these days. And the details injection (ID)-based methods can receive comparable images by sharpening the multispectral bands through adding the proper spatial details from panchromatic (PAN) images. Recently, method based on sparse representation of injected details (SR-D) which combines the SR and ID points out a new way forward for pansharpening. In this direction, manifold regularized sparse representation of injected details (MR-SR-D) which introducing a manifold regularization (MR) into the former SR-D model have improved the quality of pansharpened images greatly utilizing a graph Laplacian to incorporate the locally geometrical structure of the multispectral data. However, due to the lack of spatial information in PAN, and the use of higher-order features similarity with the original multispectral images, the resulting images still have spatial and spectral distortion. Thus, in this paper, we propose a new method to enhance the spatial resolution of aiming image by adding the weighted local geometrical structure of PAN and multispectral images, and improve the spectral resolution by joining the higher-order structure connection between multispectral and aiming images to the MR-SR-D method which can be called as weighted manifold regularized (WMR) sparse representation of featured injected details method (WMR-SR-FD). Experimental results using IKONOS, QuickBird and WorldView2 data show that the proposed method can achieve remarkable spectral and spatial quality.

Multispectral image based germination detection of potato by using supervised multiple threshold segmentation model and Canny edge detector

Whether from the perspective of agricultural production or food safety, potato germination detection is of great significance. Since the features (color, texture and context) of the germination area are similar to those of the non-germination area, the existing vision frameworks are difficult to accurately detect the germinations on the surface of potatoes. In this study, the method for detecting potato germination based on multispectral image combined with supervised multiple threshold segmentation model (SMTSM) and Canny edge detector was proposed. The SMTSM based on Genetic Programming algorithm combined with a hybrid fitness function (HF-GP) was utilized to transform the original multispectral images into multiple 2-D images for improving the contrast between region of interest (ROI) and background. A sub-mask of each transformed image was constructed using optimal segmentation threshold, and all of sub-masks were merged through pixel-multiplication to obtain segmentation mask. Meanwhile, in order to filter out the boundless areas that are misidentified as germinations, Canny edge detector was used on gray image to obtain edge mask. Finally, the segmentation mask and the edge mask were combined to complete the detection of germination of potato. Experimental results shown that the proposed method achieved the TPR of 90.91% and the precision of 89.28% for the edible potatoes, which were 4.17–19.05% and 12.39–24.62% higher than the competitive detectors in TPR and precision respectively. For the breeding potatoes, the proposed method with 89.67% of TPR and 86.37% of precision was 9.74–24.58% and 15.70–20.39% better than the competitors in TPR and precision respectively. The comparison confirms the proposed method has excellent detection effect on potato’s germination.

Epidermal growth factor receptor tyrosine kinase inhibitor remodels tumor microenvironment by upregulating LAG-3 in advanced non-small-cell lung cancer

IntroductionPrevious clinical investigations have demonstrated that patients with non-small-cell lung cancer (NSCLC) harboring epidermal growth factor receptor (EGFR) mutation have moderate responses to programmed cell death-1 and it's ligand-1 (PD-1/PD-L1) inhibitors, while some patients who failed EGFR-tyrosine kinase inhibitor (TKI) therapy could benefit from immunotherapy. As a result, we have explored alterations in the tumor immune microenvironment (TIME) before and after EGFR-TKI treatment to detect the chances and proper timing of immunotherapy among patients. MethodsWe identified 16 paired tissue samples pre- and post-EGFR-TKI treatment. Sections 4 μm thick were utilized to evaluate CD8, PD-L1, PD-1, LAG-3, and TIM-3 expressions by multiplexed fluorescent immunohistochemical staining. Five to ten representative original multispectral images of each sample were employed in the analysis. ResultsPatients with positive CD8 + T-cell infiltration accounted for 37.5 % at baseline. Positive expression of PD-L1, PD1, LAG-3, and TIM-3 cells were observed in seven (43.8 %), four (25 %), one (6.25 %) and five (31.25 %) of the patients, respectively. PD-1 expression and infiltration of CD8+PD-1+-exhausted T cells increased significantly in patients with EGFR L858R mutation compared to patients with EGFR 19DEL. Patients who acquired T790 M after TKI treatment had less infiltrations of CD8+PD-1+ T cells and CD8+TIM-3+ T cells in the TIME at baseline. Positive expression of checkpoint proteins—including PD-1, TIM-3, and LAG-3—significantly correlated with shorter progression-free survival. LAG-3 was significantly upregulated after TKI treatment (p = 0.003), while other checkpoint proteins remained stationary. ConclusionThe present study is the first work to report LAG-3 upregulation after EGFR-TKI failure in advanced NSCLC, which proposed novel insights for rational use of LAG-3 inhibitors in advanced NSCLC patients with EGFR mutation.

Edge-guided multispectral image fusion algorithm

Most existing multispectral fusion algorithms often suffer from spectral or spatial information distortion. Driven by this motivation, we propose an edge-guided multispectral (MS) image fusion algorithm. In particular, it combines the advantages of generative adversarial networks and improved fusion frameworks, so the merged image can better preserve the spectral information of the original multispectral image while injecting spatial detail information. Specifically, first, an MS image with more image detail is generated using the generated confrontation network for preliminary reconstruction. The panchromatic image edge information and the antagonistic learning strategy are introduced for the robust multispectral image reconstruction. Then, using the reconstructed MS image and the general component substitution image fusion framework, the whole fusion system of this paper is constructed. An enhancement operator is introduced to inject spatial details. Our extensive dataset evaluations show that our approach performs better in terms of high objective quality and human visual perception than several of the most advanced fusion methods.

A Generalized Tool for Accurate and Efficient Image Registration of UAV Multi-lens Multispectral Cameras by N-SURF Matching

The original multispectral (MS) images obtained from multi-lens multispectral cameras (MSCs) have significant misregistration errors, which require image registration for precise spectral measurement. However, due to the nonlinearity intensity differences among MS images, performing image matching is difficult to find sufficient correct matches (CMs) for image registration, and results in a complex coarse-to-fine solution. Based on the modification of speed-up robust feature (SURF), we proposed a normalized SURF (N-SURF) that can significantly increase the amount of CMs among different pairs of MS images and make one-step image registration possible. In this study, we first introduce N-SURF and adopt different MS datasets acquired from three representative MSCs (MCA-12, Altum, and Sequoia) to evaluate its matching ability. Meanwhile, we utilized three image transformation models—affine transform (AT), projective transform (PT), and an extended projective transform (EPT) to correct the misregistration errors of MSCs and evaluate their co-registration correctness. The results show that N-SURF can obtain 6–20 times more CMs than SURF and can successfully match all pairs of MS images, while SURF failed in the cases of significant spectral differences. Moreover, visual comparison, accuracy assessment, and residual analysis show that EPT can more accurately correct the viewpoint and lens distortion differences of MSCs than AT and PT, and it can obtain co-registration accuracy of 0.2–0.4 pixels. Subsequently, using the successful N-SURF matching and EPT model, we developed an automatic MS image registration tool that is suitable for various multilens MSCs.