SSIM Scores Research Articles

Remote sensing image dehazing using a wavelet-based generative adversarial networks

Remote sensing images often suffer from the degradation effects of atmospheric haze, which can significantly impair the quality and utility of the acquired data. A novel dehazing method leveraging generative adversarial networks is proposed to address this challenge. It integrates a generator network, designed to enhance the clarity and detail of hazy images, with a discriminator network that distinguishes between dehazed and real clear images. Initially, a dense residual block is designed to extract primary features. Subsequently, a wavelet transform block is designed to capture high and low-frequency features. Additionally, a global and local attention block is proposed to reduce the interference of redundant features and enhance the weight of important features. PixelShuffle is used as the upsampling operation, allowing for finer control of image details during the upsampling process. Finally, these designed modules are integrated to construct the generator network for image dehazing. Moreover, an improved discriminator network is proposed by adding a noise module to the conventional discriminator, enhancing the network’s robustness. A novel loss function is introduced by incorporating the color loss function and SSIM loss function into traditional loss functions, aiming to improve color accuracy and visual distortion assessment. This approach attains the highest PSNR and SSIM scores when compared to current leading methods. The proposed dehazing technique for remote sensing images successfully maintains color fidelity and detail, leading to significantly clearer images.

TDSF: Two-phase tamper detection in semi-fragile watermarking using two-level integer wavelet transform

Image watermarking is one of techniques that can be used to protect and recover from the altered images. Tamper detection in semi-fragile still does not achieve high accuracy for tamper localization, especially in malicious attacks. This study proposes two-phase tamper detection in semi-fragile image watermarking using two-level Integer Wavelet Transform (IWT). The study proposes two-phases of tamper detection to achieve a high level of accuracy in the tamper localization. The initial phase will check four segments to determine the tampered bit or not. A tampered segment is identified if it contains at least one tampered bit. The second phase will be performed if the first phase has no tampered bit. The second phase checked each segment to determine whether it was tampered with or not. This study also used Normalized Hamming Similarity (NHS) to classify malicious and incidental attacks. The proposed scheme performed two-level integer wavelets transform to obtain the LL, LH, HL, and HH sub-bands. The authentication bits of LL are embedded into the LH and HL sub-bands to generate a watermarked image. The LH and HL are compared using XOR bitwise with authentication bitto localize the tamper detection. The experimental results show that the proposed scheme achieved high quality of the recovered images with an average PSNR score of 44.1864 dB, wPSNR score of 46.2218 dB and SSIM score of 0.9945. The proposed method produced high accuracy of tamper detection of about 98.33 % under object manipulation with the tampering rate of about 12.1643 %.

Ferumoxytol-Enhanced Cardiac Cine MRI Reconstruction Using a Variable-Splitting Spatiotemporal Network.

Balanced steady-state free precession (bSSFP) imaging is commonly used in cardiac cine MRI but prone to image artifacts. Ferumoxytol-enhanced (FE) gradient echo (GRE) has been proposed as an alternative. Utilizing the abundance of bSSFP images to develop a computationally efficient network that is applicable to FE GRE cine would benefit future network development. To develop a variable-splitting spatiotemporal network (VSNet) for image reconstruction, trained on bSSFP cine images and applicable to FE GRE cine images. Retrospective and prospective. 41 patients (26 female, 53 ± 19 y/o) for network training, 31 patients (19 female, 49 ± 17 y/o) and 5 healthy subjects (5 female, 30 ± 7 y/o) for testing. 1.5T and 3T, bSSFP and GRE. VSNet was compared to VSNet with total variation loss, compressed sensing and low rank methods for 14× accelerated data. The GRAPPA×2/×3 images served as the reference. Peak signal-to-noise-ratio (PSNR), structural similarity index (SSIM), left ventricular (LV) and right ventricular (RV) end-diastolic volume (EDV), end-systolic volume (ESV), and ejection fraction (EF) were measured. Qualitative image ranking and scoring were independently performed by three readers. Latent scores were calculated based on scores of each method relative to the reference. Linear mixed-effects regression, Tukey method, Fleiss' Kappa, Bland-Altman analysis, and Bayesian categorical cumulative probit model. A P-value <0.05 was considered statistically significant. VSNet achieved significantly higher PSNR (32.7 ± 0.2), SSIM (0.880 ± 0.004), rank (2.14 ± 0.06), and latent scores (-1.72 ± 0.22) compared to other methods (rank >2.90, latent score < -2.63). Fleiss' Kappa was 0.52 for scoring and 0.61 for ranking. VSNet showed no significantly different LV and RV ESV (P = 0.938) and EF (P = 0.143) measurements, but statistically significant different (2.62 mL) EDV measurements compared to the reference. VSNet produced the highest image quality and the most accurate functional measurements for FE GRE cine images among the tested 14× accelerated reconstruction methods. 3 TECHNICAL EFFICACY: Stage 1.

Training ESRGAN with multi-scale attention U-Net discriminator

In this paper, we propose MSA-ESRGAN, a novel super-resolution model designed to enhance the perceptual quality of images. The key innovation of our approach lies in the integration of a multi-scale attention U-Net discriminator, which allows for more accurate differentiation between subject and background areas in images. By leveraging this architecture, MSA-ESRGAN surpasses traditional methods and several state-of-the-art super-resolution models in terms of Natural Image Quality Evaluator (NIQE) scores as well as Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM) across various benchmark datasets, including BSD100, Set5, Set14, Urban100, and OST300. Additionally, subjective evaluations further confirm the enhanced visual quality delivered by MSA-ESRGAN, particularly in terms of preserving texture and overall image realism. To ensure a fair comparison with Real-ESRGAN, we initialized our generator with a pre-trained Real-ESRNET model and followed the same training setup. Our model was trained on the DIV2K dataset using high-resolution image patches and the Adam optimizer, incorporating exponential moving average (EMA) for stability and performance enhancement. Evaluations on multiple benchmark datasets demonstrate that MSA-ESRGAN consistently delivers superior perceptual quality, as evidenced by higher NIQE, PSNR, and SSIM scores compared to other methods. Specifically, our model shows significant improvements in both objective and subjective measures of image quality. Furthermore, an ablation study highlighted the critical role of our multi-scale attention U-Net discriminator in enhancing the model’s performance. The results underscore the effectiveness of MSA-ESRGAN in maintaining image naturalness and perceptual quality, providing a robust benchmark for blind super-resolution tasks.

Research on Key Technologies of Image Steganography Based on Simultaneous Deception of Vision and Deep Learning Models

As machine learning continues to evolve, traditional image steganography techniques find themselves increasingly unable to meet the dual challenge of deceiving both the human visual system and machine learning models. In response, this paper introduces the Visually Robust Image Steganography (VRIS) model, specifically tailored for this dual deception task. The VRIS model elevates visual security through the use of specialized feature extraction and processing methodologies. It meticulously conducts feature-level fusion between secret images and cover images, ensuring a high level of visual similarity between them. To effectively mislead machine learning models, the VRIS model incorporates a sophisticated strategy utilizing random noise factors and discriminators. This involves adding controlled amounts of random Gaussian noise to the encrypted image, thereby enhancing the difficulty for machine learning frameworks to recognize it. Furthermore, the discriminator is trained to discern between the noise-infused encrypted image and the original cover image. Through adversarial training, the discriminator and VRIS model refine each other, successfully deceiving the machine learning systems. Additionally, the VRIS model presents an innovative method for extracting and reconstructing secret images. This approach safeguards secret information from unauthorized access while enabling legitimate users to non-destructively extract and reconstruct images by leveraging multi-scale features from the encrypted image, combined with advanced feature fusion and reconstruction techniques. Experimental results validate the effectiveness of the VRIS model, achieving high PSNR and SSIM scores on the LFW dataset and demonstrating significant deception capabilities against the ResNet50 model on the Mini-ImageNet dataset, with an impressive misclassification rate of 99.24%.

Image Stitching of Low-Resolution Retinography Using Fundus Blur Filter and Homography Convolutional Neural Network

Great advances in stitching high-quality retinal images have been made in recent years. On the other hand, very few studies have been carried out on low-resolution retinal imaging. This work investigates the challenges of low-resolution retinal images obtained by the D-EYE smartphone-based fundus camera. The proposed method uses homography estimation to register and stitch low-quality retinal images into a cohesive mosaic. First, a Siamese neural network extracts features from a pair of images, after which the correlation of their feature maps is computed. This correlation map is fed through four independent CNNs to estimate the homography parameters, each specializing in different corner coordinates. Our model was trained on a synthetic dataset generated from the Microsoft Common Objects in Context (MSCOCO) dataset; this work added an important data augmentation phase to improve the quality of the model. Then, the same is evaluated on the FIRE retina and D-EYE datasets for performance measurement using the Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM). The obtained results are promising: the average PSNR was 26.14 dB, with an SSIM of 0.96 on the D-EYE dataset. Compared to the method that uses a single neural network for homography calculations, our approach improves the PSNR by 7.96 dB and achieves a 7.86% higher SSIM score.

Detection of artificial spots in fundus images using modified U-Net based semantic segmentation

Today, deep learning algorithms are playing a very crucial role in the initial stage diagnosis of several fundus diseases like glaucoma, hypertension, and diabetic retinopathy. Lots of research is going on in this area now-a-days. During the acquisition of fundus images some artificial spots (e.g. because of the device itself, dust particles in the surroundings) have been added in captured images. In this paper, artificial spots in the fundus images that are generated due to the non-standardized conditions of scanning devices, are detected with the help of a newly proposed modified UNet (mU-Net) semantic segmentation model. Initially, preprocessing methods such as Gaussian blur, thresholding, and Hough transform have been used to create artificial spots. Now these preprocessed images have been used for training the proposed model. To make the proposed model more effective, the following modifications like, Regularization techniques (early stopping, greater weight decay, and Adam optimizer), decay learning rate scheduler, categorical cross-entropy loss function, and a significant number of filters have been modified in simple U-Net model. Apart from these mentioned modifications in the base U-Net model, a feature injecting module (FIM) has been added between the expansion and the contraction section of the simple U-Net model. FIM adds the features of the input image at the time of up-sampling. The addition of FIM to a simple U-Net model improves the detection of artificial spots and enhances the performance of the model. The mU-Net has been compared with other models, namely simple U-Net, V-Net, UNet++, ResUnet-a, WideU-Net, and Swin-Unet. The Friedman test that has been conducted on IOU, DICE, MAE, PSNR, and SSIM scores, found that mU-Net balances evaluation metrics well. It appears that the nonparametric Friedman test will improve reproducibility by demonstrating statistical significance. The IOU, DICE, MAE, PSNR, and SSIM scores of the proposed model indicate superior performance as compared to other models.

Transforming satellite imagery into vector maps using modified GANs

Vector maps find widespread utility across diverse domains due to their capacity to not only store but also represent discrete data boundaries such as building footprints, disaster impact analysis, digitization, urban planning, location points, transport links, and more. Although extensive research exists on identifying building footprints and road types from satellite imagery, the generation of vector maps from such imagery remains an area with limited exploration. Furthermore, conventional map generation techniques rely on labor-intensive manual feature extraction or rule-based approaches, which impose inherent limitations. To surmount these limitations, we propose a novel method called HPix, which utilizes modified Generative Adversarial Networks (GANs) to generate vector tile map from satellite images. HPix incorporates two hierarchical frameworks: one operating at the global level and the other at the local level, resulting in a comprehensive model. Through empirical evaluations, our proposed approach showcases its effectiveness in producing highly accurate and visually captivating vector tile maps derived from satellite images. We achieved a pixel-level accuracy of 61.04% and an SSIM score of 0.75, outperforming all other existing methods. We further extend our study’s application to include mapping of road intersections and building footprints cluster based on their area. We also show usability of our proposed architecture as a general-purpose solutions in other tasks like edges-to-photo, BW-to-color, or labels-to-street scene. GitHub:https://github.com/aditya-taparia/Satellite-Image-to-Vector-Map.

Enhancing Low-Light Medical Imaging through Deep Learning-Based Noise Reduction Techniques

Background/ Objectives: Low-light medical imaging is highly challenging in clinical diagnostics due to increased noise levels that mask or obscure important anatomical details. In this respect, conventional noise reduction methods such as Gaussian filtering and median filtering usually lead to a trade-off between noise suppression and the preservation of important features in an image, thus resulting in poor-quality images. More advanced wavelet-based denoising and Non-Local Means methods exhibit superior noise reduction but remain computationally intensive and introduce artifacts. These challenges come with a need to develop more effective and efficient noise-reduction techniques. Methods: This study proposes an end-to-end deep learning framework for low-light medical image enhancement. We present a comprehensive deep-learning framework to enhance low-light medical images by integrating Convolutional Neural Networks with denoising autoencoders to build a robust noise reduction model. The CNN extracts the feature from the noisy input images, while the autoencoder does so for the reconstruction of clean images through the encoding of a noisy input in a lower-dimensional representation for the reduction of noise while retaining critical information. Findings: This study validates the proposed model through rigorous quantitative metrics such as peak signal-to-noise ratio and structural similarity index. These metrics are designed to provide a full assessment of image quality concerning noise reduction capability and preservation of details related to structure. Our model improves traditional methods in PSNR by about 5 dB on average and SSIM by 0.15, which means better noise reduction and preservation of image details. A comparative analysis of traditional techniques for noise reduction has been included, pointing out the advantages of deep learning approaches. Experimental results depict significant improvements over previous approaches. For instance, the proposed model reduces the noise level by up to 40% and facilitates clear and sharp images by up to 30%. In terms of quantification, these improvements manifest in a PSNR value of 35 dB and an SSIM score of 0.85 compared to 30 dB and 0.70 using traditional techniques. Furthermore, the study illustrates the training dynamics, feature maps, and evolution of images to present the model's incremental learning process. Novelty: This study's findings validate the proposed model's efficacy in enhancing diagnosis accuracy and improving patient outcomes in medical imaging. Keywords: Low-light medical imaging, Noise reduction, Convolutional Neural Networks, Denoising autoencoders, Medical diagnostics

Segmentation Guided Crossing Dual Decoding Generative Adversarial Network for Synthesizing Contrast-Enhanced Computed Tomography Images.

Although contrast-enhanced computed tomography (CE-CT) images significantly improve the accuracy of diagnosing focal liver lesions (FLLs), the administration of contrast agents imposes a considerable physical burden on patients. The utilization of generative models to synthesize CE-CT images from non-contrasted CT images offers a promising solution. However, existing image synthesis models tend to overlook the importance of critical regions, inevitably reducing their effectiveness in downstream tasks. To overcome this challenge, we propose an innovative CE-CT image synthesis model called Segmentation Guided Crossing Dual Decoding Generative Adversarial Network (SGCDD-GAN). Specifically, the SGCDD-GAN involves a crossing dual decoding generator including an attention decoder and an improved transformation decoder. The attention decoder is designed to highlight some critical regions within the abdominal cavity, while the improved transformation decoder is responsible for synthesizing CE-CT images. These two decoders are interconnected using a crossing technique to enhance each other's capabilities. Furthermore, we employ a multi-task learning strategy to guide the generator to focus more on the lesion area. To evaluate the performance of proposed SGCDD-GAN, we test it on an in-house CE-CT dataset. In both CE-CT image synthesis tasks-namely, synthesizing ART images and synthesizing PV images-the proposed SGCDD-GAN demonstrates superior performance metrics across the entire image and liver region, including SSIM, PSNR, MSE, and PCC scores. Furthermore, CE-CT images synthetized from our SGCDD-GAN achieve remarkable accuracy rates of 82.68%, 94.11%, and 94.11% in a deep learning-based FLLs classification task, along with a pilot assessment conducted by two radiologists.

A Novel Two-Stage Approach for Automatic Extraction and Multi-View Generation of Litchis

Obtaining consistent multi-view images of litchis is crucial for various litchi-related studies, such as data augmentation and 3D reconstruction. This paper proposes a two-stage model that integrates the Mask2Former semantic segmentation network with the Wonder3D multi-view generation network. This integration aims to accurately segment and extract litchis from complex backgrounds and generate consistent multi-view images of previously unseen litchis. In the first stage, the Mask2Former model is utilized to predict litchi masks, enabling the extraction of litchis from complex backgrounds. To further enhance the accuracy of litchi branch extraction, we propose a novel method that combines the predicted masks with morphological operations and the HSV color space. This approach ensures accurate extraction of litchi branches even when the semantic segmentation model’s prediction accuracy is not high. In the second stage, the segmented and extracted litchi images are passed as input into the Wonder3D network to generate multi-view of the litchis. After comparing different semantic segmentation and multi-view synthesis networks, the Mask2Former and Wonder3D networks demonstrated the best performance. The Mask2Former network achieved a mean Intersection over Union (mIoU) of 79.79% and a mean pixel accuracy (mPA) of 85.82%. The Wonder3D network achieved a peak signal-to-noise ratio (PSNR) of 18.89 dB, a structural similarity index (SSIM) of 0.8199, and a learned perceptual image patch similarity (LPIPS) of 0.114. Combining the Mask2Former model with the Wonder3D network resulted in an increase in PSNR and SSIM scores by 0.21 dB and 0.0121, respectively, and a decrease in LPIPS by 0.064 compared to using the Wonder3D model alone. Therefore, the proposed two-stage model effectively achieves automatic extraction and multi-view generation of litchis with high accuracy.

Data augmentation for Gram-stain images based on Vector Quantized Variational AutoEncoder

Availability of large-scale datasets plays a significant role in segmentation and classification tasks using deep learning. However, domains such as healthcare inherently suffer from unavailability and inaccessibility of data. This leads to challenges in deploying CNN-based models for computer-aided diagnosis. This challenge extends to Gram-stain image analysis for detecting bacterial infections, which is a crucial task. The lack of datasets containing Gram-stained direct and culture smear images exacerbates the significant challenges in deep learning tasks. In this regard, we investigate a novel application of the Variational AutoEncoder. Specifically, the Vector Quantized Variational AutoEncoder model is trained to generate the Gram-stain images. Incorporating a novel loss function, where the quality loss (Lqu) is derived by integrating the LossSSIM, L1, and L2 losses with the VQ-VAE loss (Lossvq) for proposed approach for Gram-stained direct and culture smear images. This modification facilitates the creation of images closely resembling the original input, leading to notable SSIM scores of 0.92 for Gram-stained culture images and 0.88 for Gram-stained direct smear images. The current study compares the proposed method with state-of-the-art machine learning based and CNN based transformations. This work also demonstrates the classification process with and without image augmentation. It shows that the area under the curve in the case of augmentation is higher by an average of 20%.

Aligning the domains in cross domain model inversion attack

Model Inversion Attack reconstructs confidential training dataset from a target deep learning model. Most of the existing methods assume the adversary has an auxiliary dataset that has similar distribution with the private dataset. However, this assumption does not always hold in real-world scenarios. Since the private dataset is unknown, the domain divergence between the auxiliary dataset and the private dataset is inevitable. In this paper, we use Cross Domain Model Inversion Attack to represent the distribution divergence scenario in MIA. With the distribution divergence between the private images and auxiliary images, the distribution between the feature vectors of the private images and those of the auxiliary images is also different. Moreover, the outputted prediction vectors of the auxiliary images are also misclassified. The inversion attack is thus hard to be performed. We perform both the feature vector inversion task and prediction vector inversion task in this cross domain setting. For feature vector inversion, Domain Alignment MIA (DA-MIA) is proposed. While performing the reconstruction task, DA-MIA aligns the feature vectors of auxiliary images with the feature vectors of private images in an adversarial manner to mitigate the domain divergence between them. Thus, semantically meaningful images can be reconstructed. For prediction vector inversion, we further introduce an auxiliary classifier and propose Domain Alignment MIA with Auxiliary Classifier (DA-MIA-AC). The auxiliary classifier is pretrained by the auxiliary dataset and fine-tuned during the adversarial training stage. Thus, the misclassification problem caused by domain divergence can be solved, and the images can be reconstructed correctly. Various experiments are performed to show the advancement of our methods, the results show that DA-MIA can improve the SSIM score of the reconstructed images for up to 191%, DA-MIA-AC can increase the classification accuracy score of the reconstructed images from 9.18% to 81.32% in Cross Domain Model Inversion Attack.

Paired conditional generative adversarial network for highly accelerated liver 4D MRI

Purpose. 4D MRI with high spatiotemporal resolution is desired for image-guided liver radiotherapy. Acquiring densely sampling k-space data is time-consuming. Accelerated acquisition with sparse samples is desirable but often causes degraded image quality or long reconstruction time. We propose the Reconstruct Paired Conditional Generative Adversarial Network (Re-Con-GAN) to shorten the 4D MRI reconstruction time while maintaining the reconstruction quality. Methods. Patients who underwent free-breathing liver 4D MRI were included in the study. Fully- and retrospectively under-sampled data at 3, 6 and 10 times (3×, 6× and 10×) were first reconstructed using the nuFFT algorithm. Re-Con-GAN then trained input and output in pairs. Three types of networks, ResNet9, UNet and reconstruction swin transformer (RST), were explored as generators. PatchGAN was selected as the discriminator. Re-Con-GAN processed the data (3D + t) as temporal slices (2D + t). A total of 48 patients with 12 332 temporal slices were split into training (37 patients with 10 721 slices) and test (11 patients with 1611 slices). Compressed sensing (CS) reconstruction with spatiotemporal sparsity constraint was used as a benchmark. Reconstructed image quality was further evaluated with a liver gross tumor volume (GTV) localization task using Mask-RCNN trained from a separate 3D static liver MRI dataset (70 patients; 103 GTV contours). Results. Re-Con-GAN consistently achieved comparable/better PSNR, SSIM, and RMSE scores compared to CS/UNet models. The inference time of Re-Con-GAN, UNet and CS are 0.15, 0.16, and 120 s. The GTV detection task showed that Re-Con-GAN and CS, compared to UNet, better improved the dice score (3× Re-Con-GAN 80.98%; 3× CS 80.74%; 3× UNet 79.88%) of unprocessed under-sampled images (3× 69.61%). Conclusion. A generative network with adversarial training is proposed with promising and efficient reconstruction results demonstrated on an in-house dataset. The rapid and qualitative reconstruction of 4D liver MR has the potential to facilitate online adaptive MR-guided radiotherapy for liver cancer.

DF-GAM: Cross-Domain Ultrasound Image High-Quality Reconstruction Using a Dual Frequency-Domain Guided Adaptation Model

ObjectiveTo enhance the quality of low-resolution (LR) ultrasound images and mitigate artifacts and speckle noise, which can impede accurate medical diagnosis, a novel method called the dual frequency-domain guided adaptation model (DF-GAM) is proposed. The method aims to achieve high-quality image reconstruction across diverse domains, including different ultrasound machines, diseases and phantom images. MethodsDF-GAM utilizes a dual-branch network architecture combined with frequency-domain self-adaptation and self-supervised edge regression. This approach enables cross-domain enhancement by focusing on the reconstruction of clear tissue structures and speckle patterns. The model is designed to adapt to various ultrasound imaging (USI) scenarios, ensuring its applicability in real-world clinical settings. ResultsExperimental evaluations of DF-GAM were conducted using five different datasets. The results demonstrated the method's effectiveness, with DF-GAM outperforming existing enhancement techniques. The average peak signal-to-noise ratio (PSNR) achieved was 34.62, and the structural similarity index (SSIM) was 0.91, indicating a significant improvement in image quality compared to other methods. ConclusionDF-GAM shows great potential in improving medical image diagnosis and interpretation. Its ability to enhance LR ultrasound images across various domains without the need for extensive training data makes it a valuable tool for clinical use. The high PSNR and SSIM scores validate the method's effectiveness, suggesting that DF-GAM could significantly contribute to the field of USI diagnostics.

Single image dehazing using attention layers in convolutional neural networks

A hazy image is characterized by atmospheric conditions that reduce the image’s clarity and contrast, thereby making it less visible. This degradation in image quality can hinder the performance of advanced computer vision tasks such as object detection and identifying open spaces which need to perform with high accuracy in important real world applications such as security surveillance and autonomous driving. In the recent past, the use of deep learning in image processing tasks have shown a remarkable improvement in performance, in particular, Convolutional Neural Networks (CNNs) perform superior to any other type of neural network in image related tasks. In this paper, we propose the addition of Channel Attention and Pixel Attention layers to four state-of-the-art CNNs, namely, GMAN, U-Net, 123-CEDH and DMPHN, used for the task of image dehazing. We show that the addition of these layers yields a non-trivial improvement on the quality of the dehazed images which we show qualitatively with examples and quantitatively by obtaining PSNR and SSIM scores of 28.63 and 0.959 respectively. Through the experiments, we show that the addition of the mentioned attention layers to the GMAN architecture yields the best results.

TISE-LSTM: A LSTM model for precipitation nowcasting with temporal interactions and spatial extract blocks

Precipitation nowcasting has a profound impact on humanity and society, especially in areas with heavy rainfall, playing a central role in alerting against rainstorm disasters. At present, numerous deep learning-based methods have been proposed and proven superior to traditional radar echo extrapolation techniques like the Recurrent Neural Networks (RNNs). Our study introduces a novel precipitation forecasting model named TISE-LSTM, which can use the real image of the past half hour to predict the radar echo image of the next hour. By integrating the TIB and SEB into TISL-LSTM, we can alleviate the inherent issue observed in existing models, which is the decrease in forecast accuracy for high radar echo regions with extended extrapolation time. On both the CIKM23017 and AHEM real radar echo datasets, the performance of TISL-LSTM (including POD, FAR, CSI and HSS) is improved compared to the second-ranked model by 7.53%, 2.32%, 8.93%, 2.6%, and 14.84%, 6.18%, 8.92%, 18.12%, respectively, when the precipitation threshold is set to 40. Moreover, our model obtained an optimal MAE, MSE, SSIM score. Predicted images and the graphs of each metric over extrapolation time both demonstrate that our model accurately forecasts regions with high radar echoes even with a one-hour extrapolation.

UDRSNet: An unsupervised deformable registration module based on image structure similarity.

Image registration is a challenging problem in many clinical tasks, but deep learning has made significant progress in this area over the past few years. Real-time and robust registration has been made possible by supervised transformation estimation. However, the quality of registrations using this framework depends on the quality of ground truth labels such as displacement field. To propose a simple and reliable method for registering medical images based on image structure similarity in a completely unsupervised manner. We proposed a deep cascade unsupervised deformable registration approach to align images without reliable clinical data labels. Our basic network was composed of a displacement estimation module (ResUnet) and a deformation module (spatial transformer layers). We adopted -norm to regularize the deformation field instead of the traditional -norm regularization. Additionally, we utilized structural similarity (ssim) estimation during the training stage to enhance the structural consistency between the deformed images and the reference images. Experiments results indicated that by incorporating ssim loss, our cascaded methods not only achieved higher dice score of 0.9873, ssim score of 0.9559, normalized cross-correlation (NCC) score of 0.9950, and lower relative sum of squared difference (SSD) error of 0.0313 on CT images, but also outperformed the comparative methods on ultrasound dataset. The statistical -test results also proved that these improvements of our method have statisticalsignificance. In this study, the promising results based on diverse evaluation metrics have demonstrated that our model is simple and effective in deformable image registration (DIR). The generalization ability of the model was also verified through experiments on liver CT images and cardiac ultrasound images.

Spatio-temporal sequence prediction of CO2 flooding and sequestration potential under geological and engineering uncertainties

CO2 injection for subsurface hydrocarbon development not only enhances oil and gas recovery but also enables CO2 sequestration in the subsurface. It is essential to develop effective methods for evaluating the potential of CO2 flooding and sequestration. Despite the existence of methods for predicting the effectiveness of hydrocarbon development using historical production data, insufficient emphasis has been placed on adequately incorporating geological and engineering uncertainty information to enhance prediction accuracy. To address this issue, a novel spatial-temporal ResNet (ST-ResNet) model is proposed for predicting hydrocarbon production, CO2 sequestration volume and CO2 diffusion pattern in the subsurface, which represent the CO2 flooding and sequestration potential. First, high-dimensional reservoir property fields are parameterized using the combined method of principal component analysis and discrete cosine transform (PCA-DCT). Second, the spatial sequence information of various reservoir property fields is extracted with features based on residual neural network (ResNet). Then, the time series information such as dynamic well control parameters is encoded with stacked BiLSTM (SBiLSTM). Specifically, the ST-ResNet model incorporates the above modules to overcome the limitations of collaborative consideration of temporal and spatial information involved in subsurface hydrocarbon development. Comparison between simulation and prediction results on the 2D/3D reservoir model reveals a significant achievement in prediction accuracy by the ST-ResNet model (with R2 and SSIM scores of 0.947, 0.911 and 0.937, 0.922, respectively). In comparison to CNN, LSTM and their combined approach CNN-LSTM, the ST-ResNet model demonstrates an improvement of 3% to 5% in R2, along with reductions of 20% to 30% in MAE and 15% to 25% in RMSE, respectively. These results highlight the superior stability and generalization of the ST-ResNet model. The contribution of this work is to provide a more accurate and efficient prediction tool to guide the integrated development of CO2 flooding and sequestration in subsurface hydrocarbon reservoirs, which facilitates decision-making processes for engineers.

ERS-HDRI: Event-Based Remote Sensing HDR Imaging

High dynamic range imaging (HDRI) is an essential task in remote sensing, enhancing low dynamic range (LDR) remote sensing images and benefiting downstream tasks, such as object detection and image segmentation. However, conventional frame-based HDRI methods may encounter challenges in real-world scenarios due to the limited information inherent in a single image captured by conventional cameras. In this paper, an event-based remote sensing HDR imaging framework is proposed to address this problem, denoted as ERS-HDRI, which reconstructs the remote sensing HDR image from a single-exposure LDR image and its concurrent event streams. The proposed ERS-HDRI leverages a coarse-to-fine framework, incorporating the event-based dynamic range enhancement (E-DRE) network and the gradient-enhanced HDR reconstruction (G-HDRR) network. Specifically, to efficiently achieve dynamic range fusion from different domains, the E-DRE network is designed to extract the dynamic range features from LDR frames and events and perform intra- and cross-attention operations to adaptively fuse multi-modal data. A denoise network and a dense feature fusion network are then employed for the generation of the coarse, clean HDR image. Then, the G-HDRR network, with its gradient enhancement module and multiscale fusion module, performs structure enforcement on the coarse HDR image and generates a fine informative HDR image. In addition, this work introduces a specialized hybrid imaging system and a novel, real-world event-based remote sensing HDRI dataset that contains aligned remote sensing LDR images, remote sensing HDR images, and concurrent event streams for evaluation. Comprehensive experiments have demonstrated the effectiveness of the proposed method. Specifically, it improves state-of-the-art PSNR by about 30% and the SSIM score by about 9% on the real-world dataset.