Residual Modulation Research Articles

MFFC-SDN: multi-level feature fusion codec-based ship detection network in SAR images

ABSTRACT In recent years, with the development of deep learning, SAR image-based ship target-detection technology has become a current research hotspot. SAR images are characterized by complex backgrounds, significant speckle noise interference, and poor interpretability. To address this challenge, this study proposes a new Gaussian circle radius determination strategy and a multi-level feature fusion codec-based ship detection network (MFFC-SDN) for SAR images. Differing from CenterNet’s corner-oriented Gaussian circle strategy, the Gaussian circle strategy proposed in this paper takes the centre point coordinates of the real target as the origin to define a circular region. This approach allows for more precise supervision of the training model. Meanwhile, we propose MFFC-SDN to solve the problem of the poor performance of CenterNet in SAR ship detection. MFFC-SDN is an anchor-free, single-stage target-detection method with a low computation cost. In MFFC-SDN, we introduce a Multilevel Feature Recapture Codec Module (MFR-CM) to enhance the feature extraction capability of the network by using the features of the encoding stage twice because of the easy loss of image features in the lengthy codec process of the CenterNet network backbone. A Residual Attention Pyramid Feature Fusion Module (RA-PFFM) is designed to enhance the feature map and obtain multi-scale features. Experimental results on the SSDD and SAR-ship-dataset show that MFFC-SDN significantly enhances SAR image target-detection performance, effectively addressing detection challenges related to large target size variations and complex environmental conditions. Compared with existing algorithms, MFFC-SDN achieves the highest A P 50 .

Balanced spatio-spectral feature extraction for hyperspectral and multispectral image fusion

Although remote sensing sensors acquire hyperspectral images (HSI) with outstanding spectral resolution, hardware limitations prevent them from obtaining HSI with superior spatial resolution. Having high-resolution multispectral images (MSI) from the same scene has sparked the concept of HSI-MSI fusion to achieve high-resolution HSI. Recently, deep learning algorithms have shown excellent performance in HSI-MSI fusion. Deep learning algorithms can automatically extract the priors latent in the input data and reconstruct the output image. However, their performance depends on the training data’s inherent characteristics. HSI volume is generally significantly larger than MSI volume. While the high-resolution MSI provides richer spatial information, its contribution to fusion is lower due to its smaller volume. To address this volume discrepancy, a deep convolutional neural network named BASFE is proposed. This network employs a two-branch structure that balances the extracted spatio-spectral features from both HSI and MSI input images, enabling effective feature fusion. Therefore, better efficiency can be achieved by using simpler structures. Experiments showed that balancing the data improves the model’s performance significantly. Furthermore, to improve efficiency and reduce complexity and computational burden, the spatio-spectral feature extraction residual modules are embedded in a dense architecture to jointly extract these features. Comprehensive experiments on five challenging datasets show that BASFE surpasses state-of-the-art algorithms. Compared to the first- and second-best competing methods, the proposed method achieves an average improvement of 1.58 dB and 3.68 dB in peak signal-to-noise ratio (PSNR) across the five studied datasets. The source code of BASFE is available in https://github.com/rajaei-arash/BASFE_hyperspectral_fusion.git.

Nighttime road scene image enhancement based on cycle-consistent generative adversarial network

During nighttime road scenes, images are often affected by contrast distortion, loss of detailed information, and a significant amount of noise. These factors can negatively impact the accuracy of segmentation and object detection in nighttime road scenes. A cycle-consistent generative adversarial network has been proposed to address this issue to improve the quality of nighttime road scene images. The network includes two generative networks with identical structures and two adversarial networks with identical structures. The generative network comprises an encoder network and a corresponding decoder network. A context feature extraction module is designed as the foundational element of the encoder-decoder network to capture more contextual semantic information with different receptive fields. A receptive field residual module is also designed to increase the receptive field in the encoder network.The illumination attention module is inserted between the encoder and decoder to transfer critical features extracted by the encoder to the decoder. The network also includes a multiscale discriminative network to discriminate better whether the image is a real high-quality or generated image. Additionally, an improved loss function is proposed to enhance the efficacy of image enhancement. Compared to state-of-the-art methods, the proposed approach achieves the highest performance in enhancing nighttime images, making them clearer and more natural.

SSM-Net: Semi-supervised multi-task network for joint lesion segmentation and classification from pancreatic EUS images

Pancreatic cancer does not show specific symptoms, which makes the diagnosis of early stages difficult with established image-based screening methods and therefore has the worst prognosis among all cancers. Although endoscopic ultrasonography (EUS) has a key role in diagnostic algorithms for pancreatic diseases, B-mode imaging of the pancreas can be affected by confounders such as chronic pancreatitis, which can make both pancreatic lesion segmentation and classification laborious and highly specialized. To address these challenges, this work proposes a semi-supervised multi-task network (SSM-Net) to leverage unlabeled and labeled EUS images for joint pancreatic lesion classification and segmentation. Specifically, we first devise a saliency-aware representation learning module (SRLM) on a large number of unlabeled images to train a feature extraction encoder network for labeled images by computing a contrastive loss with a semantic saliency map, which is obtained by our spectral residual module (SRM). Moreover, for labeled EUS images, we devise channel attention blocks (CABs) to refine the features extracted from the pre-trained encoder on unlabeled images for segmenting lesions, and then devise a merged global attention module (MGAM) and a feature similarity loss (FSL) for obtaining a lesion classification result. We collect a large-scale EUS-based pancreas image dataset (LS-EUSPI) consisting of 9,555 pathologically proven labeled EUS images (499 patients from four categories) and 15,500 unlabeled EUS images. Experimental results on the LS-EUSPI dataset and a public thyroid gland lesion dataset show that our SSM-Net clearly outperforms state-of-the-art methods.

SCGAFusion: A skip-connecting group convolutional attention network for infrared and visible image fusion

The goal of infrared and visible image fusion is to synthesize an image that contains complementary information as much as possible in paired infrared and visible images. However, due to using the convolution operation with local receptive field to extract features, most of the existing deep learning-based fusion methods fail to effectively exploit global contextual information in source images and so obtain a limited fusion performance. To address this issue, this paper proposes an end-to-end skip-connecting group convolutional attention network, termed as SCGAFusion, to fuse infrared and visible images. In SCGAFusion, a group convolutional attention block (GAB) is purposefully developed to promote the extraction and utilization capability of the fusion network for informative features. GAB introduces group convolutional layer to obtain hierarchical features, and employs a residual non-local attention module to capture long-range dependencies between pixels. In this way, it can not only focus more on the important regions or details, but also leverage both local neighborhood and global contextual information. Additionally, a hierarchical feature compensation mechanism based on skip connections is devised to integratedly exploit both local and global features. Experimental results on several public datasets qualitatively and quantitatively demonstrate the advantage of SCGAFusion over other state-of-the-art fusion methods.

Steel Surface Defect Detection Algorithm Based on Improved YOLOv8n

The traditional detection methods of steel surface defects have some problems, such as a lack of feature extraction ability, sluggish detection speed, and subpar detection performance. In this paper, a YOLOv8-based DDI-YOLO model is suggested for effective steel surface defect detection. First, on the Backbone network, the extended residual module (DWR) is fused with the C2f module to obtain C2f_DWR, and the two-step approach is used to carry out the effective extraction of multiscale contextual information, and then fusing feature maps formed from the multiscale receptive fields to enhance the capacity for feature extraction. Also based on the above, an extended heavy parameter module (DRB) is added to the structure of C2f_DWR to make up for the lack of C2f’s ability to capture small-scale pattern defects between training to enhance the training fluency of the model. Finally, the Inner-IoU loss function is employed to enhance the regression accuracy and training speed of the model. The experimental results show that the detection of DDI-YOLO on the NEU-DET dataset improves the mAP by 2.4%, the accuracy by 3.3%, and the FPS by 59 frames/s compared with the original YOLOv8n.Therefore, this paper’s proposed model has a superior mAP, accuracy, and FPS in identifying surface defects in steel.

Hyperspectral image classification based on deep separable residual attention network

Hyperspectral image have rich spatial and spectral information, and how to fully extract and utilize the features of these two dimensions is a research hotspot in hyperspectral classification methods. At present, the unique convolutional operation and deep feature extraction structure of convolutional neural network enable them to have stronger feature representation capabilities and achieve good results in hyperspectral image classification. However, CNN methods do not assign different weights based on the importance of features in the feature extraction process, making it difficult to effectively utilize key features, and most importantly, using fixed shaped convolution kernel can easily overlook the differences between hyperspectral image features. A hyperspectral image classification method based on deep separable residual attention network is proposed to address the above issues. Firstly, to reduce the correlation between hyperspectral image data and minimize the interference of redundant information, principal component analysis is used to reduce the dimensionality of hyperspectral image. Secondly, a shallow feature extraction module is constructed, which can dynamically adjust the size of the receptive field according to the actual situation of the image, adaptively extract shallow features, and reduce the loss of original image features. Then, a depthwise separable residual attention mechanism module is proposed, based on which features are extracted. Starting from global and local features, contextual information on image features in channel and spatial domains is extracted. Finally, use a multi-scale feature fusion module to fully integrate feature maps at different scales. Using Indian Pines, Pavia University and Botswana as experimental datasets, the overall classification accuracy of this paper's method is 98.47 %, 98.70 %, 98.83 % with only 50, 50, 30 training samples per class. The Kappa coefficient is 98.25 %, 98.27 %, and 98.73 %, respectively. Compared with advanced methods, this method not only has higher classification accuracy, but also fully utilizes key features at various network levels.

SR-RDFAN-LOG: Arbitrary-scale logging image super-resolution reconstruction based on residual dense feature aggregation

In the process of oil and gas exploration and development, ultrasonic logging imaging technology visually displays the lithology, structure, fractures, pores, and other characteristics of the wellbore. This technology has been widely applied in the exploration and development of oil and gas resources. However, the low circumferential sampling rate negatively affects the imaging quality and resolution during drilling logging. To address the issues of low resolution and blurriness in ultrasonic logging images caused by these factors, as well as the limitation of existing super-resolution reconstruction algorithms that can only reconstruct fixed scales, this paper presents an Image Super-Resolution Reconstruction Network called Residual Dense Feature Aggregation (SR-RDFAN-LOG), which is specifically designed to improve ultrasonic logging image resolution. The proposed network consists of two stages, including encoding and decoding. In the encoding stage, a local feature extraction module is initially designed to integrate contextual features. Subsequently, a residual dense module is introduced for extracting deep features. Finally, a spatial attention module is constructed to focus on high-frequency details. In the decoding stage, a multilayer perceptron is used to decode the feature maps and achieve super-resolution reconstruction at arbitrary scales. The method is tested on an ultrasonic logging images test set with in-distribution scale factors of × 2 and × 4, and out-of-distribution scale factors of × 8 and × 16. When the scaling factors are × 2 and × 4, compared to the best-performing method, the peak signal-to-noise ratio (PSNR) increased by 0.4892 dB and 0.3849 dB, respectively, the structural similarity (SSIM) values increased by 0.0002 and 0.0012, respectively, and the learned perceptual image patch similarity (LPIPS) decreased by 0.0001 and 0.0008, respectively. Compared to the bicubic interpolation method, the LPIPS decreased by 0.0001 and 0.0008 at scale factors of × 8 and × 16, PSNR increased by 1.9614 dB and 2.1002 dB, respectively, and SSIM values increased by 0.0476 and 0.033, respectively. The LPIPS values decreased by 0.2233 and 0.1070, respectively. The experimental results demonstrate that our method outperforms other mainstream methods, achieves super-resolution reconstruction at arbitrary scale, and provides support for subsequent fracture-cave evaluation and oil and gas exploration.

Real-Time Attentive Dilated U-Net for Extremely Dark Image Enhancement

Images taken under low-light conditions suffer from poor visibility, color distortion, and graininess, all of which degrade the image quality and hamper the performance of downstream vision tasks, such as object detection and instance segmentation in the field of autonomous driving, making low-light enhancement an indispensable basic component of high-level visual tasks. Low-light enhancement aims to mitigate these issues, and has garnered extensive attention and research over several decades. The primary challenge in low-light image enhancement arises from the low signal-to-noise ratio caused by insufficient lighting. This challenge becomes even more pronounced in near-zero lux conditions, where noise overwhelms the available image information. Both traditional image signal processing pipeline and conventional low-light image enhancement methods struggle in such scenarios. Recently, deep neural networks have been used to address this challenge. These networks take unmodified RAW images as input and produce the enhanced sRGB images, forming a deep learning based image signal processing pipeline. However, most of these networks are computationally expensive and thus far from practical use. In this article, we propose a lightweight model called attentive dilated U-Net (ADU-Net) to tackle this issue. Our model incorporates several innovative designs, including an asymmetric U-shape architecture, dilated residual modules for feature extraction, and attentive fusion modules for feature fusion. The dilated residual modules provide strong representative capability, whereas the attentive fusion modules effectively leverage low-level texture information and high-level semantic information within the network. Both modules employ a lightweight design but offer significant performance gains. Extensive experiments demonstrate that our method is highly effective, achieving an excellent balance between image quality and computational complexity—that is, taking less than 4ms for a high-definition 4K image on a single GTX 1080Ti GPU and yet maintaining competitive visual quality. Furthermore, our method exhibits pleasing scalability and generalizability, highlighting its potential for widespread applicability.

IAE-KM3D a 3D Object Detection Method Based on an Improved KM3D Network

Deep learning-based 3D target detection methods need to solve the problem of insufficient 3D target detection accuracy. In this paper, the KM3D network is selected as the benchmark network after the experimental comparison of current mainstream algorithms, and the IAE-KM3D network algorithm based on the KM3D network is proposed. First, the Resnet V2 network is introduced, and the residual module is redesigned to improve the training capability of the new residual module with higher generalization. IBN NET is then introduced to carefully integrate instance normalization and batch normalization as building blocks to improve the model’s detection accuracy in hue- and brightness-changing scenarios without increasing time loss. Then, a parameter-free attention mechanism, Simam, is introduced to improve the detection accuracy of the model. After that, the elliptical Gaussian kernel is introduced to improve the algorithm’s ability to detect 3D targets. Finally, a new key point loss function is proposed to improve the algorithm’s ability to train. Experiments using the KITTI dataset conclude that the IAE-KM3D network model significantly improves detection accuracy and outperforms the KM3D algorithm regarding detection performance compared to the original KM3D network. The improvements for AP2D, AP3D, and APBEV are 5%, 12.5%, and 8.3%, respectively, and only a tiny amount of time loss and network parameters are added. Compared with other mainstream target detection algorithms, Monn3D, 3DOP, GS3D, and FQNet, the improved IAE-KM3D network in this paper significantly improves AP3D and APBEV, with fewer network parameters and shorter time consumption.

MpoxNet: dual-branch deep residual squeeze and excitation monkeypox classification network with attention mechanism.

While the world struggles to recover from the devastation wrought by the widespread spread of COVID-19, monkeypox virus has emerged as a new global pandemic threat. In this paper, a high precision and lightweight classification network MpoxNet based on ConvNext is proposed to meet the need of fast and safe detection of monkeypox classification. In this method, a two-branch depth-separable convolution residual Squeeze and Excitation module is designed. This design aims to extract more feature information with two branches, and greatly reduces the number of parameters in the model by using depth-separable convolution. In addition, our method introduces a convolutional attention module to enhance the extraction of key features within the receptive field. The experimental results show that MpoxNet has achieved remarkable results in monkeypox disease classification, the accuracy rate is 95.28%, the precision rate is 96.40%, the recall rate is 93.00%, and the F1-Score is 95.80%. This is significantly better than the current mainstream classification model. It is worth noting that the FLOPS and the number of parameters of MpoxNet are only 30.68% and 31.87% of those of ConvNext-Tiny, indicating that the model has a small computational burden and model complexity while efficient performance.

The fusion of lock-in phase images for the damage investigation in composites materials

Lock-in thermography (LIT) is a type of active thermography capable of detecting and evaluating the subsurface defects in composite materials. The phase or amplitude difference between the defect and sound regions can quantitatively determine the size and depth of defects. One limitation of LIT is that the optimal identification of defects located at specific depths is dispersed within phase images acquired at varying excitation frequencies. If the depth of the defects within the specimen is unknown, it is difficult to determine the excitation frequencies to achieve optimal contrast for all defects and display them in a single image. To address this challenge, a multi-frequency fused method based on lock-in thermography is proposed to improve the quality of defect detection. Initially, the optimal thermal wave excitation frequencies and the number of detection times are determined based on the theoretical solution. Subsequently, phase images at various excitation frequencies are extracted and enhanced using a specified scheme. Finally, we develop an unsupervised encoder-decoder network that combines dense connections and residual modules to fuse these phase images into a single image containing defects at various depths. An experiment is conducted to detect carbon fiber reinforced polymer (CFRP) laminate containing defects with different depths and sizes. The results demonstrate that the proposed method can broaden the detection range of defect depths and improve the quality of the inspected images, providing superior performance compared to traditional sequential image processing methods.

Multi-Branch Network for Color Image Denoising Using Dilated Convolution and Attention Mechanisms.

Image denoising is regarded as an ill-posed problem in computer vision tasks that removes additive noise from imaging sensors. Recently, several convolution neural network-based image-denoising methods have achieved remarkable advances. However, it is difficult for a simple denoising network to recover aesthetically pleasing images owing to the complexity of image content. Therefore, this study proposes a multi-branch network to improve the performance of the denoising method. First, the proposed network is designed based on a conventional autoencoder to learn multi-level contextual features from input images. Subsequently, we integrate two modules into the network, including the Pyramid Context Module (PCM) and the Residual Bottleneck Attention Module (RBAM), to extract salient information for the training process. More specifically, PCM is applied at the beginning of the network to enlarge the receptive field and successfully address the loss of global information using dilated convolution. Meanwhile, RBAM is inserted into the middle of the encoder and decoder to eliminate degraded features and reduce undesired artifacts. Finally, extensive experimental results prove the superiority of the proposed method over state-of-the-art deep-learning methods in terms of objective and subjective performances.

A novel pansharpening method based on cross stage partial network and transformer

In remote sensing image fusion, the conventional Convolutional Neural Networks (CNNs) extract local features of the image through layered convolution, which is limited by the receptive field and struggles to capture global features. Transformer utilizes self-attention to capture long-distance dependencies in images, which has a global receptive field, but the computational cost for high-resolution images is excessively high. In response to the above issues, this paper draws inspiration from the FusionNet network, harnessing the local detail acquisition capability of CNNs and the global data procuring capacity of Transformer. It presents a novel method for remote sensing image sharpening named Guided Filtering-Cross Stage Partial Network-Transformer, abbreviated as GF-CSTNet. This solution unifies the strengths of Guided Filtering (GF), Cross Stage Partial Network (CSPNet), and Transformer. Firstly, this method utilizes GF to enhance the acquired remote sensing image data. The CSPNet and Transformer structures are then combined to further enhance fusion performance by leveraging their respective advantages. Subsequently, a Rep-Conv2Former method is designed to streamline attention and extract diverse receptive field features through a multi-scale convolution modulator block. Simultaneously, a reparameterization module is constructed to integrate the multiple branches generated during training into a unified branch during inference, thereby optimizing the model’s inference speed. Finally, a residual learning module incorporating attention has been devised to augment the modeling and feature extraction capabilities of images. Experimental results obtained from the GaoFen-2 and WorldView-3 datasets demonstrate the effectiveness of the proposed GF-CSTNet approach. It effectively extracts detailed information from images while avoiding the problem of spectral distortion.

Medical image classification: Knowledge transfer via residual U-Net and vision transformer-based teacher-student model with knowledge distillation

With the widespread integration of deep learning techniques in the domain of medical image analysis, there is a prevailing consensus regarding their efficacy in handling high-dimensional and intricate medical image data. However, it is imperative to acknowledge that while complex deep models exhibit a remarkable capacity for processing high-dimensional and intricate data, they often necessitate a substantial allocation of computational resources and time. Furthermore, lightweight models, despite their computational efficiency, tend to underperform when compared to their more intricate counterparts in terms of performance. Hence, the prevailing aspiration is to transfer the cognitive prowess of complex models to their lightweight counterparts. Addressing the aforementioned concern, this study proposes a knowledge distillation approach that encompasses joint feature and soft label transfer. It entails the transference of knowledge from the teacher model’s intermediate features and predictive outcomes to the student model. The student model leverages this knowledge to emulate the behavior of the teacher model, thereby enhancing the precision of its own predictions. Building upon this foundation, we introduce a Res-Transformer teacher model based on the U-Net architecture and a student model known as ResU-Net, which is grounded in residual modules. The Res-Transformer teacher model employs multi-layer residual attention during the downsampling process to capture deep-level features of the image. Subsequently, we have incorporated a Multi-layer Perceptual Attention module (MPA) for each skip connection layer, facilitating the integration of hierarchical upsampled information to restore fine-grained details within the feature maps. The ResU-Net student model enhances network stability through the utilization of residual modules and optimizes skip connections to recover any lost image information during convolutional operations. Lastly, we conducted experimental assessments on multiple disease datasets. The results reveal that the ACC of the Res-Transformer model achieves an impressive 96.9%. Furthermore, through the knowledge distillation method, rich knowledge is effectively transferred to the ResU-Net model, resulting in a remarkable ACC improvement of 7.2%.

DeepSFM: Robust Deep Iterative Refinement for Structure From Motion.

Structure from Motion (SfM) is a fundamental computer vision problem which has not been well handled by deep learning. One of the promising solutions is to apply explicit structural constraint, e.g., 3D cost volume, into the neural network. Obtaining accurate camera poses from images alone can be challenging, especially with complicated environmental factors. Existing methods usually assume accurate camera poses from GT or other methods, which is unrealistic in practice and additional sensors are needed. In this work, we design a physical driven architecture, namely DeepSFM, inspired by traditional Bundle Adjustment, which consists of two cost volume based architectures to iteratively refine depth and pose. The explicit constraints on both depth and pose, when combined with the learning components, bring merit from both traditional BA and emerging deep learning technology. To speed up the learning and inference efficiency, we apply the Gated Recurrent Units (GRUs)-based depth and pose update modules with coarse to fine cost volumes on the iterative refinements. In addition, with the extended residual depth prediction module, our model can be adapted to dynamic scenes effectively. Extensive experiments on various datasets show that our model achieves state-of-the-art performance with superior robustness against challenging inputs.

Expansive Receptive Field and Local Feature Extraction Network: Advancing Multiscale Feature Fusion for Breast Fibroadenoma Segmentation in Sonography.

Fibroadenoma is a common benign breast disease that affects women of all ages. Early diagnosis can greatly improve the treatment outcomes and reduce the associated pain. Computer-aided diagnosis (CAD) has great potential to improve diagnosis accuracy and efficiency. However, its application in sonography is limited. A network that utilizes expansive receptive fields and local information learning was proposed for the accurate segmentation of breast fibroadenomas in sonography. The architecture comprises the Hierarchical Attentive Fusion module, which conducts local information learning through channel-wise and pixel-wise perspectives, and the Residual Large-Kernel module, which utilizes multiscale large kernel convolution for global information learning. Additionally, multiscale feature fusion in both modules was included to enhance the stability of our network. Finally, an energy function and a data augmentation method were incorporated to fine-tune low-level features of medical images and improve data enhancement. The performance of our model is evaluated using both our local clinical dataset and a public dataset. Mean pixel accuracy (MPA) of 93.93% and 86.06% and mean intersection over union (MIOU) of 88.16% and 73.19% were achieved on the clinical and public datasets, respectively. They are significantly improved over state-of-the-art methods such as SegFormer (89.75% and 78.45% in MPA and 83.26% and 71.85% in MIOU, respectively). The proposed feature extraction strategy, combining local pixel-wise learning with an expansive receptive field for global information perception, demonstrates excellent feature learning capabilities. Due to this powerful and unique local-global feature extraction capability, our deep network achieves superior segmentation of breast fibroadenoma in sonography, which may be valuable in early diagnosis.

Thermal image visualization using multi-discriminator CycleGAN with unpaired thermal–visible image training set

Thermal imaging utilizes the differences in thermal radiation within a scene to capture images, effectively compensating for the limited imaging capabilities of visible-light cameras in low-light environments. In order to further enhance thermal imaging, we propose a novel approach that transforms the visualization of thermal images into an image conversion task. Our method, named TIVNet (Thermal Images Visualization Network), is based on a multi-discriminator cycle-consistent adversarial network (CycleGAN) and operates in the YUV color space to accurately colorize and represent thermal infrared images. The utilization of color and texture discriminators in our framework ensures that the converted images exhibit appropriate color styles and capture fine details, respectively. In the generator, we employed a Dense Residual Module (DRM) to facilitate the reuse of features and enhance the flow of information. By utilizing the DRM, we aim to enhance the overall performance and efficiency of the generator network. Additionally, we incorporated the Dual-channel Self-Attention Module(DCSA) to enhance the feature representation. By leveraging self-attention, the generator can effectively learn and exploit the inherent relationships between different regions, leading to improved feature representation and, consequently, more accurate colorization and representation of thermal infrared images. Notably, our proposed method leverages unsupervised learning, eliminating the need for a fully paired thermal infrared-visible image dataset. Through extensive experimentation on publicly available datasets as well as our own, we demonstrate that our approach outperforms state-of-the-art methods in thermal image visualization. Furthermore, the converted images produced by TIVNet exhibit a high level of realism, thereby enhancing the visual impact of thermal imaging. This improvement is particularly beneficial for human perception and subsequent target recognition tasks. Our findings suggest that the application of our method yields superior results both quantitatively and qualitatively, showcasing its potential for advancing thermal imaging capabilities.

ConvSRGAN: super-resolution inpainting of traditional Chinese paintings

Existing image super-resolution methods have made remarkable advancements in enhancing the visual quality of real-world images. However, when it comes to restoring Chinese paintings, these methods encounter unique challenges. This is primarily due to the difficulty in preserving intricate non-realistic details and capturing comple semantic information with high dimensionality. Moreover, the preservation of the original artwork’s distinct style and subtle artistic nuances further amplifies this complexity. To address these challenges and effectively restore traditional Chinese paintings, we propose a Convolutional Super-Resolution Generative Adversarial Network for Chinese landscape painting super-resolution, termed ConvSRGAN. We employ Enhanced Adaptive Residual Module to delve deeply into multi-scale feature extraction in images, incorporating an Enhanced High-Frequency Retention Module that leverages an Adaptive Deep Convolution Block to capture fine-grained high-frequency details across multiple levels. By combining the Multi-Scale Structural Similarity loss with conventional losses, our ConvSRGAN ensures that the model produces outputs with improved fidelity to the original image’s texture and structure. Experimental validation demonstrates significant qualitative and quantitative results when processing traditional paintings and murals datasets, particularly excelling in high-definition reconstruction tasks for landscape paintings. The reconstruction effect showcases enhanced visual fidelity and liveliness, thus affirming the effectiveness and applicability of our approach in cultural heritage preservation and restoration.

Uncertainty estimation- and attention-based semi-supervised models for automatically delineate clinical target volume in CBCT images of breast cancer

ObjectivesAccurate segmentation of the clinical target volume (CTV) of CBCT images can observe the changes of CTV during patients' radiotherapy, and lay a foundation for the subsequent implementation of adaptive radiotherapy (ART). However, segmentation is challenging due to the poor quality of CBCT images and difficulty in obtaining target volumes. An uncertainty estimation- and attention-based semi-supervised model called residual convolutional block attention-uncertainty aware mean teacher (RCBA-UAMT) was proposed to delineate the CTV in cone-beam computed tomography (CBCT) images of breast cancer automatically.MethodsA total of 60 patients who undergone radiotherapy after breast-conserving surgery were enrolled in this study, which involved 60 planning CTs and 380 CBCTs. RCBA-UAMT was proposed by integrating residual and attention modules in the backbone network 3D UNet. The attention module can adjust channel and spatial weights of the extracted image features. The proposed design can train the model and segment CBCT images with a small amount of labeled data (5%, 10%, and 20%) and a large amount of unlabeled data. Four types of evaluation metrics, namely, dice similarity coefficient (DSC), Jaccard, average surface distance (ASD), and 95% Hausdorff distance (95HD), are used to assess the model segmentation performance quantitatively.ResultsThe proposed method achieved average DSC, Jaccard, 95HD, and ASD of 82%, 70%, 8.93, and 1.49 mm for CTV delineation on CBCT images of breast cancer, respectively. Compared with the three classical methods of mean teacher, uncertainty-aware mean-teacher and uncertainty rectified pyramid consistency, DSC and Jaccard increased by 7.89–9.33% and 14.75–16.67%, respectively, while 95HD and ASD decreased by 33.16–67.81% and 36.05–75.57%, respectively. The comparative experiment results of the labeled data with different proportions (5%, 10% and 20%) showed significant differences in the DSC, Jaccard, and 95HD evaluation indexes in the labeled data with 5% versus 10% and 5% versus 20%. Moreover, no significant differences were observed in the labeled data with 10% versus 20% among all evaluation indexes. Therefore, we can use only 10% labeled data to achieve the experimental objective.ConclusionsUsing the proposed RCBA-UAMT, the CTV of breast cancer CBCT images can be delineated reliably with a small amount of labeled data. These delineated images can be used to observe the changes in CTV and lay the foundation for the follow-up implementation of ART.