Limited Receptive Field Research Articles

Network security situation assessment based on dual attention mechanism and HHO-ResNeXt

The traditional convolutional neural network (CNN) has a limited receptive field and cannot accurately identify the importance of each channel, making it difficult to solve increasingly complex network security problems. To solve these problems, this paper combines ResNeXt with the Efficient Channel Attention (ECA) module and the Contextual Transformer (COT) block to construct a model to assess network conditions. The optimal hyperparameters of the model are selected by the Harris Hawks Optimization (HHO) algorithm. The model can accurately obtain the importance of each channel to assign weights to each channel while making full use of the rich contexts among neighbour keys, effectively enhancing the convolutional neural network. Furthermore, this paper calculates the network security situation value (NSSV) of the adopted datasets based on attack impact. Lastly, experiments on two cybersecurity datasets show that the comprehensive performance of the model on the three indicators of accuracy, precision and F-scores, as well as network security situation assessment, are superior to other models.

Progressive Transformer Machine for Natural Character Reenactment

Character reenactment aims to control a target person’s full-head movement by a driving monocular sequence that is made up of the driving character video. Current algorithms utilize convolution neural networks in generative adversarial networks, which extract historical and geometric information to iteratively generate video frames. However, convolution neural networks can merely capture local information with limited receptive fields and ignore global dependencies that play a crucial role in face synthesis, leading to generating unnatural video frames. In this work, we design a progressive transformer module that introduces multi-head self-attention with convolution refinement to simultaneously capture global-local dependencies. Specifically, we utilize the non-lapping window-based multi-head self-attention mechanism with hierarchical architecture to obtain the larger receptive fields at low-resolution feature map and thus extract global information. To better model local dependencies, we introduce the convolution operation to further refine the attentional weight in the multi-head self-attention mechanism. Finally, we use several stacked progressive transformer modules with the down-sampling operation to encode information of appearance information of previously generated frames and parameterized 3D face information of the current frame. Similarly, we use several stacked progressive transformer modules with the up-sampling operation to iteratively generate video frames. In this way, it can capture global-local information to facilitate generating video frames that are globally natural while preserving sharp outlines and rich detail information. Extensive experiments on several standard benchmarks suggest that the proposed method outperforms current leading algorithms.

3D Medical image segmentation using parallel transformers

Most recent 3D medical image segmentation methods adopt convolutional neural networks (CNNs) that rely on deep feature representation and achieve adequate performance. However, due to the convolutional architectures having limited receptive fields, they cannot explicitly model the long-range dependencies in the medical image. Recently, Transformer can benefit from global dependencies using self-attention mechanisms and learn highly expressive representations. Some works were designed based on the Transformers, but the existing Transformers suffer from extreme computational and memories, and they cannot take full advantage of the powerful feature representations in 3D medical image segmentation. In this paper, we aim to connect the different resolution streams in parallel and propose a novel network, named Transformer based High Resolution Network (TransHRNet), with an Effective Transformer (EffTrans) block, which has sufficient feature representation even at high feature resolutions. Given a 3D image, the encoder first utilizes CNN to extract the feature representations to capture the local information, and then the different feature maps are reshaped elaborately for tokens that are fed into each Transformer stream in parallel to learn the global information and repeatedly exchange the information across streams. Unfortunately, the proposed framework based on the standard Transformer needs a huge amount of computation, thus we introduce a deep and effective Transformer to deliver better performance with fewer parameters. The proposed TransHRNet is evaluated on the Multi-Atlas Labeling Beyond the Cranial Vault (BCV) dataset that consists of 11 major human organs and the Medical Segmentation Decathlon (MSD) dataset for brain tumor and spleen segmentation tasks. Experimental results show that it performs better than the convolutional and other related Transformer-based methods on the 3D multi-organ segmentation tasks. Code is available at https://github.com/duweidai/TransHRNet.

Infrared ship target segmentation based on Adversarial Domain Adaptation

Infrared ship target segmentation is one of the key technologies for automatically detecting ship targets in ocean monitoring. However, it is a challenging work to achieve accurate target segmentation from the infrared ship image. To improve its segmentation performance, we present an Adversarial Domain Adaptation Network (ADANet) for infrared ship target segmentation, where the labeled visible ship images are used as the source domain and the unlabeled infrared ship images are as the target domain. To address the issue of style difference between the two domains, we preprocess the visible images of the source domain in turn with graying and whitening to convert them into the images with the style of the target domain. For the infrared images in the target domain, we optimize them with a denoising network. Furthermore, to solve the matter of limited receptive field of the discriminator, we design a discriminator based on atrous convolution to improve its discriminative ability. Finally, for the issue of low confidence of the target domain predicted images, we add the information entropy of the target domain predicted images to the adversarial loss. Experimental results on the home-made dataset as well as a public dataset show that infrared ship target segmentation achieves higher mean intersection over union than the state-of-the-art methods without significantly increase of parameters, demonstrating its effectiveness.

Fourier-inspired neural module for real-time and high-fidelity computer-generated holography.

Learning-based computer-generated holography (CGH) algorithms appear as novel alternatives to generate phase-only holograms. However, most existing learning-based approaches underperform their iterative peers regarding display quality. Here, we recognize that current convolutional neural networks have difficulty learning cross-domain tasks due to the limited receptive field. In order to overcome this limitation, we propose a Fourier-inspired neural module, which can be easily integrated into various CGH frameworks and significantly enhance the quality of reconstructed images. By explicitly leveraging Fourier transforms within the neural network architecture, the mesoscopic information within the phase-only hologram can be more handily extracted. Both simulation and experiment were performed to showcase its capability. By incorporating it into U-Net and HoloNet, the peak signal-to-noise ratio of reconstructed images is measured at 29.16 dB and 33.50 dB during the simulation, which is 4.97 dB and 1.52 dB higher than those by the baseline U-Net and HoloNet, respectively. Similar trends are observed in the experimental results. We also experimentally demonstrated that U-Net and HoloNet with the proposed module can generate a monochromatic 1080p hologram in 0.015 s and 0.020 s, respectively.

Novel Convolutions for Semantic Segmentation of Remote Sensing Images

The networks are required to be capable of learning low-level features well when applied to remote sensing image semantic segmentation task. To capture accurate and abundant low-level semantic information, the early feature extractor layer is crucial to the whole network, because all the subsequent features are inferred from that base. To address the low-level feature extraction issue and overcome the shortcomings of traditional convolution like too many parameters or limited receptive field, some novel convolution units have been proposed in the literature. In this paper, we propose two elaborately designed and portable yet effective convolution units, i.e., directional convolution (DC) and large field convolution (LFC), combined as the extractor of low-level semantic features. DC is designed to extract directional features from specific directions, and LFC can achieve a large receptive field with few parameters. Experiment results on two public datasets provide evidence that our convolution units can help deep learning networks improve performance stably and comprehensively compared to the baseline networks.

Local Information-Enhanced Graph-Transformer for Hyperspectral Image Change Detection With Limited Training Samples

Hyperspectral image (HSI) change detection is a challenging task that focuses on identifying the differences between multi-temporal HSIs. The recent advancement of convolutional neural network (CNN) has made great progress on HSIs change detection. However, due to the limited receptive field, most CNN based change detection models trained with sufficient labeled samples cannot flexibly model the global information that is essential for distinguishing complex objects, thereby achieving relatively-low performance. In this paper, we propose a dual-branch local information enhanced graph-transformer change detection network to fully exploit the local-global spectral-spatial features of the multi-temporal HSIs with limited training samples for change recognition. Specifically, the proposed network is composed of a cascaded of local information enhanced graph-transformer (LIEG) blocks, which jointly extracts local-global features by learning local information representation to enhance the information of graph-transformer. A novel graph-transformer is developed to model global spectral–spatial correlation between graph nodes, enabling the spectral information preservation of HSIs and accurate change detection of areas with various sizes. Extensive experiments have proved that our method achieves significant performance improvement than other state-of-the-art methods on four commonly used HSI datasets.

CTCNet: A CNN-Transformer Cooperation Network for Face Image Super-Resolution.

Recently, deep convolution neural networks (CNNs) steered face super-resolution methods have achieved great progress in restoring degraded facial details by joint training with facial priors. However, these methods have some obvious limitations. On the one hand, multi-task joint learning requires additional marking on the dataset, and the introduced prior network will significantly increase the computational cost of the model. On the other hand, the limited receptive field of CNN will reduce the fidelity and naturalness of the reconstructed facial images, resulting in suboptimal reconstructed images. In this work, we propose an efficient CNN-Transformer Cooperation Network (CTCNet) for face super-resolution tasks, which uses the multi-scale connected encoder-decoder architecture as the backbone. Specifically, we first devise a novel Local-Global Feature Cooperation Module (LGCM), which is composed of a Facial Structure Attention Unit (FSAU) and a Transformer block, to promote the consistency of local facial detail and global facial structure restoration simultaneously. Then, we design an efficient Feature Refinement Module (FRM) to enhance the encoded features. Finally, to further improve the restoration of fine facial details, we present a Multi-scale Feature Fusion Unit (MFFU) to adaptively fuse the features from different stages in the encoder procedure. Extensive evaluations on various datasets have assessed that the proposed CTCNet can outperform other state-of-the-art methods significantly. Source code will be available at https://github.com/IVIPLab/CTCNet.

Defect-aware transformer network for intelligent visual surface defect detection

Surface defect detection plays an increasing role in intelligent manufacturing and product life-cycle management, such as quality inspection, process monitoring, and preventive maintenance. The existing intelligent methods almost adopt convolution architecture, and the limited receptive field hinders performance improvement of defect detection. In general, a larger receptive field can bring richer contextual information, resulting in better performance. Although operations such as dilated convolution can expand the receptive field, this improvement is still limited. Recently, benefitting from the ability to model long-range dependencies, Transformer-based models achieve great success in computer vision and image processing. However, applying Transformer-based models without modification is not desirable because there is no awareness and pertinence to defects. In this paper, an intelligent method is proposed by using defect-aware Transformer network (DAT-Net). In DAT-Net, Transformer replaces convolution in encoder to overcome the difficulty of modeling long-range dependencies. Defect-aware module assembled by basic weight matrixes is incorporated into Transformer to perceive and capture geometry and characteristic of defect. Graph position encoding by constructing a dynamic graph on tokens is designed to provide auxiliary positional information, which brings desired improved performance and fine adaptability. Specially, we carry out field experiments and painstakingly construct blade defect and tool wear datasets to compare DAT-Net with other methods. The comprehensive experiments demonstrate that DAT-Net has superior performance with 90.19 mIoU on blade defect dataset and 87.24 mIoU on tool wear dataset.

ACMTR: Attention-guided, combined multi-scale, transformer reasoning-based network for 3D CT pelvic functional bone marrow segmentation

Automatic segmentation of pelvic functional bone marrow (FBM) in computed tomography (CT) plays a critical role in the accurate diagnosis of pelvic malignancies and treatment planning for patients. Pelvic bone marrow near the mid-axis of the body, i.e., the FBM, is the most hematopoietic. Here we propose an Attention-guided, Combined Multi-scale, Transformer Reasoning-based network (ACMTR) for accurate segmentation of pelvic FBM for intensity-modulated radiation therapy. The presented Convolution-Attention Fusion Encoder overcomes the limited receptive field of convolution as well as the class imbalance issue in pelvic FBM segmentation. We also propose a Multi-scale Transformer Global Reasoning Bottleneck module for obtaining multi-scale, global attention features for extraction of small edge tissue of the FBM. We introduce a deformable attention mechanism at the decoding path to focus on key features instead of redundant information with a minimal increase in computational complexity. The pelvic FBM CT images used in this study were from a self-constructed clinical database of 92 cases and showed two main areas of FBM: the most intense FBM area (i.e., most hematopoietic area) and the second most intense FBM area. The proposed ACMTR was compared with the popular deep learning models U-Net, VGG-Net, Attention-UNet, TransUNet, CoTr, nnUNet, and nnFormer. For the most intense and second most intense FBM areas, the proposed ACMTR achieved Dice Scores of 0.799 and 0.772 and Hausdorff Distance of 6.061 and 4.791, respectively, on test data. The implementation code is available at the following Github link:https://github.com/walynlee/ACMTR.

Image Deblurring Based on an Improved CNN-Transformer Combination Network

Recently, using a CNN has been a common practice to restore blurry images due to its strong ability to learn feature information from large-scale datasets. However, CNNs essentially belong to local operations and have the defect of a limited receptive field, which reduces the naturalness of deblurring results. Moreover, CNN-based deblurring methods usually adopt many downsample operations, which hinder detail recovery. Fortunately, transformers focus on modeling the global features, so they can cooperate with CNNs to enlarge the receptive field and compensate for the details lost as well. In this paper, we propose an improved CNN-transformer combination network for deblurring, which adopts a coarse-to-fine architecture as the backbone. To extract the local features and global features simultaneously, the common methods are two blocks connected in parallel or cascaded. Different from these, we design a local-global feature combination block (LGFCB) with a new connecting structure to better use the extracted features. The LGFCB comprises multi-scale residual blocks (MRB) and a transformer block. In addition, we adopt a channel attention fusion block (CAFB) in the encoder path to integrate features. To improve the ability of feature representation, in the decoder path, we introduce a supervised attention block (SAB) operated on restoration images to refine features. Numerous experiments on GoPro and RealBlur datasets indicated that our model achieves remarkable accuracy and processing speed.

A neural harmonic-aware network with gated attentive fusion for singing melody extraction

Singing melody extraction from polyphonic musical audio is one of the most challenging tasks in music information retrieval (MIR). Recently, data-driven methods based on convolutional neural networks (CNNs) have achieved great success for this task. In the literature, harmonic relationship has been proven crucial for this task. However, few existing CNN-based singing melody extraction methods consider the harmonic relationship in the training stage. The state-of-the-art CNN based methods are not capable of capturing such long-dependency harmonic relationship due to limited receptive field and unacceptable computation cost. In this paper, we introduce a neural harmonic-aware network with gated attentive fusion (NHAN-GAF) for singing melody extraction. Specifically, in the 2-D spectrograms modeling branch, we propose to employ multiple parallel 1-D CNN kernels to capture the harmonic relations between 1–2 octaves along the frequency axis in the spectrogram. Considering the advantage of jointly using Time–Frequency (T-F) domain and time domain information, we use two-branch neural nets to learn discriminative representation for this task. A novel gated attentive fusion (GAF) network is suggested to encode potential correlations between the two branches and fuse the descriptors learned from raw waveform and T-F spectrograms. Moreover, the idea of GAF can be exploited to the multimedia applications with multimodal analysis. With the two proposed components, our proposed model is capable of learning the harmonic relationship in the spectrogram and better capturing the contextual but discriminative features for singing melody extraction. We use part of the vocal tracks of the RWC dataset and MIR-1 K dataset to train the model and evaluate the performance of the proposed model on the ADC 2004, MIREX 05 and MedleyDB datasets. The experimental results show that the proposed method outperforms the state-of-the-art ones.

MAGF-Net: A multiscale attention-guided fusion network for retinal vessel segmentation

Retinal fundus images contain plenty of morphological information, so it is particularly important to realize precise segmentation of the retinal vessels for clinical diagnosis. With the rapid development of deep convolutional neural networks (DCNNs), to replace earlier manual labeling methods and reduce the labor cost, DCNN-based automatic segmentation methods have been greatly developed. U-Net and its variant models have obtained superior performance, but segmentation tasks are still challenging for the following reasons: First, features from encoders and decoders are not sufficiently fused to retain more effective information. Second, the limited receptive field will also affect contextual information extraction. In addition, although the continuous pooling operations can speed up the segmentation network training efficiency, they also lose detailed information during the downsampling process. To address the above issues and precisely segment the vessel structures from fundus images, a multiscale attention-guided fusion network, called MAGF-Net, is presented for automatic retinal vessel segmentation. To capture multiscale contextual features, a multiscale attention (MSA) block is proposed to construct the backbone network. Furthermore, a feature enhancement (FE) block is also proposed and embedded in the bottleneck layer to acquire global multiscale contextual information. To take full advantage of the channel information from deep layers and the spatial information from shallow layers, an attention-guided fusion (AGF) block is designed to fuse features from different network layers. Moreover, a hybrid feature pooling (HFP) block is employed to preserve more information during the downsampling operation. To evaluate the segmentation performance of the proposed MAGF-Net, extensive segmentation experiments are conducted on three public datasets: the CHASE_DB1 set, the DRIVE set and the STARE set. The experimental results show that the proposed MAGF-Net can obtain remarkable segmentation performance compared with other advanced methods. In particular, the ability of the proposed MAGF-Net to segment thin blood vessels is significantly improved.

A hybrid of transformer and CNN for efficient single image super-resolution via multi-level distillation

In recent years, single image super-resolution (SISR) models based on convolutional neural networks (CNN) have made significant progress and have gradually become the mainstream method. However, they still suffer from high computational costs, heavy memory consumption, and a limited receptive field. Although Vision Transformer has a stronger modeling capability and larger receptive field, it also incurs high computing power consumption and memory occupation. To address these issues, we propose a hybrid network of Transformer and CNN with cascaded feature distillation blocks for efficient image super-resolution (TCFDN), which can take advantage of both local information and long-term interactions while being flexible enough. Concretely, TCFDN consists of cascaded Transformer-CNN feature distillation blocks (TCFDB) and an upsampling module. The feature distillation pipeline of TCFDB can help our model gradually learn refined features with better representation ability while remaining lightweight. Besides, we also designed an enhanced Swin Transformer layer (ESTL) by replacing the multi-layer perceptron (MLP) in the standard Transformer with a convolutional feed-forward layer (CFF), which is more suitable for SR tasks. Then, the enhanced spatial attention embedded in TCFDB can boost SR performance further. Moreover, we observe that using a more advanced loss function, i.e., the contrastive loss, can also bring a PSNR gain of 0.01 dB–0.03 dB on public benchmarks. Extensive experiments demonstrate that TCFDN outperforms the state-of-the-art methods in terms of a better trade-off between performance and model size. Under the 4X SR task on the public benchmark Urban100, our TCFDN outperforms the second-best model by 0.37 dB in terms of PSNR. Compared with other state-of-the-art methods, the total number of parameters in TCFDN can be reduced by up to 32 % while maintaining competitive performance.

Wave-Net: A lightweight deep network for retinal vessel segmentation from fundus images

Accurate segmentation of retinal vessels from fundus images is fundamental for the diagnosis of numerous diseases of eye, and an automated vessel segmentation method can effectively help clinicians to make accurate diagnosis for the patients and provide the appropriate treatment schemes. It is important to note that both thick and thin vessels play the key role for disease judgements. Because of complex factors, the precise segmentation of thin vessels is still a great challenge, such as the presence of various lesions, image noise, complex backgrounds and poor contrast in the fundus images. Recently, because of the advantage of context feature representation learning capabilities, deep learning has shown a remarkable segmentation performance on retinal vessels. However, it still has some shortcomings on high-precision retinal vessel extraction due to some factors, such as semantic information loss caused by pooling operations, limited receptive field, etc. To address these problems, this paper proposes a new lightweight segmentation network for precise retinal vessel segmentation, which is called as Wave-Net model on account of the whole shape. To alleviate the influence of semantic information loss problem to thin vessels, to acquire more contexts about micro structures and details, a detail enhancement and denoising block (DED) is proposed to improve the segmentation precision on thin vessels, which replaces the simple skip connections of original U-Net. On the other hand, it could well alleviate the influence of the semantic gap problem. Further, faced with limited receptive field, for multi-scale vessel detection, a multi-scale feature fusion block (MFF) is proposed to fuse cross-scale contexts to achieve higher segmentation accuracy and realize effective processing of local feature maps. Experiments indicate that proposed Wave-Net achieves an excellent performance on retinal vessel segmentation while maintaining a lightweight network design compared to other advanced segmentation methods, and it also has shown a better segmentation ability to thin vessels.

DPBET: A dual-path lung nodules segmentation model based on boundary enhancement and hybrid transformer

Accurate segmentation of lung nodules is an important basis for the subsequent differentiation of benign and malignant pathological types, which is conducive to early detection of lung cancer. Due to the local feature extraction characteristics of convolution and the limited receptive field of continuous down-sampling, the existing deep convolutional neural networks (CNN) for lung nodules segmentation cause the loss of information of lesion boundaries and locations. To address this issue, a dual-path lung nodules segmentation model based on boundary enhancement and hybrid transformer (DPBET) is proposed in this paper. The model consists of a global path, an edge path, and a feature aggregation module. In the global path, a de-redundant transformer module with explicit guidance is proposed, called Cascade-Axial-Prune Transformer (CAP-Trans). It is combined with CNN to form a hybrid architecture to generate a global representation of the target lesion. In the edge path, an edge detection operator is introduced to construct a lung nodule edge enhancement dataset, which improves the dataset utilization while providing more prior knowledge of the target lesion boundar. In addition, the Down-Attention Sample (DASample) as a basic encoding block is designed to effectively perceive local features of different ranges and scales in the down-sampling process of lung nodule feature extraction. Finally, a feature aggregation module is designed to fuse the outputs of the two paths to get the final segmentation result. Our DPBET can delineate the boundaries of various types of pulmonary nodules, with an average DSC of 89.86% and an average Sensitivity of 90.50% on the public dataset LIDC-IDRI. Compared with the state-of-the-art approaches, a substantial improvement has been achieved. The experimental results demonstrate that DPBET can use edge enhancement to promote the global-edge consistency relationship, and the network architecture is effective in lung nodule segmentation.

MSEDTNet: Multi-Scale Encoder and Decoder with Transformer for Bladder Tumor Segmentation

The precise segmentation of bladder tumors from MRI is essential for bladder cancer diagnosis and personalized therapy selection. Limited by the properties of tumor morphology, achieving precise segmentation from MRI images remains challenging. In recent years, deep convolutional neural networks have provided a promising solution for bladder tumor segmentation from MRI. However, deep-learning-based methods still face two weakness: (1) multi-scale feature extraction and utilization are inadequate, being limited by the learning approach. (2) The establishment of explicit long-distance dependence is difficult due to the limited receptive field of convolution kernels. These limitations raise challenges in the learning of global semantic information, which is critical for bladder cancer segmentation. To tackle the problem, a newly auxiliary segmentation algorithm integrating a multi-scale encoder and decoder with a transformer is proposed, which is called MSEDTNet. Specifically, the designed encoder with multi-scale pyramidal convolution (MSPC) is utilized to generate compact feature maps which capture the richly detailed local features of the image. Furthermore, the transformer bottleneck is then leveraged to model the long-distance dependency between high-level tumor semantics from a global space. Finally, a decoder with a spatial context fusion module (SCFM) is adopted to fuse the context information and gradually produce high-resolution segmentation results. The experimental results of T2-weighted MRI scans from 86 patients show that MSEDTNet achieves an overall Jaccard index of 83.46%, a Dice similarity coefficient of 92.35%, and a complexity less than that of other, similar models. This suggests that the method proposed in this article can be used as an efficient tool for clinical bladder cancer segmentation.

Lightweight lane line detection based on learnable cluster segmentation with self‐attention mechanism

AbstractPixel segmentation is one of the most commonly used deep learning methods for modern lane line detection. Although deep segmentation outperforms traditional methods, there are two main problems: slow speed and limited receptive field. In response to these problems, this paper proposes a lightweight lane line detection algorithm based on learnable cluster segmentation and self‐attention mechanism, which has extremely fast speed and the ability to adapt to real scenes. The lane detection process is considered as clustering under row segmentation. The data is processed through row segmentation and fed into a self‐attention mechanism. In addition to the benchmark dataset for lane detection, the algorithm was ported to real vehicles for real‐time road testing. Two tests show that our method performs very well on TuSimple, with an accuracy of 97.15%, an F1 score of 73.5 on CULane, and a speed of 142.7 frames per second (FPS), which solves the problem of slow cluster segmentation, while improving the accuracy of row segmentation. In the new scenario, the method has a misjudgment rate of only 6.7% for lane line points, which is suitable for the high standard requirements of autonomous driving.

Deep Image Inpainting With Enhanced Normalization and Contextual Attention

Deep learning-based image inpainting has been widely studied, leading to great success. However, many methods adopt convolution and normalization operations, which will bring up some issues to affect the performance. The vanilla normalization cannot distinguish the pixels in corrupted regions from the other valid pixels, resulting in the mean and variance shifts. In addition, the limited receptive field of convolution makes it unable to capture long-range valid information directly. In order to tackle these challenges, we propose a novel deep generative model for image inpainting with two key modules, namely, the channel and spatially adaptive batch normalization (CSA-BN) module, and the selective latent-space-mapping-based contextual attention (SLSM-CA) layer. We replace the vanilla normalization with the CSA-BN module. By channel and spatially adaptive denormalization, the CSA-BN module can mitigate the spatial mean and variance shifts in each channel in a targeted way. In addition, we also integrate the SLSM-CA layer into our model to capture the long-range correlations explicitly. By introducing dual-branch attention and a feature selection module, the SLSM-CA layer can selectively utilize the multi-scale background information to improve prediction quality. What’s more, it introduces the latent spaces to achieve the low-rank approximations of attention matrices and to reduce computational costs. Extensive quantitative and qualitative evaluations demonstrate the superiority of the proposed method compared with state-of-the-art methods.

Progressive Rain Removal Based on the Combination Network of CNN and Transformer.

The rain removal method based on CNN develops rapidly. However, convolution operation has the disadvantages of limited receptive field and inadaptability to the input content. Recently, another neural network structure Transformer has shown excellent performance in natural language processing and advanced visual tasks by modeling global relationships, but Transformer has limitations in capturing local dependencies. To address the above limitations, we propose the combination network of CNN and Transformer, which fully combines the advantages of CNN and Transformer structure to complete the task of image restoration. We use CNN to provide preliminary output and adopt Transformer architecture to further optimize the output of CNN. In addition, by using some key designs in module connection, our model strengthens feature propagation and encourages feature reuse, allowing better information and gradient flow. The experimental results show that compared with the existing methods, our method can remove the rain lines more comprehensively and achieve the state-of-the-art results. Besides, the experimental results also demonstrate that the CNN structure can be effectively combined with Transformer to fully utilize the superiority of different structures.