Parallel Encoder Research Articles

Multi-scale dual-channel feature embedding decoder for biomedical image segmentation

Background and Objective:Attaining global context along with local dependencies is of paramount importance for achieving highly accurate segmentation of objects from image frames and is challenging while developing deep learning-based biomedical image segmentation. Several transformer-based models have been proposed to handle this issue in biomedical image segmentation. Despite this, segmentation accuracy remains an ongoing challenge, as these models often fall short of the target range due to their limited capacity to capture critical local and global contexts. However, the quadratic computational complexity is the main limitation of these models. Moreover, a large dataset is required to train those models. Methods:In this paper, we propose a novel multi-scale dual-channel decoder to mitigate this issue. The complete segmentation model uses two parallel encoders and a dual-channel decoder. The encoders are based on convolutional networks, which capture the features of the input images at multiple levels and scales. The decoder comprises a hierarchy of Attention-gated Swin Transformers with a fine-tuning strategy. The hierarchical Attention-gated Swin Transformers implements a multi-scale, multi-level feature embedding strategy that captures short and long-range dependencies and leverages the necessary features without increasing computational load. At the final stage of the decoder, a fine-tuning strategy is implemented that refines the features to keep the rich features and reduce the possibility of over-segmentation. Results:The proposed model is evaluated on publicly available LiTS, 3DIRCADb, and spleen datasets obtained from Medical Segmentation Decathlon. The model is also evaluated on a private dataset from Medical College Kolkata, India. We observe that the proposed model outperforms the state-of-the-art models in liver tumor and spleen segmentation in terms of evaluation metrics at a comparative computational cost. Conclusion:The novel dual-channel decoder embeds multi-scale features and creates a representation of both short and long-range contexts efficiently. It also refines the features at the final stage to select only necessary features. As a result, we achieve better segmentation performance than the state-of-the-art models.

An efficient XOR-free implementation of polar encoder for reconfigurable hardware

— This paper presents a novel approach to implementing an XOR-Free architecture of the non-systematic polar encoder (NSPE) for 5G radio. The optimization of XOR logic for hardware (HW) implementation is essential to reduce delay and power consumption. The proposed architecture for NSPE replaces XOR operations with combinational logical patterns, and some redundant patterns are removed with the help of bit manipulation to make it more efficient. The design infers multiplexers (2:1 or 4:1) and inverters as its functional units, making the design adequate and effective in alleviating HW complexity. The XOR-Free encoder performs the same functionality as the XOR-based conventional encoder. We have written a MATLAB script that generates Verilog hardware description language (HDL) code for fully or partially parallel polar encoders tailored to specific code lengths (N) and degrees of parallelism (M). A comparative analysis of various fully and partially parallel encoders with the XOR-Free algorithm is presented. The implementation results show that the proposed architectures are more efficient in terms of HW cost, power consumption, throughput, and latency.

EAAC-Net: An Efficient Adaptive Attention and Convolution Fusion Network for Skin Lesion Segmentation.

Accurate segmentation of skin lesions in dermoscopic images is of key importance for quantitative analysis of melanoma. Although existing medical image segmentation methods significantly improve skin lesion segmentation, they still have limitations in extracting local features with global information, do not handle challenging lesions well, and usually have a large number of parameters and high computational complexity. To address these issues, this paper proposes an efficient adaptive attention and convolutional fusion network for skin lesion segmentation (EAAC-Net). We designed two parallel encoders, where the efficient adaptive attention feature extraction module (EAAM) adaptively establishes global spatial dependence and global channel dependence by constructing the adjacency matrix of the directed graph and can adaptively filter out the least relevant tokens at the coarse-grained region level, thus reducing the computational complexity of the self-attention mechanism. The efficient multiscale attention-based convolution module (EMA⋅C) utilizes multiscale attention for cross-space learning of local features extracted from the convolutional layer to enhance the representation of richly detailed local features. In addition, we designed a reverse attention feature fusion module (RAFM) to enhance the effective boundary information gradually. To validate the performance of our proposed network, we compared it with other methods on ISIC 2016, ISIC 2018, and PH2 public datasets, and the experimental results show that EAAC-Net has superior segmentation performance under commonly used evaluation metrics.

INTREPPPID-an orthologue-informed quintuplet network for cross-species prediction of protein-protein interaction.

An overwhelming majority of protein-protein interaction (PPI) studies are conducted in a select few model organisms largely due to constraints in time and cost of the associated 'wet lab' experiments. In silico PPI inference methods are ideal tools to overcome these limitations, but often struggle with cross-species predictions. We present INTREPPPID, a method that incorporates orthology data using a new 'quintuplet' neural network, which is constructed with five parallel encoders with shared parameters. INTREPPPID incorporates both a PPI classification task and an orthologous locality task. The latter learns embeddings of orthologues that have small Euclidean distances between them and large distances between embeddings of all other proteins. INTREPPPID outperforms all other leading PPI inference methods tested on both the intraspecies and cross-species tasks using strict evaluation datasets. We show that INTREPPPID's orthologous locality loss increases performance because of the biological relevance of the orthologue data and not due to some other specious aspect of the architecture. Finally, we introduce PPI.bio and PPI Origami, a web server interface for INTREPPPID and a software tool for creating strict evaluation datasets, respectively. Together, these two initiatives aim to make both the use and development of PPI inference tools more accessible to the community.

SRTRP-Net: A multi-task learning network for segmentation and prediction of stereotactic radiosurgery treatment response in brain metastases

Before the Stereotactic Radiosurgery (SRS) treatment, it is of great clinical significance to avoid secondary genetic damage and guide the personalized treatment plans for patients with brain metastases (BM) by predicting the response to SRS treatment of brain metastatic lesions. Thus, we developed a multi-task learning model termed SRTRP-Net to provide prior knowledge of BM ROI and predict the SRS treatment response of the lesion. In dual-encoder tumor segmentation Network (DTS-Net), two parallel encoders encode the original and mirrored multi-modal MRI images. The differences in the dual-encoder features between foreground and background are enhanced by the symmetrical visual difference block (SVDB). In the bottom layer of the encoder, a transformer is used to extract local contextual features in the spatial and depth dimensions of low-resolution images. Then, the decoder of DTS-Net provides the prior knowledge for predicting the response to SRS treatment by performing BM segmentation. SRS response prediction network (SRP-Net) directly utilizes shared multi-modal MRI features weighted by the signed distance map (SDM) of the masks. The bidirectional multi-dimensional feature fusion module (BMDF) fuses the shared features and the clinical text information features to obtain comprehensive tumor information for characterizing tumors and predicting SRS treatment response. Experiments based on internal and external clinical datasets have shown that SRTRP-Net achieves comparable or better results. We believe that SRTRP-Net can help clinicians accurately develop personalized first-time treatment regimens for BM patients and improve their survival.

An LDPC-RS Concatenation and Decoding Scheme to Lower the Error Floor for FTN Signaling

Faster-than-Nyquist (FTN) signaling has attracted increasing interest in the past two decades. However, when the fifth-generation (5G) communication low-density parity check (LDPC) code is applied to FTN signaling with low Bahl–Cock–Jelinek–Raviv (BCJR) states of detection and few turbo equalization iterations, an error floor near 10−5 is found, which does not exist in the original LDPC used for orthogonal signaling. This can be eliminated through many detection and decoding iterations, but this is unacceptable considering the increase in latency and storage. To solve this problem, we propose an LDPC and Reed–Solomon (RS) concatenation code, shortening, and perturbation scheme to lower the error floor. We propose a parallel encoder architecture for RS component code and a concise algorithm to calculate its constant multiplier coefficients, leveraging a traditional serial encoder, which can also be used for other parallelisms, rates, and lengths. The simulation results show that the proposed concatenation and shortening scheme can lower the error floor to about 10−7. The proposed scheme has an error correction capability for coded FTN signaling and successfully lowers the error floor with the limitation of few turbo iterations and few BCJR states.

Moiré pattern generation-based image steganography

Generative steganography has developed recently due to its high resistance against steganalysis detection. It usually synthesizes stego images directly from secret information. In contrast to this manner, we propose a generative steganographic scheme by synthesizing Moiré patterns in this paper. Moiré patterns are widely observed in captured photos. They present various shapes, colors, and frequency ranges, which are expected to help carry secret information robustly and securely. The scheme synthesizes Moiré patterns via a Moiré pattern generator. Then a secret message is encoded in these patterns by a parallel encoder. The Moiré pattern generator employs a multi-branch structure to generate sub Moiré patterns at distinct frequencies. A discriminator is further imposed to ensure the realness of synthesized Moiré patterns. The parallel encoder spreads secret information over Moiré patterns in both frequency and spatial domain. A decoder and a steganalyzer are also included to ensure the robustness and undetectability of the information hiding. Furthermore, a two-stage training strategy is designed to accomplish both Moiré pattern generation and information hiding tasks. Experimental results demonstrate that the proposed scheme presents high undetectability against existing steganalytic tools, meanwhile, it can resist common image attacks such as AWGN and JPEG compression.

LumVertCancNet: A novel 3D lumbar vertebral body cancellous bone location and segmentation method based on hybrid Swin-transformer

Lumbar vertebral body cancellous bone location and segmentation is crucial in an automated lumbar spine processing pipeline. Accurate and reliable analysis of lumbar spine image is expected to advantage practical medical diagnosis and population-based analysis of bone strength. However, the design of automated algorithms for lumbar spine processing is demanding due to significant anatomical variations and scarcity of publicly available data. In recent years, convolutional neural network (CNN) and vision transformers (Vits) have been the de facto standard in medical image segmentation. Although adept at capturing global features, the inherent bias of locality and weight sharing of CNN constrains its capacity to model long-range dependency. In contrast, Vits excel at long-range dependency modeling, but they may not generalize well with limited datasets due to the lack of inductive biases inherent to CNN. In this paper, we propose a deep learning-based two-stage coarse-to-fine solution to address the problem of automatic location and segmentation of lumbar vertebral body cancellous bone. Specifically, in the first stage, a Swin-transformer based model is applied to predict the heatmap of lumbar vertebral body centroids. Considering the characteristic anatomical structure of lumbar spine, we propose a novel loss function called LumAnatomy loss, which enforces the order and bend of the predicted vertebral body centroids. To inherit the excellence of CNN and Vits while preventing their respective limitations, in the second stage, we propose an encoder–decoder network to segment the identified lumbar vertebral body cancellous bone, which consists of two parallel encoders, i.e., a Swin-transformer encoder and a CNN encoder. To enhance the combination of CNNs and Vits, we propose a novel multi-scale attention feature fusion module (MSA-FFM), which address issues that arise when fusing features given at different encoders. To tackle the issue of lack of data, we raise the first large-scale lumbar vertebral body cancellous bone segmentation dataset called LumVBCanSeg containing a total of 185 CT scans annotated at voxel level by 3 physicians. Extensive experimental results on the LumVBCanSeg dataset demonstrate the proposed algorithm outperform other state-of-the-art medical image segmentation methods. The data is publicly available at: https://zenodo.org/record/8181250. The implementation of the proposed method is available at: https://github.com/sia405yd/LumVertCancNet.

Aided Diagnosis Model Based on Deep Learning for Glioblastoma, Solitary Brain Metastases, and Primary Central Nervous System Lymphoma with Multi-Modal MRI.

(1) Background: Diagnosis of glioblastoma (GBM), solitary brain metastases (SBM), and primary central nervous system lymphoma (PCNSL) plays a decisive role in the development of personalized treatment plans. Constructing a deep learning classification network to diagnose GBM, SBM, and PCNSL with multi-modal MRI is important and necessary. (2) Subjects: GBM, SBM, and PCNSL were confirmed by histopathology with the multi-modal MRI examination (study from 1225 subjects, average age 53 years, 671 males), 3.0 T T2 fluid-attenuated inversion recovery (T2-Flair), and Contrast-enhanced T1-weighted imaging (CE-T1WI). (3) Methods: This paper introduces MFFC-Net, a classification model based on the fusion of multi-modal MRIs, for the classification of GBM, SBM, and PCNSL. The network architecture consists of parallel encoders using DenseBlocks to extract features from different modalities of MRI images. Subsequently, an L1-norm feature fusion module is applied to enhance the interrelationships among tumor tissues. Then, a spatial-channel self-attention weighting operation is performed after the feature fusion. Finally, the classification results are obtained using the full convolutional layer (FC) and Soft-max. (4) Results: The ACC of MFFC-Net based on feature fusion was 0.920, better than the radiomics model (ACC of 0.829). There was no significant difference in the ACC compared to the expert radiologist (0.920 vs. 0.924, p = 0.774). (5) Conclusions: Our MFFC-Net model could distinguish GBM, SBM, and PCNSL preoperatively based on multi-modal MRI, with a higher performance than the radiomics model and was comparable to radiologists.

A Building Extraction Method for High-Resolution Remote Sensing Images with Multiple Attentions and Parallel Encoders Combining Enhanced Spectral Information

Accurately extracting pixel-level buildings from high-resolution remote sensing images is significant for various geographical information applications. Influenced by different natural, cultural, and social development levels, buildings may vary in shape and distribution, making it difficult for the network to maintain a stable segmentation effect of buildings in different areas of the image. In addition, the complex spectra of features in remote sensing images can affect the extracted details of multi-scale buildings in different ways. To this end, this study selects parts of Xi’an City, Shaanxi Province, China, as the study area. A parallel encoded building extraction network (MARS-Net) incorporating multiple attention mechanisms is proposed. MARS-Net builds its parallel encoder through DCNN and transformer to take advantage of their extraction of local and global features. According to the different depth positions of the network, coordinate attention (CA) and convolutional block attention module (CBAM) are introduced to bridge the encoder and decoder to retain richer spatial and semantic information during the encoding process, and adding the dense atrous spatial pyramid pooling (DenseASPP) captures multi-scale contextual information during the upsampling of the layers of the decoder. In addition, a spectral information enhancement module (SIEM) is designed in this study. SIEM further enhances building segmentation by blending and enhancing multi-band building information with relationships between bands. The experimental results show that MARS-Net performs better extraction results and obtains more effective enhancement after adding SIEM. The IoU on the self-built Xi’an and WHU building datasets are 87.53% and 89.62%, respectively, while the respective F1 scores are 93.34% and 94.52%.

Prototype Learning Guided Hybrid Network for Breast Tumor Segmentation in DCE-MRI.

Automated breast tumor segmentation on the basis of dynamic contrast-enhancement magnetic resonance imaging (DCE-MRI) has shown great promise in clinical practice, particularly for identifying the presence of breast disease. However, accurate segmentation of breast tumor is a challenging task, often necessitating the development of complex networks. To strike an optimal tradeoff between computational costs and segmentation performance, we propose a hybrid network via the combination of convolution neural network (CNN) and transformer layers. Specifically, the hybrid network consists of a encoder-decoder architecture by stacking convolution and deconvolution layers. Effective 3D transformer layers are then implemented after the encoder subnetworks, to capture global dependencies between the bottleneck features. To improve the efficiency of hybrid network, two parallel encoder sub-networks are designed for the decoder and the transformer layers, respectively. To further enhance the discriminative capability of hybrid network, a prototype learning guided prediction module is proposed, where the category-specified prototypical features are calculated through online clustering. All learned prototypical features are finally combined with the features from decoder for tumor mask prediction. The experimental results on private and public DCE-MRI datasets demonstrate that the proposed hybrid network achieves superior performance than the state-of-the-art (SOTA) methods, while maintaining balance between segmentation accuracy and computation cost. Moreover, we demonstrate that automatically generated tumor masks can be effectively applied to identify HER2-positive subtype from HER2-negative subtype with the similar accuracy to the analysis based on manual tumor segmentation. The source code is available at https://github.com/ZhouL-lab/ PLHN.

An efficient reconfigurable code rate cooperative low-density parity check codes for gigabits wide code encoder/decoder operations

In recent days, extensive digital communication process has been performed. Due to this phenomenon, a proper maintenance of authentication, communication without any overhead such as signal attenuation code rate fluctuations during digital communication process can be minimized and optimized by adopting parallel encoder and decoder operations. To overcome the above-mentioned drawbacks by using proposed reconfigurable code rate cooperative (RCRC) and low-density parity check (LDPC) method. The proposed RCRC-LDPC is capable to operate over gigabits/sec data and it effectively performs linear encoding, dual diagonal form, widens the range of code rate and optimal degree distribution of LDPC mother code. The proposed method optimize the transmission rate and it is capable to operate on 0.98 code rate. It is the highest upper bounded code rate as compared to the existing methods. The proposed method optimizes the transmission rate and is capable to operate on a 0.98 code rate. It is the highest upper bounded code rate as compared to the existing methods. the proposed method's implementation has been carried out using MATLAB and as per the simulation result, the proposed method is capable of reaching a throughput efficiency greater than 8.2 (1.9) gigabits per second with a clock frequency of 160 MHz.

A RGB-Thermal based adaptive modality learning network for day–night wildfire identification

Wildfires have long been a danger to the atmosphere and ecological environment. With the advancement of deep learning technology and sensor equipment, wildfire identification methods based on optical (RGB) or thermal infrared (TIR) images have made significant progress. However, single-modal identification technologies often experience serious performance degradation in challenging scenarios. Therefore, a multi-modal framework that combines both RGB and TIR is necessary for accurate wildfire identification. Despite this, current RGB-T based identification methods focus solely on developing fusion strategies to learn shared representations of the two modalities, while ignoring the effect of modality-specific features. To address this issue, we propose a novel wildfire identification framework that can adaptively learn modality-specific and shared features. Specifically, our model utilizes two parallel encoders to extract multi-scale RGB and TIR features, and integrates these modality-specific features into the fusion feature layer. Each modality-specific decoder and shared decoder corresponds to three independent label supervisions that oversee feature adaptive learning. To validate the effectiveness of our approach, we propose a new RGB-T paired wildfire semantic segmentation dataset that includes flames of different scales captured under varying lighting conditions from multiple UAV-based and ground-based cameras. Finally, we evaluate our proposed method alongside benchmarking single-modal methods and state-of-the-art multi-modal methods on two tasks: a common test setting under various lighting conditions, and a generalization test across three subsets. The experimental results demonstrate the superiority of our proposed method, with the average ＋6.41% (IoU) and ＋3.39% (F1-score) improvement over the second-best RGB-T semantic segmentation method.

LMQFormer: A Laplace-Prior-Guided Mask Query Transformer for Lightweight Snow Removal

Snow removal aims to locate snow areas and recover clean images without repairing traces. Unlike the regularity and semitransparency of rain, snow with various patterns and degradations seriously occludes the background. As a result, the state-of-the-art snow removal methods usually retains a large parameter size. In this paper, we propose a lightweight but high-efficient snow removal network called Laplace Mask Query Transformer (LMQFormer). Firstly, we present a Laplace-VQVAE to generate a coarse mask as prior knowledge of snow. Instead of using the mask in dataset, we aim at reducing both the information entropy of snow and the computational cost of recovery. Secondly, we design a Mask Query Transformer (MQFormer) to remove snow with the coarse mask, where we use two parallel encoders and a hybrid decoder to learn extensive snow features under lightweight requirements. Thirdly, we develop a Duplicated Mask Query Attention (DMQA) that converts the coarse mask into a specific number of queries, which constraint the attention areas of MQFormer with reduced parameters. Experimental results in popular datasets have demonstrated the efficiency of our proposed model, which achieves the state-of-the-art snow removal quality with significantly reduced parameters and the lowest running time. Codes and models are available at https://github.com/StephenLinn/LMQFormer.

Parallel encoder–decoder framework for image captioning

Recent progress in deep learning has led to successful utilization of encoder–decoder frameworks inspired by machine translation in image captioning models. The stacking of layers in encoders and decoders has made it possible to use several modules in encoders and decoders. However, just one type of module in encoder or decoder has been used in stacked models. In this research, we propose a parallel encoder–decoder framework that aims to take advantage of multiple of types modules in encoders and decoders, simultaneously. This framework contains augmented parallel blocks, which include stacking modules or non-stacked ones. Then, the results of the blocks are integrated to extract higher-level semantic concepts. This general idea is not limited to image captioning and can be customized for many applications that utilize encoder–decoder frameworks. We evaluated our proposed method on the MS-COCO dataset and achieved state-of-the-art results. We got 149.92 for CIDEr-D metric outperforming state-of-the-art image captioning models.

A new architecture combining convolutional and transformer-based networks for automatic 3D multi-organ segmentation on CT images.

Deep learning-based networks have become increasingly popular in the field of medical image segmentation. The purpose of this research was to develop and optimize a new architecture for automatic segmentation of the prostate gland and normal organs in the pelvic, thoracic, and upper gastro-intestinal (GI) regions. We developed an architecture which combines a shifted-window (Swin) transformer with a convolutional U-Net. The network includes a parallel encoder, a cross-fusion block, and a CNN-based decoder to extract local and global information and merge related features on the same scale. A skip connection is applied between the cross-fusion block and decoder to integrate low-level semantic features. Attention gates (AGs) are integrated within the CNN to suppress features in image background regions. Our network is termed "SwinAttUNet." We optimized the architecture for automatic image segmentation. Training datasets consisted of planning-CT datasets from 300 prostate cancer patients from an institutional database and 100 CT datasets from a publicly available dataset (CT-ORG). Images were linearly interpolated and resampled to a spatial resolution of (1.0×1.0×1.5) mm3 . A volume patch (192×192×96) was used for training and inference, and the dataset was split into training (75%), validation (10%), and test (15%) cohorts. Data augmentation transforms were applied consisting of random flip, rotation, and intensity scaling. The loss function comprised Dice and cross-entropy equally weighted and summed. We evaluated Dice coefficients (DSC), 95th percentile Hausdorff Distances (HD95), and Average Surface Distances (ASD) between results of our network and ground truth data. SwinAttUNet, DSC values were 86.54±1.21, 94.15±1.17, and 87.15±1.68% and HD95 values were 5.06±1.42, 3.16±0.93, and 5.54±1.63mm for the prostate, bladder, and rectum, respectively. Respective ASD values were 1.45±0.57, 0.82±0.12, and 1.42±0.38mm. For the lung, liver, kidneys and pelvic bones, respective DSC values were: 97.90±0.80, 96.16±0.76, 93.74±2.25, and 89.31±3.87%. Respective HD95 values were: 5.13±4.11, 2.73±1.19, 2.29±1.47, and 5.31±1.25mm. Respective ASD values were: 1.88±1.45, 1.78±1.21, 0.71±0.43, and 1.21±1.11mm. Our network outperformed several existing deep learning approaches using only attention-based convolutional or Transformer-based feature strategies, as detailed in the results section. We have demonstrated that our new architecture combining Transformer- and convolution-based features is able to better learn the local and global context for automatic segmentation of multi-organ, CT-based anatomy.

DepthFormer: Exploiting Long-range Correlation and Local Information for Accurate Monocular Depth Estimation

This paper aims to address the problem of supervised monocular depth estimation. We start with a meticulous pilot study to demonstrate that the long-range correlation is essential for accurate depth estimation. Moreover, the Transformer and convolution are good at long-range and close-range depth estimation, respectively. Therefore, we propose to adopt a parallel encoder architecture consisting of a Transformer branch and a convolution branch. The former can model global context with the effective attention mechanism and the latter aims to preserve the local information as the Transformer lacks the spatial inductive bias in modeling such contents. However, independent branches lead to a shortage of connections between features. To bridge this gap, we design a hierarchical aggregation and heterogeneous interaction module to enhance the Transformer features and model the affinity between the heterogeneous features in a set-to-set translation manner. Due to the unbearable memory cost introduced by the global attention on high-resolution feature maps, we adopt the deformable scheme to reduce the complexity. Extensive experiments on the KITTI, NYU, and SUN RGB-D datasets demonstrate that our proposed model, termed DepthFormer, surpasses state-of-the-art monocular depth estimation methods with prominent margins. The effectiveness of each proposed module is elaborately evaluated through meticulous and intensive ablation studies.

End-to-End Dual-Stream Transformer with a Parallel Encoder for Video Captioning

In this paper, we propose an end-to-end dual-stream transformer with a parallel encoder (DST-PE) for video captioning, which combines multimodal features and global–local representations to generate coherent captions. First, we design a parallel encoder that includes a local visual encoder and a bridge module, which simultaneously generates refined local and global visual features. Second, we devise a multimodal encoder to enhance the representation ability of our model. Finally, we adopt a transformer decoder with multimodal features as inputs and local visual features fused with textual features using a cross-attention block. Extensive experimental results demonstrate that our model achieves state-of-the-art performance with low training costs on several widely used datasets.

Multi parallel U-net encoder network for effective polyp image segmentation

Automating polyps' detection and segmentation is very helpful in achieving early detection and potentially effective treatment of colon cancer. Because of the constant rise in processing power at a reasonable cost and the availability of training data, deep learning more than ever is becoming very powerful and increasingly widespread in many domains, including computer vision and medical image processing and segmentation. In this paper, we propose a novel method for polyp image segmentation using convolutional neural networks (CNNs) and utilizing multi-parallel U-Net encoder architecture. The proposed method uses multiple encoders that utilize pre-trained CNNs to enrich the extracted features. Skip connections from these parallel encoders are concatenated and propagated to the decoder in partially and fully meshed fashions allowing for data from shallow to deeper layers in the network to be exchanged and vice versa. We trained and tested the method on five publicly available datasets: Kvasir, CVC-Clinic DB, CVC-Colon DB, CVC-T, and ETIS-Larib, which are well-known benchmark datasets for polyp image segmentation. We tested the performance of the method on these five datasets using multiple testing scenarios including the effect of using attention, vision transformers, and multiple decoders. Experimental results showed that the proposed method outperforms other state-of-the-art methods on two of the used benchmark datasets and ranked second on a third.

Aggregating transformers and CNNs for salient object detection in optical remote sensing images

Salient object detection (SOD) in optical remote sensing images (RSIs) plays a significant role in many areas such as agriculture, environmental protection, and the military. However, since the difference in imaging mode and image complexity between RSIs and natural scene images (NSIs), it is difficult to achieve remarkable results by directly extending the saliency method targeting NSIs to RSIs. Besides, we note that the convolutional neural networks (CNNs) based U-Net cannot effectively acquire the global long-range dependency, and the Transformer doesn’t adequately characterize the spatial local details of each patch. Therefore, to conduct salient object detection in RSIs, we propose a novel two-branch architecture based network for Aggregating the Transformers and CNNs, namely ATC-Net, where the local spatial details and the global semantic information are fused into the final high-quality saliency map. Specifically, our saliency model adopts an encoder-decoder architecture including two parallel encoder branches and a decoder. Firstly, the two parallel encoder branches extract global and local features by using Transformer and CNNs, respectively. Then, the decoder employs a series of feature-enhanced fusion (FF) modules to aggregate multi-level global and local features by interactive guidance and enhance the fused feature via attention mechanism. Finally, the decoder deploys the read out (RO) module to fuse the aggregated feature of FF module and the low-level CNN feature, steering the feature to focus more on spatial local details. Extensive experiments are performed on two public optical RSIs datasets, and the results show that our saliency model consistently outperforms 30 state-of-the-art methods.