Global Semantic Information Research Articles

Few-shot classification with multisemantic information fusion network

Metric-based methods aim to predict class labels by computing the similarity between samples using distance functions, which is the mainstream approach to few-shot learning. However, the limited representational space of feature vectors and appearance variations among congenetic samples still present challenges. We propose a Multisemantic Information Fusion Network (MIFN) to address these problems. A Lower-level Feature generator (LF-generator), which is an unsupervised module, adaptively activates high-response regions of objects to introduce discriminative semantic details. Meanwhile, a Higher-level Feature extractor (HF-extractor) learns global semantic information with human cognition to minimise the impact of appearance variations. We integrate the coarse outputs of these two modules, which complement each other to jointly promote more precise predictions. Furthermore, considering the importance of prototypes, we redefine the sampling strategy of the triplet loss and utilise it as an auxiliary loss to sharpen the decision boundary at the prototype level, facilitating subsequent classification. Our experimental results demonstrate the competitiveness of our approach in both general few-shot classification (mini-ImageNet and tiered-ImageNet) and cross-domain problems (CUB, Caltech-101, Stanford-dogs, and Stanford-cars) with minimal bells and whistles.

MADANet: A Lightweight Hyperspectral Image Classification Network with Multiscale Feature Aggregation and a Dual Attention Mechanism

Hyperspectral remote sensing images, with their continuous, narrow, and rich spectra, hold distinct significance in the precise classification of land cover. Deep convolutional neural networks (CNNs) and their variants are increasingly utilized for hyperspectral classification, but solving the conflict between the number of model parameters, performance, and accuracy has become a pressing challenge. To alleviate this problem, we propose MADANet, a lightweight hyperspectral image classification network that combines multiscale feature aggregation and a dual attention mechanism. By employing depthwise separable convolution, multiscale features can be extracted and aggregated to capture local contextual information effectively. Simultaneously, the dual attention mechanism harnesses both channel and spatial dimensions to acquire comprehensive global semantic information. Ultimately, techniques such as global average pooling (GAP) and full connection (FC) are employed to integrate local contextual information with global semantic knowledge, thereby enabling the accurate classification of hyperspectral pixels. The results from the experiments conducted on representative hyperspectral images demonstrate that MADANet not only attains the highest classification accuracy but also maintains significantly fewer parameters compared to the other methods. Experimental results show that our proposed framework significantly reduces the number of model parameters while still achieving the highest classification accuracy. As an example, the model has only 0.16 M model parameters in the Indian Pines (IP) dataset, but the overall accuracy is as high as 98.34%. Similarly, the framework achieves an overall accuracy of 99.13%, 99.17%, and 99.08% on the University of Pavia (PU), Salinas (SA), and WHU Hi LongKou (LongKou) datasets, respectively. This result exceeds the classification accuracy of existing state-of-the-art frameworks under the same conditions.

Learning entity-oriented representation for biomedical relation extraction

Biomedical Relation Extraction (BioRE) aims to automatically extract semantic relations for given entity pairs and is of great significance in biomedical research. Current popular methods often utilize pretrained language models to extract semantic features from individual input instances, which frequently suffer from overlapping semantics. Overlapping semantics refers to the situation in which a sentence contains multiple entity pairs that share the same context, leading to highly similar information between these entity pairs. In this study, we propose a model for learning Entity-oriented Representation (EoR) that aims to improve the performance of the model by enhancing the discriminability between entity pairs. It contains three modules: sentence representation, entity-oriented representation, and output. The first module learns the global semantic information of the input instance; the second module focuses on extracting the semantic information of the sentence from the target entities; and the third module enhances distinguishability among entity pairs and classifies the relation type. We evaluated our approach on four BioRE tasks with eight datasets, and the experiments showed that our EoR achieved state-of-the-art performance for PPI, DDI, CPI, and DPI tasks. Further analysis demonstrated the benefits of entity-oriented semantic information in handling multiple entity pairs in the BioRE task.

Deep global semantic structure-preserving hashing via corrective triplet loss for remote sensing image retrieval

With the explosive increase of remote sensing data, how to search for remote sensing data quickly and accurately in a vast dataset is an incredibly critical matter for research subjects. The deep hashing method has become the dominant method for remote sensing image retrieval because of its low-cost storage and high-speed retrieval. However, for the reason of the limitation of fixed convolutional kernels, deep hashing frameworks based on Convolutional Neural Networks (CNNs) fail to obtain the global semantic information well, which leads to the generation of suboptimal solutions. Furthermore, existing hashing methods commonly employ the random sampling strategy or hardest sample mining to build training batches, resulting in bad local minima. To remedy these problems, a novel Deep Global Semantic Structure-preserving Hashing framework via corrective triplet loss (DGSSH) is proposed for remote sensing image retrieval to learn a discriminative and stable embedding space, achieving intra-class confusion and inter-class diversity. Specifically speaking, the feature extraction module based on Swim Transformer architecture is developed to capture global semantic information and multiscale features from remote sensing images. Based on a distribution matching constraint, the corrective triplet loss for deep hashing schemes is designed to reduce the distribution shift caused by the random selection or hardest sample mining. Meanwhile, to reduce the time overhead of the model, the asymmetric learning strategy is employed to perform effective compact representation learning. Numerous experiments have been carried out on three publicly available benchmarks, which indicates that the proposed DGSSH framework could achieve optimal performance for remote sensing image retrieval applications. The source code of our DGSSH framework is hosted at https://github.com/QinLab-WFU/DGSSH.git.

3D multi-modality Transformer-GAN for high-quality PET reconstruction

Positron emission tomography (PET) scans can reveal abnormal metabolic activities of cells and provide favorable information for clinical patient diagnosis. Generally, standard-dose PET (SPET) images contain more diagnostic information than low-dose PET (LPET) images but higher-dose scans can also bring higher potential radiation risks. To reduce the radiation risk while acquiring high-quality PET images, in this paper, we propose a 3D multi-modality edge-aware Transformer-GAN for high-quality SPET reconstruction using the corresponding LPET images and T1 acquisitions from magnetic resonance imaging (T1-MRI). Specifically, to fully excavate the metabolic distributions in LPET and anatomical structural information in T1-MRI, we first use two separate CNN-based encoders to extract local spatial features from the two modalities, respectively, and design a multimodal feature integration module to effectively integrate the two kinds of features given the diverse contributions of features at different locations. Then, as CNNs can describe local spatial information well but have difficulty in modeling long-range dependencies in images, we further apply a Transformer-based encoder to extract global semantic information in the input images and use a CNN decoder to transform the encoded features into SPET images. Finally, a patch-based discriminator is applied to ensure the similarity of patch-wise data distribution between the reconstructed and real images. Considering the importance of edge information in anatomical structures for clinical disease diagnosis, besides voxel-level estimation error and adversarial loss, we also introduce an edge-aware loss to retain more edge detail information in the reconstructed SPET images. Experiments on the phantom dataset and clinical dataset validate that our proposed method can effectively reconstruct high-quality SPET images and outperform current state-of-the-art methods in terms of qualitative and quantitative metrics.

Multi‐level cross‐modality learning framework for text‐based person re‐identification

AbstractThe target of text‐based person re‐identification (Re‐ID) is to retrieve the corresponding image of a person through the given text information. However, due to the homogeneous variety and modality heterogeneity, it is challenging to simultaneously learn both global‐level and local‐level cross‐modal features and align them in the same embedding space without additional networks. To address these problems, an effective multi‐level cross‐modality learning (MCL) framework for language and vision person Re‐ID is proposed. More specifically, a multi‐branch feature extraction (MFE) module is designed to comprehensively map both global and partial semantic information for the visual and textual embedding at the same time, capturing the intra‐class semantic relationships in multi‐granularities. Besides, a cross‐modal alignment (CA) module is devised to match the multi‐grained representations and reduce the inter‐class gap from global‐level to partial‐level. Extensive experiments conducted on the CUHK‐PEDES and ICFG‐PEDES datasets suggest that this method outperforms the state‐of‐the‐art models.

Joint Global and Dynamic Pseudo Labeling for Semi-Supervised Point Cloud Sequence Segmentation

Supervised learning is a mainstay for large discriminative models in 3D computer vision, while large amounts of human-annotated data are the key to achieve state-of-the-art performance. This limitation is particularly notable for large-scale point cloud sequence segmentation tasks, because point-level annotations are very time-consuming and especially expensive. To overcome this challenge, we develop a novel semi-supervised framework for point cloud sequences segmentation. Specifically, we develop two kinds of pseudo labeling methods with extracting global semantic information from labeled frames and dynamic information from each sequence respectively. Then the two kinds of generated labels are combined as more robust pseudo labels (GD-Pseudo labels) for unlabeled frames. We finally apply an efficient iterative learning scheme to train a model with a small quantity of human-annotated data and large-scale pseudo-labeled data. Equipped with our framework, the model achieves significant performance improvement (+12—25 mIoU) on SemanticKITTI and Synthia when compared with frameworks that do not utilize large amounts of unlabeled data. Moreover, our method achieves comparable performance with only 20% annotated frames on SemanticKITTI to state-of-the-art models trained with 100% human-annotated frames.

Multi-feature fusion learning for Alzheimer's disease prediction using EEG signals in resting state.

Diagnosing Alzheimer's disease (AD) lesions via visual examination of Electroencephalography (EEG) signals poses a considerable challenge. This has prompted the exploration of deep learning techniques, such as Convolutional Neural Networks (CNNs) and Visual Transformers (ViTs), for AD prediction. However, the classification performance of CNN-based methods has often been deemed inadequate. This is primarily attributed to CNNs struggling with extracting meaningful lesion signals from the complex and noisy EEG data. In contrast, ViTs have demonstrated proficiency in capturing global signal patterns. In light of these observations, we propose a novel approach to enhance AD risk assessment. Our proposition involves a hybrid architecture, merging the strengths of CNNs and ViTs to compensate for their respective feature extraction limitations. Our proposed Dual-Branch Feature Fusion Network (DBN) leverages both CNN and ViT components to acquire texture features and global semantic information from EEG signals. These elements are pivotal in capturing dynamic electrical signal changes in the cerebral cortex. Additionally, we introduce Spatial Attention (SA) and Channel Attention (CA) blocks within the network architecture. These attention mechanisms bolster the model's capacity to discern abnormal EEG signal patterns from the amalgamated features. To make well-informed predictions, we employ a two-factor decision-making mechanism. Specifically, we conduct correlation analysis on predicted EEG signals from the same subject to establish consistency. This is then combined with results from the Clinical Neuropsychological Scale (MMSE) assessment to comprehensively evaluate the subject's susceptibility to AD. Our experimental validation on the publicly available OpenNeuro database underscores the efficacy of our approach. Notably, our proposed method attains an impressive 80.23% classification accuracy in distinguishing between AD, Frontotemporal dementia (FTD), and Normal Control (NC) subjects. This outcome outperforms prevailing state-of-the-art methodologies in EEG-based AD prediction. Furthermore, our methodology enables the visualization of salient regions within pathological images, providing invaluable insights for interpreting and analyzing AD predictions.

ConSpaS: a contrastive learning framework for identifying spatial domains by integrating local and global similarities.

Spatial transcriptomics is a rapidly growing field that aims to comprehensively characterize tissue organization and architecture at single-cell or sub-cellular resolution using spatial information. Such techniques provide a solid foundation for the mechanistic understanding of many biological processes in both health and disease that cannot be obtained using traditional technologies. Several methods have been proposed to decipher the spatial context of spots in tissue using spatial information. However, when spatial information and gene expression profiles are integrated, most methods only consider the local similarity of spatial information. As they do not consider the global semantic structure, spatial domain identification methods encounter poor or over-smoothed clusters. We developed ConSpaS, a novel node representation learning framework that precisely deciphers spatial domains by integrating local and global similarities based on graph autoencoder (GAE) and contrastive learning (CL). The GAE effectively integrates spatial information using local similarity and gene expression profiles, thereby ensuring that cluster assignment is spatially continuous. To improve the characterization of the global similarity of gene expression data, we adopt CL to consider the global semantic information. We propose an augmentation-free mechanism to construct global positive samples and use a semi-easy sampling strategy to define negative samples. We validated ConSpaS on multiple tissue types and technology platforms by comparing it with existing typical methods. The experimental results confirmed that ConSpaS effectively improved the identification accuracy of spatial domains with biologically meaningful spatial patterns, and denoised gene expression data while maintaining the spatial expression pattern. Furthermore, our proposed method better depicted the spatial trajectory by integrating local and global similarities.

MIFINN: A novel multi-information fusion and interaction neural network for aspect-based sentiment analysis

Aspect-based Sentiment Analysis (ABSA) is used to detect corresponding sentiment polarities toward different aspect terms. In recent years, graph convolutional network-based methods achieve great success for ABSA. However, previous studies ignore syntactic dependency types and aspect-related syntactic distances. Besides, effectively fusing semantic and syntactic information remains a challenging problem. Previous studies also neglect interactions between aspect terms and context. To alleviate the above problems, we develop a Multi-Information Fusion and Interaction Neural Network (MIFINN), which consists of an Aspect and Syntax-Aware Multi-Information Fusion Graph Convolutional Network (ASAMIFGCN) and a Multi-Information Interaction and Gating Network (MIIGN). The ASAMIFGCN model can integrate and fuse contextual semantics, syntactic dependency types, and aspect-related syntactic distances. Specifically, two different attentions are designed to learn global and aspect-related semantic and syntactic information, respectively. A syntactic distance gating module is proposed to acquire more precise score matrices. Besides, we propose a semantic and syntactic association module to fuse multi-information simultaneously. The MIIGN model can effectively interact aspect terms and context. Specifically, two interactive attentions are developed to learn different weights toward aspect terms and context via interactions between each other, respectively. Moreover, we propose a gating information module for controlling the information flow of aspect terms and context. We conduct experiments on multiple datasets and MIFINN achieves state-of-the-art performance. The results also demonstrate that effectively fusing and interacting multi-information can improve model performance for ABSA significantly.

A lightweight super-resolution network with skip-connections.

In some hospitals in remote areas, due to the lack of MRI scanners with high magnetic field intensity, only low-resolution MRI images can be obtained, hindering doctors from making correct diagnoses. In our study, higher-resolution images were obtained through low-resolution MRI images. Moreover, as our algorithm is a lightweight algorithm with a small number of parameters, it can be carried out in remote areas under the condition of the lack of computing resources. Moreover, our algorithm is of great clinical significance in providing references for doctors' diagnoses and treatment in remote areas. We compared different super-resolution algorithms to obtain high-resolution MRI images, including SRGAN, SPSR, and LESRCNN. A global skip connection was applied to the original network of LESRCNN to use global semantic information to get better performance. Experiments reported that our network improved SSMI by 0.8% and also achieved an obvious increase in PSNR, PI, and LPIPS compared to LESRCNN in our dataset. Similar to LESRCNN, our network has a very short running time, a small number of parameters, low time complexity, and low space complexity while ensuring high performance compared to SRGAN and SPSR. Five MRI doctors were invited for a subjective evaluation of our algorithm. All agreed on significant improvements and that our algorithm could be used clinically in remote areas and has great value. The experimental results demonstrated the performance of our algorithm in super-resolution MRI image reconstruction. It allows us to obtain high-resolution images in the absence of high-field intensity MRI scanners, which have great clinical significance. The short running time, a small number of parameters, low time complexity, and low space complexity ensure that our network can be used in grassroots hospitals in remote areas that lack computing resources. We can reconstruct high-resolution MRI images in a short time, thus saving time for patients. Our algorithm can be biased towards practical applications; however, doctors have affirmed the clinical value of our algorithm.

SiamC Transformer: Siamese coupling swin transformer Multi-Scale semantic segmentation network for vegetation extraction under shadow conditions

The accuracy of fractional vegetation cover (FVC) estimation has important significance in high-precision agricultural and ecological environment assessments. However, the presence ofshadowsunder natural light conditions can cause problems such as poor vegetation distinguishability and blurred boundaries during vegetation segmentation. This leads to missing or over-segmented vegetation inshadedareas, thus reducing the accuracy of FVC. Polarizationinformation can reflect the texture structure features, edgecharacteristics and surface state information, providing important supplementary information to light intensity information. To address the issue of shadows interfering with vegetation segmentation, a Siamese coupling swin transformer (SiamC Transformer) is proposed in this study. In this network, a dual-flow structure is used to simultaneously retrieve the feature information of vegetation degree of linear polarization (DoLP) images and light intensity images simultaneously to obtain multi-dimensional global semantic information of multi-scale vegetation. In the feature fusion stage, two fusion modules are proposed: an adaptive fusion module (AFM) and an adaptive fusion plus module (AFPM), which can fuse shallow spatial information, texture information and deep semantic information. The AFM and AFPM enable the network to better achieve object localization, region activation and edge sharpening, while improving the segmentation accuracy of small objects. The experimental results show that the mean intersection over union (mIoU) of this network is as high as 97.46% on vegetation datasets consisting of light intensity and polarization images, which is better than that of other algorithms. This network has higher accuracy and adaptability for shadow scenes and improves the accuracy of FVC calculation under shadow conditions.

TCNet: A Transformer–CNN Hybrid Network for Marine Aquaculture Mapping from VHSR Images

Precise delineation of marine aquaculture areas is vital for the monitoring and protection of marine resources. However, due to the coexistence of diverse marine aquaculture areas and complex marine environments, it is still difficult to accurately delineate mariculture areas from very high spatial resolution (VHSR) images. To solve such a problem, we built a novel Transformer–CNN hybrid Network, named TCNet, which combined the advantages of CNN for modeling local features and Transformer for capturing long-range dependencies. Specifically, the proposed TCNet first employed a CNN-based encoder to extract high-dimensional feature maps from input images. Then, a hierarchical lightweight Transformer module was proposed to extract the global semantic information. Finally, it employed a coarser-to-finer strategy to progressively recover and refine the classification results. The results demonstrate the effectiveness of TCNet in accurately delineating different types of mariculture areas, with an IoU value of 90.9%. Compared with other state-of-the-art CNN or Transformer-based methods, TCNet showed significant improvement both visually and quantitatively. Our methods make a significant contribution to the development of precision agricultural in coastal regions.

MISSU: 3D Medical Image Segmentation via Self-Distilling TransUNet.

U-Nets have achieved tremendous success in medical image segmentation. Nevertheless, it may have limitations in global (long-range) contextual interactions and edge-detail preservation. In contrast, the Transformer module has an excellent ability to capture long-range dependencies by leveraging the self-attention mechanism into the encoder. Although the Transformer module was born to model the long-range dependency on the extracted feature maps, it still suffers high computational and spatial complexities in processing high-resolution 3D feature maps. This motivates us to design an efficient Transformer-based UNet model and study the feasibility of Transformer-based network architectures for medical image segmentation tasks. To this end, we propose to self-distill a Transformer-based UNet for medical image segmentation, which simultaneously learns global semantic information and local spatial-detailed features. Meanwhile, a local multi-scale fusion block is first proposed to refine fine-grained details from the skipped connections in the encoder by the main CNN stem through self-distillation, only computed during training and removed at inference with minimal overhead. Extensive experiments on BraTS 2019 and CHAOS datasets show that our MISSU achieves the best performance over previous state-of-the-art methods. Code and models are available at: https://github.com/wangn123/MISSU.git.

Water index and Swin Transformer Ensemble (WISTE) for water body extraction from multispectral remote sensing images

ABSTRACT Automatic surface water body mapping using remote sensing technology is greatly meaningful for studying inland water dynamics at regional to global scales. Convolutional neural networks (CNN) have become an efficient semantic segmentation technique for the interpretation of remote sensing images. However, the receptive field value of a CNN is restricted by the convolutional kernel size because the network only focuses on local features. The Swin Transformer has recently demonstrated its outstanding performance in computer vision tasks, and it could be useful for processing multispectral remote sensing images. In this article, a Water Index and Swin Transformer Ensemble (WISTE) method for automatic water body extraction is proposed. First, a dual-branch encoder architecture is designed for the Swin Transformer, aggregating the global semantic information and pixel neighbor relationships captured by fully convolutional networks (FCN) and multihead self-attention. Second, to prevent the Swin Transformer from ignoring multispectral information, we construct a prediction map ensemble module. The predictions of the Swin Transformer and the Normalized Difference Water Index (NDWI) are combined by a Bayesian averaging strategy. Finally, the experimental results obtained on two distinct datasets demonstrate that the WISTE has advantages over other segmentation methods and achieves the best results. The method proposed in this study can be used for improving regional to continental surface water mapping and related hydrological studies.

Observing Individuals and Behavior of Hainan Gibbons (Nomascus hainanus) Using Drone Infrared and Visible Image Fusion Technology

The Hainan gibbon (Nomascus hainanus) is one of the most endangered primates in the world. Infrared and visible images taken by drones are an important and effective way to observe Hainan gibbons. However, a single infrared or visible image cannot simultaneously observe the movement tracks of Hainan gibbons and the appearance of the rainforest. The fusion of infrared and visible images of the same scene aims to generate a composite image which can provide a more comprehensive description of the scene. We propose a fusion method of infrared and visible images of the Hainan gibbon for the first time, termed Swin-UetFuse. The Swin-UetFuse has a powerful global and long-range semantic information extraction capability, which is very suitable for application in complex tropical rainforest environments. Firstly, the hierarchical Swin Transformer is applied as the encoder to extract the features of different scales of infrared and visible images. Secondly, the features of different scales are fused through the l1-norm strategy. Finally, the Swing Transformer blocks and patch-expanding layers are utilized as the decoder to up-sample the fusion features to obtain the fused image. We used 21 pairs of Hainan gibbon datasets to perform experiments, and the experimental results demonstrate that the proposed method achieves excellent fusion performance. The infrared and visible image fusion technology of drones provides an important reference for the observation and protection of the Hainan gibbons.

Lite‐weight semantic segmentation with AG self‐attention

AbstractDue to the large computational and GPUs memory cost of semantic segmentation, some works focus on designing a lite weight model to achieve a good trade‐off between computational cost and accuracy. A common method is to combined CNN and vision transformer. However, these methods ignore the contextual information of multi receptive fields. And existing methods often fail to inject detailed information losses in the downsampling of multi‐scale feature. To fix these issues, we propose AG Self‐Attention, which is Enhanced Atrous Self‐Attention (EASA), and Gate Attention. AG Self‐Attention adds the contextual information of multi receptive fields into the global semantic feature. Specifically, the Enhanced Atrous Self‐Attention uses weight shared atrous convolution with different atrous rates to get the contextual information under the specific different receptive fields. Gate Attention introduces gating mechanism to inject detailed information into the global semantic feature and filter detailed information by producing “fusion” gate and “update” gate. In order to prove our insight. We conduct numerous experiments in common semantic segmentation datasets, consisting of ADE20 K, COCO‐stuff, PASCAL Context, Cityscapes, to show that our method achieves state‐of‐the‐art performance and achieve a good trade‐off between computational cost and accuracy.

DPSDA-Net: Dual-Path Convolutional Neural Network with Strip Dilated Attention Module for Road Extraction from High-Resolution Remote Sensing Images

Roads extracted from high-resolution remote sensing images are widely used in many fields, such as autonomous driving, road planning, disaster relief, etc. However, road extraction from high-resolution remote sensing images has certain deficiencies in connectivity and completeness due to obstruction by surrounding ground objects, the influence of similar targets, and the slender structure of roads themselves. To address this issue, we propose a novel dual-path convolutional neural network with a strip dilated attention module, named DPSDA-Net, which adopts a U-shaped encoder–decoder structure, combining the powerful advantages of attention mechanism, dilated convolution, and strip convolution. The encoder utilizes ResNet50 as its basic architecture. A strip position attention mechanism is added between each residual block to strengthen the coherent semantic information of a road. A long-distance shortcut connection operation is introduced to preserve the spatial information characteristics of the original image during the downsampling process. At the same time, a pyramid dilated module with a strip convolution and attention mechanism is constructed between the encoder and decoder to enhance the network feature extraction ability and multi-scale extraction of road feature information, expand the model’s receptive field, and pay more attention to the global spatial semantic and connectivity information. To verify the reliability of the proposed model, road extraction was carried out on the Massachusetts dataset and the LRSNY dataset. The experimental results show that, compared with other typical road extraction methods, the proposed model achieved a higher F1 score and IOU. The DPSDA-Net model can comprehensively characterize the structural features of roads, extract roads more accurately, retain road details, and improve the connectivity and integrity of road extraction in remote sensing images.

TransCotANet: A Lung Field Image Segmentation Network with Multidimensional Global Feature Dynamic Aggregation

Chest X-ray (CXR) images can be used to diagnose a variety of lung diseases, such as tuberculosis, pneumonia, and lung cancer. However, the variation in lung morphology due to differences in age, gender, and the severity of pathology makes high-precision lung segmentation a challenging task. Traditional segmentation networks, such as U-Net, have become the standard architecture and have achieved remarkable results in lung field image segmentation tasks. However, because traditional convolutional operations can only explicitly capture local semantic information, it is difficult to obtain global semantic information, resulting in difficult performance in terms of accuracy requirements in medical practical applications. In recent years, the introduction of Transformer technology to natural language processing has achieved great success in the field of computer vision. In this paper, a new network architecture called TransCotANet is proposed. The network architecture is based on the U-Net architecture with convolutional neural networks (CNNs) as the backbone and extracts global semantic information through symmetric cross-layer connections in the encoder structure, where the encoder stage includes an upsampling module to improve the resolution of the feature map, and uses the dynamic aggregation module CotA to dynamically aggregate multi-scale feature maps and finally obtain more accurate segmentation results. The experimental results show that the method outperformed other methods for lung field image segmentation datasets.

Aggregating transformers and CNNs for salient object detection in optical remote sensing images

Salient object detection (SOD) in optical remote sensing images (RSIs) plays a significant role in many areas such as agriculture, environmental protection, and the military. However, since the difference in imaging mode and image complexity between RSIs and natural scene images (NSIs), it is difficult to achieve remarkable results by directly extending the saliency method targeting NSIs to RSIs. Besides, we note that the convolutional neural networks (CNNs) based U-Net cannot effectively acquire the global long-range dependency, and the Transformer doesn’t adequately characterize the spatial local details of each patch. Therefore, to conduct salient object detection in RSIs, we propose a novel two-branch architecture based network for Aggregating the Transformers and CNNs, namely ATC-Net, where the local spatial details and the global semantic information are fused into the final high-quality saliency map. Specifically, our saliency model adopts an encoder-decoder architecture including two parallel encoder branches and a decoder. Firstly, the two parallel encoder branches extract global and local features by using Transformer and CNNs, respectively. Then, the decoder employs a series of feature-enhanced fusion (FF) modules to aggregate multi-level global and local features by interactive guidance and enhance the fused feature via attention mechanism. Finally, the decoder deploys the read out (RO) module to fuse the aggregated feature of FF module and the low-level CNN feature, steering the feature to focus more on spatial local details. Extensive experiments are performed on two public optical RSIs datasets, and the results show that our saliency model consistently outperforms 30 state-of-the-art methods.