Deep Supervision Strategy Research Articles

Cloud Detection Using a UNet3+ Model with a Hybrid Swin Transformer and EfficientNet (UNet3+STE) for Very-High-Resolution Satellite Imagery

It is necessary to extract and recognize the cloud regions presented in imagery to generate satellite imagery as analysis-ready data (ARD). In this manuscript, we proposed a new deep learning model to detect cloud areas in very-high-resolution (VHR) satellite imagery by fusing two deep learning architectures. The proposed UNet3+ model with a hybrid Swin Transformer and EfficientNet (UNet3+STE) was based on the structure of UNet3+, with the encoder sequentially combining EfficientNet based on mobile inverted bottleneck convolution (MBConv) and the Swin Transformer. By sequentially utilizing convolutional neural networks (CNNs) and transformer layers, the proposed algorithm aimed to extract the local and global information of cloud regions effectively. In addition, the decoder used MBConv to restore the spatial information of the feature map extracted by the encoder and adopted the deep supervision strategy of UNet3+ to enhance the model’s performance. The proposed model was trained using the open dataset derived from KOMPSAT-3 and 3A satellite imagery and conducted a comparative evaluation with the state-of-the-art (SOTA) methods on fourteen test datasets at the product level. The experimental results confirmed that the proposed UNet3+STE model outperformed the SOTA methods and demonstrated the most stable precision, recall, and F1 score values with fewer parameters and lower complexity.

Gca-pvt-net: group convolutional attention and PVT dual-branch network for oracle bone drill chisel segmentation

Oracle bones (Obs) are a significant carrier of the shang dynasty civilization, primarily consisting of tortoise shells and animal bones, through the study of which we can gain a deeper understanding of the political, economic, religious, and cultural aspects of the shang dynasty. The oracle bone drill chisel (Obdc) is considered an essential non-textual material. The segmentation of Obdc assists archaeologists determine the approximate age of the Obs, which possesses considerable research value. However, the breakage of thousands of years of underground buried Obs, the blurring of the edges of the area burned by the Obdc, the different shapes, and the inconsistent number have brought challenges to the accurate segmentation of the Obdc. In this article, we propose a group convolutional attention and pvt dual-branch network (GCA-PVT-Net) for Obdc segmentation. To our knowledge, this paper is the first to research the automatic segmentation of Obdc. It is a hybrid Convolutional neural network (CNN) and Transformer framework. The work offers the following contributions: (1) The Obdc images are labeled based on the delineation criteria of different drill chisel (DC) shapes to create the Obdc dataset. (2) A convolutional attention module (CAM) is proposed as both an encoder and decoder. The feature extraction process, which effectively integrates global and local information, ensures better modeling of long-term correlations in images while preserving details. (3) A channel feature aggregation module (CFAM) is designed to enhance the effective integration of channel features, enabling feature fusion across various branches and at different levels. (4) The edge deep supervision strategy is applied to smooth the jagged edge of the predicted images at the decoder’s end. Extensive experiments on the Obdc dataset show that GCA-PVT-Net outperforms other state-of-the-art (SOTA) methods. The comparative experimental results show that the edge accuracy and segmentation accuracy of the model reach the top 1.

Edge and dense attention U-net for atrial scar segmentation in LGE-MRI

The segmentation of atrial scars in LGE-MRI images has huge potential value for clinical diagnosis and subsequent treatment. In clinical practice, atrial scars are usually manually calibrated by experienced experts, which is time-consuming and prone to errors. However, automatic segmentation also faces difficulties due to myocardial scars’ small size and variable shape. The present study introduces a dual branch network, incorporating edge attention, and deep supervision strategy. Edge attention is introduced to fully utilize the spatial relationship between the scar and the atrium. Besides, dense attention is embedded in bottom layer to solve feature disappearance. At the same time, deep supervision accelerates the convergence of the model and improves segmentation accuracy. The experiments were conducted on the 2022 atrial and scar segmentation challenge dataset. The results demonstrate that the proposed method has achieved superior performance.

LMFormer: Lightweight and multi-feature perspective via transformer for human pose estimation

The effectiveness of Token Mixer in visual tasks is well-established; however, its high computational complexity and a relatively singular spatial relationship modeling perspective present challenges. In this study, we propose LMFormer, a hybrid model based on CNN and Transformer architectures for human pose estimation. To achieve this, we first design a lightweight multi-feature perspective Token Mixer, using a lightweight feature reconstruction strategy to simultaneously aggregate the spatial and channel feature information, thereby enhancing the performance and generalization capabilities of the model. Subsequently, we explore multi-scale information interaction by developing an iterative multi-feature weighting module, coupled with the design of a multi-scale information propagation mechanism incorporated into the skip connections. Finally, we validate the effectiveness of the network on benchmark datasets, including COCO, MPII, and CrowdPose, utilizing a multi-scale deep supervision strategy. Extensive experiments demonstrate that LMFormer, with reduced computational complexity, comprehensively captures multi-scale features, resulting in significant performance improvements. Specifically, LMFormer-B achieves an AP score of 65.8 on the COCO val dataset, surpassing MobileNetV2 and ShuffleNetV2 by 1.0 and 5.6 points, respectively. Moreover, its parameters are merely 19.8% and 25% of MobileNetV2 and ShuffleNetV2, with corresponding GFLOPs at 43.8% and 50%. We aim to provide new insights into lightweight and efficient feature extraction strategies, as well as efficient Token Mixer designs.

PRT-Net: a progressive refinement transformer for dose prediction to guide ovarian transposition.

Young cervical cancer patients who require ovarian transposition usually have their ovaries moved away from the pelvic radiotherapy (RT) field before radiotherapy. The dose of ovaries during radiotherapy is closely related to the location of the ovaries. To protect ovarian function and avoid ovarian dose exceeding the limits, a safe location of transposed ovary must be determined prior to surgery. For this purpose, we input the patient's preoperative CT into a neural network model to predict the dose distribution. Surgeons were able to quickly locate low-dose regions based on the dose distribution before surgery, thus determining the safe location of the transposed ovary. In this work, we proposed a new progressive refinement transformer model PRT-Net that can generate dose prediction at multiple scale resolutions in one forward propagation, and refine the dose prediction using prediction details from low to high resolution based on a deep supervision strategy. A multi-loss function fusion algorithm was also built to fit the prediction results under different loss dimensions. The clinical feasibility of the method was verified through an actual cases. Therefore, using PRT-Net to predict the dose distribution by preoperative CT in cervical cancer patients can assist clinicians to perform ovarian transposition surgery and prevent patients' ovaries from exceeding the prescribed dose limit in postoperative radiotherapy.

A novel non-pretrained deep supervision network for polyp segmentation

In this paper, we propose a non-pretrained deep supervision network (NPD-Net) for polyp segmentation. Unlike previous deep supervision networks that rely on ground truth (GT) or pre-training with GT to supervise deep features(the prediction maps from decoder), we propose a novel deep supervision strategy that directly utilizes the GT encoder (that encodes GT to get its maps) after initialization to mitigate overfitting and enhance generalization ability without pre-training, in other words, a non-pretrained. This strategy makes up the gap of directly using GT for deep supervision while mitigates the risk of overfitting due to leverage the well-train pre-trained weights on a small polyp datasets. In addition, we introduce a simple and efficient parallel dual attention module (PDA) to enhance the global modeling ability. PDA executes spatial and channel attention in parallel, and adopts implicit positional encoding and transpose operation to reduce computational complexity. Finally, NPD-Net is able to effectively supervise deep features, expand the range of context information acquisition and improve segmentation performance, particularly in terms of generalization ability. Our experimental results on five benchmark datasets demonstrate that NPD-Net outperforms other state-of-the-art methods. The code will be available at https://github.com/guobaoxiao/NPD-Net.

Transfer-Aware Graph U-Net with Cross-Level Interactions for PolSAR Image Semantic Segmentation

Although graph convolutional networks have found application in polarimetric synthetic aperture radar (PolSAR) image classification tasks, the available approaches cannot operate on multiple graphs, which hinders their potential to generalize effective feature representations across different datasets. To overcome this limitation and achieve robust PolSAR image classification, this paper proposes a novel end-to-end cross-level interaction graph U-Net (CLIGUNet), where weighted max-relative spatial convolution is proposed to enable simultaneous learning of latent features from batch input. Moreover, it integrates weighted adjacency matrices, derived from the symmetric revised Wishart distance, to encode polarimetric similarity into weighted max-relative spatial graph convolution. Employing end-to-end trainable residual transformers with multi-head attention, our proposed cross-level interactions enable the decoder to fuse multi-scale graph feature representations, enhancing effective features from various scales through a deep supervision strategy. Additionally, multi-scale dynamic graphs are introduced to expand the receptive field, enabling trainable adjacency matrices with refined connectivity relationships and edge weights within each resolution. Experiments undertaken on real PolSAR datasets show the superiority of our CLIGUNet with respect to state-of-the-art networks in classification accuracy and robustness in handling unknown imagery with similar land covers.

Progressive deep snake for instance boundary extraction in medical images

Boundary extraction is meaningful in medical image analysis since it explicitly extracts the tissue/lesions boundary coordinates, which benefits the follow-up diagnosis processes. However, accurately extracting human tissue/lesions is challenging due to unclear target boundaries or pseudo boundaries, which could lead to inferior performance of existing boundary extraction and/or semantic segmentation methods. We propose a Progressive Deep Snake (PDS), which is designed with a multi-step contour evolution (MCE) and a sequential label generation module (SLGM) to explicitly extract the smooth and continuous boundary of different tissue/lesions even if they are unclear, with irregular shape, or occluded by pseudo boundaries. (1) MCE is designed to initialize and evolve the contours to the target boundaries, where a multi-step deep supervision strategy is designed to improve flexibility and accuracy. (2) SLGM is designed to generate easy-to-difficult supervision signals for the contour evolution, which provides targets of moderate difficulty in each step for better guiding the contour evolution. Experiments on a challenging verbal segment datasets show that our method achieves state-of-the-art performance (MR_AVBCE dataset: mIoU: 0.9234, mDice: 0.9598, mBoundF: 0.9241, mWcov: 0.9704). Our code is available at https://github.com/sysu19351118/PDS-pytorch.

Deeply Supervised Skin Lesions Diagnosis With Stage and Branch Attention.

Accurate and unbiased examinations of skin lesions are critical for the early diagnosis and treatment of skin diseases. Visual features of skin lesions vary significantly because the images are collected from patients with different lesion colours and morphologies by using dissimilar imaging equipment. Recent studies have reported that ensembled convolutional neural networks (CNNs) are practical to classify the images for early diagnosis of skin disorders. However, the practical use of these ensembled CNNs is limited as these networks are heavyweight and inadequate for processing contextual information. Although lightweight networks (e.g., MobileNetV3 and EfficientNet) were developed to achieve parameter reduction for implementing deep neural networks on mobile devices, insufficient depth of feature representation restricts the performance. To address the existing limitations, we develop a new lite and effective neural network, namely HierAttn. The HierAttn applies a novel deep supervision strategy to learn the local and global features by using multi-stage and multi-branch attention mechanisms with only one training loss. The efficacy of HierAttn was evaluated by using the dermoscopy images dataset ISIC2019 and smartphone photos dataset PAD-UFES-20 (PAD2020). The experimental results show that HierAttn achieves the best accuracy and area under the curve (AUC) among the state-of-the-art lightweight networks.

A Triplet Network Fusing Optical and SAR Images for Colored Steel Building Extraction.

The identification of colored steel buildings in images is crucial for managing the construction sector, environmental protection, and sustainable urban development. Current deep learning methods for optical remote sensing images often encounter challenges such as confusion between the roof color or shape of regular buildings and colored steel structures. Additionally, common semantic segmentation networks exhibit poor generalization and inadequate boundary regularization when extracting colored steel buildings. To overcome these limitations, we utilized the metal detection and differentiation capabilities inherent in synthetic aperture radar (SAR) data to develop a network that integrates optical and SAR data. This network, employing a triple-input structure, effectively captures the unique features of colored steel buildings. We designed a multimodal hybrid attention module in the network that discerns the varying importance of each data source depending on the context. Additionally, a boundary refinement (BR) module was introduced to extract the boundaries of the colored steel buildings in a more regular manner, and a deep supervision strategy was implemented to improve the performance of the network in the colored steel building extraction task. A BR module and deep supervision strategy were also implemented to sharpen the extraction of building boundaries, thereby enhancing the network's accuracy and adaptability. The results indicate that, compared to mainstream semantic segmentation, this method effectively enhances the precision of colored steel building detection, achieving an accuracy rate of 83.19%. This improvement marks a significant advancement in monitoring illegal constructions and supporting the sustainable development of the Beijing-Tianjin-Hebei metropolitan region.

FDR-TransUNet: A novel encoder-decoder architecture with vision transformer for improved medical image segmentation

The U-shaped and Transformer architectures have achieved exceptional performance in medical image segmentation and natural language processing, respectively. Their combination has also led to remarkable results but still suffers from enormous loss of image features during downsampling and the difficulty of recovering spatial information during upsampling. In this paper, we propose a novel encoder-decoder architecture for medical image segmentation, which has a flexibly adjustable hybrid encoder and two expanding paths decoder. The hybrid encoder incorporates the feature double reuse (FDR) block and the encoder of Vision Transformer (ViT), which can extract local and global pixel localization information, and alleviate image feature loss effectively. Meanwhile, we retain the original class-token sequence in the Vision Transformer and develop an additional corresponding expanding path. The class-token sequence and abstract image features are leveraged by two independent expanding paths with the deep-supervision strategy, which can better recover the image spatial information and accelerate model convergence. To further mitigate the feature loss and improve spatial information recovery, we introduce successive residual connections throughout the entire network. We evaluated our model on the COVID-19 lung segmentation and the infection area segmentation tasks. The mIoU index increased by 1.5 points and 3.9 points compared to other models which demonstrates a performance improvement.

An attention-guided network for bilateral ventricular segmentation in pediatric echocardiography

Accurate segmentation of pediatric echocardiograms is a challenging task, because significant heart-size changes with age and faster heart rate lead to more blurred boundaries on cardiac ultrasound images compared with adults. To address these problems, a dual decoder network model combining channel attention and scale attention is proposed in this paper. Firstly, an attention-guided decoder with deep supervision strategy is used to obtain attention maps for the ventricular regions. Then, the generated ventricular attention is fed back to multiple layers of the network through skip connections to adjust the feature weights generated by the encoder and highlight the left and right ventricular areas. Finally, a scale attention module and a channel attention module are utilized to enhance the edge features of the left and right ventricles. The experimental results demonstrate that the proposed method in this paper achieves an average Dice coefficient of 90.63% in acquired bilateral ventricular segmentation dataset, which is better than some conventional and state-of-the-art methods in the field of medical image segmentation. More importantly, the method has a more accurate effect in segmenting the edge of the ventricle. The results of this paper can provide a new solution for pediatric echocardiographic bilateral ventricular segmentation and subsequent auxiliary diagnosis of congenital heart disease.

High‐resolution optical remote sensing image change detection based on dense connection and attention feature fusion network

AbstractThe detection of ground object changes from bi‐temporal images is of great significance for urban planning, land‐use/land‐cover monitoring and natural disaster assessment. To solve the limitation of incomplete change detection (CD) entities and inaccurate edges caused by the loss of detailed information, this paper proposes a network based on dense connections and attention feature fusion, namely Siamese NestedUNet with Attention Feature Fusion (SNAFF). First, multi‐level bi‐temporal features are extracted through a Siamese network. The dense connections between the sub‐nodes of the decoder are used to compensate for the missing location information as well as weakening the semantic differences between features. Then, the attention mechanism is introduced to combine global and local information to achieve feature fusion. Finally, a deep supervision strategy is used to suppress the problem of gradient vanishing and slow convergence speed. During the testing phase, the test time augmentation (TTA) strategy is adopted to further improve the CD performance. In order to verify the effectiveness of the proposed method, two datasets with different change types are used. The experimental results indicate that, compared with the comparison methods, the proposed SNAFF achieves the best quantitative results on both datasets, in which F1, IoU and OA in the LEVIR‐CD dataset are 91.47%, 84.28% and 99.13%, respectively, and the values in the CDD dataset are 96.91%, 94.01% and 99.27%, respectively. In addition, the qualitative results show that SNAFF can effectively retain the global and edge information of the detected entity, thus achieving the best visual performance.

SMTF: Sparse transformer with multiscale contextual fusion for medical image segmentation

Medical image segmentation aims at recognizing the object of interest from surrounding tissues and structures, which is essential for the reliable diagnosis and morphological analysis of specific lesions. Automatic medical image segmentation has been significantly boosted by deep Convolutional Neural Networks (CNNs). However, CNNs usually fail to model long-range interactions due to the intrinsic locality of convolutional operations, which limits the segmentation performance. Recently, Transformer has been successfully applied in various computer visions, which leverages the self-attention mechanism for modelling long-range interactions to capture global information. Nevertheless, self-attention suffers from lacks of spatial locality and efficient computation. To address these issues, in this work, we develop a new sparse medical Transformer (SMTF) with multiscale contextual fusion for medical image segmentation. The proposed model combines convolutional operations and attention mechanisms to form a U-shaped framework to capture both local and global information. Specifically, to reduce the computational cost of traditional Transformer, we design a novel sparse attention module to construct Transformer layers by spherical Locality Sensitive Hashing method. The sparse attention partitions the feature space into different attention buckets, and the attention calculation is conducted only in the individual bucket. The designed sparse Transformer layer further incorporates a bottleneck block to construct the encoder in SMTF. It is worth noting that the proposed sparse Transformer can also aggregate the global feature information in early stages, which enables the model to learn more local and global information by incorporating CNNs at lower layers. Furthermore, we introduce a deep supervision strategy to guide the model to fuse multiscale feature information. It further enables the SMTF to effectively propagate feature information across layers to preserve more input spatial information and mitigate information attenuation. Benefiting from these, it can achieve better segmentation performance while being more robust and efficient. The proposed SMTF is evaluated on multiple medical image segmentation datasets and a clinical nasopharyngeal carcinoma dataset. Extensive experiments have demonstrated its superiority on both qualitative and quantitative evaluations. Code and models are available at https://github.com/qmx717/sparse-attention.git.

TPFR-Net: U-shaped model for lung nodule segmentation based on transformer pooling and dual-attention feature reorganization.

Accurate segmentation of lung nodules is the key to diagnosing the lesion type of lung nodule. The complex boundaries of lung nodules and the visual similarity to surrounding tissues make precise segmentation of lung nodules challenging. Traditional CNN based lung nodule segmentation models focus on extracting local features from neighboring pixels and ignore global contextual information, which is prone to incomplete segmentation of lung nodule boundaries. In the U-shaped encoder-decoder structure, variations of image resolution caused by up-sampling and down-sampling result in the loss of feature information, which reduces the reliability of output features. This paper proposes transformer pooling module and dual-attention feature reorganization module to effectively improve the above two defects. Transformer pooling module innovatively fuses the self-attention layer and pooling layer in the transformer, which compensates for the limitation of convolution operation, reduces the loss of feature information in the pooling process, and decreases the computational complexity of the Transformer significantly. Dual-attention feature reorganization module innovatively employs the dual-attention mechanism of channel and spatial to improve the sub-pixel convolution, minimizing the loss of feature information during up-sampling. In addition, two convolutional modules are proposed in this paper, which together with transformer pooling module form an encoder that can adequately extract local features and global dependencies. We use the fusion loss function and deep supervision strategy in the decoder to train the model. The proposed model has been extensively experimented and evaluated on the LIDC-IDRI dataset, the highest Dice Similarity Coefficient is 91.84 and the highest sensitivity is 92.66, indicating the model's comprehensive capability has surpassed state-of-the-art UTNet. The model proposed in this paper has superior segmentation performance for lung nodules and can provide a more in-depth assessment of lung nodules' shape, size, and other characteristics, which is of important clinical significance and application value to assist physicians in the early diagnosis of lung nodules.

OdeBERT: One-stage Deep-supervised Early-exiting BERT for Fast Inference in User Intent Classification

User intent classification is a vital task for analyzing users’ essential requirements from the users’ input query in information retrieval systems, question answering systems, and dialogue systems. Pre-trained language model Bidirectional Encoder Representation from Transformers (BERT) has been widely applied to the user intent classification task. However, BERT is compute intensive and time-consuming during inference and usually causes latency in real-time applications. To improve the inference efficiency of BERT for the user intent classification task, this article proposes a new network named one-stage deep-supervised early-exiting BERT as one-stage deep-supervised early-exiting BERT (OdeBERT). In addition, a deep supervision strategy is developed to incorporate the network with internal classifiers by one-stage joint training to improve the learning process of classifiers by extracting discriminative category features. Experiments are conducted on publicly available datasets, including ECDT, SNIPS, and FDQuestion. The results show that the OdeBERT can speed up original BERT 12 times faster at most with the same performance, outperforming state-of-the-art baseline methods.

Context–content collaborative network for building extraction from high-resolution imagery

In practical applications, different application fields have various requirements regarding the precision and completeness of building extraction. Too low precision or completeness may limit the application and promotion of building extraction. Obtaining a good trade-off between the precision and completeness of building extraction is still a challenging issue. To deal with this issue, this paper proposes a context–content collaborative network (C3Net) with an encoder–decoder structure. It consists of a context–content aware module (C2AM) and an edge residual refinement module (ER2M). In the C2AM, a context-aware block and a content-aware block complement each other and capture the localization information of buildings and long-range dependencies between the locations of each building, respectively. Thanks to the capability of the conventional filter, the ER2M can refine the features of decoder output by deploying a residual atrous spatial pyramid pooling with feature edges at the scale of the original image. To explicitly guide the function of the ER2M, we introduce a separated deep supervision strategy before and after the ER2M, which can consciously refine our C3Net towards the precision or completeness of building extraction to a certain extent, and improve the overall detection performance. Compared with several classical and state-of-the-art methods, extensive experiments on three open and challenging datasets demonstrate that the proposed C3Net not only acquires competitive performance but also achieves a better trade-off between precision and completeness of building extraction. The source code is released at https://github.com/TongfeiLiu/C3Net-for-building-extraction.

ACPA-Net: Atrous Channel Pyramid Attention Network for Segmentation of Leakage in Rail Tunnel Linings

The automatic segmentation of leakage in rail tunnel linings is a useful and challenging task. Unlike other scenarios, the complex environment inside the tunnels makes it difficult to obtain accurate results for the segmentation of leakages. Some deep learning-based methods have been used to automatically segment tunnel leakage, but these methods ignore the interdependencies between feature channels, which are very important for extracting robust leakage feature representations. In this work, we propose an atrous channel pyramid attention network (ACPA-Net) for rail tunnel lining leakage segmentation. In ACPA-Net, the proposed atrous channel pyramid attention (ACPA) module is added into a U-shaped segmentation network. The ACPA module can effectively strengthen the representation ability of ACPA-Net by explicitly modeling the dependencies between feature channels. In addition, a deep supervision strategy that helps ACPA-Net improve its discrimination ability has also been introduced into ACPA-Net. A rail tunnel leakage image dataset consisting of 1370 images with manual annotation maps is built to verify the effectiveness of ACPA-Net. The final experiment shows that ACPA-Net achieves state-of-the-art performance on the Crack500 dataset and our rail tunnel leakage image dataset, and our method has the least number of parameters of all the methods.

What happens next? Combining enhanced multilevel script learning and dual fusion strategies for script event prediction

Script event prediction (SEP), aiming at predicting next event from context event sequences (i.e., scripts), has played an important role in many real-world applications such as government decision-making. While most of the existing research only depend on the top-level event prediction, they ignore the influence of other bottom levels or other relationship modeling manners. In this paper, we focus on the problem of SEP via multilevel script learning where the goal of is to explore a multistage, multiprediction and multilevel information fusion model for SEP. This is challenging in (1) simultaneously modeling of the multilevel event relationship semantic information and (2) effectively designing multilevel information fusion strategies. In this paper, we propose a new script event prediction model based on Enhanced Multilevel script learning and Dual Fusion strategies, named EMDF-Net. Specifically, EMDF-Net designs the multilevel (event/chain/segment level) script learning to model both temporal and casual information as well as the rich structural relevance via neural stacking of self-attention mechanism and graph neural networks. Then it proposes dual fusion strategies to fully integrate different-level information by nonlinear feature composition and weighted score fusion. Finally, a deep supervision strategy is utilized to end-to-end train the whole model and provide a good initialization for information fusion. Experimental results on the popular NYT corpus demonstrate the effectiveness and superiority of EMDF-Net.

The auto segmentation for cardiac structures using a dual-input deep learning network based on vision saliency and transformer.

PurposeAccurate segmentation of cardiac structures on coronary CT angiography (CCTA) images is crucial for the morphological analysis, measurement, and functional evaluation. In this study, we achieve accurate automatic segmentation of cardiac structures on CCTA image by adopting an innovative deep learning method based on visual attention mechanism and transformer network, and its practical application value is discussed.MethodsWe developed a dual‐input deep learning network based on visual saliency and transformer (VST), which consists of self‐attention mechanism for cardiac structures segmentation. Sixty patients’ CCTA subjects were randomly selected as a development set, which were manual marked by an experienced technician. The proposed vision attention and transformer mode was trained on the patients CCTA images, with a manual contour‐derived binary mask used as the learning‐based target. We also used the deep supervision strategy by adding auxiliary losses. The loss function of our model was the sum of the Dice loss and cross‐entropy loss. To quantitatively evaluate the segmentation results, we calculated the Dice similarity coefficient (DSC) and Hausdorff distance (HD). Meanwhile, we compare the volume of automatic segmentation and manual segmentation to analyze whether there is statistical difference.ResultsFivefold cross‐validation was used to benchmark the segmentation method. The results showed the left ventricular myocardium (LVM, DSC = 0.87), the left ventricular (LV, DSC = 0.94), the left atrial (LA, DSC = 0.90), the right ventricular (RV, DSC = 0.92), the right atrial (RA, DSC = 0.91), and the aortic (AO, DSC = 0.96). The average DSC was 0.92, and HD was 7.2 ± 2.1 mm. In volume comparison, except LVM and LA (p < 0.05), there was no significant statistical difference in other structures. Proposed method for structural segmentation fit well with the true profile of the cardiac substructure, and the model prediction results closed to the manual annotation.Conclusions The adoption of the dual‐input and transformer architecture based on visual saliency has high sensitivity and specificity to cardiac structures segmentation, which can obviously improve the accuracy of automatic substructure segmentation. This is of gr