Segmentation Sub-network Research Articles

Full-field displacement measurements of structural vibrations using a novel two-stage neural network

This paper presents a novel two-stage neural network for quantifying full-field displacements of vibrating structures. A graphics software was used to simulate structural vibrations with random displacement fields and generate image datasets through batch rendering. Subsequently, a two-stage neural network model comprising an object segmentation subnetwork for extracting structural shapes and an optical-flow subnetwork for identifying displacement fields was constructed. A coordinate convolution module was embedded in the optical-flow estimation model and the two subnetworks were integrated to develop a comprehensive model. An experiment was designed to monitor the vibration displacements in a cantilever column. The results showed that the proposed model achieved optimal recognition accuracy with a mean absolute error of 0.1477 pixels and root-mean-squared error of 0.2335 pixels. Additionally, it exhibited robustness to various lighting conditions. Furthermore, case studies on real-world structural vibration videos substantiated the practical engineering application value of the proposed model.

A Multi-Task Model for Pulmonary Nodule Segmentation and Classification.

In the computer-aided diagnosis of lung cancer, the automatic segmentation of pulmonary nodules and the classification of benign and malignant tumors are two fundamental tasks. However, deep learning models often overlook the potential benefits of task correlations in improving their respective performances, as they are typically designed for a single task only. Therefore, we propose a multi-task network (MT-Net) that integrates shared backbone architecture and a prediction distillation structure for the simultaneous segmentation and classification of pulmonary nodules. The model comprises a coarse segmentation subnetwork (Coarse Seg-net), a cooperative classification subnetwork (Class-net), and a cooperative segmentation subnetwork (Fine Seg-net). Coarse Seg-net and Fine Seg-net share identical structure, where Coarse Seg-net provides prior location information for the subsequent Fine Seg-net and Class-net, thereby boosting pulmonary nodule segmentation and classification performance. We quantitatively and qualitatively analyzed the performance of the model by using the public dataset LIDC-IDRI. Our results show that the model achieves a Dice similarity coefficient (DI) index of 83.2% for pulmonary nodule segmentation, as well as an accuracy (ACC) of 91.9% for benign and malignant pulmonary nodule classification, which is competitive with other state-of-the-art methods. The experimental results demonstrate that the performance of pulmonary nodule segmentation and classification can be improved by a unified model that leverages the potential correlation between tasks.

Cross-Modality Medical Image Segmentation via Enhanced Feature Alignment and Cross Pseudo Supervision Learning.

Given the diversity of medical images, traditional image segmentation models face the issue of domain shift. Unsupervised domain adaptation (UDA) methods have emerged as a pivotal strategy for cross modality analysis. These methods typically utilize generative adversarial networks (GANs) for both image-level and feature-level domain adaptation through the transformation and reconstruction of images, assuming the features between domains are well-aligned. However, this assumption falters with significant gaps between different medical image modalities, such as MRI and CT. These gaps hinder the effective training of segmentation networks with cross-modality images and can lead to misleading training guidance and instability. To address these challenges, this paper introduces a novel approach comprising a cross-modality feature alignment sub-network and a cross pseudo supervised dual-stream segmentation sub-network. These components work together to bridge domain discrepancies more effectively and ensure a stable training environment. The feature alignment sub-network is designed for the bidirectional alignment of features between the source and target domains, incorporating a self-attention module to aid in learning structurally consistent and relevant information. The segmentation sub-network leverages an enhanced cross-pseudo-supervised loss to harmonize the output of the two segmentation networks, assessing pseudo-distances between domains to improve the pseudo-label quality and thus enhancing the overall learning efficiency of the framework. This method's success is demonstrated by notable advancements in segmentation precision across target domains for abdomen and brain tasks.

A joint model for lesion segmentation and classification of MS and NMOSD.

Multiple sclerosis (MS) and neuromyelitis optic spectrum disorder (NMOSD) are mimic autoimmune diseases of the central nervous system with a very high disability rate. Their clinical symptoms and imaging findings are similar, making it difficult to diagnose and differentiate. Existing research typically employs the T2-weighted fluid-attenuated inversion recovery (T2-FLAIR) MRI imaging technique to focus on a single task in MS and NMOSD lesion segmentation or disease classification, while ignoring the collaboration between the tasks. To make full use of the correlation between lesion segmentation and disease classification tasks of MS and NMOSD, so as to improve the accuracy and speed of the recognition and diagnosis of MS and NMOSD, a joint model is proposed in this study. The joint model primarily comprises three components: an information-sharing subnetwork, a lesion segmentation subnetwork, and a disease classification subnetwork. Among them, the information-sharing subnetwork adopts a dualbranch structure composed of a convolution module and a Swin Transformer module to extract local and global features, respectively. These features are then input into the lesion segmentation subnetwork and disease classification subnetwork to obtain results for both tasks simultaneously. In addition, to further enhance the mutual guidance between the tasks, this study proposes two information interaction methods: a lesion guidance module and a crosstask loss function. Furthermore, the lesion location maps provide interpretability for the diagnosis process of the deep learning model. The joint model achieved a Dice similarity coefficient (DSC) of 74.87% on the lesion segmentation task and accuracy (ACC) of 92.36% on the disease classification task, demonstrating its superior performance. By setting up ablation experiments, the effectiveness of information sharing and interaction between tasks is verified. The results show that the joint model can effectively improve the performance of the two tasks.

STSN-Net: Simultaneous Tooth Segmentation and Numbering Method in Crowded Environments with Deep Learning.

Accurate tooth segmentation and numbering are the cornerstones of efficient automatic dental diagnosis and treatment. In this paper, a multitask learning architecture has been proposed for accurate tooth segmentation and numbering in panoramic X-ray images. A graph convolution network was applied for the automatic annotation of the target region, a modified convolutional neural network-based detection subnetwork (DSN) was used for tooth recognition and boundary regression, and an effective region segmentation subnetwork (RSSN) was used for region segmentation. The features extracted using RSSN and DSN were fused to optimize the quality of boundary regression, which provided impressive results for multiple evaluation metrics. Specifically, the proposed framework achieved a top F1 score of 0.9849, a top Dice metric score of 0.9629, and an mAP (IOU = 0.5) score of 0.9810. This framework holds great promise for enhancing the clinical efficiency of dentists in tooth segmentation and numbering tasks.

Prototypical Learning Guided Context-Aware Segmentation Network for Few-Shot Anomaly Detection.

Few-shot anomaly detection (FSAD) denotes the identification of anomalies within a target category with a limited number of normal samples. Existing FSAD methods largely rely on pretrained feature representations to detect anomalies, but the inherent domain gap between pretrained representations and target FSAD scenarios is often overlooked. This study proposes a prototypical learning-guided context-aware segmentation network (PCSNet) to address the domain gap, thereby improving feature descriptiveness in target scenarios and enhancing FSAD performance. In particular, PCSNet comprises a prototypical feature adaption (PFA) subnetwork and a context-aware segmentation (CAS) subnetwork. PFA extracts prototypical features as guidance to ensure better feature compactness for normal data while distinct separation from anomalies. A pixel-level disparity classification (PDC) loss is also designed to make subtle anomalies more distinguishable. Then a CAS subnetwork is introduced for pixel-level anomaly localization, where pseudo anomalies are exploited to facilitate the training process. Experimental results on MVTec AD and metal part defect detection (MPDD) demonstrate the superior FSAD performance of PCSNet, with 94.9% and 80.2% image-level area under the receiver operating characteristics (AUROCs) in an eight-shot scenario, respectively. Real-world applications on automotive plastic part inspection further demonstrate that PCSNet can achieve promising results with limited training samples. The code is available at https://github.com/yuxin-jiang/PCSNet.

Bi-Fusion of Structure and Deformation at Multi-Scale for Joint Segmentation and Registration.

Medical image segmentation and registration are two fundamental and highly related tasks. However, current works focus on the mutual promotion between the two at the loss function level, ignoring the feature information generated by the encoder-decoder network during the task-specific feature mapping process and the potential inter-task feature relationship. This paper proposes a unified multi-task joint learning framework based on bi-fusion of structure and deformation at multi-scale, called BFM-Net, which simultaneously achieves the segmentation results and deformation field in a single-step estimation. BFM-Net consists of a segmentation subnetwork (SegNet), a registration subnetwork (RegNet), and the multi-task connection module (MTC). The MTC module is used to transfer the latent feature representation between segmentation and registration at multi-scale and link different tasks at the network architecture level, including the spatial attention fusion module (SAF), the multi-scale spatial attention fusion module (MSAF) and the velocity field fusion module (VFF). Extensive experiments on MR, CT and ultrasound images demonstrate the effectiveness of our approach. The MTC module can increase the Dice scores of segmentation and registration by 3.2%, 1.6%, 2.2%, and 6.2%, 4.5%, 3.0%, respectively. Compared with six state-of-the-art algorithms for segmentation and registration, BFM-Net can achieve superior performance in various modal images, fully demonstrating its effectiveness and generalization.

Syn_SegNet: A Joint Deep Neural Network for Ultrahigh-Field 7T MRI Synthesis and Hippocampal Subfield Segmentation in Routine 3T MRI.

Precise delineation of hippocampus subfields is crucial for the identification and management of various neurological and psychiatric disorders. However, segmenting these subfields automatically in routine 3T MRI is challenging due to their complex morphology and small size, as well as the limited signal contrast and resolution of the 3T images. This research proposes Syn_SegNet, an end-to-end, multitask joint deep neural network that leverages ultrahigh-field 7T MRI synthesis to improve hippocampal subfield segmentation in 3T MRI. Our approach involves two key components. First, we employ a modified Pix2PixGAN as the synthesis model, incorporating self-attention modules, image and feature matching loss, and ROI loss to generate high-quality 7T-like MRI around the hippocampal region. Second, we utilize a variant of 3D-U-Net with multiscale deep supervision as the segmentation subnetwork, incorporating an anatomic weighted cross-entropy loss that capitalizes on prior anatomical knowledge. We evaluate our method on hippocampal subfield segmentation in paired 3T MRI and 7T MRI with seven different anatomical structures. The experimental findings demonstrate that Syn_SegNet's segmentation performance benefits from integrating synthetic 7T data in an online manner and is superior to competing methods. Furthermore, we assess the generalizability of the proposed approach using a publicly accessible 3T MRI dataset. The developed method would be an efficient tool for segmenting hippocampal subfields in routine clinical 3T MRI.

Retina-TransNet: A Gradient-Guided Few-Shot Retinal Vessel Segmentation Net.

Due to the high labor cost of physicians, it is difficult to collect a rich amount of manually-labeled medical images for developing learning-based computer-aided diagnosis (CADx) systems or segmentation algorithms. To tackle this issue, we reshape the image segmentation task as an image-to-image (I2I) translation problem and propose a retinal vascular segmentation network, which can achieve good cross-domain generalizability even with a small amount of training data. We devise primarily two components to facilitate this I2I-based segmentation method. The first is the constraints provided by the proposed gradient-vector-flow (GVF) loss, and, the second is a two-stage Unet (2Unet) generator with a skip connection. This configuration makes 2Unet's first-stage play a role similar to conventional Unet, but forces 2Unet's second stage to learn to be a refinement module. Extensive experiments show that by re-casting retinal vessel segmentation as an image-to-image translation problem, our I2I translator-based segmentation subnetwork achieves better cross-domain generalizability than existing segmentation methods. Our model, trained on one dataset, e.g., DRIVE, can produce segmentation results stably on datasets of other domains, e.g., CHASE-DB1, STARE, HRF, and DIARETDB1, even in low-shot circumstances.

Determining the Dosimetric Accuracy of Deep Learning-Based Fully Automated Registration-Segmentation Approach for Thoracic Cancer Organs-at-Risk Contouring

For adaptive radiation therapy (ART), the contours on the planning CT (pCT) are frequently propagated to cone-beam CT (CBCT) via deformable image registration and manually edited, which is observer-dependent and time-consuming. To automate this process, we created a fully automated workflow by combining a deep learning (DL)-based pCT segmentation model with a CT-to-CBCT registration-segmentation DL model. The purpose of our research is to determine how using the proposed workflow's automatically generated contours affects thoracic organs-at-risk sparing (OAR). Seven patients with locally advanced non-small cell lung cancer who underwent treatment with intensity modulated radiation therapy were included in this study. Each patient's pCT was segmented using a published DL model that has been used for generating thoracic OAR segmentation and radiotherapy planning in the clinic since July of 2020. Next, pCT was deformably registered using a published recurrent deep registration-segmentation method. Whereas the original method's segmentation sub-network was only trained to segment esophagus, the registration sub-network was used to propagate contours for heart, esophagus, and the proximal bronchial tree (PBT). Geometric segmentation accuracy using the Dice Similarity Coefficient (DSC) and the 95th percentile Hausdorff Distance (HD) and dose metrics including the mean esophageal dose (MED) and D90% of the heart (D90) were computed from the total accumulated dose for the first two weeks of treatment. The esophagus had a high DSC and a low HD (0.93 and 2.85mm) and conversely, the heart had lower accuracy (DSC = 0.85, HD = 22.06mm). PBT showed relatively high performance as well, with DSC of 0.91 and HD of 2.28mm, owing to its proximity to the esophagus. The accumulated MED for manual contour was slightly lower than AI-contours (11.34 vs 11.83 Gy), suggesting reliability of the proposed workflow. The reverse is seen for the D90 of the heart (manual = 1.74 and AI-contour = 1.56 Gy), likely due to the heart not being included in the original DL framework. This study reported preliminary results on the feasibility of using a fully automated and patient-specific workflow for CBCT auto-segmentation in ART, confirming its role as a geometrically and dosimetrically accurate solution for thoracic OARs. However, because it is currently limited to the esophagus, we believe that re-training the algorithm will increase confidence in other OARs such as the heart and lungs.

MineSDS: A Unified Framework for Small Object Detection and Drivable Area Segmentation for Open-Pit Mining Scenario.

To tackle the challenges posed by dense small objects and fuzzy boundaries on unstructured roads in the mining scenario, we proposed an end-to-end small object detection and drivable area segmentation framework for open-pit mining. We employed a convolutional network backbone as a feature extractor for both two tasks, as multi-task learning yielded promising results in autonomous driving perception. To address small object detection, we introduced a lightweight attention module that allowed our network to focus more on the spatial and channel dimensions of small objects without impeding inference time. We also used a convolutional block attention module in the drivable area segmentation subnetwork, which assigned more weight to road boundaries to improve feature mapping capabilities. Furthermore, to improve our network perception accuracy of both tasks, we used weighted summation when designing the loss function. We validated the effectiveness of our approach by testing it on pre-collected mining data which were called Minescape. Our detection results on the Minescape dataset showed 87.8% mAP index, which was 9.3% higher than state-of-the-art algorithms. Our segmentation results surpassed the comparison algorithm by 1 percent in MIoU index. Our experimental results demonstrated that our approach achieves competitive performance.

Segmentation-based Decision Networks for Steel Surface Defect Detection

<p>With the advent of the Industrial 4.0 era, deep learning has been continuously applied to the task of surface defect detection, and effective progress has been made. However, the limited number of training samples and high labelling costs are considerable obstacles to the vigorous development of this task. Thus, we explore the use of different numbers of labels with various accuracies during training to achieve the maximum detection accuracy with the lowest cost. Our proposed method includes improved segmentation and decision networks. An attention mechanism is integrated into the segmentation subnetwork. Moreover, atrous convolutions are used in the segmentation and decision subnetworks. In addition, the original loss function is improved. Several experiments are carried out on the Severstal Steel Defect dataset collected in Germany, and the results show that each component improves the detection accuracy by 1% to 2%. Finally, when we add an appropriate number of pixel-level labels in the weakly supervised learning mode, the detection accuracy reaches that of the fully supervised mode with a significantly reduced annotation cost.</p> <p> </p>

Sem-Aug: Improving Camera-LiDAR Feature Fusion With Semantic Augmentation for 3D Vehicle Detection

Camera-LiDAR fusion provides precise distance measurements and fine-grained textures, making it a promising option for 3D vehicle detection in autonomous driving scenarios. Previous camera-LiDAR based 3D vehicle detection approaches mainly focused on employing image-based pre-trained models to fetch semantic features. However, these methods may perform inferior to the LiDAR-based ones when lacking semantic segmentation labels in autonomous driving tasks. Motivated by this observation, we propose a novel semantic augmentation method, namely Sem-Aug, to guide high-confidence camera-LiDAR fusion feature generation and boost the performance of multimodal 3D vehicle detection. The key novelty of semantic augmentation lies in the 2D segmentation mask auto-labeling, which provides supervision for semantic segmentation sub-network to mitigate the poor generalization performance of camera-LiDAR fusion. Using semantic-augmentation-guided camera-LiDAR fusion features, Sem-Aug achieves remarkable performance on the representative autonomous driving KITTI dataset compared to both the LiDAR-based baseline and previous multimodal 3D vehicle detectors. Qualitative and quantitative experiments demonstrate that Sem-Aug provides significant improvements in challenging <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Hard</i> detection scenarios caused by occlusion and truncation.

S[formula omitted]C-DeLeNet: A parameter transfer based segmentation-classification integration for detecting skin cancer lesions from dermoscopic images

Dermoscopic images ideally depict pigmentation attributes on the skin surface which is highly regarded in the medical community for detection of skin abnormality, disease or even cancer. The identification of such abnormality, however, requires trained eyes and accurate detection necessitates the process being time-intensive. As such, computerized detection schemes have become quite an essential, especially schemes which adopt deep learning tactics. In this paper, a convolutional deep neural network, S2C-DeLeNet, is proposed, which (i) Performs segmentation procedure of lesion based regions with respect to the unaffected skin tissue from dermoscopic images using a segmentation sub-network, (ii) Classifies each image based on its medical condition type utilizing transferred parameters from the inherent segmentation sub-network. The architecture of the segmentation sub-network contains EfficientNet-B4 backbone in place of the encoder and the classification sub-network bears a ‘Classification Feature Extraction’ system which pulls trained segmentation feature maps towards lesion prediction. Inside the classification architecture, there have been designed, (i) A ‘Feature Coalescing Module’ in order to trail and mix each dimensional feature from both encoder and decoder, (ii) A ‘3D-Layer Residuals’ block to create a parallel pathway of low-dimensional features with high variance for better classification. After fine-tuning on a publicly accessible dataset, a mean dice-score of 0.9494 during segmentation is procured which beats existing segmentation strategies and a mean accuracy of 0.9103 is obtained for classification which outperforms conventional and noted classifiers. Additionally, the already fine-tuned network demonstrates highly satisfactory results on other skin cancer segmentation datasets while cross-inference. Extensive experimentation is done to prove the efficacy of the network for not only dermoscopic images but also different medical modalities; which can show its potential in being a systematic diagnostic solution in the field of dermatology and possibly more.

Automated fatigue crack detection in steel box girder of bridges based on ensemble deep neural network

This paper constructs a novel ensemble deep neural network to improve identification accuracy of bridge cracks in complex backgrounds. In the three sub-networks of the ensemble network, the detection classifier distinguishes cracks and distractors at the scale of image patches, and then the segmentation sub-network obtains pixel-level crack details. Meanwhile, an innovative crack inference sub-network is constructed based on the prior knowledge of crack morphology. The inference sub-network can predict the probability of cracks existence using the learned image correlation between adjacent regions, and feedback to the detection and segmentation sub-networks. Then, the prediction probabilities coming from the image feature and morphological reasoning are fused to judge and correct the crack repeatedly. Therefore, the presented ensemble deep neural network can well simulate human multi-scale observation, reasoning and decision-making processes. The identification results show that the proposed algorithm can effectively reduce the misjudgment and provide more accurate crack segmentation.

A nondestructive testing method for detecting surface defects of Si3N4-Bearing cylindrical rollers based on an optimized convolutional neural network

Aiming at solving the problems of low detection efficiency, poor accuracy rate, and low applicability of the traditional detection methods for surface defects on the silicon nitride (Si3N4) bearing cylindrical rollers. In this study, to detect the surface defects on Si3N4-bearing cylindrical rollers, a nondestructive testing (NDT) method based on an optimized convolutional neural network is proposed, which uses machine vision system to detect the surface defects. The optimized convolutional neural network is a two-stage network. It combines a semantic segmentation sub-network and a decision sub-network. The semantic segmentation sub-network performs an end-to-end learning based on the features of the original image. It completes defect feature extraction to segment the surface defect area from the normal area. The decision sub-network classifies the defects in the segmented surface defect area. The experimental results show that the detection time of surface defects on Si3N4-bearing cylindrical rollers by the proposed network was 72 ms. In addition, its Accuracy (Acc), Percision (Pre), Sensitivity (Sen), and Specificity (Spe) were 97.5%, 99.9%, 99.0%, 98.6%, respectively, which significantly improved the Acc and Pre of the detection of surface defects on Si3N4-bearing cylindrical rollers compared with the traditional detection method. On the other hand, its mIoU, mPA were 84.4%, 92.9%, respectively, meaning brilliant ability of image segmentation for this network. To sum up, the ability of the proposed network to detect and classify surface defects on Si3N4-bearing cylindrical rollers is excellent.

Character-Level Street View Text Spotting Based on Deep Multisegmentation Network for Smarter Autonomous Driving

Urban scenes are full of street entities with sign boards. Therefore, in autonomous driving, street view text spotting techniques will play a significant role in the precise understanding of surrounding scenes during driving, because texts contained in the images usually provide important clues for accurate image understanding, while it is often ambiguous for existing computer vision algorithms to understand scene images without texts. In this work, we propose a <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">M</b> ulti- <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">S</b> egmentation network for character-level scene <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">T</b> ext <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">D</b> etection (MSTD). The MSTD introduces a densely connected atrous spatial pyramid pooling module to enlarge the receptive field of the feature extraction layer, so as to localize long as well as large-sized text instances. Moreover, it devises a double segmentation subnetwork to utilize two independent but inherently complementary losses to co-optimize the network and increase the reliability of the confidence scores in predicting the text/nontext areas. With the character instances detected by the MSTD, one can easily perform scene text spotting with classic object recognition networks such as ResNet and DenseNet. We carried out extensive experiments on nine scene text datasets to demonstrate the outstanding performance of the MSTD on character-level and line-level text instance localization and scene text recognition, where the MSTD significantly outperforms the state-of-the-art scene text detection methods and the sequence-to-sequence-learning-based scene text recognizers.

A-LugSeg: Automatic and explainability-guided multi-site lung detection in chest X-ray images

Large variations in anatomical shape and size, too much overlap between anatomical structures, and inconsistent anatomical shapes make accurate lung segmentation in chest x-rays (CXR) a challenging problem. In this paper, we propose an automatic method called A-LugSeg that consists of two subnetworks for lung segmentation in CXRs. The first is a segmentation subnetwork based on a deep learning model (i.e., Mask-RCNN), which completes a coarse segmentation for each input CXR image. The second is a refinement subnetwork designed to optimize the coarse segmentation result by combining an improved closed principal curve method and an enhanced machine learning, where the lung contour’s explainability-guided mathematical model is expressed by the machine learning’s parameters. The performance of the proposed method is evaluated on three public datasets, namely the ShenZhen hospital Chest X-ray dataset (SZCX), Japanese Society of Radiological Technology dataset (JSRT), and Montgomery County chest x-ray dataset (MC), which contain the 662 CXRs, 247 CXRs, and 138 CXRs, respectively. We used different datasets for training/validation (SZCX) and testing (SZCX/JSRT/MC). Furthermore, we used six evaluation metrics to evaluate the performance of our proposed method, including Dice Similarity Coefficient (DSC), Jaccard Similarity Coefficient (Ω), Accuracy (ACC), Precision, Sensitivity, and Specificity. The obtained results (DSC = 0.973, Ω = 0.958, ACC = 0.972, and p-value for DSC < 0.001) for JSRT, (DSC = 0.971, Ω = 0.955, ACC = 0.97, and p-value for DSC < 0.001) for MC, (DSC = 0.972, Ω = 0.956, and ACC = 0.97) for hybrid datasets (JSRT + MC), and (Precision, Sensitivity, and Specificity are higher than 0.98) show the superior performance of the proposed dual subnetwork segmentation algorithm compared to the existing state of the art approaches.

Weakly Supervised Video Object Segmentation via Dual-attention Cross-branch Fusion

Recently, concerning the challenge of collecting large-scale explicitly annotated videos, weakly supervised video object segmentation (WSVOS) using video tags has attracted much attention. Existing WSVOS approaches follow a general pipeline including two phases, i.e., a pseudo masks generation phase and a refinement phase. To explore the intrinsic property and correlation buried in the video frames, most of them focus on the later phase by introducing optical flow as temporal information to provide more supervision. However, these optical flow-based studies are greatly affected by illumination and distortion and lack consideration of the discriminative capacity of multi-level deep features. In this article, with the goal of capturing more effective temporal information and investigating a temporal information fusion strategy accordingly, we propose a unified WSVOS model by adopting a two-branch architecture with a multi-level cross-branch fusion strategy, named as dual-attention cross-branch fusion network (DACF-Net). Concretely, the two branches of DACF-Net, i.e., a temporal prediction subnetwork (TPN) and a spatial segmentation subnetwork (SSN), are used for extracting temporal information and generating predicted segmentation masks, respectively. To perform the cross-branch fusion between TPN and SSN, we propose a dual-attention fusion module that can be plugged into the SSN flexibly. We also pose a cross-frame coherence loss (CFCL) to achieve smooth segmentation results by exploiting the coherence of masks produced by TPN and SSN. Extensive experiments demonstrate the effectiveness of proposed approach compared with the state-of-the-arts on two challenging datasets, i.e., Davis-2016 and YouTube-Objects.

Deep Symmetric Adaptation Network for Cross-Modality Medical Image Segmentation.

Unsupervised domain adaptation (UDA) methods have shown their promising performance in the cross-modality medical image segmentation tasks. These typical methods usually utilize a translation network to transform images from the source domain to target domain or train the pixel-level classifier merely using translated source images and original target images. However, when there exists a large domain shift between source and target domains, we argue that this asymmetric structure, to some extent, could not fully eliminate the domain gap. In this paper, we present a novel deep symmetric architecture of UDA for medical image segmentation, which consists of a segmentation sub-network, and two symmetric source and target domain translation sub-networks. To be specific, based on two translation sub-networks, we introduce a bidirectional alignment scheme via a shared encoder and two private decoders to simultaneously align features 1) from source to target domain and 2) from target to source domain, which is able to effectively mitigate the discrepancy between domains. Furthermore, for the segmentation sub-network, we train a pixel-level classifier using not only original target images and translated source images, but also original source images and translated target images, which could sufficiently leverage the semantic information from the images with different styles. Extensive experiments demonstrate that our method has remarkable advantages compared to the state-of-the-art methods in three segmentation tasks, such as cross-modality cardiac, BraTS, and abdominal multi-organ segmentation.