Semi-supervised Image Segmentation Research Articles

Dual structure-aware image filterings for semi-supervised medical image segmentation

Semi-supervised image segmentation has attracted great attention recently. The key is how to leverage unlabeled images in the training process. Most methods maintain consistent predictions of the unlabeled images under variations (e.g., adding noise/perturbations, or creating alternative versions) in the image and/or model level. In most image-level variation, medical images often have prior structure information, which has not been well explored. In this paper, we propose novel dual structure-aware image filterings (DSAIF) as the image-level variations for semi-supervised medical image segmentation. Motivated by connected filtering that simplifies image via filtering in structure-aware tree-based image representation, we resort to the dual contrast invariant Max-tree and Min-tree representation. Specifically, we propose a novel connected filtering that removes topologically equivalent nodes (i.e. connected components) having no siblings in the Max/Min-tree. This results in two filtered images preserving topologically critical structure. Applying the proposed DSAIF to mutually supervised networks decreases the consensus of their erroneous predictions on unlabeled images. This helps to alleviate the confirmation bias issue of overfitting to noisy pseudo labels of unlabeled images, and thus effectively improves the segmentation performance. Extensive experimental results on three benchmark datasets demonstrate that the proposed method significantly/consistently outperforms some state-of-the-art methods. The source codes will be publicly available.

Bidirectional Efficient Attention Parallel Network for Segmentation of 3D Medical Imaging

Currently, although semi-supervised image segmentation has achieved significant success in many aspects, further improvement in segmentation accuracy is necessary for practical applications. Additionally, there are fewer networks specifically designed for segmenting 3D images compared to those for 2D images, and their performance is notably inferior. To enhance the efficiency of network training, various attention mechanisms have been integrated into network models. However, these networks have not effectively extracted all the useful spatial or channel information. Particularly for 3D medical images, which contain rich spatial and channel information with tightly interconnected relationships between them, there remains a wealth of spatial and channel-specific information waiting to be explored and utilized. This paper proposes a bidirectional and efficient attention parallel network (BEAP-Net). Specifically, we introduce two modules: Supreme Channel Attention (SCA) and Parallel Spatial Attention (PSA). These modules aim to extract more spatial and channel-specific feature information and effectively utilize it. We combine the principles of consistency training and entropy regularization to enable mutual learning among sub-models. We evaluate the proposed BEAP-Net on two public 3D medical datasets, LA and Pancreas. The network outperforms the current state of the art in eight algorithms and is better suited for 3D medical images. It achieves the new best semi-supervised segmentation performance on the LA database. Ablation studies further validate the effectiveness of each component of the proposed model. Moreover, the SCA and PSA modules proposed can be seamlessly integrated into other 3D medical image segmentation networks to yield significant performance gains.

The student-teacher framework guided by self-training and consistency regularization for semi-supervised medical image segmentation.

Due to the high suitability of semi-supervised learning for medical image segmentation, a plethora of valuable research has been conducted and has achieved noteworthy success in this field. However, many approaches tend to confine their focus to a singular semi-supervised framework, thereby overlooking the potential enhancements in segmentation performance offered by integrating several frameworks. In this paper, we propose a novel semi-supervised framework named Pesudo-Label Mean Teacher (PLMT), which synergizes the self-training pipeline with pseudo-labeling and consistency regularization techniques. In particular, we integrate the student-teacher structure with consistency loss into the self-training pipeline to facilitate a mutually beneficial enhancement between the two methods. This structure not only generates remarkably accurate pseudo-labels for the self-training pipeline but also furnishes additional pseudo-label supervision for the student-teacher framework. Moreover, to explore the impact of different semi-supervised losses on the segmentation performance of the PLMT framework, we introduce adaptive loss weights. The PLMT could dynamically adjust the weights of different semi-supervised losses during the training process. Extension experiments on three public datasets demonstrate that our framework achieves the best performance and outperforms the other five semi-supervised methods. The PLMT is an initial exploration of the framework that melds the self-training pipeline with consistency regularization and offers a comparatively innovative perspective in semi-supervised image segmentation.

MDT: semi-supervised medical image segmentation with mixup-decoupling training.

Objective. In the field of medicine, semi-supervised segmentation algorithms hold crucial research significance while also facing substantial challenges, primarily due to the extreme scarcity of expert-level annotated medical image data. However, many existing semi-supervised methods still process labeled and unlabeled data in inconsistent ways, which can lead to knowledge learned from labeled data being discarded to some extent. This not only lacks a variety of perturbations to explore potential robust information in unlabeled data but also ignores the confirmation bias and class imbalance issues in pseudo-labeling methods.Approach. To solve these problems, this paper proposes a semi-supervised medical image segmentation method 'mixup-decoupling training (MDT)' that combines the idea of consistency and pseudo-labeling. Firstly, MDT introduces a new perturbation strategy 'mixup-decoupling' to fully regularize training data. It not only mixes labeled and unlabeled data at the data level but also performs decoupling operations between the output predictions of mixed target data and labeled data at the feature level to obtain strong version predictions of unlabeled data. Then it establishes a dual learning paradigm based on consistency and pseudo-labeling. Secondly, MDT employs a novel categorical entropy filtering approach to pick high-confidence pseudo-labels for unlabeled data, facilitating more refined supervision.Main results. This paper compares MDT with other advanced semi-supervised methods on 2D and 3D datasets separately. A large number of experimental results show that MDT achieves competitive segmentation performance and outperforms other state-of-the-art semi-supervised segmentation methods.Significance. This paper proposes a semi-supervised medical image segmentation method MDT, which greatly reduces the demand for manually labeled data and eases the difficulty of data annotation to a great extent. In addition, MDT not only outperforms many advanced semi-supervised image segmentation methods in quantitative and qualitative experimental results, but also provides a new and developable idea for semi-supervised learning and computer-aided diagnosis technology research.

Feature similarity learning based on fuzziness minimization for semi-supervised medical image segmentation

Deep learning has advanced the automation and intelligence levels of medical image segmentation, but the acquisition of annotations for medical images proves to be very challenging. Deep semi-supervised learning provides an effective approach to tackling this challenge. In this paper, we propose a method of feature similarity learning for deep semi-supervised image segmentation. The method performs a weighted transformation on the latent features of deep convolution network, and the objective of this transformation is designed to minimize the uncertainty of feature similarity. Based on fuzzy sets of similarity, we define the fuzziness to measure the uncertainty in the objective, and demonstrate that the optimization objective is equivalent to strengthening the discriminant property of those transformed features. This similarity learning is independent of the pseudo-labels, boosting the robustness against noisy pseudo-labels. Our method is implemented through a plug-and-play neural network layer, which can be optimized alongside the segmentation model in an end-to-end manner. The model validation focuses on the left atrium segmentation which is a challenging task for atrial fibrillation treatment. Comprehensive experiments demonstrate the effectiveness of the uncertainty-guided similarity learning on semi-supervised segmentation tasks.

Semi-supervised image segmentation using a residual-driven mean teacher and an exponential Dice loss

Semi-supervised segmentation plays an important role in computer vision and medical image analysis and can alleviate the burden of acquiring abundant expert-annotated images. In this paper, we developed a residual-driven semi-supervised segmentation method (termed RDMT) based on the classical mean teacher (MT) framework by introducing a novel model-level residual perturbation and an exponential Dice (eDice) loss. The introduced perturbation was integrated into the exponential moving average (EMA) scheme to enhance the performance of the MT, while the eDice loss was used to improve the detection sensitivity of a given network to object boundaries. We validated the developed method by applying it to segment 3D Left Atrium (LA) and 2D optic cup (OC) from the public LASC and REFUGE datasets based on the V-Net and U-Net, respectively. Extensive experiments demonstrated that the developed method achieved the average Dice score of 0.8776 and 0.7751, when trained on 10% and 20% labeled images, respectively for the LA and OC regions depicted on the LASC and REFUGE datasets. It significantly outperformed the MT and can compete with several existing semi-supervised segmentation methods (i.e., HCMT, UAMT, DTC and SASS).

RCPS: Rectified Contrastive Pseudo Supervision for Semi-Supervised Medical Image Segmentation.

Medical image segmentation methods are generally designed as fully-supervised to guarantee model performance, which requires a significant amount of expert annotated samples that are high-cost and laborious. Semi-supervised image segmentation can alleviate the problem by utilizing a large number of unlabeled images along with limited labeled images. However, learning a robust representation from numerous unlabeled images remains challenging due to potential noise in pseudo labels and insufficient class separability in feature space, which undermines the performance of current semi-supervised segmentation approaches. To address the issues above, we propose a novel semi-supervised segmentation method named as Rectified Contrastive Pseudo Supervision (RCPS), which combines a rectified pseudo supervision and voxel-level contrastive learning to improve the effectiveness of semi-supervised segmentation. Particularly, we design a novel rectification strategy for the pseudo supervision method based on uncertainty estimation and consistency regularization to reduce the noise influence in pseudo labels. Furthermore, we introduce a bidirectional voxel contrastive loss in the network to ensure intra-class consistency and inter-class contrast in feature space, which increases class separability in the segmentation. The proposed RCPS segmentation method has been validated on two public datasets and an in-house clinical dataset. Experimental results reveal that the proposed method yields better segmentation performance compared with the state-of-the-art methods in semi-supervised medical image segmentation. The source code is available at https://github.com/hsiangyuzhao/RCPS.

Contour-aware consistency for semi-supervised medical image segmentation

In medical images, the edges of organs are often blurred and unclear. Existing semi-supervised image segmentation methods rarely model edges explicitly. Thus most methods produce inaccurate predictions in target edge regions. In this paper, we propose a contour-aware consistency framework for semi-supervised medical image segmentation. The framework consists of a shared encoder, a vanilla primary decoder and a contour-enhanced auxiliary decoder. The contour-enhanced decoder is designed to enhance the features of the target contour region. The predictions from the primary decoder and the auxiliary decoder are combined to create pseudo labels, enabling the unlabeled data for supervision. For the inconsistent regions in predictions, we propose a self-contrast strategy that further improves the performance by reducing the discrepancy of the dual decoder for the same pixel. We conducted extensive experiments on three publicly available datasets and verified that our approach outperforms other methods for boundary quality. Specifically, with 5% labeled data on Left Atrial (LA) dataset, our proposed approach achieved a Boundary IoU 3.76% higher than the state-of-the-art methods. Code is available at https://github.com/SmileJET/CAC4SSL.

Anatomically-aware uncertainty for semi-supervised image segmentation

Semi-supervised learning relaxes the need of large pixel-wise labeled datasets for image segmentation by leveraging unlabeled data. A prominent way to exploit unlabeled data is to regularize model predictions. Since the predictions of unlabeled data can be unreliable, uncertainty-aware schemes are typically employed to gradually learn from meaningful and reliable predictions. Uncertainty estimation methods, however, rely on multiple inferences from the model predictions that must be computed for each training step, which is computationally expensive. Moreover, these uncertainty maps capture pixel-wise disparities and do not consider global information. This work proposes a novel method to estimate segmentation uncertainty by leveraging global information from the segmentation masks. More precisely, an anatomically-aware representation is first learnt to model the available segmentation masks. The learnt representation thereupon maps the prediction of a new segmentation into an anatomically-plausible segmentation. The deviation from the plausible segmentation aids in estimating the underlying pixel-level uncertainty in order to further guide the segmentation network. The proposed method consequently estimates the uncertainty using a single inference from our representation, thereby reducing the total computation. We evaluate our method on two publicly available segmentation datasets of left atria in cardiac MRIs and of multiple organs in abdominal CTs. Our anatomically-aware method improves the segmentation accuracy over the state-of-the-art semi-supervised methods in terms of two commonly used evaluation metrics.

Adaptive feature aggregation based multi-task learning for uncertainty-guided semi-supervised medical image segmentation

Automatic segmentation of medical images is a necessary prerequisite for diagnosing related diseases. Magnetic resonance imaging (MRI) is a widely used non-invasive method in clinical practice, but obtaining reliable labeled training data for deep learning models is time-consuming and challenging. In this study, we propose an adaptive feature aggregation-based multi-task learning for uncertainty-guided semi-supervised image segmentation, dubbed AFAM-Net. The framework leverages a reconstruction task to capture anatomical information from raw images and assist the segmentation task. Moreover, we propose an adaptive feature aggregation strategy that selectively transfers useful features while filtering out irrelevant information, considering the associations between the two tasks. To better utilize unlabeled data, we incorporate dual uncertainty-aware methods to improve the segmentation performance. We evaluate the proposed AFAM-Net on clinical liver, 3D heart, and Cine cardiac MRI datasets. According to the experimental results, our AFAM-Net significantly outperformed other state-of-the-art semi-supervised medical image segmentation algorithms.

Mutually aided uncertainty incorporated dual consistency regularization with pseudo label for semi-supervised medical image segmentation

Semi-supervised learning has contributed plenty to promoting computer vision tasks. Especially concerning medical images, semi-supervised image segmentation can significantly reduce the labor and time cost of labeling images. Among the existing semi-supervised methods, pseudo-labelling and consistency regularization prevail; however, the current related methods still need to achieve satisfactory results due to the poor quality of the pseudo-labels generated and needing more certainty awareness the models. To address this problem, we propose a novel method that combines pseudo-labelling with dual consistency regularization based on a high capability of uncertainty awareness. This method leverages a cycle-loss regularized to lead to a more accurate uncertainty estimate. Followed by the uncertainty estimation, the certain region with its pseudo-label is further trained in a supervised manner. In contrast, the uncertain region is used to promote the dual consistency between the student and teacher networks. The developed approach was tested on three public datasets and showed that: 1) The proposed method achieves excellent performance improvement by leveraging unlabeled data; 2) Compared with several state-of-the-art (SOTA) semi-supervised segmentation methods, ours achieved better or comparable performance.

Semi-supervised medical image segmentation using adversarial consistency learning and dynamic convolution network.

Popular semi-supervised medical image segmentation networks often suffer from error supervision from unlabeled data since they usually use consistency learning under different data perturbations to regularize model training. These networks ignore the relationship between labeled and unlabeled data, and only compute single pixel-level consistency leading to uncertain prediction results. Besides, these networks often require a large number of parameters since their backbone networks are designed depending on supervised image segmentation tasks. Moreover, these networks often face a high over-fitting risk since a small number of training samples are popular for semi-supervised image segmentation. To address the above problems, in this paper, we propose a novel adversarial self-ensembling network using dynamic convolution (ASE-Net) for semi-supervised medical image segmentation. First, we use an adversarial consistency training strategy (ACTS) that employs two discriminators based on consistency learning to obtain prior relationships between labeled and unlabeled data. The ACTS can simultaneously compute pixel-level and image-level consistency of unlabeled data under different data perturbations to improve the prediction quality of labels. Second, we design a dynamic convolution-based bidirectional attention component (DyBAC) that can be embedded in any segmentation network, aiming at adaptively adjusting the weights of ASE-Net based on the structural information of input samples. This component effectively improves the feature representation ability of ASE-Net and reduces the overfitting risk of the network. The proposed ASE-Net has been extensively tested on three publicly available datasets, and experiments indicate that ASE-Net is superior to state-of-the-art networks, and reduces computational costs and memory overhead. The code is available at: https://github.com/SUST-reynole/ASE-Net.

Weak-to-Strong Consistency Learning for Semisupervised Image Segmentation

Supervised remote sensing (RS) image segmentation has achieved remarkable success with large amounts of manually labeled data, which may be difficult to acquire in some practical application scenarios. Semisupervised RS image segmentation can efficiently utilize the knowledge embedded in unlabeled data to improve recognition performance, which is of great significance for the generalization application of segmentation models. In this work, we propose an end-to-end semisupervised RS image segmentation method based on weak-to-strong consistency learning, denoted as WSCL. Specifically, a common strong data augmentation technique for image segmentation is introduced to provide powerful input perturbation to decouple self-biased cognition. By forcing weakly augmented, and strongly augmented perspectives from the same sample to be consistent, WSCL not only enables the model to steadily learn knowledge contained in unlabeled data but also alleviates overfitting. In addition, a novel sparse dual-view cross-sample image generation method is presented to generate new training samples, which helps provide a more comprehensive diversity of perturbations. Furthermore, an adaptive re-weighting strategy based on the entropy maps of the outputs of strongly perturbed samples is proposed to suppress noise, guiding the training process in a positive direction. Extensive experiments demonstrate the significant advantage of WSCL over other advanced methods, achieving new state-of-the-art under several evaluation metrics on DFC22, iSAID, MER, MSL, Vaihingen, and GID-15 datasets. The source code is open-sourced at https://github.com/xiaoqiang-lu/WSCL.

Deep Tri-Training for Semi-Supervised Image Segmentation

Semantic segmentation is of great value to autonomous driving and many robotic applications, while it highly depends on costly and time-consuming pixel-level annotation. To make full use of unlabeled data, this work proposes a deep tri-training framework (dubbed DTT) to utilize labeled along with unlabeled data for training in a semi-supervised manner. Concretely, in the DTT framework, three networks are initialized with the same structure but different parameters. The networks are optimized circularly, where one network is trained in each optimization step with the guidance of the other two networks. A simple yet effective voting mechanism is adopted to construct reliable training sets from unlabeled data for the training stage and fusing multi-experts prediction in the testing stage. Exhaustive experiments on Cityscapes and PASCAL VOC 2012 demonstrate that the proposed DTT realizes state-of-the-art performance in the semi-supervised segmentation task. The source code is made publicly available.

An Effective Semi-Supervised Approach for Liver CT Image Segmentation.

Despite the substantial progress made by deep networks in the field of medical image segmentation, they generally require sufficient pixel-level annotated data for training. The scale of training data remains to be the main bottleneck to obtain a better deep segmentation model. Semi-supervised learning is an effective approach that alleviates the dependence on labeled data. However, most existing semi-supervised image segmentation methods usually do not generate high-quality pseudo labels to expand training dataset. In this paper, we propose a deep semi-supervised approach for liver CT image segmentation by expanding pseudo-labeling algorithm under the very low annotated-data paradigm. Specifically, the output features of labeled images from the pretrained network combine with corresponding pixel-level annotations to produce class representations according to the mean operation. Then pseudo labels of unlabeled images are generated by calculating the distances between unlabeled feature vectors and each class representation. To further improve the quality of pseudo labels, we adopt a series of operations to optimize pseudo labels. A more accurate segmentation network is obtained by expanding the training dataset and adjusting the contributions between supervised and unsupervised loss. Besides, the novel random patch based on prior locations is introduced for unlabeled images in the training procedure. Extensive experiments show our method has achieved more competitive results compared with other semi-supervised methods when fewer labeled slices of LiTS dataset are available.

Image Segmentation with a Priori Conditions: Applications to Medical and Geophysical Imaging

In this paper, we propose a method for semi-supervised image segmentation based on geometric active contours. The main novelty of the proposed method is the initialization of the segmentation process, which is performed with a polynomial approximation of a user defined initialization (for instance, a set of points or a curve to be interpolated). This work is related to many potential applications: the geometric conditions can be useful to improve the quality the segmentation process in medicine and geophysics when it is required (weak contrast of the image, missing parts in the image, non-continuous contour…). We compare our method to other segmentation algorithms, and we give experimental results related to several medical and geophysical applications.

Graph Convolutional Networks for Semi-Supervised Image Segmentation

The problem of image segmentation is one of the most significant ones in computer vision. Recently, deep-learning methods have dominated state-of-the-art solutions that automatically or interactively divide an image into subregions. However, the limitation of deep-learning approaches is that they require a substantial amount of training data, which is costly to prepare. An alternative solution is semi-supervised image segmentation. It requires rough denotations to define constraints that are next generalized to precisely delimit relevant image regions without using train examples. Among semi-supervised strategies for image segmentation, the leading are graph-based techniques that define image segmentation as a result of pixel or region affinity graph partitioning. This paper revisits the problem of graph-based image segmentation. It approaches the problem as semi-supervised node classification in the SLIC superpixels region adjacency graph using a graph convolutional network (GCN). The performance of both spectral and spatial graph convolution operators is considered, represented by Chebyshev convolution operator and GraphSAGE respectively. The results of the proposed method applied to binary and multi-label segmentation are presented, numerically assessed, and analyzed. In its best variant, the proposed method scored the average DICE of 0.86 in the binary segmentation task and 0.79 in the multi-label segmentation task. Comparison with state-of-the-art graph-based methods, including Random Walker and GrabCut, shows that graph convolutional networks can represent an attractive alternative to the existing solutions to graph-based semi-supervised image segmentation.

Minimizing Estimated Risks on Unlabeled Data: A New Formulation for Semi-Supervised Medical Image Segmentation.

Supervised segmentation can be costly, particularly in applications of biomedical image analysis where large scale manual annotations from experts are generally too expensive to be available. Semi-supervised segmentation, able to learn from both the labeled and unlabeled images, could be an efficient and effective alternative for such scenarios. In this work, we propose a new formulation based on risk minimization, which makes full use of the unlabeled images. Different from most of the existing approaches which solely explicitly guarantee the minimization of prediction risks from the labeled training images, the new formulation also considers the risks on unlabeled images. Particularly, this is achieved via an unbiased estimator, based on which we develop a general framework for semi-supervised image segmentation. We validate this framework on three medical image segmentation tasks, namely cardiac segmentation on ACDC2017, optic cup and disc segmentation on REFUGE dataset and 3D whole heart segmentation on MM-WHS dataset. Results show that the proposed estimator is effective, and the segmentation method achieves superior performance and demonstrates great potential compared to the other state-of-the-art approaches. Our code and data will be released via https://zmiclab.github.io/projects.html, once the manuscript is accepted for publication.

Boosting Semi-supervised Image Segmentation with Global and Local Mutual Information Regularization

The scarcity of labeled data often impedes the application of deep learning to the segmentation of medical images. Semi-supervised learning seeks to overcome this limitation by leveraging unlabeled examples in the learning process. In this paper, we present a novel semi-supervised segmentation method that leverages mutual information (MI) on categorical distributions to achieve both global representation invariance and local smoothness. In this method, we maximize the MI for intermediate feature embeddings that are taken from both the encoder and decoder of a segmentation network. We first propose a global MI loss constraining the encoder to learn an image representation that is invariant to geometric transformations. Instead of resorting to computationally-expensive techniques for estimating the MI on continuous feature embeddings, we use projection heads to map them to a discrete cluster assignment where MI can be computed efficiently. Our method also includes a local MI loss to promote spatial consistency in the feature maps of the decoder and provide a smoother segmentation. Since mutual information does not require a strict ordering of clusters in two different assignments, we incorporate a final consistency regularization loss on the output which helps align the cluster labels throughout the network. We evaluate the method on four challenging publicly-available datasets for medical image segmentation. Experimental results show our method to outperform recently-proposed approaches for semi-supervised segmentation and provide an accuracy near to full supervision while training with very few annotated images.

Self-paced and self-consistent co-training for semi-supervised image segmentation

Deep co-training has recently been proposed as an effective approach for image segmentation when annotated data is scarce. In this paper, we improve existing approaches for semi-supervised segmentation with a self-paced and self-consistent co-training method. To help distillate information from unlabeled images, we first design a self-paced learning strategy for co-training that lets jointly-trained neural networks focus on easier-to-segment regions first, and then gradually consider harder ones. This is achieved via an end-to-end differentiable loss in the form of a generalized Jensen Shannon Divergence (JSD). Moreover, to encourage predictions from different networks to be both consistent and confident, we enhance this generalized JSD loss with an uncertainty regularizer based on entropy. The robustness of individual models is further improved using a self-ensembling loss that enforces their prediction to be consistent across different training iterations. We demonstrate the potential of our method on three challenging image segmentation problems with different image modalities, using a small fraction of labeled data. Results show clear advantages in terms of performance compared to the standard co-training baselines and recently proposed state-of-the-art approaches for semi-supervised segmentation.