Unsupervised Semantic Segmentation Research Articles

Bridging knowledge distillation gap for few-sample unsupervised semantic segmentation

Due to privacy, security, and costly labeling of images, unsupervised semantic segmentation with very few samples has become a promising direction, but still remains unexplored. This inspires us to introduce the few-sample unsupervised semantic segmentation task, which is very challenging because generalizing the segmentation model from only a few unlabeled images is far from sufficient. We address this problem in the knowledge distillation perspective, by proposing a medium-sized auxiliary network as the bridge, which narrows down the semantic knowledge gap between teacher network (large) and student network (small). To this end, we develop the Knowledge Distillation Bridge (KDB) framework for few-sample unsupervised semantic segmentation. In particular, it consists of the teacher-auxiliary-student architecture, which adopts the block-wise distillation that encourages the auxiliary to imitate the teacher and the student to imitate the auxiliary. In this way, the knowledge gap between the source feature distribution and the target one is reduced, allowing the student with the smaller network to be readily deployed in highly-demanding environment. Meanwhile, each channel characterizes different semantics in feature map, which motivates us to distill the features of decoder in a channel-wise manner. Extensive experiments on two benchmarks including Pascal VOC2012 and Cityscapes demonstrate the promising performance of the proposed method, which strikes a good balance between precision and speed, e.g., it achieves the inference speed of 230 fps for a 512×512 image.

An unsupervised semantic segmentation method that combines the ImSE-Net model with SLICm superpixel optimization

ABSTRACT In the field of remote sensing, using a large amount of labeled image data to supervise the training of fully convolutional networks for the semantic segmentation of images is expensive. However, using a small amount of labeled data can lead to reduced network performance. This paper proposes an unsupervised semantic segmentation method that combines the ImSE-Net model with SLICm superpixel optimization. First, the ImSE-Net model is used to extract semantic features from the image to obtain rough semantic segmentation results. Then, the SLICm superpixel segmentation algorithm is used to segment the input image into superpixel images. Finally, an unsupervised semantic segmentation model (UGLS) is used to combine high-level abstract semantic features with detailed information on superpixels to obtain edge-optimized semantic segmentation results. Experimental results show that compared with other semantic segmentation algorithms, our method more effectively handles unbalanced areas, such as object boundaries, and achieves better segmentation results, with higher semantic consistency.

Expand-and-Quantize: Unsupervised Semantic Segmentation Using High-Dimensional Space and Product Quantization

Unsupervised semantic segmentation (USS) aims to discover and recognize meaningful categories without any labels. For a successful USS, two key abilities are required: 1) information compression and 2) clustering capability. Previous methods have relied on feature dimension reduction for information compression, however, this approach may hinder the process of clustering. In this paper, we propose a novel USS framework called Expand-and-Quantize Unsupervised Semantic Segmentation (EQUSS), which combines the benefits of high-dimensional spaces for better clustering and product quantization for effective information compression. Our extensive experiments demonstrate that EQUSS achieves state-of-the-art results on three standard benchmarks. In addition, we analyze the entropy of USS features, which is the first step towards understanding USS from the perspective of information theory.

An evolutionary vehicle scanning method for bridges based on time series segmentation and change point detection

In this paper, the problem of bridge health monitoring is formulated as an unsupervised semantic segmentation where the aim is to identify the onset of structural damage by detecting unexpected irregularities in the time series data. In particular, the main objective of this study is to approach this problem through an indirect sensing framework based on the concept of vehicle-mounted scanning. Thus, the sole source of information is the acceleration responses of the moving vehicle passing over the bridge. This is achieved through automatic feature learning by adopting the concept of matrix profile to conduct an all-pair-similarity search for a given time series, where the location of the contextual change is determined by the corresponding Corrected Arc Crossings (CAC) profile. The performance of the method is first evaluated through numerical investigations for different damage cases where the effect of operational uncertainties such as road roughness, and variability in vehicle properties are considered. Further, the method is experimentally evaluated using a lab-scale bridge model with different damage cases. Through these investigations, it is consistently observed that the location of the change point always coincides with the minimum point of the CAC profile. The proposed framework has three major advantages making it more appealing compared to the competing techniques. First, it has been fully formulated in the context of unsupervised learning where no access to labeled data is required. Second, it is capable of working with a limited amount of data yet providing promising results. Finally, it can be easily applied to online settings where data is streaming in real-time. These successful results demonstrate the potential of the proposed method as a cost-effective and rigorous bridge inspection tool and open up opportunities in the field of indirect bridge health monitoring.

Unsupervised corneal contour extraction algorithm with shared model for dynamic deformation videos: improving accuracy and noise resistance

BackgroundIn this study, an automatic corneal contour extraction algorithm with a shared model is developed to extract contours from dynamic corneal videos containing noise, which improves the accuracy of corneal biomechanical evaluation and clinical diagnoses. The algorithm does not require manual labeling and completes the unsupervised semantic segmentation of each frame in corneal dynamic deformation videos based on a fully convolutional deep-learning network using corneal geometry and texture information.ResultsWe included 1027 corneal videos at Tianjin Eye Hospital (Nankai University Affiliated Eye Hospital) from May 2020 to November 2021. The videos were obtained by the ultra-high-speed Scheimpflug camera, and then we used the shared model mechanism to accelerate the segmentation of corneal regions in videos, effectively resist noise, determine corneal regions based on shape factors, and finally achieve automatic and accurate extraction of corneal region contours. The Intersection over Union (IoU) of the extracted and real corneal contours using this algorithm reached 95%, and the average overlap error was 0.05, implying that the extracted corneal contour overlapped almost completely with the real contour.ConclusionsCompared to other algorithms, the method introduced in this study does not require manual annotation of corneal contour data in advance and can still extract accurate corneal contours from noisy corneal videos with good repeatability.

Crop Identification of UAV Images Based on an Unsupervised Semantic Segmentation Method

Crop Identification of UAV Images Based on an Unsupervised Semantic Segmentation Method

Unsupervised Semantic Segmentation of PolSAR Images Based on Multiview Similarity

Unsupervised Semantic Segmentation of PolSAR Images Based on Multiview Similarity

DatUS: Data-driven Unsupervised Semantic Segmentation with Pre-trained Self-supervised Vision Transformer

DatUS: Data-driven Unsupervised Semantic Segmentation with Pre-trained Self-supervised Vision Transformer

Bayesian-optimized unsupervised semantic segmentation model for structural crack detection

To reduce the reliance of existing mainstream vision-based crack detection algorithms on annotated datasets, this paper presents an unsupervised semantic segmentation-based approach. The proposed method first employs the Felzenszwalb-Huttenlocher algorithm for pre-segmenting the image, generating superpixels. Subsequently, a with an autoencoder model is designed to progressively approximate the superpixel segmentation results, and the optimal model is obtained by Bayesian optimization. Through comparative experiments with existing algorithms, it has been demonstrated that the proposed method performance comparable to supervised algorithms, even without the need for labeled data. As a result, the deployment complexity of the algorithm is significantly reduced, while expanding its applicability.

Pixel-level clustering network for unsupervised image segmentation

While image segmentation is crucial in various computer vision applications, such as autonomous driving, grasping, and robot navigation, annotating all objects at the pixel-level for training is nearly impossible. Therefore, the study of unsupervised image segmentation methods is essential. In this paper, we present a pixel-level clustering framework for segmenting images into regions without using ground truth annotations. The proposed framework includes feature embedding modules with an attention mechanism, a feature statistics computing module, image reconstruction, and superpixel segmentation to achieve accurate unsupervised segmentation. Additionally, we propose a training strategy that utilizes intra-consistency within each superpixel, inter-similarity/dissimilarity between neighboring superpixels, and structural similarity between images. To avoid potential over-segmentation caused by superpixel-based losses, we also propose a post-processing method. Furthermore, we present an extension of the proposed method for unsupervised semantic segmentation. We conducted experiments on three publicly available datasets (Berkeley segmentation dataset, PASCAL VOC 2012 dataset, and COCO-Stuff dataset) to demonstrate the effectiveness of the proposed framework. The experimental results show that the proposed framework outperforms previous state-of-the-art methods.

Extracting Semantic Knowledge From GANs With Unsupervised Learning.

Recently, unsupervised learning has made impressive progress on various tasks. Despite the dominance of discriminative models, increasing attention is drawn to representations learned by generative models and in particular, Generative Adversarial Networks (GANs). Previous works on the interpretation of GANs reveal that GANs encode semantics in feature maps in a linearly separable form. In this work, we further find that GAN's features can be well clustered with the linear separability assumption. We propose a novel clustering algorithm, named KLiSH, which leverages the linear separability to cluster GAN's features. KLiSH succeeds in extracting fine-grained semantics of GANs trained on datasets of various objects, e.g., car, portrait, animals, and so on. With KLiSH, we can sample images from GANs along with their segmentation masks and synthesize paired image-segmentation datasets. Using the synthesized datasets, we enable two downstream applications. First, we train semantic segmentation networks on these datasets and test them on real images, realizing unsupervised semantic segmentation. Second, we train image-to-image translation networks on the synthesized datasets, enabling semantic-conditional image synthesis without human annotations.

Pull and concentrate: improving unsupervised semantic segmentation adaptation with cross- and intra-domain consistencies

Pull and concentrate: improving unsupervised semantic segmentation adaptation with cross- and intra-domain consistencies

Large-scale Unsupervised Semantic Segmentation.

Empowered by large datasets, e.g., ImageNet and MS COCO, unsupervised learning on large-scale data has enabled significant advances for classification tasks. However, whether the large-scale unsupervised semantic segmentation can be achieved remains unknown. There are two major challenges: i) we need a large-scale benchmark for assessing algorithms; ii) we need to develop methods to simultaneously learn category and shape representation in an unsupervised manner. In this work, we propose a new problem of large-scale unsupervised semantic segmentation (LUSS) with a newly created benchmark dataset to help the research progress. Building on the ImageNet dataset, we propose the ImageNet-Sdataset with 1.2 million training images and 50k high-quality semantic segmentation annotations for evaluation. Our benchmark has a high data diversity and a clear task objective. We also present a simple yet effective method that works surprisingly well for LUSS. In addition, we benchmark related un/weakly/fully supervised methods accordingly, identifying the challenges and possible directions of LUSS. The benchmark and source code is publicly available at https://github.com/LUSSeg.

Latent space unsupervised semantic segmentation.

The development of compact and energy-efficient wearable sensors has led to an increase in the availability of biosignals. To effectively and efficiently analyze continuously recorded and multidimensional time series at scale, the ability to perform meaningful unsupervised data segmentation is an auspicious target. A common way to achieve this is to identify change-points within the time series as the segmentation basis. However, traditional change-point detection algorithms often come with drawbacks, limiting their real-world applicability. Notably, they generally rely on the complete time series to be available and thus cannot be used for real-time applications. Another common limitation is that they poorly (or cannot) handle the segmentation of multidimensional time series. Consequently, the main contribution of this work is to propose a novel unsupervised segmentation algorithm for multidimensional time series named Latent Space Unsupervised Semantic Segmentation (LS-USS), which was designed to easily work with both online and batch data. Latent Space Unsupervised Semantic Segmentation addresses the challenge of multivariate change-point detection by utilizing an autoencoder to learn a 1-dimensional latent space on which change-point detection is then performed. To address the challenge of real-time time series segmentation, this work introduces the Local Threshold Extraction Algorithm (LTEA) and a "batch collapse" algorithm. The "batch collapse" algorithm enables Latent Space Unsupervised Semantic Segmentation to process streaming data by dividing it into manageable batches, while Local Threshold Extraction Algorithm is employed to detect change-points in the time series whenever the computed metric by Latent Space Unsupervised Semantic Segmentation exceeds a predefined threshold. By using these algorithms in combination, our approach is able to accurately segment time series data in real-time, making it well-suited for applications where timely detection of changes is critical. When evaluating Latent Space Unsupervised Semantic Segmentation on a variety of real-world datasets the Latent Space Unsupervised Semantic Segmentation systematically achieves equal or better performance than other state-of-the-art change-point detection algorithms it is compared to in both offline and real-time settings.

Unsupervised Segmentation of Muscle Precursor Cell Images In Situ

In vitro culture of muscle stem cells on a large scale could bring light to the treatment of muscle-related diseases. However, the current work related to muscle stem cell culture is still only performed in specialized biological laboratories that are very much limited by manual experience. There are still some difficulties to achieve an automated culture of complex morphological cells in terms of live cell observation and morphological analysis. In this paper, a set of bright-field cell in situ imaging devices is designed to perform non-contact and invasive imaging of muscle precursor cells in vitro, and a neural network structured lightweight unsupervised semantic segmentation algorithm is proposed for the acquired images to achieve online extraction of cell regions of interest without manual annotation and pre-training. The algorithm first uses a graph-based super-pixel segmentation to obtain a coarse segmentation, then aggregates the coarse segmentation results with the help of Laplace operators as a reference to a four-layer convolutional neural network (CNN). The CNN parameters learn to refine the boundaries of the cells which helps the final segmentation accuracy and mean intersection–merge ratio reach 88% and 77%, respectively.

Unsupervised defect segmentation with pose priors

AbstractSingle shot, semantic bounding box detectors, trained in a supervised manner are popular in computer vision‐aided visual inspections. These methods have several key limitations: (1) bounding boxes capture too much background, especially when images experience perspective transformation; (2) insufficient domain‐specific data and cost to label; and (3) redundant or incorrect detection results on videos or multi‐frame data; where it is a nontrivial task to select the best detection and check for outliers. Recent developments in commercial augmented reality and robotic hardware can be leveraged to support inspection tasks. A common capability of the previous is the ability to obtain image sequences and camera poses. In this work, the authors leverage pose information as “prior” to address the limitations of existing supervised learned, single‐shot, semantic detectors for the application of visual inspection. The authors propose an unsupervised semantic segmentation method (USP), based on unsupervised learning for image segmentation inspired by differentiable feature clustering coupled with a novel outlier rejection and stochastic consensus mechanism for mask refinement. USP was experimentally validated for a spalling quantification task using a mixed reality headset (Microsoft HoloLens 2). Also, a sensitivity study was conducted to evaluate the performance of USP under environmental or operational variations.

Improving GAN Learning Dynamics for Thyroid Nodule Segmentation.

Thyroid nodules are lesions requiring diagnosis and follow-up. Tools for detecting and segmenting nodules can help physicians with this diagnosis. Besides immediate diagnosis, automated tools can also enable tracking of the probability of malignancy over time. This paper demonstrates a new algorithm for segmenting thyroid nodules in ultrasound images. The algorithm combines traditional supervised semantic segmentation with unsupervised learning using GANs. The hybrid approach has the potential to upgrade the semantic segmentation model's performance, but GANs have the well-known problems of unstable learning and mode collapse. To stabilize the training of the GAN model, we introduce the concept of closed-loop control of the gain on the loss output of the discriminator. We find gain control leads to smoother generator training and avoids the mode collapse that typically occurs when the discriminator learns too quickly relative to the generator. We also find that the combination of the supervised and unsupervised learning styles encourages both low-level accuracy and high-level consistency. As a test of the concept of controlled hybrid supervised and unsupervised semantic segmentation, we introduce a new model named the StableSeg GAN. The model uses DeeplabV3+ as the generator, Resnet18 as the discriminator, and uses PID control to stabilize the GAN learning process. The performance of the new model in terms of IoU is better than DeeplabV3+, with mean IoU of 81.26% on a challenging test set. The results of our thyroid nodule segmentation experiments show that StableSeg GANs have flexibility to segment nodules more accurately than either comparable supervised segmentation models or uncontrolled GANs.

Unsupervised Semantic Segmentation Inpainting Network Using a Generative Adversarial Network with Preprocessing

The generative adversarial neural network has shown a novel result in the image generation area. However, applying it to a semantic segmentation inpainting task exhibits instability due to the different data distribution. To solve this problem, we propose an unsupervised semantic segmentation inpainting method using an adversarial deep neural network with a newly introduced preprocessing method and loss function. For stabilizing the adversarial training for semantic segmentation inpainting, we match the probability distribution of the segmentation maps with the developed preprocessing method. In addition, a new cross-entropy total variation loss for the probability map is introduced to improve the segmentation inpainting work by smoothing the segmentation map. The experimental results demonstrate the proposed algorithm’s effectiveness on both synthetic and real datasets.

A hybrid domain learning framework for unsupervised semantic segmentation

Supervised semantic segmentation often fails to generalize well in unseen scenarios due to the domain gap between the source and the target domains. Unsupervised domain adaptation is one possible way to solve this problem. However, the existing methods suffer two limitations. First, the number limitation of samples may lead to decreasing generalization. Second, only the source dataset contains pixel-level annotations, which provide stronger supervision in the source domain and result in overfitting to the source domain. To tackle these issues, we propose a hybrid domain learning (HDL) framework where the hybrid domain acts as the intermediate domain between the source domain and the target domain. Specifically, we first generate the hybrid domain feature (HDF) by a deep feature interpolation method and discuss the characteristics of the hybrid domain feature. Then, we further design a triple domain strategy to align the distribution of the source domain, the hybrid domain, and the target domain. The experiments in the tasks of GTA5 to Cityscapes and SYNTHIA to Cityscapes demonstrate that the proposed HDL framework is robust to domain adaptation and outperforms the state-of-the-art approaches.

Attention-Based Transformation from Latent Features to Point Clouds

In point cloud generation and completion, previous methods for transforming latent features to point clouds are generally based on fully connected layers (FC-based) or folding operations (Folding-based). However, point clouds generated by FC-based methods are usually troubled by outliers and rough surfaces. For folding-based methods, their data flow is large, convergence speed is slow, and they are also hard to handle the generation of non-smooth surfaces. In this work, we propose AXform, an attention-based method to transform latent features to point clouds. AXform first generates points in an interim space, using a fully connected layer. These interim points are then aggregated to generate the target point cloud. AXform takes both parameter sharing and data flow into account, which makes it has fewer outliers, fewer network parameters, and a faster convergence speed. The points generated by AXform do not have the strong 2-manifold constraint, which improves the generation of non-smooth surfaces. When AXform is expanded to multiple branches for local generations, the centripetal constraint makes it has properties of self-clustering and space consistency, which further enables unsupervised semantic segmentation. We also adopt this scheme and design AXformNet for point cloud completion. Considerable experiments on different datasets show that our methods achieve state-of-the-art results.