Scene Segmentation Research Articles

Improved automated parallel implementation of GMM background subtraction on a multicore digital signal processor

<p><span>Scene segmentation is an essential step in a wide range of video processing applications, for instance, object recognition and tracking. The Gaussian mixture model (GMM) for background subtraction (BS) has gained widespread usage in scene segmentation, despite its known computational intensity. To tackle this challenge, we propose a practical solution to accelerate processing through a parallel implementation on an embedded multicore platform. In this paper, we present an improved automated parallel implementation of the GMM algorithm using the Orphan directive provided by open multiprocessing (OpenMP). Experimental assessments conducted on the eight cores of the C6678 digital signal processor (DSP) demonstrate significant advancements in parallel efficiency, particularly when handling high-resolution frames, including high-definition (HD) and full-HD resolutions. The achieved parallel efficiency surpasses the results obtained with classical OpenMP scheduling modes, encompassing dynamic, static, and guided approaches. Specifically, the parallel efficiency reaches approximately 82% for full-HD resolution frames and, 99.3% for low-resolution frames, respectively.</span></p>

Domain adaptation for semantic segmentation of road scenes via two-stage alignment of traffic elements

Unsupervised domain adaptation has been used to reduce the domain shift, which would improve the performance of semantic segmentation on unlabeled real-world data. However, existing methodologies fall short in effectively addressing the domain shift issue prevalent in traffic scenarios, leading to less than satisfactory segmentation results. In this paper, we propose a novel domain adaptation method for semantic segmentation via unsupervised alignment of traffic elements. Firstly, we introduce a two-stage self-training framework that leverages a blended set of training samples to enhance the training process. In the first stage, we leverage generated mixup training samples as inputs within our two-stage self-training framework and have developed corresponding loss functions for both the source and target domains to direct the training process. Then, the alignment modules for dynamic and static traffic elements are designed to achieve accurate matching between the source and the target domain images. The cosine similarity maximization is applied to the alignment of dynamic traffic elements, while the prototype learning is utilized for the static traffic elements. Additionally, we present a new technique for reducing noise in pseudo labels by constructing thresholds that adjust to each class. Meanwhile, we formulate the associated target domain loss function for vacant pseudo label pixels. The experimental results demonstrate that the proposed method is superior to the existing methods on five different domain adaptation tasks, which is more applicable to semantic segmentation of road scenes.

Towards full autonomous driving: challenges and frontiers

With the rapid advancement of information technology and intelligent systems, autonomous driving has garnered significant attention and research in recent years. Key technologies, such as Simultaneous Localization and Mapping (SLAM), Perception and Localization, and Scene Segmentation, have proven to be essential in this field. These technologies not only evolve independently, each with its own research focus and application paths, but also complement and rely on one another in various complex autonomous driving scenarios. This paper provides a comprehensive review of the development and current state of these technologies, along with a forecast of their future trends.

A point cloud segmentation algorithm based on multi-feature training and weighted random forest

Abstract Point cloud segmentation is the process of dividing point cloud data into a series of coherent subsets according to its attributes. It has been widely used in target recognition, digital protection of cultural relics, medical research and other fields. To improve the classification accuracy of point cloud and achieve accurate segmentation of objects or scenes, a point cloud segmentation algorithm based on multi–features training and weighted random forest (RF) is proposed. Firstly, the feature vector composed of 3D coordinate value, RGB value, echo intensity, point cloud density, normal direction and average curvature is used to train the SVM classifier, and the ‘one–to–one’ strategy is adopted to achieve the initial multivariate rough segmentation of point cloud. Then, the maximum information coefficient and sample correlation coefficient (SCC) are used to evaluate the correlation of the decision tree, and the decision tree is weighted accordingly to build a weak correlation weighted RF, so as to achieve further accurate segmentation of the point cloud. The experiment verifies the effectiveness of the proposed algorithm by segmenting the outdoor scene point cloud data model. The results show that the segmentation algorithm based on multi–features training and weighted RF can achieve accurate point cloud segmentation, and is an effective point cloud segmentation method.

Mesh Convolution With Continuous Filters for 3-D Surface Parsing.

Geometric feature learning for 3-D surfaces is critical for many applications in computer graphics and 3-D vision. However, deep learning currently lags in hierarchical modeling of 3-D surfaces due to the lack of required operations and/or their efficient implementations. In this article, we propose a series of modular operations for effective geometric feature learning from 3-D triangle meshes. These operations include novel mesh convolutions, efficient mesh decimation, and associated mesh (un)poolings. Our mesh convolutions exploit spherical harmonics as orthonormal bases to create continuous convolutional filters. The mesh decimation module is graphics processing unit (GPU)-accelerated and able to process batched meshes on-the-fly, while the (un)pooling operations compute features for upsampled/downsampled meshes. We provide an open-source implementation of these operations, collectively termed Picasso. Picasso supports heterogeneous mesh batching and processing. Leveraging its modular operations, we further contribute a novel hierarchical neural network for perceptual parsing of 3-D surfaces, named PicassoNet++. It achieves highly competitive performance for shape analysis and scene segmentation on prominent 3-D benchmarks. The code, data, and trained models are available at https://github.com/EnyaHermite/Picasso.

A Patch Diversity Transformer for Domain Generalized Semantic Segmentation.

Domain generalization (DG) is one of the critical issues for deep learning in unknown domains. How to effectively represent domain-invariant context (DIC) is a difficult problem that DG needs to solve. Transformers have shown the potential to learn generalized features, since the powerful ability to learn global context. In this article, a novel method named patch diversity Transformer (PDTrans) is proposed to improve the DG for scene segmentation by learning global multidomain semantic relations. Specifically, patch photometric perturbation (PPP) is proposed to improve the representation of multidomain in the global context information, which helps the Transformer learn the relationship between multiple domains. Besides, patch statistics perturbation (PSP) is proposed to model the feature statistics of patches under different domain shifts, which enables the model to encode domain-invariant semantic features and improve generalization. PPP and PSP can help to diversify the source domain at the patch level and feature level. PDTrans learns context across diverse patches and takes advantage of self-attention to improve DG. Extensive experiments demonstrate the tremendous performance advantages of the PDTrans over state-of-the-art DG methods.

CAFA: Cross-Modal Attentive Feature Alignment for Cross-Domain Urban Scene Segmentation

CAFA: Cross-Modal Attentive Feature Alignment for Cross-Domain Urban Scene Segmentation

MSF2Net: multi-stage feature fusion network for real-time semantic segmentation in road scenes

MSF2Net: multi-stage feature fusion network for real-time semantic segmentation in road scenes

MRAM: Multi-scale Regional Attribute-weighting via Meta-learning for Personalized Image Aesthetics Assessment

Image aesthetics assessment (IAA) is inherently subjective, with generic image aesthetics assessment (GIAA) often overlooking individual user preferences. Personalized aesthetic assessment (PIAA) addresses this by accounting for user-specific inconsistencies. However, the limited annotated data from individual users and the scarcity of user-jointly annotated image data pose challenges. To cope with these problems, we propose a multi-scale regional attribute-weighting via meta-learning for PIAA, namely MRAM, which aims to simulate how the brain computes aesthetic values from visual stimuli. The human brain is to predict the subjective aesthetic value of the individual to the image by combining the features of the visual scene segmentation through the weighting method. MRAM introduces a multi-scale attention module to capture visual scene regions segmented at different scales, adjusting feature selection based on user attention. Additionally, a regional attribute-weighting module (RAM) is proposed to weigh these features using aesthetic attribute representations. In the meta-training phase, MRAM learns aesthetic prior knowledge from a diverse set of PIAA tasks. Transitioning to the personalization phase, MRAM is fine-tuned with only a small number of specific user samples, enabling it to rapidly adapt to the unique aesthetic preferences of individuals. Experimental results demonstrate the superior performance of the proposed MRAM compared to state-of-the-art PIAA methods.

Understanding the time course and spatial biases of natural scene segmentation

Understanding the time course and spatial biases of natural scene segmentation

Nighttime scene understanding with label transfer scene parser

Semantic segmentation plays a crucial role in traffic scene understanding, especially in nighttime conditions. This paper tackles the task of semantic segmentation in nighttime scenes. The largest challenge of this task is the lack of annotated nighttime images to train a deep learning-based scene parser. The existing annotated datasets are abundant in daytime conditions but scarce in nighttime due to the high cost. Thus, we propose a novel Label Transfer Scene Parser (LTSP) framework for nighttime scene semantic segmentation by leveraging daytime annotation transfer. Our framework performs segmentation in the dark without training on real nighttime annotated data. In particular, we propose translating daytime images to nighttime conditions to obtain more data with annotation in an efficient way. In addition, we utilize the pseudo-labels inferred from unlabeled nighttime scenes to further train the scene parser. The novelty of our work is the ability to perform nighttime segmentation via daytime annotated labels and nighttime synthetic versions of the same set of images. The extensive experiments demonstrate the improvement and efficiency of our scene parser over the state-of-the-art methods with a similar semi-supervised approach on the benchmark of Nighttime Driving Test dataset. Notably, our proposed method utilizes only one-tenth of the amount of labeled and unlabeled data in comparison with the previous methods. Code is available at https://github.com/danhntd/Label_Transfer_Scene_Parser.git.

Multi-scale deep learning and clustering-based tabletop object instance segmentation for robot manipulation

3D object instance segmentation plays a vital role in various applications such as autonomous driving, robotics and virtual reality. However, tabletop scenes exhibit diverse object complexities and size variations. The challenge is to enhance the accuracy of segmenting these scenes for multiple object instances. This limitation directly impacts robots’ capabilities to effectively grasp and manipulate objects. In this paper, we propose a multi-scale deep learning and clustering-based approach for object instance segmentation in tabletop scenes. Our approach incorporates a multi-scale neighborhood feature sampling (MNFS) module specifically designed to extract local features, and a clustering algorithm to eliminate noise and preserve instance integrity. Furthermore, we combine the strength of both methods through ScoreNet and non-maximal suppression. We conducted extensive experiments on TO-Scene, the first large-scale dataset of 3D tabletop scenes, and observed an average mIoU improvement of approximately 4.07% compared to existing methods. This highlights the superior performance of our proposed method. In addition, we tested our algorithm on a real-scene robotics platform and showed that it has good performance and generalization capabilities to support future applications such as robot grasping.

Semantic Segmentation of Point Cloud Scene via Multi-Scale Feature Aggregation and Adaptive Fusion

Point cloud semantic segmentation is a key step in 3D scene understanding and analysis. In recent years, deep learning–based point cloud semantic segmentation methods have received extensive attention from researchers. Multi-scale neighborhood feature learning methods are suitable for inhomogeneous density point clouds, but different scale branching feature learning increases the computational complexity and makes it difficult to accurately fuse different scale features to express local information. In this study, a point cloud semantic segmentation network based on RandLA-Net with multi-scale local feature aggregation and adaptive fusion is proposed. The designed structure can reduce computational complexity and accurately express local features. The mean intersection-over-union is improved by 1.1% on the SemanticKITTI data set with an inference speed of nine frames per second, while the mean intersection-over-union is improved by 0.9% on the S3DIS data set, compared with RandLA-Net. We also conduct ablation studies to validate the effectiveness of the proposed key structure.

DIPO: Differentiable Parallel Operation Blocks for Surgical Neural Architecture Search.

Deep learning has been used across a large number of computer vision tasks, however designing the network architectures for each task is time consuming. Neural Architecture Search (NAS) promises to automatically build neural networks, optimised for the given task and dataset. However, most NAS methods are constrained to a specific macro-architecture design which makes it hard to apply to different tasks (classification, detection, segmentation). Following the work in Differentiable NAS (DNAS), we present a simple and efficient NAS method, Differentiable Parallel Operation (DIPO), that constructs a local search space in the form of a DIPO block, and can easily be applied to any convolutional network by injecting it in-place of the convolutions. The DIPO block's internal architecture and parameters are automatically optimised end-to-end for each task. We demonstrate the flexibility of our approach by applying DIPO to 4 model architectures (U-Net, HRNET, KAPAO and YOLOX) across different surgical tasks (surgical scene segmentation, surgical instrument detection, and surgical instrument pose estimation) and evaluated across 5 datasets. Results show significant improvements in surgical scene segmentation (+10.5% in CholecSeg8K, +13.2% in CaDIS), instrument detection (+1.5% in ROBUST-MIS, +5.3% in RoboKP), and instrument pose estimation (+9.8% in RoboKP).

VPA-Net: A visual perception assistance network for 3d lidar semantic segmentation

The semantic segmentation of 3D point clouds holds paramount importance in visual perception tasks of automatic driving, including obstacle avoidance, decision control, path planning, and map construction, etc. Multi-sensor fusion is a rational and pivotal technique for implementing LiDAR semantic segmentation. However, effectively fusing and utilizing multi-source data remains a challenging task. In this work, we introduce a novel visual perception-assisted point cloud segmentation network, termed VPA-Net. This network architecture employs a dual-branch design to effectively combine spatial information from point clouds and visual cues from images, thereby bolstering the performance of 3D LiDAR semantic segmentation. More specifically, the dual-branch network structure processes and fuses multi-modal data from both point clouds and RGB images. Subsequently, the intermediate features from the two branches are merged via the proposed attention-based feature fusion module. Furthermore, to address the challenge of precise boundary prediction in large-scale point cloud scene segmentation, we introduce a refinement module based on 3D sparse convolution to enhance the spatial information of the LiDAR point clouds. The effectiveness of our method is validated on SemanticKITTI and a more challenging 3D semantic segmentation dataset. Experimental results underscore significant improvement on SemanticKITTI, with our approach surpassing the state-of-the-art method, achieving a 7.1% higher mIoU than SalsaNext.

MA-SAM: Modality-agnostic SAM adaptation for 3D medical image segmentation

The Segment Anything Model (SAM), a foundation model for general image segmentation, has demonstrated impressive zero-shot performance across numerous natural image segmentation tasks. However, SAM’s performance significantly declines when applied to medical images, primarily due to the substantial disparity between natural and medical image domains. To effectively adapt SAM to medical images, it is important to incorporate critical third-dimensional information, i.e., volumetric or temporal knowledge, during fine-tuning. Simultaneously, we aim to harness SAM’s pre-trained weights within its original 2D backbone to the fullest extent. In this paper, we introduce a modality-agnostic SAM adaptation framework, named as MA-SAM, that is applicable to various volumetric and video medical data. Our method roots in the parameter-efficient fine-tuning strategy to update only a small portion of weight increments while preserving the majority of SAM’s pre-trained weights. By injecting a series of 3D adapters into the transformer blocks of the image encoder, our method enables the pre-trained 2D backbone to extract third-dimensional information from input data. We comprehensively evaluate our method on five medical image segmentation tasks, by using 11 public datasets across CT, MRI, and surgical video data. Remarkably, without using any prompt, our method consistently outperforms various state-of-the-art 3D approaches, surpassing nnU-Net by 0.9%, 2.6%, and 9.9% in Dice for CT multi-organ segmentation, MRI prostate segmentation, and surgical scene segmentation respectively. Our model also demonstrates strong generalization, and excels in challenging tumor segmentation when prompts are used. Our code is available at: https://github.com/cchen-cc/MA-SAM.

Altered Use of Context During Visual Perception in Psychotic Psychopathology: A Neurophysiological Investigation of Tuned and Untuned Suppression During Contrast Perception.

The human visual system streamlines visual processing by suppressing responses to textures that are similar to their surrounding context. Surround suppression is weaker in individuals with schizophrenia (ISZ); this altered use of visuospatial context may relate to the characteristic visual distortions they experience. To understand atypical surround suppression in psychotic psychopathology, we investigated neurophysiological responses in ISZ, healthy controls (HC), individuals with bipolar disorder (IBP), and first-degree relatives (ISZR/IBPR). Participants performed a contrast judgment task on a circular target with annular surrounds, with concurrent electroencephalography. Orientation-independent (untuned) suppression was estimated from responses to central targets with orthogonal surrounds; the orientation-dependence of suppression was estimated by fitting an exponential function to the increase in suppression as surrounds became more aligned with the center. ISZ exhibited weakened untuned suppression coupled with enhanced orientation-dependence of suppression. The N1 visual evoked potential was associated with the orientation-dependence of suppression, with ISZ and ISZR (but not IBP or IBPR) showing enhanced orientation-dependence of the N1. Collapsed across orientation conditions, the N1 for ISZ lacked asymmetry toward the right hemisphere; this reduction in N1 asymmetry was associated with reduced untuned suppression, real-world perceptual anomalies, and psychotic psychopathology. The overall amplitude of the N1 was reduced in ISZ and IBP. Key measures of symptomatology for ISZ are associated with reductions in untuned suppression. Increased sensitivity for ISZ to the relative orientation of suppressive surrounds is reflected in the N1 VEP, which is commonly associated with higher-level visual functions such as allocation of spatial attention or scene segmentation.

HRDLNet: a semantic segmentation network with high resolution representation for urban street view images

Semantic segmentation of urban street scenes has attracted much attention in the field of autonomous driving, which not only helps vehicles perceive the environment in real time, but also significantly improves the decision-making ability of autonomous driving systems. However, most of the current methods based on Convolutional Neural Network (CNN) mainly use coding the input image to a low resolution and then try to recover the high resolution, which leads to problems such as loss of spatial information, accumulation of errors, and difficulty in dealing with large-scale changes. To address these problems, in this paper, we propose a new semantic segmentation network (HRDLNet) for urban street scene images with high-resolution representation, which improves the accuracy of segmentation by always maintaining a high-resolution representation of the image. Specifically, we propose a feature extraction module (FHR) with high-resolution representation, which efficiently handles multi-scale targets and high-resolution image information by efficiently fusing high-resolution information and multi-scale features. Secondly, we design a multi-scale feature extraction enhancement (MFE) module, which significantly expands the sensory field of the network, thus enhancing the ability to capture correlations between image details and global contextual information. In addition, we introduce a dual-attention mechanism module (CSD), which dynamically adjusts the network to more accurately capture subtle features and rich semantic information in images. We trained and evaluated HRDLNet on the Cityscapes Dataset and the PASCAL VOC 2012 Augmented Dataset, and verified the model’s excellent performance in the field of urban streetscape image segmentation. The unique advantages of our proposed HRDLNet in the field of semantic segmentation of urban streetscapes are also verified by comparing it with the state-of-the-art methods.

Point cloud semantic segmentation of grotto scenes using the knowledge-guided deep learning method

ABSTRACT In recent years, the deep learning-based semantic segmentation for point clouds has demonstrated remarkable capabilities in processing 3D urban scenes for applications such as three-dimensional reconstruction, semantic modeling, and augmented reality. However, research on grottoes scenes is very limited. It is currently unclear how existing neural architectures for point cloud semantic segmentation perform in grotto scenes, and how to effectively incorporate the unique characteristics of grotto scenes to enhance the performance of deep neural networks. This study proposed a method for point cloud semantic segmentation of grotto scenes, combining knowledge with deep learning approaches. The method adopted knowledge to guide the creation of benchmark datasets, the design of a neural network called GSS-Net, and the correction of segmentation errors in the results of deep learning. The results show that the proposed method outperforms four existing mainstream models without the correction of segmentation results. Moreover, a set of ablation studies verified the effectiveness of each proposed module. This method not only improves the accuracy of point cloud semantic segmentation in grotto scenes but also enhances the interpretability of network designs. It provides new insights into the application of knowledge-guided deep learning models in grotto scenes.

Gated feature aggregate and alignment network for real-time semantic segmentation of street scenes

Gated feature aggregate and alignment network for real-time semantic segmentation of street scenes