Scene Segmentation Research Articles

MOVES: Movable and moving LiDAR scene segmentation in label-free settings using static reconstruction

Accurate static structure reconstruction and segmentation of non-stationary objects is of vital importance for autonomous navigation applications. These applications assume a LiDAR scan to consist of only static structures. In the real world however, LiDAR scans consist of non-stationary dynamic structures — moving and movable objects. Current solutions use segmentation information to isolate and remove moving structures from LiDAR scan. This strategy fails in several important use-cases where segmentation information is not available. In such scenarios, moving objects and objects with high uncertainty in their motion i.e. movable objects, may escape detection. This violates the above assumption. We present MOVES, a novel GAN based adversarial model that segments out moving as well as movable objects in the absence of segmentation information. We achieve this by accurately transforming a dynamic LiDAR scan to its corresponding static scan. This is obtained by replacing dynamic objects and corresponding occlusions with static structures which were occluded by dynamic objects. We leverage corresponding static-dynamic LiDAR pairs. We design a novel discriminator, coupled with a contrastive loss on a smartly selected LiDAR scan triplet. For datasets lacking paired information, we propose MOVES-MMD that integrates Unsupervised Domain Adaptation into the network. We perform rigorous experiments to demonstrate state of the art dynamic to static translation performance on a sparse real world industrial dataset, an urban and a simulated dataset. MOVES also segments out movable and moving objects without using segmentation information. Without utilizing segmentation labels, MOVES performs better than segmentation based navigation baseline in highly dynamic and long LiDAR sequences. The code is available here.

Point-State Segmentation for Dynamic Scenes via Probability Model and Orientation Consistency

Point-State Segmentation for Dynamic Scenes via Probability Model and Orientation Consistency

Problems and related results in algebraic vision and multiview geometry

This article is a survey of results in algebraic vision and multiview geometry. The starting point is the study of projective algebraic varieties which are critical for scene reconstruction. Initially studied for reconstructing static scenes in three-dimensional spaces, these critical loci are later investigated for dynamic and segmented scenes in higher-dimensional projective spaces. The formal analysis of the ideals of critical loci employs Grassmann tensors, introduced as crucial tools for determining these ideals and aiding the reconstruction process away from critical loci. A long-term goal of the authors with other co-authors involves two main aspects: firstly studying properties of Grassmann tensors, as rank, multi-rank and core, along with the varieties that parameterize these tensors; concurrently conducting an analysis of families of critical loci in various scenarios.

BSNet: A bilateral real-time semantic segmentation network based on multi-scale receptive fields

The rise of autonomous driving and mobile robots has drawn attention to real-time road scene segmentation. However, category confusion, incomplete segmentation and inaccurate detail contours are common issues encountered in road traffic scene segmentation tasks. To tackle these road segmentation challenges, we proposes a bilateral real-time semantic segmentation network based on multi-scale receptive fields(BSNet). To enhance the segmentation accuracy of detail contours, an auxiliary detail detection module is introduced in the spatial branch. In the semantic branch, strip pooling and channel attention are added to the short-term dense concatenate module to enrich multi-scale receptive fields and strengthen channel features, addressing issues such as incomplete segmentation and category confusion. The final design of the complementary dual-guided fusion module improves feature fusion effectiveness. Experimental results on two benchmark datasets demonstrate that BSNet significantly improves these road segmentation difficulties and achieves competitive results in both speed and accuracy compared to the current state-of-the-art methods.

Real-time segmentation of short videos under VR technology in dynamic scenes

Abstract This work addresses the challenges of scene segmentation and low segmentation accuracy in short videos by employing virtual reality (VR) technology alongside a 3D DenseNet model for real-time segmentation in dynamic scenes. First, this work extracted short videos by frame and removed redundant background information. Then, the volume rendering algorithm in VR technology was used to reconstruct short videos in dynamic scenes in 3D. It enriched the detailed information of short videos, and finally used the 3D DenseNet model for real-time segmentation of short videos in dynamic scenes, improving the accuracy of segmentation. The experiment compared the performance of High resolution network, Mask region based convolutional neural network, 3D U-Net, Efficient neural network models on the Densely annotation video segmentation dataset. The experimental results showed that the segmentation accuracy of the 3D DenseNet model has reached 99.03%, which was 15.11% higher than that of the ENet model. The precision rate reached 98.33%, and the average segmentation time reached 0.64 s, improving the segmentation accuracy and precision rate. It can adapt to various scene situations and has strong robustness. The significance of this research lies in its innovative approach in tackling these issues. By integrating VR technology with advanced deep learning models, we can achieve more precise segmentation of dynamic scenes in short videos, enabling real-time processing. This has significant practical implications for fields such as video editing, VR applications, and intelligent surveillance. Furthermore, the outcomes of this research contribute to advancing computer vision in video processing, providing valuable insights for the development of future intelligent video processing systems.

Surgical optomics: hyperspectral imaging and deep learning towards precision intraoperative automatic tissue recognition-results from the EX-MACHYNA trial.

Hyperspectral imaging (HSI), combined with machine learning, can help to identify characteristic tissue signatures enabling automatic tissue recognition during surgery. This study aims to develop the first HSI-based automatic abdominal tissue recognition with human data in a prospective bi-center setting. Data were collected from patients undergoing elective open abdominal surgery at two international tertiary referral hospitals from September 2020 to June 2021. HS images were captured at various time points throughout the surgical procedure. Resulting RGB images were annotated with 13 distinct organ labels. Convolutional Neural Networks (CNNs) were employed for the analysis, with both external and internal validation settings utilized. A total of 169 patients were included, 73 (43.2%) from Strasbourg and 96 (56.8%) from Verona. The internal validation within centers combined patients from both centers into a single cohort, randomly allocated to the training (127 patients, 75.1%, 585 images) and test sets (42 patients, 24.9%, 181 images). This validation setting showed the best performance. The highest true positive rate was achieved for the skin (100%) and the liver (97%). Misclassifications included tissues with a similar embryological origin (omentum and mesentery: 32%) or with overlaying boundaries (liver and hepatic ligament: 22%). The median DICE score for ten tissue classes exceeded 80%. To improve automatic surgical scene segmentation and to drive clinical translation, multicenter accurate HSI datasets are essential, but further work is needed to quantify the clinical value of HSI. HSI might be included in a new omics science, namely surgical optomics, which uses light to extract quantifiable tissue features during surgery.

Research on Efficient Asymmetric Attention Module for Real-Time Semantic Segmentation Networks in Urban Scenes

Currently, numerous high-precision models have been proposed for semantic segmentation, but the model parameters are large and the segmentation speed is slow. Real-time semantic segmentation for urban scenes necessitates a balance between accuracy, inference speed, and model size. In this paper, we present an efficient solution to this challenge, efficient asymmetric attention module net (EAAMNet) for the semantic segmentation of urban scenes, which adopts an asymmetric encoder–decoder structure. The encoder part of the network utilizes an efficient asymmetric attention module to form the network backbone. In the decoding part, we propose a lightweight multi-feature fusion decoder that can maintain good segmentation accuracy with a small number of parameters. Our extensive evaluations demonstrate that EAAMNet achieves a favorable equilibrium between segmentation efficiency, model parameters, and segmentation accuracy, rendering it highly suitable for real-time semantic segmentation in urban scenes. Remarkably, EAAMNet attains a 73.31% mIoU at 128 fps on Cityscapes and a 69.32% mIoU at 141 fps on CamVid without any pre-training. Compared to state-of-the-art models, our approach not only matches their model parameters but also enhances accuracy and increases speed.

Temporal Scene Montage for Self-Supervised Video Scene Boundary Detection

Once a video sequence is organized as basic shot units, it is of great interest to temporally link shots into semantic-compact scene segments to facilitate long video understanding. However, it still challenges existing video scene boundary detection methods to handle various visual semantics and complex shot relations in video scenes. We proposed a novel self-supervised learning method, Video Scene Montage for Boundary Detection (VSMBD), to extract rich shot semantics and learn shot relations using unlabeled videos. More specifically, we present Video Scene Montage (VSM) to synthesize reliable pseudo scene boundaries, which learns task-related semantic relations between shots in a self-supervised manner. To lay a solid foundation for modeling semantic relations between shots, we decouple visual semantics of shots into foreground and background. Instead of costly learning from scratch as in most previous self-supervised learning methods, we build our model upon large-scale pre-trained visual encoders to extract the foreground and background features. Experimental results demonstrate VSMBD trains a model with strong capability in capturing shot relations, surpassing previous methods by significant margins. The code is available at https://github.com/mini-mind/VSMBD.

Semantic scene segmentation for indoor autonomous vision systems: leveraging an enhanced and efficient U-NET architecture

Semantic scene segmentation for indoor autonomous vision systems: leveraging an enhanced and efficient U-NET architecture

RGB‐D road segmentation based on cross‐modality feature maintenance and encouragement

AbstractDeep images can provide rich spatial structure information, which can effectively exclude the interference of illumination and road texture in road scene segmentation and make better use of the prior knowledge of road area. This paper first proposes a new cross‐modal feature maintenance and encouragement network. It includes a quantization statistics module as well as a maintenance and encouragement module for effective fusion between multimodal data. Meanwhile, for the problem that if the road segmentation is performed directly using a segmentation network, there will be a lack of supervised guidance with clear physical meaningful information and poor interpretability of learning features, this paper proposes two road segmentation models based on prior knowledge of deep image: disparity information and surface normal vector information. Then, a two‐branch neural network is used to process the colour image and the processed depth image separately, to achieve the full utilization of the complementary features of the two modalities. The experimental results on the KITTI road dataset and Cityscapes dataset show that the method in this paper has good road segmentation performance and high computational efficiency.

Dynamic Object Recognition in Video Streams

Dynamic Object Recognition in Video Streams, essen- tial for advancements in video surveillance, autonomous systems, and robotics. Focusing on challenging scenarios like crowded- ness and rapid scene changes, the study introduces novel ob- ject detection and tracking techniques. Object segmentation in crowded scenes and trajectory analysis for simultaneous tracking in dense environments are explored, considering both immediate interactions and long-term movements. Adaptive object recognition algorithms, integrating event detection for rapid scene transitions, are investigated for their responsiveness. Additionally, the study incorporates advanced deep learning frameworks like Temporal Convolutional Networks (TCNs) and one-shot learning methods, enhancing the system’s adaptability to dynamic environments. Real- time processing efficiency is addressed through hardware accelera- tion and online learning strategies, contributing to the development of more efficient and adaptive dynamic object recognition systems for crowded and dynamic scenes. Index Terms—Computer Vision, Object recognition, Video, TCN, Robotics

One model to use them all: training a segmentation model with complementary datasets

PurposeUnderstanding surgical scenes is crucial for computer-assisted surgery systems to provide intelligent assistance functionality. One way of achieving this is via scene segmentation using machine learning (ML). However, such ML models require large amounts of annotated training data, containing examples of all relevant object classes, which are rarely available. In this work, we propose a method to combine multiple partially annotated datasets, providing complementary annotations, into one model, enabling better scene segmentation and the use of multiple readily available datasets.MethodsOur method aims to combine available data with complementary labels by leveraging mutual exclusive properties to maximize information. Specifically, we propose to use positive annotations of other classes as negative samples and to exclude background pixels of these binary annotations, as we cannot tell if a positive prediction by the model is correct.ResultsWe evaluate our method by training a DeepLabV3 model on the publicly available Dresden Surgical Anatomy Dataset, which provides multiple subsets of binary segmented anatomical structures. Our approach successfully combines 6 classes into one model, significantly increasing the overall Dice Score by 4.4% compared to an ensemble of models trained on the classes individually. By including information on multiple classes, we were able to reduce the confusion between classes, e.g. a 24% drop for stomach and colon.ConclusionBy leveraging multiple datasets and applying mutual exclusion constraints, we developed a method that improves surgical scene segmentation performance without the need for fully annotated datasets. Our results demonstrate the feasibility of training a model on multiple complementary datasets. This paves the way for future work further alleviating the need for one specialized large, fully segmented dataset but instead the use of already existing datasets.

Semantic segmentation of large-scale point cloud scenes via dual neighborhood feature and global spatial-aware

As a core task in 3D scene information extraction, point cloud semantic segmentation is crucial for understanding 3D scenes and environmental perception. While extracting local geometric structural features from point clouds, existing research often overlooks the long-range dependencies present in the scene, making it challenging to fully uncover the long-range contextual features hidden within point clouds. On this basis, we propose a segmentation algorithm (DG-Net) that integrates dual neighborhood features with global spatial-aware. Initially, the local structure information encoding module is designed to learn about local geometric shapes by encoding spatial position and directional features, thus supplementing structural information. Subsequently, a dual neighborhood features complementary module is introduced to merge the geometric structural and semantic features within local neighborhoods, learning local dependencies and capturing distinguishable local contextual features. Finally, these features are relayed to a global spatial-aware module equipped with a gated unit, which dynamically adjusts the weights of features at different stages, effectively modeling long-range dependencies between local structures and finely extracting long-range contextual features. We conducted experiments on benchmark datasets of point cloud scenes, and both quantitative and qualitative results demonstrate that our algorithm can accurately identify small-scale objects with complex geometric structures within scenes, surpassing other mainstream networks in segmentation performance. The mIoU on the S3DIS, Toronto3D, and SensatUrban datasets are 71.9 %, 82.1 %, and 59.8 %, respectively.

Research on trajectory control of multi‐degree‐of‐freedom industrial robot based on visual image

AbstractIn order to improve the trajectory control effect of multi‐degree‐of‐freedom industrial robots, this paper combines visual image technology to conduct research on trajectory control of multi‐degree‐of‐freedom industrial robots. Aiming at the problem of video segmentation under sudden illumination changes, this paper uses a Gaussian mixture model based on the global illumination function to adopt a variety of illumination invariant features, and proposes a scene segmentation algorithm suitable for sudden illumination changes. Moreover, this paper compares and verifies the algorithm from the subjective and objective perspectives through experiments, which shows that the algorithm in this paper can segment the scene more accurately even in the environment of sudden changes in illumination. In addition, the results of the accuracy test and the trajectory control test show that the research method of the multi‐degree‐of‐freedom industrial robot trajectory control based on the visual image proposed in this paper can effectively improve the trajectory control effect of the robot.

Multisensor Fusion Network for Unstructured Scene Segmentation With Surface Normal Incorporated

Multisensor Fusion Network for Unstructured Scene Segmentation With Surface Normal Incorporated

Cataract-1K Dataset for Deep-Learning-Assisted Analysis of Cataract Surgery Videos

In recent years, the landscape of computer-assisted interventions and post-operative surgical video analysis has been dramatically reshaped by deep-learning techniques, resulting in significant advancements in surgeons’ skills, operation room management, and overall surgical outcomes. However, the progression of deep-learning-powered surgical technologies is profoundly reliant on large-scale datasets and annotations. In particular, surgical scene understanding and phase recognition stand as pivotal pillars within the realm of computer-assisted surgery and post-operative assessment of cataract surgery videos. In this context, we present the largest cataract surgery video dataset that addresses diverse requisites for constructing computerized surgical workflow analysis and detecting post-operative irregularities in cataract surgery. We validate the quality of annotations by benchmarking the performance of several state-of-the-art neural network architectures for phase recognition and surgical scene segmentation. Besides, we initiate the research on domain adaptation for instrument segmentation in cataract surgery by evaluating cross-domain instrument segmentation performance in cataract surgery videos. The dataset and annotations are publicly available in Synapse.

Alignment and fusion for adaptive domain nighttime semantic segmentation

In the field of autonomous driving technology, both daytime and nighttime scenes are common. However, due to the poor illumination and difficulty in manual annotation of nighttime images, semantic segmentation of nighttime scenes is more challenging compared to daytime scenes. Therefore, achieving significant progress in nighttime semantic segmentation would greatly enhance the effectiveness of the application of autonomous driving scenarios. In this work, we investigate the problem of domain-adaptive nighttime semantic segmentation (DANSS). The problem aims to learn semantic segmentation in nighttime scenes by leveraging a labeled Cityscapes dataset and unlabeled but roughly aligned day-night image pairs. To address this, we propose an Align and Fusion Network (AAFnet), a network for adaptive domain nighttime semantic segmentation. AAFnet utilizes a novel DAFormer as the backbone to separately compute features for dynamic and static objects during the training process. It also employs methods such as small category sampling and image blending to improve learning effectiveness. Additionally, we propose utilizing local image patches to enhance the results during the testing process. Experimental results demonstrate a significant improvement of 10.4% compared to the previous method, DANNet. Extensive experiments conducted on the Dark Zurich dataset and Nightdriving dataset validate the effectiveness of the proposed approach. Our method outperforms previous backbones and achieves top-ranking results.

Research on multitask model of object detection and road segmentation in unstructured road scenes

With the rapid development of artificial intelligence and computer vision technology, autonomous driving technology has become a hot area of concern. The driving scenarios of autonomous vehicles can be divided into structured scenarios and unstructured scenarios. Compared with structured scenes, unstructured road scenes lack the constraints of lane lines and traffic rules, and the safety awareness of traffic participants is weaker. Therefore, there are new and higher requirements for the environment perception tasks of autonomous vehicles in unstructured road scenes. The current research rarely integrates the target detection and road segmentation to achieve the simultaneous processing of target detection and road segmentation of autonomous vehicle in unstructured road scenes. Aiming at the above issues, a multitask model for object detection and road segmentation in unstructured road scenes is proposed. Through the sharing and fusion of the object detection model and road segmentation model, multitask model can complete the tasks of multi-object detection and road segmentation in unstructured road scenes while inputting a picture. Firstly, MobileNetV2 is used to replace the backbone network of YOLOv5, and multi-scale feature fusion is used to realize the information exchange layer between different features. Subsequently, a road segmentation model was designed based on the DeepLabV3+ algorithm. Its main feature is that it uses MobileNetV2 as the backbone network and combines the binary classification focus loss function for network optimization. Then, we fused the object detection algorithm and road segmentation algorithm based on the shared MobileNetV2 network to obtain a multitask model and trained it on both the public dataset and the self-built dataset NJFU. The training results demonstrate that the multitask model significantly enhances the algorithm’s execution speed by approximately 10 frames per scond while maintaining the accuracy of object detection and road segmentation. Finally, we conducted validation of the multitask model on an actual vehicle.

Occluded Multi-Human Target Segmentation in Dynamic Indoor Scenes Based on Thermopile Array Sensor

Occluded Multi-Human Target Segmentation in Dynamic Indoor Scenes Based on Thermopile Array Sensor

LSKANet: Long Strip Kernel Attention Network for Robotic Surgical Scene Segmentation.

Surgical scene segmentation is a critical task in Robotic-assisted surgery. However, the complexity of the surgical scene, which mainly includes local feature similarity (e.g., between different anatomical tissues), intraoperative complex artifacts, and indistinguishable boundaries, poses significant challenges to accurate segmentation. To tackle these problems, we propose the Long Strip Kernel Attention network (LSKANet), including two well-designed modules named Dual-block Large Kernel Attention module (DLKA) and Multiscale Affinity Feature Fusion module (MAFF), which can implement precise segmentation of surgical images. Specifically, by introducing strip convolutions with different topologies (cascaded and parallel) in two blocks and a large kernel design, DLKA can make full use of region- and strip-like surgical features and extract both visual and structural information to reduce the false segmentation caused by local feature similarity. In MAFF, affinity matrices calculated from multiscale feature maps are applied as feature fusion weights, which helps to address the interference of artifacts by suppressing the activations of irrelevant regions. Besides, the hybrid loss with Boundary Guided Head (BGH) is proposed to help the network segment indistinguishable boundaries effectively. We evaluate the proposed LSKANet on three datasets with different surgical scenes. The experimental results show that our method achieves new state-of-the-art results on all three datasets with improvements of 2.6%, 1.4%, and 3.4% mIoU, respectively. Furthermore, our method is compatible with different backbones and can significantly increase their segmentation accuracy. Code is available at https://github.com/YubinHan73/LSKANet.