Scene Understanding Research Articles

TAMC: Textual Alignment and Masked Consistency for Open-Vocabulary 3D Scene Understanding.

Three-dimensional (3D) Scene Understanding achieves environmental perception by extracting and analyzing point cloud data with wide applications including virtual reality, robotics, etc. Previous methods align the 2D image feature from a pre-trained CLIP model and the 3D point cloud feature for the open vocabulary scene understanding ability. We believe that existing methods have the following two deficiencies: (1) the 3D feature extraction process ignores the challenges of real scenarios, i.e., point cloud data are very sparse and even incomplete; (2) the training stage lacks direct text supervision, leading to inconsistency with the inference stage. To address the first issue, we employ a Masked Consistency training policy. Specifically, during the alignment of 3D and 2D features, we mask some 3D features to force the model to understand the entire scene using only partial 3D features. For the second issue, we generate pseudo-text labels and align them with the 3D features during the training process. In particular, we first generate a description for each 2D image belonging to the same 3D scene and then use a summarization model to fuse these descriptions into a single description of the scene. Subsequently, we align 2D-3D features and 3D-text features simultaneously during training. Massive experiments demonstrate the effectiveness of our method, outperforming state-of-the-art approaches.

Attention redirection transformer with semantic oriented learning for unbiased scene graph generation

Scene Graph Generation (SGG) plays an important role in scene understanding because all of the objects and relations in an image can be abstracted into a concise topological graph. Due to the complexity of visual scenes, including mutual occlusion between objects and semantic ambiguity, SGG is still a challenging task. Most of the existing models only focused on the context of a single object while contexts provided by paired objects are ignored. In this paper, we propose an Attention Redirection Transformer (ART) to extract pair-level contexts specifically, which is divided into an attention distraction stage and an attention integration stage. In this way, the attention of the model is forced to be redirected, which explores the implicit information in the background. In addition, to incorporate the semantic information of predicates, a Semantic Oriented Learning Module (SOL) is designed, which may assist in getting better textual semantics and also prompts cross-modal information fusion. At last, a self-diversity driven Dual Translation Embedding Module (DTM) is designed, which refines representations of subject and object and makes them distinct. Experimental results on the Visual Genome dataset demonstrate the effectiveness of our proposed method. Moreover, our method outperforms state-of-the-art methods on the mR@K metric. The source codes are released on Github: https://github.com/Nora-Zhang98/ART-SOL.

Tangled Physics: Knots Strain Intuitive Physical Reasoning.

Whereas decades of research have cataloged striking errors in physical reasoning, a resurgence of interest in intuitive physics has revealed humans' remarkable ability to successfully predict the unfolding of physical scenes. A leading interpretation intended to resolve these opposing results is that physical reasoning recruits a general-purpose mechanism that reliably models physical scenarios (explaining recent successes), but overly contrived tasks or impoverished and ecologically invalid stimuli can produce poor performance (accounting for earlier failures). But might there be tasks that persistently strain physical understanding, even in naturalistic contexts? Here, we explore this question by introducing a new intuitive physics task: evaluating the strength of knots and tangles. Knots are ubiquitous across cultures and time-periods, and evaluating them correctly often spells the difference between safety and peril. Despite this, 5 experiments show that observers fail to discern even very large differences in strength between knots. In a series of two-alternative forced-choice tasks, observers viewed a variety of simple "bends" (knots joining two pieces of thread) and decided which would require more force to undo. Though the strength of these knots is well-documented, observers' judgments completely failed to reflect these distinctions, across naturalistic photographs (E1), idealized renderings (E2), dynamic videos (E3), and even when accompanied by schematic diagrams of the knots' structures (E4). Moreover, these failures persisted despite accurate identification of the topological differences between the knots (E5); in other words, even when observers correctly perceived the underlying structure of the knot, they failed to correctly judge its strength. These results expose a blindspot in physical reasoning, placing new constraints on general-purpose theories of scene understanding.

CRFormer: A cross-region transformer for shadow removal

Image shadow removal is a fundamental task in computer vision, which aims to restore damaged signals caused by shadows, thereby improving image quality and scene understanding. Recently, transformers have demonstrated strong capabilities in various applications by capturing global pixel interactions, a capability highly desirable for shadow removal. However, applying transformers to promote shadow removal is non-trivial for the following two reasons: 1) The patchify operation is not suitable for shadow removal due to irregular shadow shapes; 2) Shadow removal only requires one-way interaction from the non-shadow region to the shadow region instead of the common two-way interactions among all pixels in the image. In this paper, we propose a novel Cross-Region transFormer (CRFormer) for shadow removal which differs from existing transformers by only considering the pixel interactions from the non-shadow region to the shadow region without splitting images into patches. This is achieved by a carefully designed region-aware cross-attention mechanism that aggregates the recovered shadow region features, conditioned on the non-shadow region features. Extensive experiments on the ISTD, AISTD, SRD, and Video Shadow Removal datasets demonstrate the superiority of our method compared to other state-of-the-art methods.

Combining Generative Adversarial Networks (GANs) with behavior and brain recordings to study scene understanding

Combining Generative Adversarial Networks (GANs) with behavior and brain recordings to study scene understanding

Lightweight multi-scale feature dense cascade neural network for scene understanding of intelligent autonomous platform

Understanding the surrounding environment is an essential part of intelligent autonomous platform to complete autonomous driving or other autonomous tasks. Thereinto, scene understanding is a key technology to understand environmental information. For intelligent autonomous platform, it not only requires high accuracy, but also has a strict requirement for the inference speed. Moreover, it should be able to perform normally in complex environments such as target occlusion, the influence of illumination, and objects at different sizes. For tackling these problems, in this study, a lightweight multi-scale feature dense cascade neural network (LMFDCNet) is proposed in real-time scene understanding in complex scenes. Furthermore, trilateral fusion module is adopted to enhance low-level detailed features or high-level semantic features of each branch to describe the object better in different environments. A multi-scale upsampling module that fuses contextual information at different scales allows objects to be represented at different size. The results indicate that LMFDCNet produces 99.1 % classification accuracy on the tobacco dataset. On the Cityscapes dataset, we achieve 74.2 % Mean Intersection over Union (MIoU) with 13.6 M parameters at the speed of 38.2 FPS on a single 1070Ti card. To test the performance on the real-time application in the real world, LMFDCNet is deployed on the autonomous platform for real-time vision task such as classification of tobacco leaf state in curing process and semantic segmentation in the driving process of autonomous platform. The results reveal that LMFDCNet perform well on the AI embedded device and could be applied into intelligent autonomous platform.

Nighttime scene understanding with label transfer scene parser

Semantic segmentation plays a crucial role in traffic scene understanding, especially in nighttime conditions. This paper tackles the task of semantic segmentation in nighttime scenes. The largest challenge of this task is the lack of annotated nighttime images to train a deep learning-based scene parser. The existing annotated datasets are abundant in daytime conditions but scarce in nighttime due to the high cost. Thus, we propose a novel Label Transfer Scene Parser (LTSP) framework for nighttime scene semantic segmentation by leveraging daytime annotation transfer. Our framework performs segmentation in the dark without training on real nighttime annotated data. In particular, we propose translating daytime images to nighttime conditions to obtain more data with annotation in an efficient way. In addition, we utilize the pseudo-labels inferred from unlabeled nighttime scenes to further train the scene parser. The novelty of our work is the ability to perform nighttime segmentation via daytime annotated labels and nighttime synthetic versions of the same set of images. The extensive experiments demonstrate the improvement and efficiency of our scene parser over the state-of-the-art methods with a similar semi-supervised approach on the benchmark of Nighttime Driving Test dataset. Notably, our proposed method utilizes only one-tenth of the amount of labeled and unlabeled data in comparison with the previous methods. Code is available at https://github.com/danhntd/Label_Transfer_Scene_Parser.git.

High‐order multilayer attention fusion network for 3D object detection

AbstractThree‐dimensional object detection based on the fusion of 2D image data and 3D point clouds has become a research hotspot in the field of 3D scene understanding. However, different sensor data have discrepancies in spatial position, scale, and alignment, which severely impact detection performance. Inappropriate fusion methods can lead to the loss and interference of valuable information. Therefore, we propose the High‐Order Multi‐Level Attention Fusion Network (HMAF‐Net), which takes camera images and voxelized point clouds as inputs for 3D object detection. To enhance the expressive power between different modality features, we introduce a high‐order feature fusion module that performs multi‐level convolution operations on the element‐wise summed features. By incorporating filtering and non‐linear activation, we extract deep semantic information from the fused multi‐modal features. To maximize the effectiveness of the fused salient feature information, we introduce an attention mechanism that dynamically evaluates the importance of pooled features at each level, enabling adaptive weighted fusion of significant and secondary features. To validate the effectiveness of HMAF‐Net, we conduct experiments on the KITTI dataset. In the “Car,” “Pedestrian,” and “Cyclist” categories, HMAF‐Net achieves mAP performances of 81.78%, 60.09%, and 63.91%, respectively, demonstrating more stable performance compared to other multi‐modal methods. Furthermore, we further evaluate the framework's effectiveness and generalization capability through the KITTI benchmark test, and compare its performance with other published detection methods on the 3D detection benchmark and BEV detection benchmark for the “Car” category, showing excellent results. The code and model will be made available on https://github.com/baowenzhang/High‐order‐Multilayer‐Attention‐Fusion‐Network‐for‐3D‐Object‐Detection.

Deep scene understanding with extended text description for human object interaction detection

Human–object interaction (HOI) detection plays a pivotal role in scene understanding, enabling the identification, localization, and behavioral intention prediction of humans and objects within a visual scene. Conventional approaches, such as graph-based networks, have demonstrated effectiveness in capturing spatio-temporal interaction cues. However, relying solely on visual information within these networks limits their ability to comprehensively grasp the intricate aspects of human interactions. To address this shortcoming, we propose a novel deep scene understanding graph network that harnesses the power of extended text descriptions to represent and interpret interactions between humans and objects effectively. Text serves as a rich source of information, directly conveying the nature of interactions within a visual scene. Our model seamlessly integrates text descriptions with visual features extracted from the entire video sequence, enabling it to capture the context and nuances of human–object interactions. This approach significantly enhances the model’s ability to accurately predict ambiguous human actions and anticipate future interactions. Extensive experiments on two benchmark datasets, CAD-120, and Something-Else, demonstrate that our proposed method achieves remarkable improvements in the F1-Score for both HOI detection and anticipation tasks. This work paves the way for more accurate and comprehensive HOI understanding in various real-world scenarios, highlighting the potential of text-augmented graph networks for effective interaction modeling.

Scene understanding method utilizing global visual and spatial interaction features for safety production

Risk identification in power operations is crucial for both personal safety and power production. Existing risk identification methods mainly use target detection models to identify the common risks but the scene specificity of risk occurrence. For example, not wearing a safety harness, not wearing insulated gloves, etc. Since most methods for detecting safety gears make sense only under specific scene. But the power electric work is a complex object involving many elements such as personnel, equipment and safety tools. Therefore, this paper proposes a scene understanding method that integrates visual features and spatial relationship features among scene elements. This method constructs a scenean undirected scene graph to represent the interactive relationship among the elements, extracts the interactive features by using a graph encoder-decoder convolution module, and fuse perceived high-dimensional visual features and spatial topological features for scene recognition, in order to effectively solve addressing the power operation scene understanding problem under multi-element interaction. Finally, a power inspection operation scenario was chosen as the test case. The outcome of the evaluation indicates results indicate that the proposed approach suggested in this study exhibits superior precision in scene identification and shows ademonstrates strong generalization ability.

Vehicle Ego-Trajectory Segmentation Using Guidance Cues

Computer vision has significantly influenced recent advancements in autonomous driving by providing cutting-edge solutions for various challenges, including object detection, semantic segmentation, and comprehensive scene understanding. One specific challenge is ego-vehicle trajectory segmentation, which involves learning the vehicle’s path and describing it with a segmentation map. This can play an important role in both autonomous driving and advanced driver assistance systems, as it enhances the accuracy of perceiving and forecasting the vehicle’s movements across different driving scenarios. In this work, we propose a deep learning approach for ego-trajectory segmentation that leverages a state-of-the-art segmentation network augmented with guidance cues provided through various merging mechanisms. These mechanisms are designed to direct the vehicle’s path as intended, utilizing training data obtained with a self-supervised approach. Our results demonstrate the feasibility of using self-supervised labels for ego-trajectory segmentation and embedding directional intentions within the network’s decisions through image and guidance input concatenation, feature concatenation, or cross-attention between pixel features and various types of guidance cues. We also analyze the effectiveness of our approach in constraining the segmentation outputs and prove that our proposed improvements bring major boosts in the segmentation metrics, increasing IoU by more than 12% and 5% compared with our two baseline models. This work paves the way for further exploration into ego-trajectory segmentation methods aimed at better predicting the behavior of autonomous vehicles.

Monocular BEV Perception of Road Scenes via Front-to-Top View Projection.

HD map reconstruction is crucial for autonomous driving. LiDAR-based methods are limited due to expensive sensors and time-consuming computation. Camera-based methods usually need to perform road segmentation and view transformation separately, which often causes distortion and missing content. To push the limits of the technology, we present a novel framework that reconstructs a local map formed by road layout and vehicle occupancy in the bird's-eye view given a front-view monocular image only. We propose a front-to-top view projection (FTVP) module, which takes the constraint of cycle consistency between views into account and makes full use of their correlation to strengthen the view transformation and scene understanding. In addition, we apply multi-scale FTVP modules to propagate the rich spatial information of low-level features to mitigate spatial deviation of the predicted object location. Experiments on public benchmarks show that our method achieves various tasks on road layout estimation, vehicle occupancy estimation, and multi-class semantic estimation, at a performance level comparable to the state-of-the-arts, while maintaining superior efficiency.

Semantic Segmentation of Point Cloud Scene via Multi-Scale Feature Aggregation and Adaptive Fusion

Point cloud semantic segmentation is a key step in 3D scene understanding and analysis. In recent years, deep learning–based point cloud semantic segmentation methods have received extensive attention from researchers. Multi-scale neighborhood feature learning methods are suitable for inhomogeneous density point clouds, but different scale branching feature learning increases the computational complexity and makes it difficult to accurately fuse different scale features to express local information. In this study, a point cloud semantic segmentation network based on RandLA-Net with multi-scale local feature aggregation and adaptive fusion is proposed. The designed structure can reduce computational complexity and accurately express local features. The mean intersection-over-union is improved by 1.1% on the SemanticKITTI data set with an inference speed of nine frames per second, while the mean intersection-over-union is improved by 0.9% on the S3DIS data set, compared with RandLA-Net. We also conduct ablation studies to validate the effectiveness of the proposed key structure.

Beyond "Taming Electric Scooters": Disentangling Understandings of Micromobility Naturalistic Riding

Electric(e)-scooters have emerged as a popular, ubiquitous, and first/last-mile micromobility transportation option within and across many cities worldwide. With the increasing situation-awareness and on-board computational capability, such intelligent micromobility has become a critical means of understanding the rider's interactions with other traffic constituents (called Rider-to-X Interactions, RXIs), such as pedestrians, cars, and other micromobility vehicles, as well as road environments, including curbs, road infrastructures, and traffic signs. How to interpret these complex, dynamic, and context-dependent RXIs, particularly for the rider-centric understandings across different data modalities --- such as visual, behavioral, and textual data --- is essential for enabling safer and more comfortable micromobility riding experience and the greater good of urban transportation networks. Under a naturalistic riding setting (i.e., without any unnatural constraint on rider's decision-making and maneuvering), we have designed, implemented, and evaluated a pilot Cross-modality E-scooter Naturalistic Riding Understanding System, namely CENRUS, from a human-centered AI perspective. We have conducted an extensive study with CENRUS in sensing, analyzing, and understanding the behavioral, visual, and textual annotation data of RXIs during naturalistic riding. We have also designed a novel, efficient, and usable disentanglement mechanism to conceptualize and understand the e-scooter naturalistic riding processes, and conducted extensive human-centered AI model studies. We have performed multiple downstream tasks enabled by the core model within CENRUS to derive the human-centered AI understandings and insights of complex RXIs, showcasing such downstream tasks as efficient information retrieval and scene understanding. CENRUS can serve as a foundational system for safe and easy-to-use micromobility rider assistance as well as accountable use of micromobility vehicles.

Enhancing image caption generation through context-aware attention mechanism

Image captioning, the process of generating natural language descriptions based on image content, has garnered attention in AI research for its implications in scene understanding and human-computer interaction. While much prior research has focused on caption generation for English, addressing low-resource languages like Bengali presents challenges, particularly in producing coherent captions linking visual objects with corresponding words. This paper proposes a context-aware attention mechanism over semantic attention to accurately diagnose objects for image captioning in Bengali. The proposed architecture consists of an encoder and a decoder block. We chose ResNet-50 over the other pre-trained models for encoding the image features due to its ability to solve the vanishing gradient problem and recognize complex object features. For decoding generated captions, a bidirectional Gated Recurrent Unit (GRU) architecture combined with an attention mechanism captures contextual dependencies in both directions, resulting in more accurate captions. The paper also highlights the challenge of transferring knowledge between domains, especially with culturally specific images. Evaluation of three Bengali benchmark datasets, namely BAN-Cap, BanglaLekhaImageCaption, and Bornon, demonstrates significant performance improvement in METEOR score over existing methods by approximately 30%, 18%, and 45%, respectively. The proposed context-aware, attention-based image captioning system significantly outperforms current state-of-the-art models in Bengali caption generation despite limitations in reference captions on certain datasets.

ChineseMPD: A Semantic Segmentation Dataset of Chinese Martial Arts Classic Movie Props

Recent advances in computer vision and deep learning techniques have facilitated significant progress in video scene understanding, thus helping film and television practitioners achieve accurate video editing. However, so far, publicly available semantic segmentation datasets are mostly limited to indoor scenes, city streets, and natural images, often ignoring example objects in action movies, which is a research gap that needs to be urgently filled. In this paper, we introduce a large-scale, high-precision semantic segmentation dataset of props in Chinese martial arts movie clips, named ChineseMPD. Specifically, this dataset first establishes segmentation rules and general review criteria for audiovisual data, and then provides semantic segmentation annotations for six weapon props (Gun, Sword, Stick, Knife, Hook, and Arrow) with a summary of 32,992 objects.To the best of our knowledge, this dataset is the largest semantic segmentation dataset for movie props to date. ChineseMPD dataset not only significantly expands the application of traditional tasks of computer vision such as object detection and scene understanding, but also opens up new avenues for interdisciplinary research.

A rich RGBD images captioning for scene understanding

A rich RGBD images captioning for scene understanding

Competing with autonomous model vehicles: a software stack for driving in smart city environments

This article introduces an open-source software stack designed for autonomous 1:10 scale model vehicles. Initially developed for the Bosch Future Mobility Challenge (BFMC) student competition, this versatile software stack is applicable to a variety of autonomous driving competitions. The stack comprises perception, planning, and control modules, each essential for precise and reliable scene understanding in complex environments such as a miniature smart city in the context of BFMC. Given the limited computing power of model vehicles and the necessity for low-latency real-time applications, the stack is implemented in C++, employs YOLO Version 5 s for environmental perception, and leverages the state-of-the-art Robot Operating System (ROS) for inter-process communication. We believe that this article and the accompanying open-source software will be a valuable resource for future teams participating in autonomous driving student competitions. Our work can serve as a foundational tool for novice teams and a reference for more experienced participants. The code and data are publicly available on GitHub.

TSG-Seg: Temporal-selective guidance for semi-supervised semantic segmentation of 3D LiDAR point clouds

LiDAR-based semantic scene understanding holds a pivotal role in various applications, including remote sensing and autonomous driving. However, the majority of LiDAR segmentation models rely on extensive and densely annotated training datasets, which is extremely laborious to annotate and hinder the widespread adoption of LiDAR systems. Semi-supervised learning (SSL) offers a promising solution by leveraging only a small amount of labeled data and a larger set of unlabeled data, aiming to train robust models with desired accuracy comparable to fully supervised learning. A typical pipeline of SSL involves the initial use of labeled data to train segmentation models, followed by the utilization of predictions generated from unlabeled data, which are used as pseudo-ground truths for model retraining. However, the scarcity of labeled data limits the capture of comprehensive representations, leading to the constraints of these pseudo-ground truths in reliability. We observed that objects captured by LiDAR sensors from varying perspectives showcase diverse data characteristics due to occlusions and distance variation, and LiDAR segmentation models trained with limited labels prove susceptible to these viewpoint disparities, resulting in inaccurately predicted pseudo-ground truths across viewpoints and the accumulation of retraining errors. To address this problem, we introduce the Temporal-Selective Guided Learning (TSG-Seg) framework. TSG-Seg explores temporal cues inherent in LiDAR frames to bridge the cross-viewpoint representations, fostering consistent and robust segmentation predictions across differing viewpoints. Specifically, we first establish point-wise correspondences across LiDAR frames with different time stamps through point registration. Subsequently, reliable point predictions are selected and propagated to points from adjacent views to the current view, serving as strong and refined supervision signals for subsequent model re-training to achieve better segmentation. We conducted extensive experiments on various SSL labeling setups across multiple public datasets, including SemanticKITTI and SemanticPOSS, to evaluate the effectiveness of TSG-Seg. Our results demonstrate its competitive performance and robustness in diverse scenarios, from data-limited to data-abundant settings. Notably, TSG-Seg achieves a mIoU of 48.6% using only 5% of and 62.3% with 40% of labeled data in the sequential split on SemanticKITTI. This consistently outperforms state-of-the-art segmentation methods, including GPC and LaserMix. These findings underscore TSG-Seg’s superior capability and potential for real-world applications. The project can be found at https://tsgseg.github.io.

Detection of pedestrians and vehicles on foggy roads based on improved YOLOv7

Abstract Nowadays, object detection technology plays an indispensable role in the direction of automatic vehicle driving. However, inclement weather, such as fog and haze, poses severe challenges to the ability to understand visual perception and scene understanding of automatic driving technology, making it extremely difficult. Therefore, this paper proposes an improved YOLOv7 algorithm. Firstly, Gaussian blur and Gaussian noise are used to process the original data set, reducing the sensitivity of our model to image information, improving the ability to adapt to changing environments and tasks, and enhancing the generalization performance. Then, using the SIOU loss function, we optimize the algorithm to accelerate the convergence speed while considering the vector angle problem. Add Coordinate Attention integrates accurate position information into the coding network, optimizes the recognition accuracy of the model for a specific position, significantly improves its overall feature extraction ability, and captures key features more accurately, thereby improving the overall performance, capturing key information, and reducing the possibility of missed detection. The introduction of CoordConv into ELAN in the backbone allows for a more sensitive perception of the position of objects during image processing, thus improving the accuracy of its recognition and positioning. The RepGhostNet structure is introduced into the neck to realize the implicit reuse of features. According to the experiments in this paper, our improved YOLOv7 algorithm achieves 87.3% map in the RTTS data set and partial synthetic fog KITTI data set, which improves by 3.9% based on the original model.