Scene Flow Research Articles

Stereo disparity estimation based on Siamese network

Abstract This paper proposes an approach for estimating stereo disparity based on a Siamese network. The existing Top-k disparity regression strategy may overlook the true disparity at object edges. To address this issue, an additional maximum disparity estimation value is introduced to optimize the Top-k strategy, creating the Top-k+ strategy. Furthermore, this research improves the MobileNetv2 (MV2) block by introducing an attention mechanism in the frequency domain, resulting in a more efficient FMV2 block that extracts high-frequency information such as textures and edges. Considering the excellent performance of the gated recurrent unit (GRU) in disparity optimization, the algorithm adopts an iterative optimization method based on GRU. Comparative studies pertaining to the Scene Flow, KITTI 2015, and ETH3D benchmark datasets show outstanding results in disparity estimation, with an end-point error of 0.58 pixels on the Scene Flow dataset, a D1-all error of 1.67% on the KITTI 2015 dataset, and an average disparity error of 0.22 pixels on the ETH3D dataset. Compared to other GRU-based iterative optimization algorithms, the proposed method not only exhibits significant performance advantages but also demonstrates a lightweight design.

Spatial-frequency attention-based optical and scene flow with cross-modal knowledge distillation

Spatial-frequency attention-based optical and scene flow with cross-modal knowledge distillation

Optical Flow as Spatial-Temporal Attention Learners.

Optical flow is an indispensable building block for various important computer vision tasks, including motion estimation, object tracking, and disparity measurement. To date, the dominant methods are CNN-based, leaving plenty of room for improvement. Inthiswork, we propose TransFlow, a transformer architecture for optical flow estimation. Compared to dominant CNN-based methods, TransFlow demonstrates three advantages. First, it provides more accurate correlation and trustworthy matching in flow estimation by utilizing spatial self-attention and cross-attention mechanisms between adjacent frames to effectively capture global dependencies; Second, it recovers more compromised information (e.g. occlusion and motion blur) in flow estimation through long-range temporal association in dynamic scenes; Third, it introduces a concise self-learning paradigm, eliminating the need for complex and laborious multi-stage pre-training procedures. The versatility and superiority of TransFlow extend seamlessly to 3D scene motion, yielding competitive outcomes in 3D scene flow estimation. Our approach attains state-of-the-art results on benchmark datasets such as Sintel and KITTI-15, while also exhibiting exceptional performance on downstream tasks, including video object detection using the ImageNet VID dataset, video frame interpolation using the GoPro dataset, and video stabilization using the DeepStab dataset. We believe that the effectiveness of TransFlow positions it as a flexible baseline for both optical flow and scene flow estimation, offering promising avenues for future research and development.

TCANet: Three-dimensional cross-attention mechanism for stereo-matching

Abstract Effective disparity estimation is a current hotspot in stereo vision research, and cost aggregation is an important part of disparity prediction. How to perform more effective cost aggregation is the core step to improve the accuracy of disparity prediction. In previous studies, 3DCNN with a stacked hourglass structure is often used for cost aggregation. In this research, we propose an effective 3D cross-attention stereo network that utilizes the attention mechanism to obtain contextual information for cost aggregation in a more efficient way. Specifically, the 3D cross-attention module in TCANet acquires the geometric information of all pixels on the 3D cross path. By repeating this operation twice, each pixel can eventually obtain global dependencies from all other pixels. Using the 3D cross-attention module in a stacked hourglass-structured 3DCNN only increases the number of parameters by a very small amount, which can effectively improve the performance of the model. Experimental results show that TCANet performs well on virtual dataset Scene Flow and realistic KITTI datasets.

Learning to use landmarks for navigation amplifies their representation in retrosplenial cortex.

Visual landmarks provide powerful reference signals for efficient navigation by altering the activity of spatially tuned neurons, such as place cells, head direction cells, and grid cells. To understand the neural mechanism by which landmarks exert such strong influence, it is necessary to identify how these visual features gain spatial meaning. In this study, we characterized visual landmark representations in mouse retrosplenial cortex (RSC) using chronic two-photon imaging of the same neuronal ensembles over the course of spatial learning. We found a pronounced increase in landmark-referenced activity in RSC neurons that, once established, remained stable across days. Changing behavioral context by uncoupling treadmill motion from visual feedback systematically altered neuronal responses associated with the coherence between visual scene flow speed and self-motion. To explore potential underlying mechanisms, we modeled how burst firing, mediated by supralinear somatodendritic interactions, could efficiently mediate context- and coherence-dependent integration of landmark information. Our results show that visual encoding shifts to landmark-referenced and context-dependent codes as these cues take on spatial meaning during learning.

GloFP-MSF: monocular scene flow estimation with global feature perception

GloFP-MSF: monocular scene flow estimation with global feature perception

SFD-SLAM: a novel dynamic RGB-D SLAM based on saliency region detection

Abstract In dynamic environments, several simultaneous localization and mapping (SLAM) systems effectively utilize optical flow fields to distinguish dynamic from static feature points. Commonly, these systems leverage the amplitude information within the optical flow field to develop adaptive thresholding segmentation models for identifying dynamic scene regions. Nevertheless, designing adaptive thresholding models typically necessitates meticulous planning and extensive experimentation. This study introduces a dynamic RGBD SLAM system, SFD-SLAM, which innovates by employing a saliency detection network for the direct extraction of dynamic regions via scene flow. This approach notably streamlines the design process associated with conventional adaptive thresholding models. Furthermore, SFD-SLAM incorporates a geometric module that merges depth residuals with hyperpixel segmentation to enhance the refinement of the dynamic mask. This is followed by integration with FCM clustering for the precise identification of moving objects. The efficacy of SFD-SLAM is assessed using the widely recognized TUM dynamic dataset. Experimental results demonstrate that the proposed system surpasses DGFlow-SLAM, which relies on an adaptive thresholding model for dynamic object segmentation, in terms of trajectory accuracy. It also achieves comparable localization accuracy to DynaSLAM. Moreover, SFD-SLAM maintains robust tracking capabilities, even in scenarios where DynaSLAM experiences tracking loss, thereby augmenting the robustness of RGBD-SLAM in dynamic settings.

Bridging the Domain Gap in Scene Flow Estimation via Hierarchical Smoothness Refinement

This article introduces SmoothFlowNet3D, an innovative encoder-decoder architecture specifically designed for bridging the domain gap in scene flow estimation. To achieve this goal, SmoothFlowNet3D divides the scene flow estimation task into two stages: initial scene flow estimation and smoothness refinement. Specifically, SmoothFlowNet3D comprises a hierarchical encoder that extracts multi-scale point cloud features from two consecutive frames, along with a hierarchical decoder responsible for predicting the initial scene flow and further refining it to achieve smoother estimation. To generate the initial scene flow, a cross-frame nearest-neighbor search operation is performed between the features extracted from two consecutive frames, resulting in forward and backward flow embeddings. These embeddings are then combined to form the bidirectional flow embedding, serving as input for predicting the initial scene flow. Additionally, a flow smoothing module based on the self-attention mechanism is proposed to predict the smoothing error and facilitate the refinement of the initial scene flow for more accurate and smoother estimation results. Extensive experiments demonstrate that the proposed SmoothFlowNet3D approach achieves state-of-the-art performance on both synthetic datasets and real LiDAR point clouds, confirming its effectiveness in enhancing scene flow smoothness.

Self-supervised rigid object 3-D motion estimation from monocular video

Independent object 3D motion estimation is a fundamental problem in 3D computer vision. Directly segmenting and estimating rigid object 3D motion from a consistent video frame is an ill-posed problem. We present a self-supervised framework for segmenting moving independent rigid objects and estimating their motion information including location, driving direction, and speed from a monocular video. Specifically, we first estimate depth, optical flow, and camera pose between a pair of video frames, and then synthesize a new 3D viewpoint from this pair. Subsequently, the Motion Recurrent All-Pairs Field Transforms (MRAFT) module is introduced to extract 3D scene flow and a motion area binary mask from a pair of images and depth. After that, a Rigid Object Motion Estimation Module (ROMEM) with a slot attention mechanism is proposed to extract rigid object motion masks from a multi-layer motion field, including optical flow, depth changes, refined scene flow, and motion masks. Finally, 2D images and 3D scene reconstruction errors are used to facilitate self-supervised training for rigid object motion. Experiments on the FlyingThings3D and KITTI datasets demonstrate that our method outperforms other advanced algorithms in estimating depth, optical flow, scene flow, and rigid moving object masks, demonstrating the benefits of our approach.

Unsupervised 3D Object Segmentation of Point Clouds by Geometry Consistency.

In this paper, we study the problem of 3D object segmentation from raw point clouds. Unlike existing methods which usually require a large amount of human annotations for full supervision, we propose the first unsupervised method, called OGC, to simultaneously identify multiple 3D objects in a single forward pass, without needing any type of human annotations. The key to our approach is to fully leverage the dynamic motion patterns over sequential point clouds as supervision signals to automatically discover rigid objects. Our method consists of three major components, 1) the object segmentation network to directly estimate multi-object masks from a single point cloud frame,2)the auxiliary self-supervised scene flow estimator,and 3)our core object geometry consistency component. By carefully designing a series of loss functions, we effectively take into account the multi-object rigid consistency and the object shape invariance in both temporal and spatial scales. This allows our method to truly discover the object geometry even in the absence of annotations. We extensively evaluate our method on five datasets, demonstrating the superior performance for object part instance segmentation and general object segmentation in both indoor and the challenging outdoor scenarios. Our code and data are available at https://github.com/vLAR-group/OGC.

Optimizing Object Classification in Robotic Perception Environments Exploring Late Fusion Strategies

Robotic perception systems often include approaches that can extract valuable features or information from the studied dataset. These methods often involve the application of deep learning approaches, such as convolutional neural networks (CNNs), for processing of images, as well as the incorporation of 3D data. The notion of image categorization is well delineated via the use of networks that include convolutional networks. However, some network topologies exhibit a substantial scope and need significant amounts of time and memory resources. On the other hand, the neural networks FlowNet3D and PointFlowNet have the capability to accurately predict scene flow. Specifically, these networks are capable of estimating the three-dimensional movements of point clouds (PCs) within a dynamic environment. When using PCs in robotic applications, it is crucial to examine the robustness of accurately recognizing the points that belong to the object. This article examines the use of robotic perception systems inside autonomous vehicles and the inherent difficulties linked to the analysis and processing of information obtained from diverse sensors. The researchers put out a late fusion methodology that integrates the results of many classifiers in order to enhance the accuracy of categorization. Additionally, the authors propose a weighted fusion technique that incorporates the proximity to objects as a significant factor. The findings indicate that the fusion methods described in this study exhibit superior performance compared to both single modality classification and classic fusion strategies.

RMS-FlowNet++: Efficient and Robust Multi-scale Scene Flow Estimation for Large-Scale Point Clouds

The proposed RMS-FlowNet++ is a novel end-to-end learning-based architecture for accurate and efficient scene flow estimation that can operate on high-density point clouds. For hierarchical scene flow estimation, existing methods rely on expensive Farthest-Point-Sampling (FPS) to sample the scenes, must find large correspondence sets across the consecutive frames and/or must search for correspondences at a full input resolution. While this can improve the accuracy, it reduces the overall efficiency of these methods and limits their ability to handle large numbers of points due to memory requirements. In contrast to these methods, our architecture is based on an efficient design for hierarchical prediction of multi-scale scene flow. To this end, we develop a special flow embedding block that has two advantages over the current methods: First, a smaller correspondence set is used, and second, the use of Random-Sampling (RS) is possible. In addition, our architecture does not need to search for correspondences at a full input resolution. Exhibiting high accuracy, our RMS-FlowNet++ provides a faster prediction than state-of-the-art methods, avoids high memory requirements and enables efficient scene flow on dense point clouds of more than 250K points at once. Our comprehensive experiments verify the accuracy of RMS-FlowNet++ on the established FlyingThings3D data set with different point cloud densities and validate our design choices. Furthermore, we demonstrate that our model has a competitive ability to generalize to the real-world scenes of the KITTI data set without fine-tuning.

Hourglass cascaded recurrent stereo matching network

Stereo matching acts a crucial role in computer vision and robotics applications. An accurate cost volume and robust disparity regression method are essential for stereo matching of high accuracy. Following GCNet and PSMNet, constructing 4D cost volume and then using the soft argmin method to regress has been dominated. However, it will encounter many difficulties due to the multi-modal distribution of cost volume. One of the reasons for this multi-modal distribution is the occlusion area which not be possible to find a matching region on the reference image and rarely discussed. In this paper, we propose to use global context information could improve the performance of model in occluded regions. Recently, novel recurrent neural network regression methods are proposed, but most of them regress disparity maps from 3D cost volume. In this paper, we propose the new combinatorial paradigm that combine stacked hourglass modules and recurrent neural networks to further aggregate 4D cost volume and regress disparity respectively. The proposed method can be seamlessly integrated into most stereo matching networks, we improved the accuracy by 45% for PSMNet and 38% for GwcNet in our experiment. Experimental results on Scene Flow, KITTI2012, KITTI2015, and ETH3D datasets show our method is competitive. The code is available at: https://github.com/truman1211/HCRnet.

Accuracy and efficiency stereo matching network with adaptive feature modulation.

Feature enhancement plays a crucial role in improving the quality and discriminative power of features used in matching tasks. By enhancing the informative and invariant aspects of features, the matching process becomes more robust and reliable, enabling accurate predictions even in challenging scenarios, such as occlusion and reflection in stereo matching. In this paper, we propose an end-to-end dual-dimension feature modulation network called DFMNet to address the issue of mismatches in interference areas. DFMNet utilizes dual-dimension feature modulation (DFM) to capture spatial and channel information separately. This approach enables the adaptive combination of local features with more extensive contextual information, resulting in an enhanced feature representation that is more effective in dealing with challenging scenarios. Additionally, we introduce the concept of cost filter volume (CFV) by utilizing guide weights derived from group-wise correlation. CFV aids in filtering the concatenated volume adaptively, effectively discarding redundant information, and further improving matching accuracy. To enable real-time performance, we designed a fast version named Fast-GFM. Fast-GFM employs the global feature modulation (GFM) block to enhance the feature expression ability, improving the accuracy and stereo matching robustness. The accurate DFMNet and the real-time Fast-GFM achieve state-of-the-art performance across multiple benchmarks, including Scene Flow, KITTI, ETH3D, and Middlebury. These results demonstrate the effectiveness of our proposed methods in enhancing feature representation and significantly improving matching accuracy in various stereo matching scenarios.

End-to-end information fusion method for transformer-based stereo matching

In stereo matching, the application of transformers can overcome the limitations of disparity range and capture long-range matching information. However, the lack of cross-epipolar context information often leads to numerous mismatches, especially in low-texture regions. An end-to-end information fusion stereo matching method is proposed to address this issue. In the proposed method, a feature extraction method that combines dense connections and a residual block is proposed. Global and local semantic information can be effectively fused by incorporating dense connections among multiscale feature maps. Additionally, the inclusion of a residual block helps extract more representative feature maps. The idea of criss-cross attention is introduced in the transformer implicit matching process. Criss-cross attention enables the capture of cross-epipolar context information by combining horizontal and vertical attention mechanisms. This method improves the matching accuracy from the perspective of multi-path information fusion. According to the matching results, the disparity regression layer and the context adjustment layer are used to generate the initial and final disparity maps, respectively. The proposed method is evaluated on the Scene Flow, KITTI 2012, and Middlebury 2014 datasets. Experimental results indicate that the proposed method effectively enhances matching accuracy. Moreover, the proposed method exhibits strong generalization ability, allowing for direct application to synthetic, real outdoor, and real indoor scene images.

Accurate and Efficient Stereo Matching via Attention Concatenation Volume.

Stereo matching is a fundamental building block for many vision and robotics applications. An informative and concise cost volume representation is vital for stereo matching of high accuracy and efficiency. In this article, we present a novel cost volume construction method, named attention concatenation volume (ACV), which generates attention weights from correlation clues to suppress redundant information and enhance matching-related information in the concatenation volume. The ACV can be seamlessly embedded into most stereo matching networks, the resulting networks can use a more lightweight aggregation network and meanwhile achieve higher accuracy. We further design a fast version of ACV to enable real-time performance, named Fast-ACV, which generates high likelihood disparity hypotheses and the corresponding attention weights from low-resolution correlation clues to significantly reduce computational and memory cost and meanwhile maintain a satisfactory accuracy. Furthermore, we design a highly accurate network ACVNet and a real-time network Fast-ACVNet based on our ACV and Fast-ACV respectively, which achieve state-of-the-art performance on several benchmarks.

Scene Flow Prior Based Point Cloud Completion with Masked Transformer (Student Abstract)

It is necessary to explore an effective point cloud completion mechanism that is of great significance for real-world tasks such as autonomous driving, robotics applications, and multi-target tracking. In this paper, we propose a point cloud completion method using a self-supervised transformer model based on the contextual constraints of scene flow. Our method uses the multi-frame point cloud context relationship as a guide to generate a series of token proposals, this priori condition ensures the stability of the point cloud completion. The experimental results show that the method proposed in this paper achieves high accuracy and good stability.

Dual-stream pyramid and attention network for stereo matching

In order to accurately estimate the disparity of ill-posed regions, such as weak texture and occlusion regions, we propose DSPANet, a stereo matching network that incorporates a dual-stream pyramid module and a channel and spatial attention module. The dual-stream pyramid module captures numerous complementary features from different layers by utilizing multi-resolution inputs and feature extraction blocks. This approach enables the learning of local detailed features at various scales. These features at various scales are then combined to calculate the stereo matching cost. By incorporating channel and spatial attention module into the feature extraction process to obtain richer and more concise contextual information, the matching cost can be constructed more accurately, which provides powerful conditions for subsequent cost aggregation. In the cost aggregation stage, we utilize the stacked hourglass module for both encoding and decoding. Additionally, we incorporate 3D global attention upsampling during the decoding stage, which enables high-level features to provide guidance information to low-level features in a simple way. We evaluate our proposed method on the Scene Flow dataset, as well as the KITTI2012 and KITTI2015 datasets. The experimental results demonstrate that our DSPANet achieves superior performance and effectively enhances the matching results in ill-posed regions. Our code has been implemented using PyTorch and will be released after paper publication at https://github.com/Shi-LiQing/DSPANet.

Kalman-Based Scene Flow Estimation for Point Cloud Densification and 3D Object Detection in Dynamic Scenes

Point cloud densification is essential for understanding the 3D environment. It provides crucial structural and semantic information for downstream tasks such as 3D object detection and tracking. However, existing registration-based methods struggle with dynamic targets due to the incompleteness and deformation of point clouds. To address this challenge, we propose a Kalman-based scene flow estimation method for point cloud densification and 3D object detection in dynamic scenes. Our method effectively tackles the issue of localization errors in scene flow estimation and enhances the accuracy and precision of shape completion. Specifically, we introduce a Kalman filter to correct the dynamic target’s position while estimating long sequence scene flow. This approach helps eliminate the cumulative localization error during the scene flow estimation process. Extended experiments on the KITTI 3D tracking dataset demonstrate that our method significantly improves the performance of LiDAR-only detectors, achieving superior results compared to the baselines.

Self-Supervised 3D Scene Flow Estimation and Motion Prediction using Local Rigidity Prior.

In this article, we investigate self-supervised 3D scene flow estimation and class-agnostic motion prediction on point clouds. A realistic scene can be well modeled as a collection of rigidly moving parts, therefore its scene flow can be represented as a combination of rigid motion of these individual parts. Building upon this observation, we propose to generate pseudo scene flow labels for self-supervised learning through piecewise rigid motion estimation, in which the source point cloud is decomposed into local regions and each region is treated as rigid. By rigidly aligning each region with its potential counterpart in the target point cloud, we obtain a region-specific rigid transformation to generate its pseudo flow labels. To mitigate the impact of potential outliers on label generation, when solving the rigid registration for each region, we alternately perform three steps: establishing point correspondences, measuring the confidence for the correspondences, and updating the rigid transformation based on the correspondences and their confidence. As a result, confident correspondences will dominate label generation, and a validity mask will be derived for the generated pseudo labels. By using the pseudo labels together with their validity mask for supervision, models can be trained in a self-supervised manner. Extensive experiments on FlyingThings3D and KITTI datasets demonstrate that our method achieves new state-of-the-art performance in self-supervised scene flow learning, without any ground truth scene flow for supervision, even performing better than some supervised counterparts. Additionally, our method is further extended to class-agnostic motion prediction and significantly outperforms previous state-of-the-art self-supervised methods on nuScenes dataset.