Abstract
In order to address the issue of edge-blurring and improve the accuracy and robustness of scene flow estimation under motion occlusions, we in this article propose a piecewise 3D scene flow estimation approach with semantic segmentation, named SS-SF. First, we utilize the semantic optical flow to initialize the 3D plane and its rigid motion parameters, and then produce the initial mappings of pixel-to-segment and segment-to-plane of the input left and right image sequences. Second, we plan a novel energy function to optimize the initial mappings by using a semantic segmentation constraint term to regularize the classical scene flow model, which the optimized mappings are employed to update the assignment and motion parameters of each pixel. Third, we adopt the semantic label to extract the occlusion pixels and exploit an occlusion handling constraint to enhance the robustness of the scene flow estimation. Finally, we compare the proposed SS-SF model with several state-of-the-art approaches by using the KITTI and MPI-Sintel databases. The experimental results demonstrate that the proposed method has the advanced accuracy and robustness in scene flow estimation, especially owns the capacities of edge-preserving and occlusion handling.
Highlights
For a straightforward online evaluation of scene flow, the KITTI benchmark counts the outliers in either optical flow or disparity results to indicate the performance of scene flow as following: scene flow error (SF) − all = P1 ∪ P2 × 100%, (17)
The comparison results between the SS-SF method and the different modeling choices indicate that the proposed occlusion-aware constraint and semantic segmentation scheme are beneficial for improving the performance of scene flow estimation, especially in regions of occlusions and motion boundaries
In this report, we started by reviewing the progress and several previous approaches in the fields of scene flow and optical flow estimation
Summary
Dense motion estimation from consecutive frames is a focus of research in image processing and computer vision, with broad applications in human posture estimation and recognition [1], moving target segmentation and tracking [2], obstacle detection and identification [3], foreground prediction and navigation [4], facial expression recognition [5], video deblurring and coding [6], and many other fields [7], [8]. C. Feng et al.: SS-SF: Piecewise 3D Scene Flow Estimation With Semantic Segmentation regions, which is robust to the motion occlusion [14]–[16]. The existing piecewise scene flow approaches may cause the issue of edge-blurring around image and motion boundaries because these models only use a random superpixel segmentation scheme to initialize the motion parameters and ignore the boundary differences of various objectives. To address the abovementioned issue of edge-blurring and ensure the accuracy and robustness of scene flow estimation under motion occlusions, we present a piecewise rigid scene flow estimation with semantic segmentation optimization, named SS-SF. We exploit an occlusion handling constraint by using the semantic labels of pixels to cope with the motion occlusions between the consecutive frames, by which the presented occlusion handling scheme can effectively develop the robustness of scene flow estimation in regions of occlusions.
Sign-in/Register to view highlights & summary Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
