Occlusion Handling Research Articles

Registration and occlusion handling based on the FAST ICP-ORB method for augmented reality systems

Spatial position consistency and occlusion consistency are two important problems in augmented reality systems. In this paper, we proposed a novel method that can address the registration problem and occlusion problem simultaneously by using an RGB-D camera. First, to solve the image alignment errors caused by the imaging mode of the RGB-D camera, we developed a depth map inpainting method that combines the FMM and RGB-D information. Second, we established an automatic method to judge the close-range mode based on the depth histogram to solve the registration failure problem caused by hardware limitations. In the close-range mode, the registration method combining the fast ICP and ORB was adopted to calculate the camera pose. Third, we developed an occlusion handling method based on the geometric analysis of the scene. Several experiments were performed to validate the performance of the proposed method. The experimental results indicate that our method can obtain stable and accurate registration and occlusion handling results in both the close-range and non-close-range modes. Moreover, the mutual occlusion problem can be handled effectively, and the proposed method can satisfy the real-time requirements of augmented reality systems.

Ellipse R-CNN: Learning to Infer Elliptical Object From Clustering and Occlusion.

Images of heavily occluded objects in cluttered scenes, such as fruit clusters in trees, are hard to segment. To further retrieve the 3D size and 6D pose of each individual object in such cases, bounding boxes are not reliable from multiple views since only a little portion of the object's geometry is captured. We introduce the first CNN-based ellipse detector, called Ellipse R-CNN, to represent and infer occluded objects as ellipses. We first propose a robust and compact ellipse regression based on the Mask R-CNN architecture for elliptical object detection. Our method can infer the parameters of multiple elliptical objects even they are occluded by other neighboring objects. For better occlusion handling, we exploit refined feature regions for the regression stage, and integrate the U-Net structure for learning different occlusion patterns to compute the final detection score. The correctness of ellipse regression is validated through experiments performed on synthetic data of clustered ellipses. We further quantitatively and qualitatively demonstrate that our approach outperforms the state-of-the-art model (i.e., Mask R-CNN followed by ellipse fitting) and its three variants on both synthetic and real datasets of occluded and clustered elliptical objects.

Vision-Based Human Detection Techniques: A Descriptive Review

Cameras are being used everywhere for the safety and security of citizens in different countries. Using a machine to detect humans in a photo or a video frame is a very complicated and challenging task. Various techniques have been developed for this purpose, which mainly rely on Artificial Intelligence. This article aims to provide a comprehensive review and analysis of the literatures from a descriptive perspective, which is its main differentiator from the existing survey papers in this area. Firstly, the vision-based human detection techniques and classifiers are elucidated in conjunction with the variants of feature extraction techniques. Secondly, various pros and cons of such techniques are discussed. Then, an investigation has been conducted and reported based on the state-of-the-art human detection descriptors (e.g. Log-Average Miss Rate and accuracy). Although techniques such as Viola-Jones and Speeded-Up Robust Features can detect objects in real-time and overcome Scale-Invariant Feature Transform (SIFT) limitations, they are still sensitive to illuminated conditions. Other techniques such as SIFT, Bag of Words, Orthogonal Moments, and Histogram of oriented Gradients provide other interesting benefits which include insensitivity to occlusion and clutters, simplicity, low-order element construction and invariance to illuminated conditions; nevertheless, they are computationally expensive and sensitive to image rotation. A meticulous review along similar lines revealed that the Deformable Part-based Model performs relatively better due to its ability to deal with particular pose variations and multiple views, occlusion handling (partial) and is application-free while its counterparts focus on only a single aspect. This article highlights and provides a brief description of each available data-sets for human detection research. Various use-cases of human detection systems are also elaborated. Finally, various conclusions are derived based on the conducted review followed by recommendations for future directions and possibilities to further improve the speed and accuracy of human detection systems.

Occlusion Handling by Successively Excluding Foregrounds for Light Field Depth Estimation Based on Foreground-Background Separation

This paper proposes a depth from light field (DFLF) method specifically to deal with occlusion based on the foreground-background separation (FBS). The FBS-based methods infer the disparity maps by accumulating the binary maps which divide whether each pixel is the foreground or background. Although there have been widely studied to handle the occlusion problem with the cost-based method, there are not enough researches to handle the occlusion problem with the FBS-based methods yet. We found that errors around the occlusion boundary in the resulting disparity maps of the FBS-based methods arise from the fattened foreground by the light field reprameterization. To avoid fattened foregrounds, the inferred foreground maps in the front region with respect to the disparity axis could be utilized in the back region in the three-dimensional volume construction, which corresponds to the cost volume construction in the cost-based methods. With the front-to-back scanning manner of the FBS-based method, by successively excluding inferred foreground maps, errors around occlusion boundary could be effectively reduced in the resulting disparity maps. With synthetic and real LF images, the proposed method shows reasonable performance compared to the existing methods and better performance than existing FBS-based methods.

SS-SF: Piecewise 3D Scene Flow Estimation With Semantic Segmentation

In order to address the issue of edge-blurring and improve the accuracy and robustness of scene flow estimation under motion occlusions, we in this article propose a piecewise 3D scene flow estimation approach with semantic segmentation, named SS-SF. First, we utilize the semantic optical flow to initialize the 3D plane and its rigid motion parameters, and then produce the initial mappings of pixel-to-segment and segment-to-plane of the input left and right image sequences. Second, we plan a novel energy function to optimize the initial mappings by using a semantic segmentation constraint term to regularize the classical scene flow model, which the optimized mappings are employed to update the assignment and motion parameters of each pixel. Third, we adopt the semantic label to extract the occlusion pixels and exploit an occlusion handling constraint to enhance the robustness of the scene flow estimation. Finally, we compare the proposed SS-SF model with several state-of-the-art approaches by using the KITTI and MPI-Sintel databases. The experimental results demonstrate that the proposed method has the advanced accuracy and robustness in scene flow estimation, especially owns the capacities of edge-preserving and occlusion handling.

Improved Exemplar based Image Inpainting for Partial Instance Occlusion Handling with K-means Clustering and YCbCr Color Space

The images acquired in real time outdoor environment are often subject to uneven illumination conditions, cloudy weather, lighting conditions. The instances of partial occlusions deteriorate the background modeling of such scenes. Varying illumination outdoor scene set of images with partial occlusions are addressed through this investigative work. There is a need for a restoration method that finds improved subjective perception and execution time. The proposed work focuses on novel amended exemplar model to improve the subjective perception. The exemplar inpainting method is improved through the color quantization with K-means clustering approach in YCbCr color space. Experimental validations and proposed method results show better improvement in qualitative and objective measures than existing methods. The average “Peak Signal to Noise Ratio (PSNR)” as 28.2869 and “Structural SIMilarity index (SSIM)” as 0.9759 of the proposed method has shown better results respectively both visually and with a tradeoff in time.

A Novel Occlusion-Aware Vote Cost for Light Field Depth Estimation.

Capturing the directions of light by light field cameras powers next-generation immersive multimedia applications. A critical problem in taking advantage of the rich visual information in light field images is depth estimation. Conventional light field depth estimation methods build a cost volume that measures the photo-consistency of pixels refocused to a range of depths, and the highest consistency indicates the correct depth. This strategy works well in most regions but usually generates blurry edges in the estimated depth map due to occlusions. Recent work shows that integrating occlusion models to light field depth estimation can largely reduce blurry edges. However, existing occlusion handling methods rely on complex edge-aided processing and post-refinement, and this reliance limits the resultant depth accuracy and impacts on the computational performance. In this paper, we propose a novel occlusion-aware vote cost (OAVC) which is able to accurately preserve edges in the depth map. Instead of using photo-consistency as an indicator of the correct depth, we construct a novel cost from a new perspective that counts the number of refocused pixels whose deviations from the central-view pixel are less than a small threshold, and utilizes that number to select the correct depth. The pixels from occluders are thus excluded in determining the correct depth. Without the use of any explicit occlusion handling methods, the proposed method can inherently preserve edges and produces high-quality depth estimates. Experimental results show that the proposed OAVC outperforms state-of-the-art light field depth estimation methods in terms of depth estimation accuracy and computational complexity.

Dense feature-based motion estimation in MV fluoroscopy during dynamic tumor tracking treatment: preliminary study on reduced aperture and partial occlusion handling

Quality assurance solutions to complement available motion compensation technologies are central for their safe routine implementation and success of treatment. This work presents a dense feature-based method for soft-tissue tumor motion estimation in megavoltage (MV) beam’s-eye-view (BEV) projections for potential intra-treatment monitoring during dynamic tumor tracking (DTT). Dense sampling and matching principles were employed to track a gridded set of features landmarks (FLs) in MV-BEV projections and estimate tumor motion, capable to overcome reduced field aperture and partial occlusion challenges. The algorithm’s performance was evaluated by retrospectively applying it to fluoroscopic sequences acquired at ∼2 frames s−1 (fps) for a dynamic phantom and two lung stereotactic body radiation therapy (SBRT) patients treated with DTT on the Vero SBRT system. First, a field-specific train image is initialized by sampling the tumor region at, S, pixel intervals on a grid using a representative frame from a stream of query frames. Sampled FLs are locally characterized in the form of descriptor vectors and geometric attributes representing the target. For motion tracking, subsequent query frames are likewise sampled, corresponding feature descriptors determined, and then patch-wise matched to the training set based on their descriptors and geometric relationships. FLs with high correspondence are pruned and used to estimate tumor displacement. In scenarios of partial occlusions, position is estimated from the set of correctly (visible) FLs on past observations. Reconstructed trajectories were benchmarked against ground-truth manual tracking using the root-mean-square (RMS) as a metric of positional accuracy. A total of 19 fluoroscopy sequences were analyzed. This included scenarios of field aperture obstruction during three-dimensional conformal, as well as step-and-shoot intensity modulated radiotherapy (IMRT) delivery assisted with DTT. The algorithm resolved target motion satisfactorily. The RMS was <1.2 mm and <1.8 mm for the phantom and the clinical dataset, respectively. Dense tracking showed promising results to overcome localization challenges at the field penumbra and partial obstruction by multi-leaf collimator (MLC). Motion retrieval was possible in ∼66% of the control points studied. In addition to MLC obstruction, changes in the external/internal breathing dynamics and baseline drifts were a major source of estimation bias. Dense feature-based tracking is a viable alternative. The algorithm is rotation-/scale-invariant and robust to photometric changes. Tracking multiple features may help overcome partial occlusion challenges by the MLC. This in turn opens up new possibilities for motion detection and intra-treatment monitoring during IMRT and potentially VMAT.

Coupled Network for Robust Pedestrian Detection With Gated Multi-Layer Feature Extraction and Deformable Occlusion Handling.

Pedestrian detection methods have been significantly improved with the development of deep convolutional neural networks. Nevertheless, detecting ismall-scaled pedestrians and occluded pedestrians remains a challenging problem. In this paper, we propose a pedestrian detection method with a couple-network to simultaneously address these two issues. One of the sub-networks, the gated multi-layer feature extraction sub-network, aims to adaptively generate discriminative features for pedestrian candidates in order to robustly detect pedestrians with large variations on scale. The second sub-network targets on handling the occlusion problem of pedestrian detection by using deformable regional region of interest (RoI)-pooling. We investigate two different gate units for the gated sub-network, namely, the channel-wise gate unit and the spatio-wise gate unit, which can enhance the representation ability of the regional convolutional features among the channel dimensions or across the spatial domain, repetitively. Ablation studies have validated the effectiveness of both the proposed gated multi-layer feature extraction sub-network and the deformable occlusion handling sub-network. With the coupled framework, our proposed pedestrian detector achieves promising results on both two pedestrian datasets, especially on detecting small or occluded pedestrians. On the CityPersons dataset, the proposed detector achieves the lowest missing rates (i.e. 40.78% and 34.60%) on detecting small and occluded pedestrians, surpassing the second best comparison method by 6.0% and 5.87%, respectively.

AeroRIT: A New Scene for Hyperspectral Image Analysis

We investigate applying convolutional neural network (CNN) architecture to facilitate aerial hyperspectral scene understanding and present a new hyperspectral dataset-AeroRIT-that is large enough for CNN training. To date the majority of hyperspectral airborne have been confined to various sub-categories of vegetation and roads and this scene introduces two new categories: buildings and cars. To the best of our knowledge, this is the first comprehensive large-scale hyperspectral scene with nearly seven million pixel annotations for identifying cars, roads, and buildings. We compare the performance of three popular architectures - SegNet, U-Net, and Res-U-Net, for scene understanding and object identification via the task of dense semantic segmentation to establish a benchmark for the scene. To further strengthen the network, we add squeeze and excitation blocks for better channel interactions and use self-supervised learning for better encoder initialization. Aerial hyperspectral image analysis has been restricted to small datasets with limited train/test splits capabilities and we believe that AeroRIT will help advance the research in the field with a more complex object distribution to perform well on. The full dataset, with flight lines in radiance and reflectance domain, is available for download at https://github.com/aneesh3108/AeroRIT. This dataset is the first step towards developing robust algorithms for hyperspectral airborne sensing that can robustly perform advanced tasks like vehicle tracking and occlusion handling.

Integration of the 3D Environment for UAV Onboard Visual Object Tracking

Single visual object tracking from an unmanned aerial vehicle (UAV) poses fundamental challenges such as object occlusion, small-scale objects, background clutter, and abrupt camera motion. To tackle these difficulties, we propose to integrate the 3D structure of the observed scene into a detection-by-tracking algorithm. We introduce a pipeline that combines a model-free visual object tracker, a sparse 3D reconstruction, and a state estimator. The 3D reconstruction of the scene is computed with an image-based Structure-from-Motion (SfM) component that enables us to leverage a state estimator in the corresponding 3D scene during tracking. By representing the position of the target in 3D space rather than in image space, we stabilize the tracking during ego-motion and improve the handling of occlusions, background clutter, and small-scale objects. We evaluated our approach on prototypical image sequences, captured from a UAV with low-altitude oblique views. For this purpose, we adapted an existing dataset for visual object tracking and reconstructed the observed scene in 3D. The experimental results demonstrate that the proposed approach outperforms methods using plain visual cues as well as approaches leveraging image-space-based state estimations. We believe that our approach can be beneficial for trafficmonitoring, video surveillance, and navigation.

A Bayesian Filter for Multi-View 3D Multi-Object Tracking With Occlusion Handling.

This paper proposes an online multi-camera multi-object tracker that only requires monocular detector training, independent of the multi-camera configurations, allowing seamless extension/deletion of cameras without retraining effort. The proposed algorithm has a linear complexity in the total number of detections across the cameras, and hence scales gracefully with the number of cameras. It operates in the 3D world frame, and provides 3D trajectory estimates of the objects. The key innovation is a high fidelity yet tractable 3D occlusion model, amenable to optimal Bayesian multi-view multi-object filtering, which seamlessly integrates, into a single Bayesian recursion, the sub-tasks of track management, state estimation, clutter rejection, and occlusion/misdetection handling. The proposed algorithm is evaluated on the latest WILDTRACKS dataset, and demonstrated to work in very crowded scenes on a new dataset.

A novel multi-view pedestrian detection database for collaborative Intelligent Transportation Systems

Recent advances in machine-learning, especially in deep neural networks have significantly accelerated the development and deployment of transport-oriented intelligent designs with increasingly high efficiency. While these technologies are exceptionally promising toward revolutionizing our current mobility and reducing the number of road accidents, the way to safe Intelligent Transportation Systems (ITS) remains long. Since pedestrians are the most vulnerable road users, designing accurate pedestrian detection methods is a priority task. However, traditional monocular pedestrian detection methods are limited, especially in occlusion handling. Hence, a collaborative perception scheme in which vehicles no longer restrict their input data to their immediate embedded sensors and rather exploit data from remote sensors is necessary to achieve a more comprehensive environment perception. In this work, we propose a novel public dataset: Infrastructure to Vehicle Multi-View Pedestrian Detection Database (I2V-MVPD) that combines synchronized images from both a mobile camera embedded in a car and a static camera in the road infrastructure. We also propose a new multi-view pedestrian detection framework based on collaborative intelligence between vehicles and infrastructure. Our results show a significant improvement in detection performance over monocular detection.

Image-Based Approach for Parking-Spot Detection with Occlusion Handling

AbstractWith the aid of advanced information technology, car parking space management is evolving dramatically toward an automatic way. The most efficient approach for parking-spot detection is bas...

Radius Nearest Neighbour Based Feature Classification for Occlusion Handling

Radius Nearest Neighbour Based Feature Classification for Occlusion Handling

Nighttime Vehicle Detection and Tracking with Occlusion Handling by Pairing Headlights and Taillights

In recent years, vision-based vehicle detection has received considerable attention in the literature. Depending on the ambient illuminance, vehicle detection methods are classified as daytime and nighttime detection methods. In this paper, we propose a nighttime vehicle detection and tracking method with occlusion handling based on vehicle lights. First, bright blobs that may be vehicle lights are segmented in the captured image. Then, a machine learning-based method is proposed to classify whether the bright blobs are headlights, taillights, or other illuminant objects. Subsequently, the detected vehicle lights are tracked to further facilitate the determination of the vehicle position. As one vehicle is indicated by one or two light pairs, a light pairing process using spatiotemporal features is applied to pair vehicle lights. Finally, vehicle tracking with occlusion handling is applied to refine incorrect detections under various traffic situations. Experiments on two-lane and four-lane urban roads are conducted, and a quantitative evaluation of the results shows the effectiveness of the proposed method.

High-Quality Video Generation from Static Structural Annotations

This paper proposes a novel unsupervised video generation that is conditioned on a single structural annotation map, which in contrast to prior conditioned video generation approaches, provides a good balance between motion flexibility and visual quality in the generation process. Different from end-to-end approaches that model the scene appearance and dynamics in a single shot, we try to decompose this difficult task into two easier sub-tasks in a divide-and-conquer fashion, thus achieving remarkable results overall. The first sub-task is an image-to-image (I2I) translation task that synthesizes high-quality starting frame from the input structural annotation map. The second image-to-video (I2V) generation task applies the synthesized starting frame and the associated structural annotation map to animate the scene dynamics for the generation of a photorealistic and temporally coherent video. We employ a cycle-consistent flow-based conditioned variational autoencoder to capture the long-term motion distributions, by which the learned bi-directional flows ensure the physical reliability of the predicted motions and provide explicit occlusion handling in a principled manner. Integrating structural annotations into the flow prediction also improves the structural awareness in the I2V generation process. Quantitative and qualitative evaluations over the autonomous driving and human action datasets demonstrate the effectiveness of the proposed approach over the state-of-the-art methods. The code has been released: https://github.com/junting/seg2vid .

Collaborative model tracking with robust occlusion handling

Currently, the discriminative correlation filter-based trackers have achieved higher tracking accuracy. However, visual tracking still faces challenges in terms of heavy occlusion, scale variation and so on. In this study, the authors intend to solve heavy occlusion by introducing collaborative model into classifier-box. Firstly, they introduce complex colour features into correlation filter tracker to improve the effect of the tracker. Secondly, they introduce a multi-scale method into their tracker to ease the scale problem. Thirdly, in order to solve the heavy occlusion in the tracking process, they adopt the locally weighted distance and classifier-box. Their algorithm achieves distance precision rates of 81.7 and 77.4% on OTB2013 dataset and OTB2015 dataset, respectively. Their contribution focuses on solving heavy occlusion by using colour features, locally weighted distance and classifier-box. The experimental results on OTB2013 and OTB2015 datasets demonstrate their algorithm to perform better than state-of-the-art methods.

Intertarget Occlusion Handling in Multiextended Target Tracking Based on Labeled Multi-Bernoulli Filter Using Laser Range Finder

Multiextended target tracking (METT) has important applications in many fields, including autonomous vehicles, traffic flow monitoring, etc. However, as an acknowledged METT algorithm, the labeled multi-Bernoulli (LMB) filter cannot handle the intertarget occlusion problem, which frequently observed among METT, leading to an estimated trajectory break and even target loss. To tackle this problem, this article proposes an improved LMB filter with intertarget occlusion handling ability for METT. First, an intertarget occlusion probability (IOP) model based on a labeled random finite set using a laser range finder is developed. The IOP model theoretically describes the probability that each tracked target being occluded, considering both partial and full occlusion. Second, a true detection probability (TDP) model is presented to describe the probability that a target will generate corresponding measurement of a sensor in the presence of occlusion. The TDPs of tracked targets provide a way to integrate their estimated occlusion probabilities into the LMB filter to improve its capability of handling occluded tracks. We carried out multiple vehicle tracking simulation and field test using a laser range finder. Both simulation and experiment results illustrate that our improved LMB filter with occlusion handling ability can successfully tackle the intertarget occlusion problem and outperform conventional algorithms.

Overcoming Occlusion in the Automotive Environment—A Review

Accurate and consistent vulnerable road user detection remains one of the most challenging perception tasks for autonomous vehicles. One of the most complex outstanding issues is partial occlusion, where a sensor has only a partial view of the target object due to a foreground object that partially obscures the target. A review of occlusion detection and handling solutions for the automotive environment is presented by this research. This article first discusses object detection by the human visual system, provides an overview of occlusion reasoning in computer vision, presents a summary of occlusion handling strategies in pedestrian, vehicle and object detection applications in the automotive environment. A selection of the remaining challenges to achieving the required level of object detection performance for safe autonomous driving are also discussed.