Occlusion Handling Research Articles

The Missing Data Encoder: Cross-Channel Image Completion with Hide-and-Seek Adversarial Network

Image completion is the problem of generating whole images from fragments only. It encompasses inpainting (generating a patch given its surrounding), reverse inpainting/extrapolation (generating the periphery given the central patch) as well as colorization (generating one or several channels given other ones). In this paper, we employ a deep network to perform image completion, with adversarial training as well as perceptual and completion losses, and call it the “missing data encoder” (MDE). We consider several configurations based on how the seed fragments are chosen. We show that training MDE for “random extrapolation and colorization” (MDE-REC), i.e. using random channel-independent fragments, allows a better capture of the image semantics and geometry. MDE training makes use of a novel “hide-and-seek” adversarial loss, where the discriminator seeks the original non-masked regions, while the generator tries to hide them. We validate our models qualitatively and quantitatively on several datasets, showing their interest for image completion, representation learning as well as face occlusion handling.

Dynamic Graph Representation for Occlusion Handling in Biometrics

The generalization ability of Convolutional neural networks (CNNs) for biometrics drops greatly due to the adverse effects of various occlusions. To this end, we propose a novel unified framework integrated the merits of both CNNs and graphical models to learn dynamic graph representations for occlusion problems in biometrics, called Dynamic Graph Representation (DGR). Convolutional features onto certain regions are re-crafted by a graph generator to establish the connections among the spatial parts of biometrics and build Feature Graphs based on these node representations. Each node of Feature Graphs corresponds to a specific part of the input image and the edges express the spatial relationships between parts. By analyzing the similarities between the nodes, the framework is able to adaptively remove the nodes representing the occluded parts. During dynamic graph matching, we propose a novel strategy to measure the distances of both nodes and adjacent matrixes. In this way, the proposed method is more convincing than CNNs-based methods because the dynamic graph method implies a more illustrative and reasonable inference of the biometrics decision. Experiments conducted on iris and face demonstrate the superiority of the proposed framework, which boosts the accuracy of occluded biometrics recognition by a large margin comparing with baseline methods.

CNN and HOG based comparison study for complete occlusion handling in human tracking

Applications in the Human Machine Interaction (HMI) field are increasing since people and robots share the same environment and perform activities together. Therefore, robust detection and strong tracking methods are required. This study is about human detection and tracking. For detection, two different approaches, Histogram of Oriented Gradients (HOG)-Support Vector Machine (SVM) and Convolutional Neural Network (CNN) are used. The video used in the application is recorded in a noisy environment, and in some frames, the human is completely covered by other objects in the environment, i.e. exposed to complete occlusion. For this reason, Kalman Filter (KF) and Particle Filter (PF) based tracking is performed. Different from these two traditional filters, a hybrid Kalman-Particle Filter (KPF) has also been proposed. The contribution of this study is to compare CNN with HOG-SVM, which is described as the most successful human detection method. As a result of the study, it was found that CNN provides successful results especially in case of occlusion. This study is also important in terms of proposing a hybrid KPF and comparing the filters in the case of complete occlusion. The results showed that for human tracking, CNN using KF performed better performance throughout the video. The proposed KPF also outperformed PF and this superiority became much more prominent in the case of complete occlusion.

Applying R-spatiogram in Object Tracking for Occlusion Handling

Object tracking is one of the most important problems in computer vision. The aim of video tracking is to extract the trajectories of a target or object of interest, i.e. accurately locate a moving target in a video sequence and discriminate target from non-targets in the feature space of the sequence. So, feature descriptors can have significant effects on such discrimination. In this paper, we use the basic idea of many trackers which consists of three main components of the reference model, i.e., object modeling, object detection and localization, and model updating. However, there are major improvements in our system. Our forth component, occlusion handling, utilizes the r-spatiogram to detect the best target candidate. While spatiogram contains some moments upon the coordinates of the pixels, r-spatiogram computes region-based compactness on the distribution of the given feature in the image that captures richer features to represent the objects. The proposed research develops an efficient and robust way to keep tracking the object throughout video sequences in the presence of significant appearance variations and severe occlusions. The proposed method is evaluated on the Princeton RGBD tracking dataset considering sequences with different challenges and the obtained results demonstrate the effectiveness of the proposed method.

Pyramidal Part-Based Model for Partial Occlusion Handling in Pedestrian Classification

Pedestrian detection and classification are of increased interest in the intelligent transportation system (ITS), and among the challenging issues, we can find limitations of tiny and occluded appearances, large variation of human pose, cluttered background, and complex environment. In fact, a partial occlusion handling is important in the case of detecting pedestrians, in order to avoid accidents between pedestrians and vehicles, since it is difficult to detect when pedestrians are suddenly crossing the road. To solve the partial occlusion problem, a pyramidal part-based model (PPM) is proposed to obtain a more accurate prediction based on the majority vote of the confidence score of the visible parts by cascading the pyramidal structure. The experimental results on the proposed scheme achieved 96.25% accuracy on the INRIA dataset and 81% accuracy on the PSU (Prince of Songkla University) dataset. Our proposed model can be applied in the real-world environment to classify the occluded part of pedestrians with the various information of part representation at each pyramid layer.

Vehicle Make and Model Recognition using Bag of Expressions.

Vehicle make and model recognition (VMMR) is a key task for automated vehicular surveillance (AVS) and various intelligent transport system (ITS) applications. In this paper, we propose and study the suitability of the bag of expressions (BoE) approach for VMMR-based applications. The method includes neighborhood information in addition to visual words. BoE improves the existing power of a bag of words (BOW) approach, including occlusion handling, scale invariance and view independence. The proposed approach extracts features using a combination of different keypoint detectors and a Histogram of Oriented Gradients (HOG) descriptor. An optimized dictionary of expressions is formed using visual words acquired through k-means clustering. The histogram of expressions is created by computing the occurrences of each expression in the image. For classification, multiclass linear support vector machines (SVM) are trained over the BoE-based features representation. The approach has been evaluated by applying cross-validation tests on the publicly available National Taiwan Ocean University-Make and Model Recognition (NTOU-MMR) dataset, and experimental results show that it outperforms recent approaches for VMMR. With multiclass linear SVM classification, promising average accuracy and processing speed are obtained using a combination of keypoint detectors with HOG-based BoE description, making it applicable to real-time VMMR systems.

Attention guided neural network models for occluded pedestrian detection

Occluded pedestrian detection has been a research difficulty in computer vision for a long time. The conventional approach to solve this problem is to learn partial detectors, which can be properly integrated for occlusion handling. However, the efficiency of this type of methods is limited in practical applications since the partial detectors can not cover all occlusion patterns. In this paper, an attention guided neural network model (AGNN) is proposed for the occlusion handling of pedestrian detections, which is inspired by the approaches of sentiment classification. Firstly, a fixed-size window slides on a still image without overlapping to generate a set of sub-images. Secondly, a convolutional neural network is employed to extract the high-level features from resulting sub-images. Then, the attention network performs local feature weighting, from which the features representing the body parts of pedestrians are selected. Finally, the feature sequences are classified by recurrent neural network in proper order based on the weighted results. In addition, we explore different mechanisms of attention guidance on the detector for the detections. Compared with the state-of-the-art methods on two standard pedestrian datasets, experimental results demonstrate the comparable performance of our approach in terms of miss rates.

A Lightweight Neural Network for Monocular View Generation With Occlusion Handling.

In this article, we present a very lightweight neural network architecture, trained on stereo data pairs, which performs view synthesis from one single image. With the growing success of multi-view formats, this problem is indeed increasingly relevant. The network returns a prediction built from disparity estimation, which fills in wrongly predicted regions using a occlusion handling technique. To do so, during training, the network learns to estimate the left-right consistency structural constraint on the pair of stereo input images, to be able to replicate it at test time from one single image. The method is built upon the idea of blending two predictions: a prediction based on disparity estimation and a prediction based on direct minimization in occluded regions. The network is also able to identify these occluded areas at training and at test time by checking the pixelwise left-right consistency of the produced disparity maps. At test time, the approach can thus generate a left-side and a right-side view from one input image, as well as a depth map and a pixelwise confidence measure in the prediction. The work outperforms visually and metric-wise state-of-the-art approaches on the challenging KITTI dataset, all while reducing by a very significant order of magnitude (5 or 10 times) the required number of parameters (6.5 M).

A novel change-detection scheduler for a network of depth sensors

In many monitoring applications such as smart home and surveillance, deployment of multiple depth sensors increases monitoring area and offers better occlusion handling which is not sensitive to illumination condition in comparison with RGB sensors. However, multiple sensors also increase the volume of data associated with signal processing alongside the associated computational complexity and power consumption. In order to address these drawbacks, this paper proposes a novel change detection algorithm that can be used as a part of a sensor scheduler in a centralized (e.g. star) network configuration. Initially, each sensor in the network performs a unique single scan of the common environment in order to detect any incremental changes in the sensed depth signal. This initial change detection is then used as a basis for several follow-up tasks such as foreground segmentation, background detection, target detection, and tracking for monitoring tasks. Here, instead of processing a complete depth frame, we proposed to utilize a collection of 1D scans of the depth frames. A confidence function is defined that can be used to estimate the reliability of the detected changes in each sensor and to reduce any false positive events which can be triggered by the noise and outliers. Analysis of the proposed confidence function is carried out through performance analysis in the presence of sensor noise and other parameters which can affect the reliability of the sensed data of each sensor. Finally, a score function is defined based on the confidence of the detected parameters and sensor resolution in order to rank and match sensors with the associated objects to be tracked. It results in tracking target(s) by a sensor (or sensors) that offer a high tracking score. This approach offers many advantages such as decreasing the overall system power consumption by placing the sensors with a low confidence value on standby mode and reducing the overall computational overheads.

Occlusion detection and processing using optical flow and particle filter

Object tracking systems continue to be an intensive area of research, for which detection and processing of occlusion is a well-known challenge. This paper proposes a new approach to detection and handling of occlusion based on the integration of two known techniques, optical flow and particle filtering. Results of preliminary experiments show that the proposed method can detect and overcome the occlusion problem successfully during the tracking process.

Spatio-temporal silhouette sequence reconstruction for gait recognition against occlusion

Gait-based features provide the potential for a subject to be recognized even from a low-resolution image sequence, and they can be captured at a distance without the subject’s cooperation. Person recognition using gait-based features (gait recognition) is a promising real-life application. However, several body parts of the subjects are often occluded because of beams, pillars, cars and trees, or another walking person. Therefore, gait-based features are not applicable to approaches that require an unoccluded gait image sequence. Occlusion handling is a challenging but important issue for gait recognition. In this paper, we propose silhouette sequence reconstruction from an occluded sequence (sVideo) based on a conditional deep generative adversarial network (GAN). From the reconstructed sequence, we estimate the gait cycle and extract the gait features from a one gait cycle image sequence. To regularize the training of the proposed generative network, we use adversarial loss based on triplet hinge loss incorporating Wasserstein GAN (WGAN-hinge). To the best of our knowledge, WGAN-hinge is the first adversarial loss that supervises the generator network during training by incorporating pairwise similarity ranking information. The proposed approach was evaluated on multiple challenging occlusion patterns. The experimental results demonstrate that the proposed approach outperforms the existing state-of-the-art benchmarks.

Rotation invariant person tracker using top view

Person tracking is considered an important application in the field of video surveillance. A top view camera provides a wide coverage of scene and better handling of occlusion as compared to frontal view. The proposed top view based person tracking method mainly contains four modules, namely BLOB detection, standardisation, size estimation and tracking. In blob detection, foreground is extracted using segmentation, statistical operation, connected component labelling and morphological operations. The top view is radial symmetric thus using this property, the extracted blob is transformed to upright position using geometric transformations. This effectively makes the blob/person rotation invariant in the scene. Basic shape based features including width, height and body ratio are measured for each blob. On the basis of these features, the algorithm effectively distinguishes a person, no person as well as merged person blob. A simple tracker for each blob is formerly created to maintain different parameters for example, blob name, blob width & height, blob x & y coordinates, blob time, blob history, blob size and blob ratio. The proposed algorithm along with the seven attractive/popular tracking methods have been tested on different test sequences. The experimental results show that the proposed algorithm significantly improves results by achieving true detection rate of 95%.

Image Composition of Partially Occluded Objects

AbstractImage composition extracts the content of interest (COI) from a source image and blends it into a target image to generate a new image. In the majority of existing works, the COI is manually extracted and then overlaid on top of the target image. However, in practice, it is often necessary to deal with situations in which the COI is partially occluded by the target image content. In this regard, both tasks of extracting the COI and cropping its occluded part require intensive user interactions, which are laborious and seriously reduce the composition efficiency. This paper addresses the aforementioned challenges by proposing an efficient image composition method. First, we extract the semantic contents of the images by using state‐of‐the‐art deep learning methods. Therefore, the COI can be selected with clicks only, which can greatly reduce the demanded user interactions. Second, according to the user's operations (such as translation or scale) on the COI, we can effectively infer the occlusion relationships between the COI and the contents of the target image. Thus, the COI can be adaptively embedded into the target image without concern about cropping its occluded part. Therefore, the procedures of content extraction and occlusion handling can be significantly simplified, and work efficiency is remarkably improved. Experimental results show that compared to existing works, our method can reduce the number of user interactions to approximately one‐tenth and increase the speed of image composition by more than ten times.

Deep Learning for Pedestrian Detection

Pedestrian detection has so far worked efficiently using four basic important components namely: feature extraction, deformation handling, occlusion handling, and individual or sequential classification proposed in existing methods.

A Framework for Learning Depth From a Flexible Subset of Dense and Sparse Light Field Views.

In this paper, we propose a learning-based depth estimation framework suitable for both densely and sparsely sampled light fields. The proposed framework consists of three processing steps: initial depth estimation, fusion with occlusion handling, and refinement. The estimation can be performed from a flexible subset of input views. The fusion of initial disparity estimates, relying on two warping error measures, allows us to have an accurate estimation in occluded regions and along the contours. In contrast with methods relying on the computation of cost volumes, the proposed approach does not need any prior information on the disparity range. Experimental results show that the proposed method outperforms state-of-the-art light fields depth estimation methods, including prior methods based on deep neural architectures.

Offset Aperture: A Passive Single-Lens Camera for Depth Sensing

Numerous camera sensors have been recently proposed to estimate the depth of objects in the scene. However, many of these sensors either require multiple imaging elements or an active illuminant. In this paper, we propose a new depth camera, termed offset aperture (OA) camera, which provides depth of the scene with a single shot without an active illuminant. The camera is based on a 4-color sensor that senses the infrared (IR) signal along with the visible RGB signals. Separate OAs are used for IR and RGB images. A depth-dependent disparity is generated between the IR and RGB images. There is no disparity across the R, G, and B channel images, which allows the RGB image to be well-aligned. Numerous techniques are used in this work to observe good-quality depth maps and images. Leakage compensation has been proposed to remove crosstalk between the IR and RGB images. A comparison with another 4-color sensor called Dual Aperture camera shows that OA camera provides 88% improvement in depth sensitivity and 45% improvement in color quality, on average. The proposed OA camera can be used for numerous applications including occlusion handling in stereo, selective focus, and night vision. The advantages of the proposed OA camera include competitive image quality, capability of depth extraction, well-aligned RGB images, and small footprint of the camera, which make OA camera a suitable candidate for mobile applications.

Transmission Estimation by Complex Assumption with Occlusion Handling

Transmission estimation is a critical step in single-image dehazing. The estimate of each pixel describes the portion of the scene radiance that is degraded by hazing and finally reaches the image sensor. Transmission estimation is an underconstrained problem, and, thus, various assumptions, priors, and models are employed to make it solvable. However, most of the previous methods did not consider the different assumptions simultaneously, which, therefore, did not correctly reflect the previous assumptions in the final result. This paper focuses on this problem and proposes a method using an energy function that clearly defines the optimal transmission map and combines the assumptions from three aspects: fidelity, smoothness, and occlusion handling, simultaneously. Fidelity is measured by a novel principle derived from the dark channel prior, smoothness is described by the assumption of piecewise smoothening, and occlusion handling is achieved based on a new proposed feature. The transmissions are estimated by searching for the optimal solution of the function that can retain all the employed assumptions simultaneously. The proposed method is evaluated on the synthetic images of two datasets and various natural images. The results show that there is remarkable fidelity and smoothness in the transmission and that a good performance is exhibited for haze removal.

Segment-Based Disparity Refinement With Occlusion Handling for Stereo Matching.

In this paper, we propose a disparity refinement method that directly refines the winner-take-all (WTA) disparity map by exploring its statistical significance. According to the primary steps of the segment-based stereo matching, the reference image is over-segmented into superpixels and a disparity plane is fitted for each superpixel by an improved random sample consensus (RANSAC). We design a two-layer optimization to refine the disparity plane. In the global optimization, mean disparities of superpixels are estimated by Markov random field (MRF) inference, and then, a 3D neighborhood system is derived from the mean disparities for occlusion handling. In the local optimization, a probability model exploiting Bayesian inference and Bayesian prediction is adopted and achieves second-order smoothness implicitly among 3D neighbors. The two-layer optimization is a pure disparity refinement method because no correlation information between stereo image pairs is demanded during the refinement. Experimental results on the Middlebury and KITTI datasets demonstrate that the proposed method can perform accurate stereo matching with a faster speed and handle the occlusion effectively. It can be indicated that the "matching cost computation + disparity refinement" framework is a possible solution to produce accurate disparity map at low computational cost.

A labeled random finite set online multi-object tracker for video data

This paper proposes an online multi-object tracking algorithm for image observations using a top-down Bayesian formulation that seamlessly integrates state estimation, track management, handling of false positives, false negatives and occlusion into a single recursion. This is achieved by modeling the multi-object state as labeled random finite set and using the Bayes recursion to propagate the multi-object filtering density forward in time. The proposed filter updates tracks with detections but switches to image data when detection loss occurs, thereby exploiting the efficiency of detection data and the accuracy of image data. Furthermore the labeled random finite set framework enables the incorporation of prior knowledge that detection loss in the middle of the scene are likely to be due to occlusions. Such prior knowledge can be exploited to improve occlusion handling, especially long occlusions that can lead to premature track termination in on-line multi-object tracking. Tracking performance is compared to state-of-the-art algorithms on synthetic data and well-known benchmark video datasets.

An improved long-term correlation tracking method with occlusion handling

By improving the long-term correlation tracking (LCT) algorithm, an effective object tracking method, improved LCT (ILCT), is proposed to address the issue of occlusion. If the object is judged being occluded by the designed criterion, which is based on the characteristic of response value curve, an added re-detector will perform re-detection, and the tracker is ordered to stop. Besides, a filtering and adoption strategy of re-detection results is given to choose the most reliable one for the re-initialization of the tracker. Extensive experiments are carried out under the conditions of occlusion, and the results demonstrate that ILCT outperforms some state-of-the-art methods in terms of accuracy and robustness.