Occluded Regions Research Articles

Sniffer-Net: quantitative evaluation of smoke in the wild based on spatial–temporal motion spectrum

Smoke detection plays an essential role in the wild video surveillance systems for abnormal events warning. In this paper, we introduced a dedicated neural network structure named Sniffer-Net to simultaneously extract smoke dynamic feature robustly and evaluate the smoke concentration accurately. Firstly, we utilize an improved LiteFlowNet to estimate the global optical flow from image sequence. Meanwhile, a Marr–Hildreth method is brought up and fused into this network to distinguish and eliminate occluded regions from global flow map. Then, an evaluation module based on Context-Encoder network is put forward specially to quantify smoke concentration levels. This network, following the improved LiteFlowNet, is modified through replacing the loss function and removing the multiscale scheme and trained to infer approximate smoke optical flow behind occlusion regions. Starting from the statistical view, the irregular RGB/HSV feature spaces are converted into a specific quantitative evaluation space. As a result, the whole evaluation system is responsible to transform the distribution of irregular smoke motion feature into a quantified form of representation. In turn, this transformation endows the system with a novel numerical standard for smoke concentration evaluation. Finally, an accuracy assessment method is applied to compare the results of detected smoke concentration with the human experience prior model, which feedback the accuracy and false detection rate of system algorithm. In the experiments of five smoke datasets, our proposed smoke detection approach is superior to other state-of-the-art methods, and concentration algorithm achieves the satisfactory performance of 97.3% accuracy on some specialized dataset.

Flow-guided feature propagation with occlusion aware detail enhancement for hand segmentation in egocentric videos

Recently, deep convolutional networks have achieved great success in image hand segmentation. However, such CNN-based methods mainly focus on per-frame inference, which is inefficient for hand segmentation in videos as continuity and redundancy exist in contiguous video frames. We present an approach to extending CNN-based hand segmentation methods from still image to video. It consists of two main branches: flow-guided feature propagation and light-weight occlusion aware detail enhancement. The flow-guided feature propagation branch runs an image segmentation network only on sparse frames and warps intermediate features to other frames according to the cross-frame flow field. Compared with the per-frame inference, it achieves significant speedup while causing large accuracy degradation due to distortion and occlusion issues in the warping. By introducing the light-weight occlusion aware detail enhancement branch, which extracts low-level detail features from each frame with spatial attention on occlusion regions, our approach becomes more robust against the distortion and occlusion issues and achieves a better accuracy–latency tradeoff. Experiments on three public egocentric video datasets, namely Egohands, GTEA and EDSH, have demonstrated the effectiveness and efficiency of our approach.

An FPGA-based real-time occlusion robust stereo vision system using semi-global matching

Stereo matching approaches are an appealing choice for acquiring depth information in a number of video processing applications. It is desirable that these solutions generate dense, robust disparity maps in real time. However, occlusion regions may disturb the applications that need these maps. Among the best of these approaches is the semi-global matching (SGM) technique. This paper presents an FPGA-based stereo vision system based on SGM. This system calculates disparity maps by streaming, which are scalable to several resolutions and disparity ranges. To increase the robustness of the SGM technique even further, the present work has implemented a combination of the gradient filter and the sampling-insensitive absolute difference in the pre-processing phase. Furthermore, as a post-processing step, this paper proposes a novel streaming architecture to detect noisy and occluded regions. The FPGA-based implementations of the proposed stereo matching system in two distinct heterogeneous architecture (GPP—general purpose processor, and FPGA) were evaluated using the Middlebury stereo vision benchmark. The achieved results reported a frame rate of 25 FPS for the disparity maps processing in HD resolution (1024 $$\times$$ 768 pixels), with 256 disparity levels. The results have demonstrated that the memory utilization, processing performance, and accuracy are among the best of FPGA-based stereo vision systems.

Occlusion removal based on epipolar plane images in integral imaging system

Generally, the resolution of reconstructed three-dimensional (3D) images in integral imaging can be seriously degraded by undesired occlusions, where the undesired information of the occlusion overlaps the 3D images to be reconstructed. To solve the problem of occlusion, we proposed an occlusion removal system based on epipolar plane images (EPIs) to remove occlusions in integral imaging. In the system, we proposed a contour based object extraction method to extract the occlusions, which is robust to pose variation of the 3D occlusion objects. Due to the traditional image restoration method is not suitable for the occlusion region with significant textured structure, we proposed a new restoration method based on the EPI. In our proposed restoration method, we utilize the spinning parallelogram operator to analyze the linear structure of the EPI to restore the images. Finally, the enhanced elemental images (EIs) without occlusion are reconstructed in the integral imaging system. The preliminary experiments are presented to verify the feasibility of the proposed system.

Learning a Deep Convolutional Network for Colorization in Monochrome-Color Dual-Lens System

In the monochrome-color dual-lens system, the gray image captured by the monochrome camera has better quality than the color image from the color camera, but does not have color information. To get high-quality color images, it is desired to colorize the gray image with the color image as reference. Related works usually use hand-crafted methods to search for the best-matching pixel in the reference image for each pixel in the input gray image, and copy the color of the best-matching pixel as the result. We propose a novel deep convolution network to solve the colorization problem in an end-to-end way. Based on our observation that, for each pixel in the input image, there usually exist multiple pixels in the reference image that have the correct colors, our method performs weighted average of colors of the candidate pixels in the reference image to utilize more candidate pixels with correct colors. The weight values between pixels in the input image and the reference image are obtained by learning a weight volume using deep feature representations, where an attention operation is proposed to focus on more useful candidate pixels and a 3-D regulation is performed to learn with context information. In addition, to correct wrongly colorized pixels in occlusion regions, we propose a color residue joint learning module to correct the colorization result with the input gray image as guidance. We evaluate our method on the Scene Flow, Cityscapes, Middlebury, and Sintel datasets. Experimental results show that our method largely outperforms the state-of-the-art methods.

Joint view synthesis and disparity refinement for stereo matching

Typical stereo algorithms treat disparity estimation and view synthesis as two sequential procedures. In this paper, we consider stereo matching and view synthesis as two complementary components, and present a novel iterative refinement model for joint view synthesis and disparity refinement. To achieve the mutual promotion between view synthesis and disparity refinement, we apply two key strategies, disparity maps fusion and disparity-assisted plane sweep-based rendering (DAPSR). On the one hand, the disparity maps fusion strategy is applied to generate disparity map from synthesized view and input views. This strategy is able to detect and counteract disparity errors caused by potential artifacts from synthesized view. On the other hand, the DAPSR is used for view synthesis and updating, and is able to weaken the interpolation errors caused by outliers in the disparity maps. Experiments on Middlebury benchmarks demonstrate that by introducing the synthesized view, disparity errors due to large occluded region and large baseline are eliminated effectively and the synthesis quality is greatly improved.

Low Complexity, Hardware-Efficient Neighbor-Guided SGM Optical Flow for Low-Power Mobile Vision Applications

Accurate, low-latency, and energy-efficient optical flow estimation is a fundamental kernel function to enable several real-time vision applications on mobile platforms. This paper presents neighbor-guided semi-global matching (NG-fSGM), a new low-complexity optical flow algorithm tailored for low-power mobile applications. NG-fSGM obtains high accuracy optical flow by aggregating local matching costs over a semi-global region, successfully resolving local ambiguity in texture-less and occluded regions. The proposed NG-fSGM aggressively prunes the search space based on neighboring pixels’ information to significantly lower the algorithm complexity from the original fSGM. As a result, NG-fSGM achieves $17.9{\times}$ reduction in the number of computations and $8.37{\times}$ reduction in memory space compared to the original fSGM without compromising its algorithm accuracy. A multicore architecture for NG-fSGM is implemented in hardware to quantify algorithm complexity and power consumption. The proposed architecture realizes NG-fSGM with overlapping blocks processed in parallel to enhance throughput and to lower power consumption. The eight-core architecture achieves 20 M pixel/s (66 frames/s for VGA) throughput with 9.6 mm2 area at 679.2-mW power consumption in 28-nm node.

Detection and Restoration of Image from Multi-Color Fence Occlusions

In Image De-fencing, segmentation and restoration of occluded fence region from the images are very challenging, when the desired object is occluded with an undesired object. This undesired object may be scattered in the whole image region. Existing algorithms can only detect single colored fences at a time from the digital images. This paper presents a multi-colored fence detection algorithm. Multi-threshold segmentation technique is used to segment the fence in the image. The segmented mask is amended by using morphological operations. To restore the fence occluded area in an image hybrid inpainting technique is used. Obtained results after experimentation are compared with the start-of-art image de-fencing technique.

A Framework for Learning Depth From a Flexible Subset of Dense and Sparse Light Field Views.

In this paper, we propose a learning-based depth estimation framework suitable for both densely and sparsely sampled light fields. The proposed framework consists of three processing steps: initial depth estimation, fusion with occlusion handling, and refinement. The estimation can be performed from a flexible subset of input views. The fusion of initial disparity estimates, relying on two warping error measures, allows us to have an accurate estimation in occluded regions and along the contours. In contrast with methods relying on the computation of cost volumes, the proposed approach does not need any prior information on the disparity range. Experimental results show that the proposed method outperforms state-of-the-art light fields depth estimation methods, including prior methods based on deep neural architectures.

Unsupervised Learning for Depth, Ego-Motion, and Optical Flow Estimation Using Coupled Consistency Conditions

Herein, we propose an unsupervised learning architecture under coupled consistency conditions to estimate the depth, ego-motion, and optical flow. Previously invented learning techniques in computer vision adopted a large amount of the ground truth dataset for network training. A ground truth dataset, including depth and optical flow collected from the real world, requires tremendous effort in pre-processing due to the exposure to noise artifacts. In this paper, we propose a framework that trains networks while using a different type of data with combined losses that are derived from a coupled consistency structure. The core concept is composed of two parts. First, we compare the optical flows, which are estimated from both the depth plus ego-motion and flow estimation network. Subsequently, to prevent the effects of the artifacts of the occluded regions in the estimated optical flow, we compute flow local consistency along the forward–backward directions. Second, synthesis consistency enables the exploration of the geometric correlation between the spatial and temporal domains in a stereo video. We perform extensive experiments on the depth, ego-motion, and optical flow estimation on the Karlsruhe Institute of Technology and Toyota Technological Institute (KITTI) dataset. We verify that the flow local consistency loss improves the optical flow accuracy in terms of the occluded regions. Furthermore, we also show that the view-synthesis-based photometric loss enhances the depth and ego-motion accuracy via scene projection. The experimental results exhibit the competitive performance of the estimated depth and the optical flow; moreover, the induced ego-motion is comparable to that obtained from other unsupervised methods.

3D geometry-based face recognition in presence of eye and mouth occlusions

This study proposes a novel occlusions detection and restoration strategy. The aim is to success with 3D face recognition even when faces are partially occluded by external objects. The method, which relies on geometrical facial properties, is designed for managing two types of facial occlusions (eye and mouth occlusions due to hands). First occlusions are detected and (if present) classified, by considering their effects on the 3D points cloud. Then, the occluded regions are progressively removed, and finally, the non-occluded symmetrical regions are used to restore the missing information. After the restoration process, face recognition is performed relying on the restored facial information and on the localized landmarks. The landmarking methodology relies on derivatives and on 12 differential geometry descriptors. The discriminating features adopted for facial comparison include shape index histograms, Euclidean and geodetical distances between landmarks, facial curves, and nose volume. Obtained recognition rates, evaluated on the whole Bosphorus database and on our private dataset, ranging from 92.55 to 97.20% depending on the completeness of data.

Face detection based on occlusion area detection and recovery

Face detection is an important part of face image processing. In many cases, face images have occlusion problems. In this paper, the POOA (positioning the optimal occlusion area) algorithm is proposed for the problem of occlusion face detection. After obtaining the data of the saliency detection processing, firstly, the algorithm computes an average gray value according to the face image, and multiplies the appropriate coefficient as a threshold to obtain a binary image. Then, using the idea of the Haar feature, the two features of “large rectangle” and “large T shape” are used for retrieval, and the occlusion region of the face is obtained by combining the binary images. Finally, a robust principal component analysis (PCA) method is used to obtain the best projection of the occlusion face, and the face occlusion area is filled. The algorithm proposed in this paper is fast. The Adaboost method has achieved good results in terms of occlusion area, size and shape, and the detection precision has also been improved to varying degrees.

Peeking behind objects: Layered depth prediction from a single image

While conventional depth estimation can infer the geometry of a scene from a single RGB image, it fails to estimate scene regions that are occluded by foreground objects. This limits the use of depth prediction in augmented and virtual reality applications, that aim at scene exploration by synthesizing the scene from a different vantage point, or at diminished reality. To address this issue, we shift the focus from conventional depth map prediction to the regression of a specific data representation called Layered Depth Image (LDI), which contains information about the occluded regions in the reference frame and can fill in occlusion gaps in case of small view changes. We propose a novel approach based on Convolutional Neural Networks (CNNs) to jointly predict depth maps and foreground separation masks used to condition Generative Adversarial Networks (GANs) for hallucinating plausible color and depths in the initially occluded areas. We demonstrate the effectiveness of our approach for novel scene view synthesis from a single image.

Ultra Wideband Positioning Technology for Accident Ships under Adverse Sea Condition

ABSTRACT Zhang, K.; Shen, C.; Gao, Q.; Zheng, L.; Wang, H., and Li, Z., 2018. Ultra wideband positioning technology for accident ships under adverse sea condition. In: Liu, Z.L. and Mi, C. (eds.), Advances in Sustainable Port and Ocean Engineering. Journal of Coastal Research, Special Issue No. 83, pp. 902–907. Coconut Creek (Florida), ISSN 0749-0208. Aiming at the problem that the current location method has the locating error in the base station and its occlusion region of accident ship nearby, an ultra wideband localization technology for accident ship under adverse sea conditions was proposed. Firstly, this method carried out the NLOS identification of ship accident, and measured and determined the NLOS weighting factor of accident ship through contrasting the proximity of the probability density function of distance measurement under line of sight situation and the probability density function under non line of sight situation. The weighted factor reflected the proportion of NLOS nodes and reference ...

Image gradient orientations embedded structural error coding for face recognition with occlusion

Partially occluded faces are very common in automatic face recognition (FR) in the real world. We explore the problem of FR with occlusion by embedding Image Gradient Orientations (IGO) into robust error coding. The existing works usually put stress on the error distribution in the non-occluded region but neglect the one in the occluded region due to its unpredictability incurred by irregular occlusion. However, in the IGO domain, the error distribution in the occluded region can be built simply and elegantly by a uniform distribution in the interval $$\left[ -\pi ,\pi \right)$$, and the one in the occluded region can be well built by a weight-conditional Gaussian distribution. By incorporating the two error distributions and a Markov random field for the priori distribution of the occlusion support, we propose a joint probabilistic generative model for a novel IGO-embedded Structural Error Coding (IGO-SEC) model. Two methods, a new reconstruction method and a new robust structural error metric, are further presented to boost the performance of IGO-SEC. Extensive experiments on 8 popular robust FR methods and 4 benchmark face databases demonstrate the effectiveness and robustness of IGO-SEC in dealing with facial occlusion and occlusion-like variations.

BBC Net: Bounding-Box Critic Network for Occlusion-Robust Object Detection

Object detection has received significant interest in the research field of computer vision and is widely used in human-centric applications. The occlusion problem is a frequent obstacle that degrades detection quality. In this paper, we propose a novel object detection framework targeting robust object detection in occlusion. The proposed deep learning-based network consists mainly of two parts: 1) object detection framework, which classifies the object categories and localizes the object location and 2) plug-in bounding-box (BB) estimator, which estimates the object and occlusion region from the feature map of the backbone network and the corresponding critic network for evaluating the predicted BB map. The BB estimator and the critic network are the plug-in modules added to the object detection framework and learned competitively with adversarial manner. As the plug-in BB estimator is learned to estimate the BB map containing the object and occlusion pattern information, the backbone network can embed this information to enable robust detection under occlusion in the test phase. The comprehensive experimental results on the PASCAL VOC, MS COCO, and KITTI dataset showed that the performance is improved with the plug-in BB-Critic network by predicting and criticizing object and occlusion in general generic object detection framework.

Multi-Layer Frame Rate Up-Conversion Using Trilateral Filtering for Multi-View Video

Frame rate up-conversion (FRUC) is a technology for generating a higher frame rate video from one with a lower frame rate. The multi-view plus depth map (MVD) system has been recently considered a promising solution for three-dimensional video systems. Accordingly, an FRUC efficient solution for the MVD system is greatly needed. In this paper, we thus propose an FRUC approach that considers the MVD system characteristics. It divides a frame into multiple layers using a depth map, and it reconstructs occluded regions using neighboring views and depth map information. With reconstructed information, it accurately performs motion estimation when a region occlusion occurs. Experimental results demonstrated that the proposed method significantly improved the quality of interpolated frames, and it can effectively complement other conventional FRUC approaches.

One-view occlusion detection for stereo matching with a fully connected CRF model.

In this paper, we extend the standard belief propagation (BP) sequential technique proposed in the tree-reweighted sequential method [15] to the fully connected CRF models with the geodesic distance affinity. The proposed method has been applied to the stereo matching problem. Also a new approach to the BP marginal solution is proposed that we call one-view occlusion detection (OVOD). In contrast to the standard winner takes all (WTA) estimation, the proposed OVOD solution allows to find occluded regions in the disparity map and simultaneously improve the matching result. As a result we can perform only one energy minimization process and avoid the cost calculation for the second view and the left-right check procedure. We show that the OVOD approach considerably improves results for cost augmentation and energy minimization techniques in comparison with the standard one-view affinity space implementation. We apply our method to the Middlebury data set and reach state-of-the-art especially for median, average and mean squared error metrics.

An Extended Occlusion Detection Approach for Video Processing

Occlusions become conspicuous as failure regions in video processing when unified over time because the contraventions of the restriction of brightness have accumulated and evolved in occluded regions. The accuracy at the boundaries of the moving objects is one of the challenging areas that required further exploration and research. This paper presents the work in process approach that can detect occlusion regions by using pixel-wise coherence, segment-wise confidence and interpolation technique. Our method can get the same result as usual methods by solving only one Partial Differential Equations (PDE) problem; it is superior to existing methods because it is faster and provides better coverage rates for occlusion regions than variation techniques when tested against a varied number of benchmark datasets. With these improved results, we can apply and extend our approach to a wider range of applications in computer vision, such as background subtraction, tracking, 3D reconstruction, video surveillance, video compression.

An Efficient Dense Stereo Matching Method for Planetary Rover

Stereo matching is one of the most important and challenging subjects in the field of planetary rover with a stereo vision system. The generated disparity map can make rover to avoid the obstacle and explore the planetary surface automatically. In this paper, we propose an efficient dense stereo matching method to generate disparity maps for a planetary rover, which relies on 3-D plane fitting, adaptive penalties, and coarse-to-fine disparity constraint. In order to achieve efficient stereo matching at the coarsest level of the pyramid, we present a 3-D plane fitting to reduce the disparity search range and propose adaptive penalties in the more-global matching method for obtaining an accurate disparity map. At the finer level, our method then estimates the disparity search range based on coarse-to-fine disparity constraint and utilizes adaptive penalties to obtain an accurate disparity map. The extensive experiments with stereo images of Chang'e-3 rover demonstrate that our approach can generate disparity maps efficiently and accurately compared with the most state-of-the-art semi-global matching methods, especially at low texture regions, depth discontinuities, and occlusion regions.