Occlusion Boundaries Research Articles

Binary Opacity Grids: Capturing Fine Geometric Detail for Mesh-Based View Synthesis

While surface-based view synthesis algorithms are appealing due to their low computational requirements, they often struggle to reproduce thin structures. In contrast, more expensive methods that model the scene's geometry as a volumetric density field (e.g. NeRF) excel at reconstructing fine geometric detail. However, density fields often represent geometry in a "fuzzy" manner, which hinders exact localization of the surface. In this work, we modify density fields to encourage them to converge towards surfaces, without compromising their ability to reconstruct thin structures. First, we employ a discrete opacity grid representation instead of a continuous density field, which allows opacity values to discontinuously transition from zero to one at the surface. Second, we anti-alias by casting multiple rays per pixel, which allows occlusion boundaries and subpixel structures to be modelled without using semi-transparent voxels. Third, we minimize the binary entropy of the opacity values, which facilitates the extraction of surface geometry by encouraging opacity values to binarize towards the end of training. Lastly, we develop a fusion-based meshing strategy followed by mesh simplification and appearance model fitting. The compact meshes produced by our model can be rendered in real-time on mobile devices and achieve significantly higher view synthesis quality compared to existing mesh-based approaches. Our interactive webdemo is available at https://binary-opacity-grid.github.io.

Multi-Scenario Mask Detection Method and Implementation Based on Deep Learning

Object detection is one of the core issues in the field of computer vision, with the task of accurately locating and recognizing all objects of interest in images, determining the categories and positions of objects. Regarding masks as targets in object detection, utilizing deep learning techniques to detect the wearing of masks on faces can greatly improve issues such as high manual supervision costs, low efficiency, and subjective differences. In this paper, based on the YOLOV5 algorithm, we propose a mask detection technology that can accurately detect the wearing of masks in real-time, and experimentally determine the occlusion boundaries.

Occlusion Boundary Prediction and Transformer Based Depth-Map Refinement From Single Image

Due to the numerous applications of boundary maps and occlusion orientation maps (ORI-maps) in high-level vision problems, accurate estimation of these maps is a crucial task. The existing deep networks employ a single-stream network to estimate the relation between boundary map and ORI-map estimation. However, these networks fail to explore significant individual information separately. To resolve this problem, in this paper, we propose a novel two-stream generative adversarial network (GAN) for boundary map and ORI-map estimation, named OBP-GAN. The proposed OBP-GAN consists of two streams known as BP-GAN and OR-GAN. The BP-GAN estimates the boundary map, and the OR-GAN predicts the ORI-map. The boundary and ORI-map can also be useful cues for the task of depth-map refinement from single images. Therefore, in this work, we propose a transformer-based depth-map refinement network (TRANSDMR-GAN) for refining the depth estimated from monocular images using boundary and ORI-map. We conducted extensive analyses on indoor and outdoor datasets to validate our proposed OBP-GAN and TRANSDMR-GAN. The extensive experimental analysis and ablation study demonstrate the ability of the proposed OBP-GAN to generate state-of-the-art occlusion boundary maps. Furthermore, we show that the proposed network, TRANSDMR-GAN, can generate an edge-enhanced depth map without degrading the accuracy of the initial depth map.

Beyond Photometric Consistency: Geometry-Based Occlusion-Aware Unsupervised Light Field Disparity Estimation.

Although learning-based light field disparity estimation has achieved great progress in the most recent years, the performance of unsupervised light field learning is still hindered by occlusions and noises. By analyzing the overall strategy underlying the unsupervised methodology and the light field geometry implied in epipolar plane images (EPIs), we look beyond the photometric consistency assumption, and design an occlusion-aware unsupervised framework to deal with the situations of photometric consistency conflict. Specifically, we present a geometry-based light field occlusion modeling, which predicts a group of visibility masks and occlusion maps, respectively, by forward warping and backward EPI-line tracing. In order to learn better the noise-and occlusion-invariant representations of the light field, we propose two occlusion-aware unsupervised losses: occlusion-aware SSIM and statistics-based EPI loss. Experiment results demonstrate that our method can improve the estimation accuracy of light field depth over the occluded and noisy regions, and preserve the occlusion boundaries better.

Occlusion relationship reasoning with a feature separation and interaction network

Occlusion relationship reasoning aims to locate where an object occludes others and estimate the depth order of these objects in three-dimensional (3D) space from a two-dimensional (2D) image. The former sub-task demands both the accurate location and the semantic indication of the objects, while the latter sub-task needs the depth order among the objects. Although several insightful studies have been proposed, a key characteristic of occlusion relationship reasoning, i.e., the specialty and complementarity between occlusion boundary detection and occlusion orientation estimation, is rarely discussed. To verify this claim, in this paper, we integrate these properties into a unified end-to-end trainable network, namely the feature separation and interaction network (FSINet). It contains a shared encoder-decoder structure to learn the complementary property between the two sub-tasks, and two separated paths to learn specialized properties of the two sub-tasks. Concretely, the occlusion boundary path contains an image-level cue extractor to capture rich location information of the boundary, a detail-perceived semantic feature extractor, and a contextual correlation extractor to acquire refined semantic features of objects. In addition, a dual-flow cross detector has been customized to alleviate false-positive and false-negative boundaries. For the occlusion orientation estimation path, a scene context learner has been designed to capture the depth order cue around the boundary. In addition, two stripe convolutions are built to judge the depth order between objects. The shared decoder supplies the feature interaction, which plays a key role in exploiting the complementarity of the two paths. Extensive experimental results on the PIOD and BSDS ownership datasets reveal the superior performance of FSINet over state-of-the-art alternatives. Additionally, abundant ablation studies are offered to demonstrate the effectiveness of our design.

Occlusion-Aware Instance Segmentation Via BiLayer Network Architectures.

Segmenting highly-overlapping image objects is challenging, because there is typically no distinction between real object contours and occlusion boundaries on images. Unlike previous instance segmentation methods, we model image formation as a composition of two overlapping layers, and propose Bilayer Convolutional Network (BCNet), where the top layer detects occluding objects (occluders) and the bottom layer infers partially occluded instances (occludees). The explicit modeling of occlusion relationship with bilayer structure naturally decouples the boundaries of both the occluding and occluded instances, and considers the interaction between them during mask regression. We investigate the efficacy of bilayer structure using two popular convolutional network designs, namely, Fully Convolutional Network (FCN) and Graph Convolutional Network (GCN). Further, we formulate bilayer decoupling using the vision transformer (ViT), by representing instances in the image as separate learnable occluder and occludee queries. Large and consistent improvements using one/two-stage and query-based object detectors with various backbones and network layer choices validate the generalization ability of bilayer decoupling, as shown by extensive experiments on image instance segmentation benchmarks (COCO, KINS, COCOA) and video instance segmentation benchmarks (YTVIS, OVIS, BDD100 K MOTS), especially for heavy occlusion cases.

Multi-scale foreground-background separation for light field depth estimation with deep convolutional networks

There are three viewpoints to divide the depth from light field (DFLF) methods: light field (LF) representation, signal processing, and learning viewpoints. In the signal processing viewpoint in details, the DFLF methods can be divided into three groups: cost-based, foreground-background separation (FBS)-based, and depth model-based methods. Although the deep learning techniques are widely implemented in the cost-based and depth model-based methods for the DFLF, they are not implemented in the FBS-based method, yet. In this paper, we propose a DFLF method based on the FBS with the deep learning techniques. Based on an observation that the angular aliasing artifacts are reduced in the resulting disparity maps obtained from spatially low-scale LF images, multi-scale (MS) FBS score maps are used in the proposed methods. To combine the MS FBS score maps into a single FBS score maps, four directional LF gradients, features from the 4-D LF, and features from the center view images are utilized. Based on the sensitivity and specificity loss, the gradient matching loss between the predicted and ground truth disparity maps is used to improve the performance around the occlusion boundary in the resulting disparity maps. The experimental results show that the proposed method performs reasonably in both the synthetic and real LF datasets. Especially, in the texture-less region in the real dataset, the resulting disparity maps obtained by the proposed method show that the artifacts are reduced compared to those by the existing DFLF methods.

Locally available achromatic cues can reliably distinguish occlusion boundaries from cast shadows

Locally available achromatic cues can reliably distinguish occlusion boundaries from cast shadows

LFRSNet: A robust light field semantic segmentation network combining contextual and geometric features

Light field (LF) semantic segmentation is a newly arisen technology and is widely used in many smart city applications such as remote sensing, virtual reality and 3D photogrammetry. Compared with RGB images, LF images contain multi-layer contextual information and rich geometric information of real-world scenes, which are challenging to be fully exploited because of the complex and highly inter-twined structure of LF. In this paper, LF Contextual Feature (LFCF) and LF Geometric Feature (LFGF) are proposed respectively for occluded area perception and segmentation edge refinement. With exploitation of all the views in LF, LFCF provides glimpse of some occluded areas from other angular positions besides the superficial color information of the target view. The multi-layer information of the occluded area enhances the classification of partly occluded objects. Whereas LFGF is extracted from Ray Epipolar-Plane Images (RayEPIs) in eight directions for geometric information embedding. The solid geometric information refines object edges, especially for occlusion boundaries with similar colors. At last, Light Field Robust Segmentation Network (LFRSNet) is designed to integrate LFCF and LFGF. Multi-layer contextual information and geometric information are effectively incorporated through LFRSNet, which brings significant improvement for segmentation of the occluded objects and the object edges. Experimental results on both realworld and synthetic datasets proves the state-of-the-art performance of our method. Compared with other methods, LFRSNet produces more accurate segmentation under occlusion, especially in the edge regions.

CellNet: A Lightweight Model towards Accurate LOC-Based High-Speed Cell Detection

Label-free cell separation and sorting in a microfluidic system, an essential technique for modern cancer diagnosis, resulted in high-throughput single-cell analysis becoming a reality. However, designing an efficient cell detection model is challenging. Traditional cell detection methods are subject to occlusion boundaries and weak textures, resulting in poor performance. Modern detection models based on convolutional neural networks (CNNs) have achieved promising results at the cost of a large number of both parameters and floating point operations (FLOPs). In this work, we present a lightweight, yet powerful cell detection model named CellNet, which includes two efficient modules, CellConv blocks and the h-swish nonlinearity function. CellConv is proposed as an effective feature extractor as a substitute to computationally expensive convolutional layers, whereas the h-swish function is introduced to increase the nonlinearity of the compact model. To boost the prediction and localization ability of the detection model, we re-designed the model’s multi-task loss function. In comparison with other efficient object detection methods, our approach achieved state-of-the-art 98.70% mean average precision (mAP) on our custom sea urchin embryos dataset with only 0.08 M parameters and 0.10 B FLOPs, reducing the size of the model by 39.5× and the computational cost by 4.6×. We deployed CellNet on different platforms to verify its efficiency. The inference speed on a graphics processing unit (GPU) was 500.0 fps compared with 87.7 fps on a CPU. Additionally, CellNet is 769.5-times smaller and 420 fps faster than YOLOv3. Extensive experimental results demonstrate that CellNet can achieve an excellent efficiency/accuracy trade-off on resource-constrained platforms.

Object-Aware Monocular Depth Prediction With Instance Convolutions

With the advent of deep learning, estimating depth from a single RGB image has recently received a lot of attention, being capable of empowering many different applications ranging from path planning for robotics to computational cinematography. Nevertheless,while the depth maps are in their entirety fairly reliable, the estimates around object discontinuities are still far from satisfactory. This can beattributed to the fact that the convolutional operator naturally aggregates features across object discontinuities, resulting in smooth transitions rather than clear boundaries. Therefore, in order to circumvent this issue, we propose a novel convolutional operator which is explicitly tailored to avoid feature aggregation of different object parts. In particular, our method is based on estimating per-part depth values by means of super-pixels. The proposed convolutional operator, which we dub “Instance Convolution,” then only considers each object part individually on the basis of the estimated super-pixels. Our evaluation with respect to the NYUv2, iBims and KITTI datasets demonstrate the advantages of Instance Convolutions over the classical convolution at estimating depth around occlusion boundaries, while producing comparable results elsewhere. Our code is available at github.com/enisimsar/instance-conv.

Where to draw the line?

We often take people's ability to understand and produce line drawings for granted. But where should we draw lines, and why? We address psychological principles that underlie efficient representations of complex information in line drawings. First, 58 participants with varying degree of artistic experience produced multiple drawings of a small set of scenes by tracing contours on a digital tablet. Second, 37 independent observers ranked the drawings by how representative they are of the original photograph. Matching contours between drawings of the same scene revealed that the most consistently drawn contours tend to be drawn earlier. We generated half-images with the most- versus least-consistently drawn contours and asked 25 observers categorize the quickly presented scenes. Observers performed significantly better for the most compared to the least consistent half-images. The most consistently drawn contours were more likely to depict occlusion boundaries, whereas the least consistently drawn contours frequently depicted surface normals.

Occlusion-aware light field depth estimation using side window angular coherence.

Depth estimation is crucial in many light field applications. However, the accuracy of light field depth estimation is prone to be affected by occlusions. In this paper, a method of side window angular coherence is proposed to handle different types of occlusions, and the ability of the proposed method to resist occlusions is theoretically analyzed. The angular patch is divided into several discrete side window subsets. These subsets are a pure occluder-type subset, a pure object point-type subset, and a hybrid-type subset. The photo-consistency of the pure object point-type subset can reflect the true depth. Meanwhile, the occlusion edges can be detected to identify occluded points and nonoccluded points so the robustness of the algorithm can be further enhanced by processing the two types of points. Moreover, fast guided filtering is applied to cost volume for improving the accuracy of depth estimation. Experimental results demonstrate that our method outperforms the state-of-the-art depth estimation methods on both synthetic and real scenes, especially near occlusion boundaries.

Occlusion Handling by Successively Excluding Foregrounds for Light Field Depth Estimation Based on Foreground-Background Separation

This paper proposes a depth from light field (DFLF) method specifically to deal with occlusion based on the foreground-background separation (FBS). The FBS-based methods infer the disparity maps by accumulating the binary maps which divide whether each pixel is the foreground or background. Although there have been widely studied to handle the occlusion problem with the cost-based method, there are not enough researches to handle the occlusion problem with the FBS-based methods yet. We found that errors around the occlusion boundary in the resulting disparity maps of the FBS-based methods arise from the fattened foreground by the light field reprameterization. To avoid fattened foregrounds, the inferred foreground maps in the front region with respect to the disparity axis could be utilized in the back region in the three-dimensional volume construction, which corresponds to the cost volume construction in the cost-based methods. With the front-to-back scanning manner of the FBS-based method, by successively excluding inferred foreground maps, errors around occlusion boundary could be effectively reduced in the resulting disparity maps. With synthetic and real LF images, the proposed method shows reasonable performance compared to the existing methods and better performance than existing FBS-based methods.

The Chan-Vese Model With Elastica and Landmark Constraints for Image Segmentation

In order to completely separate objects with large sections of occluded boundaries in an image, we devise a new variational level set model for image segmentation combining the Chan-Vese model with elastica and landmark constraints. For computational efficiency, we design its Augmented Lagrangian Method (ALM) or Alternating Direction Method of Multiplier (ADMM) method by introducing some auxiliary variables, Lagrange multipliers, and penalty parameters. In each loop of alternating iterative optimization, the sub-problems of minimization can be easily solved via the Gauss-Seidel iterative method and generalized soft thresholding formulas with projection, respectively. Numerical experiments show that the proposed model can not only recover larger broken boundaries but can also improve segmentation efficiency, as well as decrease the dependence of segmentation on parameter tuning and initialization.

Glossy probe reprojection for interactive global illumination

Recent rendering advances dramatically reduce the cost of global illumination. But even with hardware acceleration, complex light paths with multiple glossy interactions are still expensive; our new algorithm stores these paths in precomputed light probes and reprojects them at runtime to provide interactivity. Combined with traditional light maps for diffuse lighting our approach interactively renders all light paths in static scenes with opaque objects. Naively reprojecting probes with glossy lighting is memory-intensive, requires efficient access to the correctly reflected radiance, and exhibits problems at occlusion boundaries in glossy reflections. Our solution addresses all these issues. To minimize memory, we introduce an adaptive light probe parameterization that allocates increased resolution for shinier surfaces and regions of higher geometric complexity. To efficiently sample glossy paths, our novel gathering algorithm reprojects probe texels in a view-dependent manner using efficient reflection estimation and a fast rasterization-based search. Naive probe reprojection often sharpens glossy reflections at occlusion boundaries, due to changes in parallax. To avoid this, we split the convolution induced by the BRDF into two steps: we precompute probes using a lower material roughness and apply an adaptive bilateral filter at runtime to reproduce the original surface roughness. Combining these elements, our algorithm interactively renders complex scenes while fitting in the memory, bandwidth, and computation constraints of current hardware.

A Neurally Inspired Model of Figure Ground Organization with Local and Global Cues

Figure Ground Organization (FGO)-inferring spatial depth ordering of objects in a visual scene-involves determining which side of an occlusion boundary is figure (closer to the observer) and which is ground (further away from the observer). A combination of global cues, like convexity, and local cues, like T-junctions are involved in this process. A biologically motivated, feed forward computational model of FGO incorporating convexity, surroundedness, parallelism as global cues and spectral anisotropy (SA), T-junctions as local cues is presented. While SA is computed in a biologically plausible manner, the inclusion of T-Junctions is biologically motivated. The model consists of three independent feature channels, Color, Intensity and Orientation, but SA and T-Junctions are introduced only in the Orientation channel as these properties are specific to that feature of objects. The effect of adding each local cue independently and both of them simultaneously to the model with no local cues is studied. Model performance is evaluated based on figure-ground classification accuracy (FGCA) at every border location using the BSDS 300 figure-ground dataset. Each local cue, when added alone, gives statistically significant improvement in the FGCA of the model suggesting its usefulness as an independent FGO cue. The model with both local cues achieves higher FGCA than the models with individual cues, indicating SA and T-Junctions are not mutually contradictory. Compared to the model with no local cues, the feed-forward model with both local cues achieves ≥8.78% improvement in terms of FGCA.

Detecting the occluding contours of the uterus to automatise augmented laparoscopy: score, loss, dataset, evaluation and user study.

The registration of a preoperative 3D model, reconstructed, for example, from MRI, to intraoperative laparoscopy 2D images, is the main challenge to achieve augmented reality in laparoscopy. The current systems have a major limitation: they require that the surgeon manually marks the occluding contours during surgery. This requires the surgeon to fully comprehend the non-trivial concept of occluding contours and surgeon time, directly impacting acceptance and usability. To overcome this limitation, we propose a complete framework for object-class occluding contour detection (OC2D), with application to uterus surgery. Our first contribution is a new distance-based evaluation score complying with all the relevant performance criteria. Our second contribution is a loss function combining cross-entropy and two new penalties designed to boost 1-pixel thickness responses. This allows us to train a U-Net end to end, outperforming all competing methods, which tends to produce thick responses. Our third contribution is a dataset of 3818 carefully labelled laparoscopy images of the uterus, which was used to train and evaluate our detector. Evaluation shows that the proposed detector has a similar false false-negative rate to existing methods but substantially reduces both false-positive rate and response thickness. Finally, we ran a user study to evaluate the impact of OC2D against manually marked occluding contours in augmented laparoscopy. We used 10 recorded gynecologic laparoscopies and involved 5 surgeons. Using OC2D led to a reduction of 3 min and 53 s in surgeon time without sacrificing registration accuracy. We provide a new set of criteria and a distance-based measure to evaluate an OC2D method. We propose an OC2D method which outperforms the state-of-the-art methods. The results obtained from the user study indicate that fully automatic augmented laparoscopy is feasible.

Occlusion Boundary: A Formal Definition & Its Detection via Deep Exploration of Context.

Occlusion boundaries contain rich perceptual information about the underlying scene structure and provide important cues in many visual perception-related tasks such as object recognition, segmentation, motion estimation, scene understanding, and autonomous navigation. However, there is no formal definition of occlusion boundaries in the literature, and state-of-the-art occlusion boundary detection is still suboptimal. With this in mind, in this paper we propose a formal definition of occlusion boundaries for related studies. Further, based on a novel idea, we develop two concrete approaches with different characteristics to detect occlusion boundaries in video sequences via enhanced exploration of contextual information (e.g, local structural boundary patterns, observations from surrounding regions, and temporal context) with deep models and conditional random fields. Experimental evaluations of our methods on two challenging occlusion boundary benchmarks (CMU and VSB100) demonstrate that our detectors significantly outperform the current state-of-the-art. Finally, we empirically assess the roles of several important components of the proposed detectors to validate the rationale behind these approaches.

TransCut2: Transparent Object Segmentation From a Light-Field Image

Transparent object segmentation can be very useful in computer vision applications. However, because the transparent objects borrow texture from their background and have a similar appearance to their surroundings, they are not handled well by regular image segmentation methods. In this paper, we propose a method that overcomes these problems using the consistency and distortion properties in a light-field image. The light-field linearity is used to estimate the likelihood of a pixel belonging to the transparent object or Lambertian background, and the occlusion detector is used to find the occlusion boundary. Graph-cut optimization is applied for the pixel labeling problem. We acquire light field datasets from both camera array and lenslet camera for the transparent object, and use these datasets to evaluate our method. The results demonstrate that the proposed method successfully segments transparent objects from the background under various conditions.