Occlusion Handling Research Articles

Mutually coupled detection and tracking of trucks for monitoring construction material arrival delays

Construction project is sensitive to material arrival delays, which can cause schedule delays and budget overruns. The prompt detection of construction material arrival delays is necessary to recover disrupted projects in time. This paper explores computer vision-based (CVB) mutually coupled detection and tracking of transport trucks for monitoring construction material arrival delays. Through the mutually coupled mechanism, the truck detection and tracking algorithms complement each other in reducing accumulated tracking errors, modifying false positives and false negatives in detection, and stabilizing the detected bounding boxes. The experimental results indicate integrating a deep Convolutional Neural Network (CNN), a Kanade-Lucas-Tomasi (KLT) corner feature tracker, and a hash-based occlusion handling strategy can achieve high tracking precision compared to state-of-the-art trackers in construction. The field application results indicate that the proposed method can achieve the automatic monitoring of construction material arrival delays.

Future landscape visualization using a city digital twin: integration of augmented reality and drones with implementation of 3D model-based occlusion handling

Abstract The integration of augmented reality and drones allows past and future landscapes to be visualized from an aerial perspective. However, these visualizations still suffer from the occlusion problem, where the three-dimensional (3D) virtual model displayed in the real world is in front of a real-world object. Currently, city digital twins are essential for the sustainable development of cities and the development of detailed 3D models of cities. By visualizing the city digital twin, augmented reality can facilitate the participation of nonexpert citizens in the decision-making process of urban design, but research examples are limited. Here, using detailed city 3D models, we develop a digital-twin approach to outdoor augmented reality with occlusion handling for both first-person and bird’s-eye views. In a verification experiment, the occlusion handling accuracy of the prototype system was measured to be about 0.8 using intersection over union. The frame rate of the entire prototype system was about 30 fps, and the delay between the controller and the augmented reality device was about 3 s. The internet-based system architecture was developed to integrate augmented reality and drone systems. Our system allows multiple stakeholders involved in building construction projects to observe aerial perspectives of those projects, both on-site and off-site via an internet browser, using augmented reality with occlusion handling.

Deep Learning-Based Occlusion Handling of Overlapped Plants for Robotic Grasping

Instance segmentation of overlapping plants to detect their grasps for possible robotic grasping presents a challenging task due to the need to address the problem of occlusion. We addressed the problem of occlusion using a powerful convolutional neural network for segmenting objects with complex forms and occlusions. The network was trained with a novel dataset named the “occluded plants” dataset, containing real and synthetic images of plant cuttings on flat surfaces with differing degrees of occlusion. The synthetic images were created using the novel framework for synthesizing 2D images by using all plant cutting instances of available real images. In addition to the method for occlusion handling for overlapped plants, we present a novel method for determining the grasps of segmented plant cuttings that is based on conventional image processing. The result of the employed instance segmentation network on our plant dataset shows that it can accurately segment the overlapped plants, and it has a robust performance for different levels of occlusions. The presented plants’ grasp detection method achieved 94% on the rectangle metric which had an angular deviation of 30 degrees and an IoU of 0.50. The achieved results show the viability of our approach on plant species with an irregular shape and provide confidence that the presented method can provide a basis for various applications in the food and agricultural industries.

A robust object detection system with occlusion handling for mobile devices

AbstractDeep learning techniques help computer vision automatically learn the intrinsic patterns within complex data. This research mainly concentrates on creating a mobile application based on augmented reality for elderly mobile users that helps in identifying the traffic signals and other signboards in real‐time using deep learning techniques. TensorFlow serves as an implementation platform to build the object detection system with deep learning. The single shot multibox detector (SSD) model and the two‐stage faster‐regional convolutional neural network (RCNN) models from TensorFlow's object detection application programming interface (API) are compared in this study. The SSD model with MobileNet as a backbone network serves well for this study as it is faster than the RCNN model with comparable accuracy. However, unconstrained environments like occlusions can be an obstacle to the effective performance of an object detection system. This research provides a solution to handle occlusions by developing a robust object detection system through image segmentation techniques. The model introduced is based on the SSD MobileNet model which enables it to be deployable on mobile devices for real‐time offline detection. The developed model exhibits faster performance than the state‐of‐the‐art instance segmentation model, Mask RCNN with comparable accuracy. Elaborated implementation of this system and results are presented in further sections.

Real-time occlusion handling for augmented reality assistance assembly systems with monocular images

The application of Augmented Reality (AR) can effectively reduce the cognitive load of operators by presenting complex assembly work instruction with virtual augmented contents. If the virtual augmented contents superimposed on physical product without handling occlusion, the distortion may appear in AR guidance video. This paper proposes a monocular image based real-time occlusion handling method for virtual part model and physical product. In AR assistance assembly process, the virtual part model can be occluded by assembly scene or assembly objects. First, the monocular SLAM is used to reconstruct the assembly scene, and the reconstructed sparse 3D points are converted into depth points in the depth map. We use the color information of the assembly scene to control the propagation of parse depth points to the remaining pixels. With GPU acceleration calculation, a densification depth map of the assembly scene can be obtained. Then, the 3D models of the assembly objects are rendered as depth map according to the registration coordinates in AR assistance assembly system. The depth map of the assembly scene and assembly objects are synthesized as the final depth map for AR occlusion handling. Finally, the depth relationship between real assembly scenes, assembly objects, and virtual augmented contents are compared to determine the display pixels for AR effect. The results show that the proposed occlusion handling method shows high accuracy and fast speed in comparison with conventional methods, which demonstrates that our method can effectively solve the problem of occlusion handing for AR assistance assembly systems.

Occlusion Handling and Multi-Scale Pedestrian Detection Based on Deep Learning: A Review

Pedestrian detection is an important branch of computer vision, and has important applications in the fields of autonomous driving, artificial intelligence and video surveillance. With the rapid development of deep learning and the proposal of large-scale datasets, pedestrian detection has reached a new stage and has achieved better performance. However, the performance of state-of-the-art methods is far behind expectations, especially when occlusion and scale variance exist. Therefore, many works focused on occlusion and scale variance have been proposed in the past few years. The purpose of this article is to make a detailed review of recent progress in pedestrian detection. First, a brief progress of pedestrian detection in the past two decades is summarized. Second, recent deep learning methods focusing on occlusion and scale variance are analyzed. Moreover, the popular datasets and evaluation methods for pedestrian detection are introduced. Finally, the development trends in pedestrian detection are discussed.

AOH: Online Multiple Object Tracking With Adaptive Occlusion Handling

Multiple object tracking has improved drastically in recent years due to one-shot tracking methods. These methods design joint-detection-and-tracking structures to achieve real-time tracking performance and introduce more powerful detectors to deal with missed objects. However, most of them perform poorly in crowded scenes because of frequent occlusions. Several previous works have attempted to alleviate the occlusion issue, but they hardly involve the essence of the problem. In this letter, we suppose that occlusion is closely related to crowd density, so the degree of occlusion can be estimated. Therefore, we propose a Potential Object Mining strategy to adaptively obtain occluded objects for reducing broken trajectories, which re-weights detections based on the predicted density map. Additionally, for the strategy, Dense Estimator is designed to predict the density of each region in an image by employing a Transformer-based structure. Combining them together forms our Adaptive Occlusion Handling (AOH) tracking framework. Extensive experiments on MOTChallenge benchmarks (MOT17 and MOT20) demonstrate that our AOH achieves the state-of-the-art performance, especially on the heavily occluded MOT20.

Efficient Online Object Tracking Scheme for Challenging Scenarios.

Visual object tracking (VOT) is a vital part of various domains of computer vision applications such as surveillance, unmanned aerial vehicles (UAV), and medical diagnostics. In recent years, substantial improvement has been made to solve various challenges of VOT techniques such as change of scale, occlusions, motion blur, and illumination variations. This paper proposes a tracking algorithm in a spatiotemporal context (STC) framework. To overcome the limitations of STC based on scale variation, a max-pooling-based scale scheme is incorporated by maximizing over posterior probability. To avert target model from drift, an efficient mechanism is proposed for occlusion handling. Occlusion is detected from average peak to correlation energy (APCE)-based mechanism of response map between consecutive frames. On successful occlusion detection, a fractional-gain Kalman filter is incorporated for handling the occlusion. An additional extension to the model includes APCE criteria to adapt the target model in motion blur and other factors. Extensive evaluation indicates that the proposed algorithm achieves significant results against various tracking methods.

Neural frame interpolation for rendered content

The demand for creating rendered content continues to drastically grow. As it often is extremely computationally expensive and thus costly to render high-quality computer-generated images, there is a high incentive to reduce this computational burden. Recent advances in learning-based frame interpolation methods have shown exciting progress but still have not achieved the production-level quality which would be required to render fewer pixels and achieve savings in rendering times and costs. Therefore, in this paper we propose a method specifically targeted to achieve high-quality frame interpolation for rendered content. In this setting, we assume that we have full input for every n -th frame in addition to auxiliary feature buffers that are cheap to evaluate (e.g. depth, normals, albedo) for every frame. We propose solutions for leveraging such auxiliary features to obtain better motion estimates, more accurate occlusion handling, and to correctly reconstruct non-linear motion between keyframes. With this, our method is able to significantly push the state-of-the-art in frame interpolation for rendered content and we are able to obtain production-level quality results.

Parallel multiscale context-based edge-preserving optical flow estimation with occlusion detection

Although convolutional neural network (CNN)-based optical flow approaches have exhibited good performance in terms of computational accuracy and efficiency in recent years, the issue of edge-blurring caused by motion occlusions remains. In this paper, we propose a parallel multiscale context-based edge-preserving optical flow estimation method with occlusion detection, named PMC-PWC. First, we exploit a parallel multiscale context (PMC) network for occlusion detection, in which the proposed PMC model is able to aggregate the multiscale context information to develop the performance of occlusion detection near motion boundaries. Second, we combine the PMC model with a context network to plan an occlusion estimation module and incorporate it into a pyramid, warping, and cost volume model to construct an edge-preserving optical flow computation network. Third, we design a novel loss function including an endpoint error (EPE)-based loss, a binary cross-entropy loss and an edge loss to supervise the proposed PMC-PWC network to produce optical flow and occlusion simultaneously. Finally, we run the proposed PMC-PWC method on the MPI-Sintel and KITTI datasets to conduct a comprehensive comparison with several state-of-the-art approaches. The experimental results indicate that the proposed PMC-PWC method performed well in terms of both accuracy and robustness, especially due to the significant benefits of edge preservation and occlusion handling.

A bi-network architecture for occlusion handling in Person re-identification

A bi-network architecture for occlusion handling in Person re-identification

3D shape sensing and deep learning-based segmentation of strawberries

Automation and robotisation of the agricultural sector are seen as a viable solution to socio-economic challenges faced by this industry. This technology often relies on intelligent perception systems providing information about crops, plants and the entire environment. The challenges faced by traditional 2D vision systems can be addressed by modern 3D vision systems which enable straightforward localisation of objects, size and shape estimation, or handling of occlusions. So far, the use of 3D sensing was mainly limited to indoor or structured environments. In this paper, we evaluate modern sensing technologies including stereo and time-of-flight cameras for 3D perception of shape in agriculture and study their usability for segmenting out soft fruit from background based on their shape. To that end, we propose a novel 3D deep neural network which exploits the organised nature of information originating from the camera-based 3D sensors. We demonstrate the superior performance and efficiency of the proposed architecture compared to the state-of-the-art 3D networks. Through a simulated study, we also show the potential of the 3D sensing paradigm for object segmentation in agriculture and provide insights and analysis of what shape quality is needed and expected for further analysis of crops. The results of this work should encourage researchers and companies to develop more accurate and robust 3D sensing technologies to assure their wider adoption in practical agricultural applications.

Homography-guided stereo matching for wide-baseline image interpolation

Image interpolation has a wide range of applications such as frame rate-up conversion and free viewpoint TV. Despite significant progresses, it remains an open challenge especially for image pairs with large displacements. In this paper, we first propose a novel optimization algorithm for motion estimation, which combines the advantages of both global optimization and a local parametric transformation model. We perform optimization over dynamic label sets, which are modified after each iteration using the prior of piecewise consistency to avoid local minima. Then we apply it to an image interpolation framework including occlusion handling and intermediate image interpolation. We validate the performance of our algorithm experimentally, and show that our approach achieves state-of-the-art performance.

Design Guidelines on Deep Learning–based Pedestrian Detection Methods for Supporting Autonomous Vehicles

Intelligent transportation systems (ITS) enable transportation participants to communicate with each other by sending and receiving messages, so that they can be aware of their surroundings and facilitate efficient transportation through better decision making. As an important part of ITS, autonomous vehicles can bring massive benefits by reducing traffic accidents. Correspondingly, much effort has been paid to the task of pedestrian detection, which is a fundamental task for supporting autonomous vehicles. With the progress of computational power in recent years, adopting deep learning–based methods has become a trend for improving the performance of pedestrian detection. In this article, we present design guidelines on deep learning–based pedestrian detection methods for supporting autonomous vehicles. First, we will introduce classic backbone models and frameworks, and we will analyze the inherent attributes of pedestrian detection. Then, we will illustrate and analyze representative pedestrian detectors from occlusion handling, multi-scale feature extraction, multi-perspective data utilization, and hard negatives handling these four aspects. Last, we will discuss the developments and trends in this area, followed by some open challenges.

Spatio-Temporal Context, Correlation Filter and Measurement Estimation Collaboration Based Visual Object Tracking

Despite eminent progress in recent years, various challenges associated with object tracking algorithms such as scale variations, partial or full occlusions, background clutters, illumination variations are still required to be resolved with improved estimation for real-time applications. This paper proposes a robust and fast algorithm for object tracking based on spatio-temporal context (STC). A pyramid representation-based scale correlation filter is incorporated to overcome the STC’s inability on the rapid change of scale of target. It learns appearance induced by variations in the target scale sampled at a different set of scales. During occlusion, most correlation filter trackers start drifting due to the wrong update of samples. To prevent the target model from drift, an occlusion detection and handling mechanism are incorporated. Occlusion is detected from the peak correlation score of the response map. It continuously predicts target location during occlusion and passes it to the STC tracking model. After the successful detection of occlusion, an extended Kalman filter is used for occlusion handling. This decreases the chance of tracking failure as the Kalman filter continuously updates itself and the tracking model. Further improvement to the model is provided by fusion with average peak to correlation energy (APCE) criteria, which automatically update the target model to deal with environmental changes. Extensive calculations on the benchmark datasets indicate the efficacy of the proposed tracking method with state of the art in terms of performance analysis.

Single-scale siamese network based RGB-D object tracking with adaptive bounding boxes

Most of the excellent methods for visual object tracking are based on RGB videos. With the popularity of depth cameras, the research on RGB-D(RGB+depth) tracking has gradually gained extensive attention. The depth map provides more available information for dealing with intractable tracking problems. How to make full use of depth maps to construct a better tracker is the foremost problem to be settled. The fully-convolutional siamese network shows excellent performance in 2D tracking, but still cannot achieve satisfying tracking performance in complex scenarios. Therefore, we have proposed the RGB-D tracker integrated with the single-scale siamese network as well as the adaptive bounding boxes, which achieves stable tracking performance under the challenges such as occlusion, scale variation and background clutter. Our proposed adaptive strategy enables the bounding box to adjust automatically when the target appearance changes during the tracking, instead of multi-scale input in the siamese network. We design an effective algorithm to quickly obtain the target depth and construct the 3D local visual field to eliminate the interference from background and similar objects. In addition, the total occlusion handling approach combined with RGB and depth information has achieved more reliable occlusion detection and target recovery. Our presented object tracker, including the strategies of 3D local visual field, adaptive bounding boxes and occlusion handling, has been evaluated on two widely utilized RGB-D tracking benchmarks and achieves suprior performance especially for the situations of occlusion and pedestrian detection.

Guided Attention in CNNs for Occluded Pedestrian Detection and Re-identification

Pedestrian detection and re-identification have progressed significantly in the last few years. However, occluded people are notoriously hard to detect and recognize, as their appearance varies substantially depending on a wide range of occlusion patterns. In this paper, we aim to propose a simple and compact method based on CNNs for occlusion handling. We start with interpreting CNN channel features of a pedestrian detector, and we find that different channels activate responses for different body parts respectively. These findings motivate us to employ an attention mechanism across channels to represent various occlusion patterns in one single model, as each occlusion pattern can be formulated as some specific combination of body parts. Therefore, an attention network with self or external guidances is proposed as an add-on to the baseline CNN method. Also, we propose an attention guided self-paced learning method to balance the optimization across different occlusion levels. Our proposed method shows significant improvements over the baseline methods for both pedestrian detection and re-identification tasks. For pedestrian detection, we achieve a considerable improvement of 8pp to the baseline FasterRCNN detector on the heavy occlusion subset of CityPersons and on Caltech we outperform the state-of-the-art method by 5pp. For pedestrian re-identification, our method surpasses the baseline and achieves state-of-the-art performance on multiple re-identification benchmarks.

An Occlusion Handling Evaluation Criterion for Deep Learning Object Segmentation

This paper introduces a novel evaluation criterion to occlusion handling for deep learning object segmentation. The occlusion is defined as objects blocking each other on an image. It affects deep learning object segmentation. More and more researches focus on occlusion handling for object segmentation. However, these researches do not clearly show the evaluation of their occlusion handling method, because there is no suitable evaluation criterion. Traditionally, people just use images results or use the entire object boundary accuracy to show the occlusion handling of their methods. Conversely, these ideas cannot give a numerical evaluation focusing on occlusion handling. This research reports an evaluation criterion to measure occlusion handling performance. This evaluation criterion uses the shortest distances between the pixels from the ground truth occlusion edges and the segmentation (result) shape. The shortest distances are segmentation errors. Then, the average value of these errors is the final parameter for occlusion handling evaluation criterion. The experiment uses a deep learning based segmentation model as an example. It shows that this criterion (or method) successfully measures the occlusion handling for deep learning based object segmentation.

A novel augmented reality visualization in jaw surgery: enhanced ICP based modified rotation invariant and modified correntropy

Image registration, accuracy, processing time and occlusions are the main limitations of augmented reality (AR) based jaw surgery. Therefore, the main aim of this paper is to reduce the registration error, which will help in improving the accuracy and reducing the processing time. Also, it aims to remove outliers and remove the registration outcomes trapped in local minima to improve the alignment problems and remove the occlusion caused by surgery instrument. The enhanced Iterative Closest Point (ICP) algorithm with rotation invariant and correntropy was used for the proposed system. Markerless image registration technique was used for AR-based jaw surgery. The problem of occlusion caused by surgical tools and blood is solved by using stereo based tracing with occlusion handling techniques. This research reduced alignment error 0.59 mm ~ 0.62 mm against 0.69 ~ 0.72 mm of state-of-the-art solution. The processing time of video frames was enhanced to 11.9 ~ 12.8 fps against 8 ~ 9.15 fps in state-of-the-art solution. This paper is focused on providing fast and accurate AR-based system for jaw surgery. The proposed system helps in improving the AR visualization during jaw surgery. The combination of methods and technology helped in improving AR visualization for jaw surgery and to overcome the failure caused by a large rotation angle and provides an initial parameter for better image registration. It also enhances performance by removing outliers and noises. The pose refinement stage provides a better result in terms of processing time and accuracy.

Assessing future landscapes using enhanced mixed reality with semantic segmentation by deep learning

Architecture, engineering, and construction projects need to be promoted in harmony with the natural environment and with the aim of preserving people’s living environment. At the planning and design stage, decision-makers and stakeholders share and assess landscape images during and after construction in order to avoid as much uncertainty as possible when performing environmental impact assessment. Given the lack of a standard visualization method for future landscapes that do not yet exist, mixed reality (MR), which overlays virtual content onto a real scene, has attracted attention in the field of landscape design. One challenge in MR is occlusion, which occurs when virtual objects obscure physical objects that should be rendered in the foreground. In MR-based landscape visualization, the distance between the MR camera and real objects located in front of the virtual objects might vary and might be large, causing difficulty for existing occlusion handling methods. In the process of landscape design, an evidence-based approach has also become important. Landscape index estimation using semantic segmentation by deep learning, which can recognize the surrounding environment, has been actively studied for landscape assessment. In this study, semantic segmentation by deep learning was integrated into an MR system to enable dynamic occlusion handling and landscape index estimation for both existing and designed landscape assessment. This system can be operated on a mobile device with video communication over the internet by connecting to real-time semantic segmentation on a high-performance personal computer. The applicability of the developed system is demonstrated through accuracy verification and case studies.