Presence Of Partial Occlusions Research Articles

A Deep Regression Approach for Human Activity Recognition Under Partial Occlusion.

In real-life scenarios, Human Activity Recognition (HAR) from video data is prone to occlusion of one or more body parts of the human subjects involved. Although it is common sense that the recognition of the majority of activities strongly depends on the motion of some body parts, which when occluded compromise the performance of recognition approaches, this problem is often underestimated in contemporary research works. Currently, training and evaluation is based on datasets that have been shot under laboratory (ideal) conditions, i.e. without any kind of occlusion. In this work, we propose an approach for HAR in the presence of partial occlusion, in cases wherein up to two body parts are involved. We assume that human motion is modeled using a set of 3D skeletal joints and also that occluded body parts remain occluded during the whole duration of the activity. We solve this problem using regression, performed by a novel deep Convolutional Recurrent Neural Network (CRNN). Specifically, given a partially occluded skeleton, we attempt to reconstruct the missing information regarding the motion of its occluded part(s). We evaluate our approach using four publicly available human motion datasets. Our experimental results indicate a significant increase of performance, when compared to baseline approaches, wherein networks that have been trained using only nonoccluded or both occluded and nonoccluded samples are evaluated using occluded samples. To the best of our knowledge, this is the first research work that formulates and copes with the problem of HAR under occlusion as a regression task.

Multiple object tracking with appearance feature prediction and similarity fusion

Object tracking is a crucial research area within the field of intelligent transportation, providing a vital foundation for anomalous behavior analysis and traffic statistics. Although pedestrian detectors have shown impressive results, leading to the advancement of detection-based tracking methods, target association in complex scenarios remains a difficult and less efficient task due to the lack of feature robustness in the presence of partial occlusions. In the proposed tracking method, we extract convolutional features on each entire object and its local blocks, segmented by the superpixel algorithm. Aiming to emphasize the global and local information respectively, the global features for each entire object are extracted from the last layer of the backbone network, while local features are derived from a specific intermediate layer of the backbone network. The association between tracked targets and detected pedestrian candidates relies on fused similarity degrees. Furthermore, we use the transformer’s self-attention mechanism to predict features for the current frame based on the information within past frames, aiming to eliminate the effects of target appearance variations. Additionally, we remove redundant background pixels in the detected rectangles of pedestrian candidates by using a background modeling algorithm. Experimental results demonstrate that the tracker proposed in this paper outperforms other trackers across five publicly available datasets, indicating its effectiveness and potential for further development.

Semi-supervised learning approach for localization and pose estimation of texture-less objects in cluttered scenes

3D object recognition and 6D pose estimation are crucial and fundamental endeavours for industrial assembly line automation such as robotic controlled pick-and-place. While the problem on textured objects is extensively studied, it is still an open research topic for texture-less industrial parts, e.g, solid cylinder and hollow tube, which are symmetric and appear similar in shapes from many viewing perspectives, causing pose ambiguity. Also, the industrial assembly line environment is usually cluttered and the captured data is noisy, which makes this task even more challenging. In this paper, we propose a novel object localization and pose estimation technique using RGB images and depth maps of industrial assembly parts. Our segmentation model is fully morphological and unsupervised for localizing the region of interest containing the target object extracted from the depth map. Our segmentation technique is effective in the presence of partial occlusion, multiple objects, and cluttered scenes. We use a model based approach for object recognition based on Stochastic Gradient Descent trained on features of Histogram of Oriented Gradients (HOG) and invariant moments of the region of interest containing the target object. We generate synthetic training images automatically from the CAD models of the industrial parts. We use a contour matching strategy based on Dynamic Time Warping (DTW) algorithm to estimate the optimal 6D pose of the object from a set of candidates. Experimental results show that our proposed approach competes and demonstrates advantages on the challenging T-LESS dataset compared to the state-of-the-art methods. • We introduce a strategy for generating multiple viewpoint synthetic images from a CAD model. • We introduce a non-supervised clustering process for object segmentation based on a depth map that does not require a training stage. • We demonstrate how different HOG feature parameters can be tuned to achieve better prediction and performance. • We present a new real-time strategy for object pose estimation using Dynamic Time Warping (DTW) between the target object contour and best prior candidates. • We show that our method achieves better time performance than other works for pose estimation.

Hogel-Free Holography

Holography is a promising avenue for high-quality displays without requiring bulky, complex optical systems. While recent work has demonstrated accurate hologram generation of 2D scenes, high-quality holographic projections of 3D scenes has been out of reach until now. Existing multiplane 3D holography approaches fail to model wavefronts in the presence of partial occlusion while holographic stereogram methods have to make a fundamental tradeoff between spatial and angular resolution. In addition, existing 3D holographic display methods rely on heuristic encoding of complex amplitude into phase-only pixels which results in holograms with severe artifacts. Fundamental limitations of the input representation, wavefront modeling, and optimization methods prohibit artifact-free 3D holographic projections in today’s displays. To lift these limitations, we introduce hogel-free holography which optimizes for true 3D holograms, supporting both depth- and view-dependent effects for the first time. Our approach overcomes the fundamental spatio-angular resolution tradeoff typical to stereogram approaches. Moreover, it avoids heuristic encoding schemes to achieve high image fidelity over a 3D volume. We validate that the proposed method achieves 10 dB PSNR improvement on simulated holographic reconstructions. We also validate our approach on an experimental prototype with accurate parallax and depth focus effects.

Robust optimum design of maximum power point tracker for photovoltaic power generation systems

The photovoltaic array’s power-voltage curve usually appears multiple peaks in the presence of partial occlusion and fault conditions, and the habitual maximum power point tracking (MPPT) algorithm is not easy to discover the MPP of the global space, thus falling into the local range, which extremely decreases the efficiency of the photovoltaic power generation system. In this paper, a robust optimum design technology employing an enhanced terminal attractor (ETA) and a monkey-king genetic algorithm (MKGA) are put forward. The proposed technology can be adopted for the maximum power point tracker of the photovoltaic power generation system, therefore yielding fast limited-time convergence and global extremum. The proposed photovoltaic power generation system is also implemented on the basis of Texas Instruments digital signal processor, to achieve the improvement of the tracking speed and good performance in the transient as well as steady-state. Simulation and experimental results have been provided to demonstrate the virtue of the proposed technology.

Pixel-Wise Weighted Region-Based 3D Object Tracking Using Contour Constraints.

Region-based methods are currently achieving state-of-the-art performance for monocular 3D object tracking. However, they are still prone to fail in cases of partial occlusions and ambiguous colors. We propose a novel region-based method to tackle these problems. The key idea is to derive a pixel-wise weighted region-based cost function using contour constraints. First, we propose a novel region-based cost function using search lines around the object contour, which is more efficient than previous region-based cost functions using signed distance transform, and in the meantime can deal with partial occlusions and ambiguous colors more effectively. Second, we propose an optimal searching strategy to search the object contour points in cluttered scenes, and then use the object contour points to detect partial occlusions and ambiguous colors. Third, we propose a pixel-wise weight function based on color and distance constraints of the object contour points, and integrate it into the proposed region-based cost function to reduce the negative impact of partial occlusions and ambiguous colors. We verify the effectiveness and efficiency of our method on challenging public datasets. Experiments demonstrate that our method outperforms the recent state-of-the-art region-based methods in complex scenarios, especially in the presence of partial occlusions and ambiguous colors.

Deep learning integral imaging for three-dimensional visualization, object detection, and segmentation

A depth slice image that is computationally reconstructed from an integral imaging system consists of focused and out of focus areas. The unfocused areas affect three-dimensional (3D) image analyses and visualization including 3D object detection, extraction, and tracking. In this work, we present a deep learning integral imaging system that can reconstruct a 3D image without the out of focus areas and can accomplish target detection and segmentation at the same time. A Mask-Regional Convolutional Neural Network (Mask-RCNN) deep learning algorithm was trained using a public dataset and applied to detect and segment multiple targets in two-dimensional (2D) elemental images in the integral imaging system. The 3D images were then reconstructed using segmented elemental images with the target detected. The proposed method works well in the presence of partial occlusions. Experimental results show the performance of the proposed scheme.

Structured object tracking with discriminative patch attributed relational graph

Local features have been widely used in visual tracking to improve robustness in the presence of partial occlusion, deformation, and rotation. In this paper, a structured object tracking algorithm, which uses local discriminative color (LoDC) patch representation and discriminative patch attributed relational graph (DPARG) matching, is proposed. Unlike several existing local feature-based algorithms that divide an object into some rectangular patches of fixed sizes while separately locating each patch to track the object, the proposed algorithm relies on a discriminative color model to distinguish the outstanding colors of the given object. Thus, the multimodal color object is represented by multiple unimodal, homogeneous, and discriminative patches. Moreover, these patches are assembled in a structured DPARG, in which vertexes describe the object’s local discriminative patches while encoding the appearance information, and edges express the relations between vertexes while encoding inner geometric structure information. The object tracking is then formulated as inexact matching of the dynamic undirected graph. The changes of DPARG, along with dynamic environments, are used to filter out invalid patches at the current frame, which usually correspond to those abnormal patches emerging from partial occlusion, similar color disturbances, etc. Finally, the valid patches are assembled to locate the object. The experimental results on the popular tracking benchmark datasets exhibit that the proposed algorithm is reliable enough in tracking even in the presence of serious appearance changes, partial occlusion, and background clutter.

Fundamentals of automated human gesture recognition using 3D integral imaging: a tutorial

Automated human gesture recognition is receiving significant research interest, with applications ranging from novel acquisition techniques to algorithms, data processing, and classification methodologies. This tutorial presents an overview of the fundamental components and basics of the current 3D optical image acquisition technologies for gesture recognition, including the most promising algorithms. Experimental results illustrate some examples of 3D integral imaging, which are compared to conventional 2D optical imaging. Examples of classifying human gestures under normal and degraded conditions, such as low illumination and the presence of partial occlusions, are provided. This tutorial is aimed at an audience who may or may not be familiar with gesture recognition approaches, current 3D optical image acquisition techniques, and classification algorithms and methodologies applied to human gesture recognition.

Gaussian-response correlation filter for robust visual object tracking

This paper presents a novel correlation filter-based tracking method for robust visual object tracking in the presence of partial occlusion, large-scale variation and model drift. To do this, first, we develop a correlation filter for predicting the target location based on the distribution of correlation response. In this formulation, the correlation response of the target image follows Gaussian distribution to estimate the target location efficiently. Second, the constraints are derived using kernel ridge regression to mitigate the target failure in object tracking. Third, we propose an adaptive scale estimation method to detect the target scale changes during the tracking. In addition, two feature integration is elaborately designed to improve the discriminative strength of the correlation filter. Finally, extensive experimental results on OTB2013, OTB2015, TempleColor128 and UAV123 datasets demonstrate that the proposed method performs favourably against several state-of-the-art methods.

Cluttered TextSpotter: An End-to-End Trainable Light-Weight Scene Text Spotter for Cluttered Environment

Scene text spotting aims at simultaneously localizing and recognizing text instances, symbols, and logos in natural scene images. Scene text detection and recognition approaches have received immense attention in computer vision research community. The presence of partial occlusion or truncation artifact due to the cluttered background of scene images creates an obstacle in perceiving the text instances, which makes the process of spotting very complex. In this paper, we propose a light-weight scene text spotter that can address the issue of cluttered environment of scene images. It is an end-to-end trainable deep neural network that uses local part information, global structural features, and context cue information of oriented region proposals for spotting text instances. It helps to localize in scene images with background clutters, where partially occluded text parts, truncation artifacts, and perspective distortions are present. We mitigate the problem of misclassification caused by inter-class interference by exploring inter-class separability and intra-class compactness. We also incorporate multi-language character segmentation and word-level recognition in a light-weight recognition module. We have used six publicly available benchmark datasets in different smart devices to illustrate the efficacy of the network.

In-situ Online Measurement of Rhombic Distortion in Billets

DD (diagonal-difference) is considered as measure of rhombic distortion, aka Rhomboidity, which is a shape related defect in square cross-section billets. Rhomboidity in billets starts with non-uniform shell solidification in the mold primarily due to inconsistent cooling causing irregular heat transfer. The higher diagonal difference greatly impacts the quality of billets to be rolled at various mills. Rhomboidity at or over 4% leads to billet twisting in the roughing stands of the rolling mill. Currently, billet rhomboidity is measured manually at end of casting operation. The presented work describes an optical, online & real-time image processing based method to determine the rhomboidity induced in each strand and alerts the operator to take corrective actions. The online Rhomboidity Measurement System employs sophisticated image acquisition & processing techniques to determine face contours of the billet with sub-pixel accuracy. The key features of RMS are the construction of a gaussian penalty function for selection of suitable 4-lines combination that precisely fits the billet face and use of a highly efficient and accurate statistical indicator, based on KL-Divergence measure, to estimate the rhomboidity even in presence of partial occlusion of billet face by scales. The expected savings are to the tune of 0.27 Million USD.

3D object recognition for anthropomorphic robots performing tracking tasks

Object recognition is still a major research issue of particular relevance in robotics. The new trend in industrial and mainly in service robotics is to perform manipulation tasks in an unstructured environment working in synergy with humans. To perform tasks in an environment that is not perfectly controlled, robots need adequate perceptual capabilities. Among various types of sensors available for robotic systems, the time-of-flight (ToF) camera is one of the most utilized since it simultaneously provides intensity and depth data at a high frame rate. Our proposal makes use of this technology exploiting both, depth and grey-scale information. Therefore, intensity and geometric features are fused together to allow 3D object recognition in real scenes in presence of partial occlusions. As a case study, an object tracking task for an anthropomorphic robot is presented. Experimental results demonstrate the effectiveness of the proposed method, not only providing reliable visual information about the object to be tracked but also recognizing potential obstacles which should be avoided during the robot arm movement.

Automatic Ear Detection and Segmentation over Partially Occluded Profile Face Images

Automated, non invasive ear detection in images and video is becoming increasingly required in several contexts, including nonivasive biometric identification, biomedical analysis, forensics, and many others. In biometric recognition systems, fast and robust ear detection is a crucial step within the recognition pipeline. Existing approaches to ear detection are susceptible to fail in the presence of typical everyday situations that prevent a crisp imaging of the ears, like partial occlusions, ear accessories, or uncontrolled camera and illumination conditions. Even more, most of the proposed solutions work efficiently only within a previously detected rectangular region of interest, which limits their applicability and lowers the accuracy of the overall detection. In this paper we evaluate the use of Convolutional Neural Networks (CNNs) together with Geometric Morphometrics (GM) for automatic ear detection in the presence of partial occlusions, and a Convex Hull algorithm for the ear area segmentation. A CNN was trained with a set of ear images landmarked by experts using GM to achieve high consistency. After training, the CNN is able to detect ears over profile faces, even in the presence of partial occlusions. We analyze the performance of the proposed ear detection and segmentation method over partially occluded ear images using the CVL Dataset

Spatial-temporal human gesture recognition under degraded conditions using three-dimensional integral imaging.

We present spatial-temporal human gesture recognition in degraded conditions including low light levels and occlusions using passive sensing three-dimensional (3D) integral imaging (InIm) system and 3D correlation filters. The 4D (lateral, longitudinal, and temporal) reconstructed data is processed using a variety of algorithms including linear and non-linear distortion-invariant filters; and compared with previously reported space-time interest points (STIP) feature detector, 3D histogram of oriented gradients (3D HOG) feature descriptor, with a standard bag-of-features support vector machine (SVM) framework, etc. The gesture recognition results with different classification algorithms are compared using a variety of performance metrics such as receiver operating characteristic (ROC) curves, area under the curve (AUC), SNR, the probability of classification errors, and confusion matrix. Integral imaging video sequences of human gestures are captured under degraded conditions such as low light illumination and in the presence of partial occlusions. A four-dimensional (4D) reconstructed video sequence is computed that provides lateral and depth information of a scene over time i.e. (x, y, z, t). The total-variation denoising algorithm is applied to the signal to further reduce noise and preserve data in the video frames. We show that the 4D signal consists of decreased scene noise, partial occlusion removal, and improved SNR due to the computational InIm and/or denoising algorithms. Finally, gesture recognition is processed with classification algorithms, such as distortion-invariant correlation filters; and STIP, 3D HOG with SVM, which are applied to the reconstructed 4D gesture signal to classify the human gesture. Experiments are conducted using a synthetic aperture InIm system in ambient light. Our experiments indicate that the proposed approach is promising in detection of human gestures in degraded conditions such as low illumination conditions with partial occlusion. To the best of our knowledge, this is the first report on spatial-temporal human gesture recognition in degraded conditions using passive sensing 4D integral imaging with nonlinear correlation filters.

Robust Multi-scale Prior L-K Tracking Based on Local Features

Due to the limitation of algorithm principle, inverse compositional L-K algorithm always fails to track fast moving targets. An improved tracking method based on multi-scale motion prior information is proposed in this paper. The multi-scale priori error Jacobi matrix is calculated offline by artificially constructed multi-scale disturbance. The target searching strategy based on multi-scale stratification can greatly increase the convergence range of the algorithm. At the same time, in order to improve the stability of global feature tracking in the case of local occlusion, the pyramid iterative L-K tracking method based on local feature is employed. Experimental results demonstrate outstanding stability of our method for fast moving targets. And by taking advantage of local gray information around feature points, our algorithm can still effectively track the target in the presence of partial occlusion.

TransHist: Occlusion-robust shape detection in cluttered images

Shape matching plays an important role in various computer vision and graphics applications such as shape retrieval, object detection, image editing, image retrieval, etc. However, detecting shapes in cluttered images is still quite challenging due to the incomplete edges and changing perspective. In this paper, we propose a novel approach that can efficiently identify a queried shape in a cluttered image. The core idea is to acquire the transformation from the queried shape to the cluttered image by summarising all point-to-point transformations between the queried shape and the image. To do so, we adopt a point-based shape descriptor, the pyramid of arc-length descriptor (PAD), to identify point pairs between the queried shape and the image having similar local shapes. We further calculate the transformations between the identified point pairs based on PAD. Finally, we summarise all transformations in a 4D transformation histogram and search for the main cluster. Our method can handle both closed shapes and open curves, and is resistant to partial occlusions. Experiments show that our method can robustly detect shapes in images in the presence of partial occlusions, fragile edges, and cluttered backgrounds.

Multi-modal imaging, model-based tracking, and mixed reality visualisation for orthopaedic surgery.

Orthopaedic surgeons are still following the decades old workflow of using dozens of two-dimensional fluoroscopic images to drill through complex 3D structures, e.g. pelvis. This Letter presents a mixed reality support system, which incorporates multi-modal data fusion and model-based surgical tool tracking for creating a mixed reality environment supporting screw placement in orthopaedic surgery. A red–green–blue–depth camera is rigidly attached to a mobile C-arm and is calibrated to the cone-beam computed tomography (CBCT) imaging space via iterative closest point algorithm. This allows real-time automatic fusion of reconstructed surface and/or 3D point clouds and synthetic fluoroscopic images obtained through CBCT imaging. An adapted 3D model-based tracking algorithm with automatic tool segmentation allows for tracking of the surgical tools occluded by hand. This proposed interactive 3D mixed reality environment provides an intuitive understanding of the surgical site and supports surgeons in quickly localising the entry point and orienting the surgical tool during screw placement. The authors validate the augmentation by measuring target registration error and also evaluate the tracking accuracy in the presence of partial occlusion.

Structured fragment-based object tracking using discrimination, uniqueness, and validity selection

Local features have widely been used in visual tracking to improve robustness in the presence of partial occlusion, deformation, and rotation. In this paper, a local fragment-based object tracking algorithm is proposed. Unlike many existing fragment-based algorithms using all the fragments and allocating the weight to each fragment according to similarity, the proposed algorithm only selects discriminative, unique, and valid fragments for tracking. First, discrimination and uniqueness metric are defined for each local fragment, and an automatic pre-selection mechanism is proposed for all these fragments. Second, a Harris-SIFT filter is used to select the current valid fragments and exclude the occluded or highly deformed fragments. By selecting the discriminative, unique, and valid fragments, these fragments are used to construct a structured description for the object. Finally, the object tracking is performed using the selected fragments combining the displacement and similarity, as well as spatial constraint of the selected fragments. The object template can be updated by fusing feature similarity and structural consistency. The experimental results on a recent OTB 2013 tracking benchmark data set demonstrate that the proposed algorithm can achieve reliable tracking results even in the presence of significant appearance changes, partial occlusion, and similar disturbances.

Three-dimensional integral imaging and object detection using long-wave infrared imaging.

We propose a passive three-dimensional (3D) imaging technique based on integral imaging using a long-wave infrared (LWIR) camera. 3D imaging can improve visualization and detection of objects in adverse environments, such as low light levels and the presence of partial occlusions, along with depth estimation by reconstructing the scene at the plane of the object. This is achieved by capturing multiple two-dimensional images, known as elemental images (EI), of a scene with each image having a unique perspective of the 3D objects. Moreover, LWIR imaging performs well in photon-limited environments due to detection of thermal radiation from an object rather than the reflected light. Once the EIs have been captured, image restoration is performed on the captured images. A 3D scene is then reconstructed and object detection using correlation filters and support vector machines is performed. Our experiments with human face detection show that 2D imaging may fail to detect occluded humans, whereas passive 3D imaging with LWIR could be successful. To the best of our knowledge, this is the first report of passive 3D integral imaging with LWIR for object detection, and in particular, in low light environments.