Self-occlusion Problem Research Articles

Extended target tracking under multitarget tracking framework for convex polytope shapes

This paper discusses the problem of extended target tracking for a single 2D extended target with a convex polytopic shape and known dynamics. Extended targets are those that produce multiple measurements for a single frame. One of the major challenges in extended target tracking is the joint uncertainty in the shape and the kinematics of the target. Another challenge is the lack of visibility due to self-occlusion in targets with a finite extent (as opposed to zero extent for point targets). To address these challenges, we develop a framework for tracking single (or widely separated) extended targets. This framework is based on the existing point multitarget tracking framework by modeling different parts of an extended target as separate targets. An algorithm is developed using the proposed framework for tracking convex polytope-shaped targets. The proposed shape function consists only of the boundary of the target since the center may not be observable. The algorithm is capable of dynamically changing the number of parameters used to describe the shape as more parts of the target become visible over time. The performance of the algorithm is evaluated for various scenarios using root mean square error (RMSE) of velocity, center, and intersection over union (IoU) metrics. It is seen that the algorithm is able to handle the self-occlusion problem and estimate the whole target shape even when different parts of the target are visible at different frames, for various shapes, and for various conditions of measurement noise covariance and number of measurements. New faces are added to the shape estimate as more parts of the target become visible. The algorithm is able to conserve the parts of the target that were visible in the previous frames but are no longer visible.

On the Use of Circadian Cycles to Monitor Individual Young Plants

Occlusion is a very common problem in computer vision. The presence of objects seen as overlapped under a camera negatively impacts object recognition, object counting or shape estimation. This problem is especially important in plant imaging because plants are very self-similar objects which produce a lot of self-occlusions. A possible way to disentangle apparent occlusions is to acquire the same scene from different points of view when the object is motionless. Such a realization is not necessary if the objects move themselves under the camera and thus offer different points of view for free. This is the case in plant imagery, since plants have their own natural movements, including the so-called circadian rhythms. We propose to use these movements to solve some self-occlusion problems with a set of simple yet innovative sampling algorithms to monitor the growth of individualized young plants. The proposed sampling methods make it possible to monitor the growth of the individual plants until their overlap is definitive. The gain is significant with an average maximum duration of observation increase from 3 days to more than 10 days by comparison with a sampling method that would stop when the first overlap occurs.

Flight Planning for Survey-Grade 3D Reconstruction of Truss Bridges

Autonomous UAV 3D reconstruction has been widely used for infrastructure inspections and asset management. However, its applications on truss structures remain a challenging task due to geometric complexity and the severe self-occlusion problem of the truss structures when constrained by camera FOV, safety clearance, and flight duration. This paper proposes a new flight planning method to effectively address the self-occlusion problem to enable autonomous and efficient data acquisition for survey-grade 3D truss reconstruction. The proposed method contains two steps: First, identifying a minimal set of viewpoints achieves the maximal reconstruction quality at every observed surface of the truss geometry through an iterative optimization schema. Second, converting the optimal viewpoints into the shortest, collision-free flight trajectories while considering the UAV constraints. The computed flight path can also be implemented in a multi-UAV fashion. Evaluations of the proposed method include a synthetic truss bridge and a real-world truss bridge. The evaluation results suggested that the proposed approach outperforms the existing works in terms of 3D reconstruction quality while taking less time in both the inflight image acquisition and the post-flight 3D reconstruction.

3D Human Pose Estimation With Spatial Structure Information

Estimating 3D human poses from 2D poses is a challenging problem due to joints self-occlusion, weak generalization, and inherent ambiguity of recovering depth. Actually, there exists spatial structure dependence on human body key points which can be used to alleviate the problem of joints self-occlusion. Therefore, we represent human pose as a directed graph and propose a network implemented with graph convolution to predict 3D poses from the given 2D poses. In the digraph, we determine the connection weight of each edge according to the error distribution of joints estimation. This makes our model robust to noise. By optimizing coarse 3D estimation and adversarial learning, our algorithm can successfully improve the accuracy of estimation and relieve the ambiguity of mapping. Through testing on Human 3.6M and MPI-INF-3DHP datasets, we achieve excellent quantitative performance. More importantly, our algorithm also has a superior generalization to outdoor dataset MPII by the pre-training process.

Geometry-Driven Self-Supervised Method for 3D Human Pose Estimation

The neural network based approach for 3D human pose estimation from monocular images has attracted growing interest. However, annotating 3D poses is a labor-intensive and expensive process. In this paper, we propose a novel self-supervised approach to avoid the need of manual annotations. Different from existing weakly/self-supervised methods that require extra unpaired 3D ground-truth data to alleviate the depth ambiguity problem, our method trains the network only relying on geometric knowledge without any additional 3D pose annotations. The proposed method follows the two-stage pipeline: 2D pose estimation and 2D-to-3D pose lifting. We design the transform re-projection loss that is an effective way to explore multi-view consistency for training the 2D-to-3D lifting network. Besides, we adopt the confidences of 2D joints to integrate losses from different views to alleviate the influence of noises caused by the self-occlusion problem. Finally, we design a two-branch training architecture, which helps to preserve the scale information of re-projected 2D poses during training, resulting in accurate 3D pose predictions. We demonstrate the effectiveness of our method on two popular 3D human pose datasets, Human3.6M and MPI-INF-3DHP. The results show that our method significantly outperforms recent weakly/self-supervised approaches.

Surface Scanning-Based Texture Atlas for Voxelized 3D Object

The representation of voxel colors has become crucial because voxelized 3D data are used in a variety of applications. In particular, texture atlases are widely used for representing voxel colors. The texture atlas is typically utilized for encoding voxel colors using image/video encoders, so it is important to design a texture atlas while maintaining the connectivity between voxels. In this paper, we present a texture atlas based on a surface scanning scheme. To this end, we represent a 3D object as slices, and texture strips are obtained by scanning voxels along the surface for each slice. The texture strips represent the voxel colors while preserving the connectivity of the voxels. Besides, because the texture strips are highly correlated, we construct a texture atlas by aligning texture strips. We also propose a partitioning method for a 3D object to solve the self-occlusion problem that occurs in the alignment process. As an application of the proposed texture atlases, we utilize it for encoding voxel colors and propose encoding strategies. Experimental results indicate that the proposed texture atlases outperform the previous methods. To the best of our knowledge, this is the first attempt to use a surface scanning scheme for a texture atlas.

Occlusion-robust bimanual gesture recognition by fusing multi-views

Human hands are dexterous and always be an intuitive way to instruct or communicate with peers. In recent years, hand gesture is widely used as a novel way for human computer interaction as well. However, existing approaches target solely to recognize single-handed gesture, but not gestures with two hands in close proximity (bimanual gesture). Thus, this paper tries to tackle the problems in bimanual gestures recognition which are not well studied from the literature. To overcome the critical issue of hand-hand self-occlusion problem in bimanual gestures, multiple cameras from different view points are used. A tailored multi-camera system is constructed to acquire multi-views bimanual gesture data. By employing both shape and color features, classifiers are trained with our bimanual gestures dataset. A weighted sum fusion scheme is employed to ensemble results predicted from different classifiers. While, the weightings in the fusion are optimized according to how well the recognition performed on a particular view. Our experiments show that multiple-view results outperform single-view results. The proposed method is especially suitable to interactive multimedia applications, such as our two demo programs: a video game and a sign language learner.

Face Frontalization Using an Appearance-Flow-Based Convolutional Neural Network.

Facial pose variation is one of the major factors making face recognition (FR) a challenging task. One popular solution is to convert non-frontal faces to frontal ones on which FR is performed. Rotating faces causes facial pixel value changes. Therefore, existing CNN-based methods learn to synthesize frontal faces in color space. However, this learning problem in a color space is highly non-linear, causing the synthetic frontal faces to lose fine facial textures. In this paper, we take the view that the nonfrontal-frontal pixel changes are essentially caused by geometric transformations (rotation, translation, and so on) in space. Therefore, we aim to learn the nonfrontal-frontal facial conversion in the spatial domain rather than the color domain to ease the learning task. To this end, we propose an appearance-flow-based face frontalization convolutional neural network (A3F-CNN). Specifically, A3F-CNN learns to establish the dense correspondence between the non-frontal and frontal faces. Once the correspondence is built, frontal faces are synthesized by explicitly "moving" pixels from the non-frontal one. In this way, the synthetic frontal faces can preserve fine facial textures. To improve the convergence of training, an appearance-flow-guided learning strategy is proposed. In addition, generative adversarial network loss is applied to achieve a more photorealistic face, and a face mirroring method is introduced to handle the self-occlusion problem. Extensive experiments are conducted on face synthesis and pose invariant FR. Results show that our method can synthesize more photorealistic faces than the existing methods in both the controlled and uncontrolled lighting environments. Moreover, we achieve a very competitive FR performance on the Multi-PIE, LFW and IJB-A databases.

Dynamic Pose Estimation Using Multiple RGB-D Cameras.

Human poses are difficult to estimate due to the complicated body structure and the self-occlusion problem. In this paper, we introduce a marker-less system for human pose estimation by detecting and tracking key body parts, namely the head, hands, and feet. Given color and depth images captured by multiple red, green, blue, and depth (RGB-D) cameras, our system constructs a graph model with segmented regions from each camera and detects the key body parts as a set of extreme points based on accumulative geodesic distances in the graph. During the search process, local detection using a supervised learning model is utilized to match local body features. A final set of extreme points is selected with a voting scheme and tracked with physical constraints from the unified data received from the multiple cameras. During the tracking process, a Kalman filter-based method is introduced to reduce positional noises and to recover from a failure of tracking extremes. Our system shows an average of 87% accuracy against the commercial system, which outperforms the previous multi-Kinects system, and can be applied to recognize a human action or to synthesize a motion sequence from a few key poses using a small set of extremes as input data.

Human Action Recognition Using Motion History Image Based Temporal Segmentation

A human action recognition system based on image depth is proposed in this paper. Depth information features are not easily disturbed by noise; and due to this characteristic, the system can quickly extract foreground targets. Moreover, the target data, namely, depth and two-dimensional (2D) data, are projected to three orthogonal planes. In this manner, the action featured in the depth motion along the optical axis can clearly describe the trajectory. Based on the change of motion energy and the angle variations of motion orientations, the temporal segmentation (TS) method automatically segments the complex action into several simple movements. Three-dimensional (3D) data is further applied to acquire the three-viewpoint (3V) motion history trajectory, whereby a target’s motion is described through the motion history images (MHIs) from the 3Vs. The weightings corresponding to the gradients of the MHIs are included for determining the viewpoint that bests describe the target’s motion. In terms of feature extraction, the application of multi-resolution motion history histograms can effectively reduce the computational load and achieve a high recognition rate. Experimental results demonstrate that the proposed method can effectively solve the self-occlusion problem.

3-D Object Recognition via Aspect Graph Aware 3-D Object Representation

This letter addresses the problem of 3-D object recognition, whose aim is to recognize and estimate the pose of user-defined 3-D object when given an image. One difficult problem for 3-D object recognition is false correspondences between input image and 3-D model. To overcome this problem, we propose a novel aspect graph aware 3-D object representation method which enable us to output continuous pose and deal with self-occlusion problem. We also propose a two-stage 2-D to 3-D false correspondence filter based on proposed 3-D representation to achieve more consistent 2-D to 3-D matching pairs. We evaluate our proposed algorithm on Weizman Cars Viewpoint dataset and it demonstrates obvious improvement on localization and pose estimation accuracy compared with traditional methods. Besides, our proposed method accelerates computation time.

An adaptively weighted algorithm for camera calibration with 1D objects

Camera calibration is a necessary step to extract 3D metric information from 2D images. 1D object-based camera calibration has recently received attention because there is no self-occlusion problem for a 1D object. In this paper, an adaptively weighted 1D calibration algorithm with high accuracy is presented. The innovation and contributions of such an algorithm are two-fold: one is that data normalization is used to improve estimation accuracy of the relative depths of marker points on the 1D calibration object; the other one is that a weighted coefficient is adaptively assigned to each constraint on the camera parameters by analyzing the data error involved in the constraint. Experiments with synthetic and real image data show that the accuracy of the proposed algorithm is much higher than that of classical 1D calibration. In some cases, it outperforms the nonlinear optimal 1D algorithm and is even comparable with the normalized 2D and 3D calibration.

Method for User Interface of Large Displays Using Arm Pointing and Finger Counting Gesture Recognition

Although many three-dimensional pointing gesture recognition methods have been proposed, the problem of self-occlusion has not been considered. Furthermore, because almost all pointing gesture recognition methods use a wide-angle camera, additional sensors or cameras are required to concurrently perform finger gesture recognition. In this paper, we propose a method for performing both pointing gesture and finger gesture recognition for large display environments, using a single Kinect device and a skeleton tracking model. By considering self-occlusion, a compensation technique can be performed on the user's detected shoulder position when a hand occludes the shoulder. In addition, we propose a technique to facilitate finger counting gesture recognition, based on the depth image of the hand position. In this technique, the depth image is extracted from the end of the pointing vector. By using exception handling for self-occlusions, experimental results indicate that the pointing accuracy of a specific reference position was significantly improved. The average root mean square error was approximately 13 pixels for a 1920 × 1080 pixels screen resolution. Moreover, the finger counting gesture recognition accuracy was 98.3%.

Adaptive Self-Occlusion Behavior Recognition Based on pLSA

Human action recognition is an important area of human action recognition research. Focusing on the problem of self-occlusion in the field of human action recognition, a new adaptive occlusion state behavior recognition approach was presented based on Markov random field and probabilistic Latent Semantic Analysis (pLSA). Firstly, the Markov random field was used to represent the occlusion relationship between human body parts in terms an occlusion state variable by phase space obtained. Then, we proposed a hierarchical area variety model. Finally, we use the topic model of pLSA to recognize the human behavior. Experiments were performed on the KTH, Weizmann, and Humaneva dataset to test and evaluate the proposed method. The compared experiment results showed that what the proposed method can achieve was more effective than the compared methods.

Trinocular Calibration Method Based on Binocular Calibration

In order to solve the self-occlusion problem in plane-based multi-camera calibration system and expand the measurement range, a tri-camera vision system based on binocular calibration is proposed. The three cameras are grouped into two pairs, while the public camera is taken as the reference to build the global coordinate. By calibration of the measured absolute distance and the true absolute distance, global calibration is realized. The MRE (mean relative error) of the global calibration of the two camera pairs in the experiments can be as low as 0.277% and 0.328% respectively. Experiment results show that this method is feasible, simple and effective, and has high precision. DOI: http://dx.doi.org/10.11591/telkomnika.v10i6.1417 Full Text: PDF

Single view-based 3D face reconstruction robust to self-occlusion

Abstract State-of-the-art 3D morphable model (3DMM) is used widely for 3D face reconstruction based on a single image. However, this method has a high computational cost, and hence, a simplified 3D morphable model (S3DMM) was proposed as an alternative. Unlike the original 3DMM, S3DMM uses only a sparse 3D facial shape, and therefore, it incurs a lower computational cost. However, this method is vulnerable to self-occlusion due to head rotation. Therefore, we propose a solution to the self-occlusion problem in S3DMM-based 3D face reconstruction. This research is novel compared with previous works, in the following three respects. First, self-occlusion of the input face is detected automatically by estimating the head pose using a cylindrical head model. Second, a 3D model fitting scheme is designed based on selected visible facial feature points, which facilitates 3D face reconstruction without any effect from self-occlusion. Third, the reconstruction performance is enhanced by using the estimated pose as the initial pose parameter during the 3D model fitting process. The experimental results showed that the self-occlusion detection had high accuracy and our proposed method delivered a noticeable improvement in the 3D face reconstruction performance compared with previous methods.

Initialization of 3D Human Hand Model and Its Applications in Human Hand Tracking

Initialization of 3D human hand gesture is one of the fundamental and key steps in the study of 3D hand tracking, and a novel approach to initialize 3D human hand gesture is put forward in this paper. This paper will cover the following points. First, a new approach to selecting a human hand gesture from the hand postures database is presented. Second, both techniques of visualization and human-computer interaction are used into the initialization process, through which the 3D human hand model is fine-tuned time after time until the required accuracy is satisfied. Lastly, the proposed initialization method is applied to 3D human hand tracking system based on PF (Particle Filtering), with real video data under complex background. In order to address high dimensional problem of 3D hand structures, a new concept, which is called key factor in this paper, is introduced to guide sampling; to improve robustness to changing light conditions, a new skin model is proposed. The main contributions of this paper include: (1) combine the techniques of the interaction between operator and computer and the visualization to achieve initialization of 3D human hand model. (2) use the key factors to address the issue of high dimensionality of 3D hand model, and (3) a new hand skin model is presented, as well as self-occlusion problem is effective addressed. Our experimental results show that the proposed approach is not only fast, accurate and robust, but also direct, natural and convenient for operators to handle.

Posture Symmetry based Motion Capture System for Analysis of Lower -limbs Rehabilitation Training

This paper presents a motion capture based rehabilitation training system for a lower-limb paretic patient. The system evaluates the rehabilitation status of the patient by using the bend posture of the knees and the weight balance of the body. The posture of both legs is captured with a single camera using the planar mirror. The weight distribution is obtained by the Wii Balance Board. Self-occlusion problem in the tracking of the legs is resolved by using k-nearest neighbor based clustering with body symmetry and local-linearity of the posture data. To do this, we present data normalization and its symmetric property in the normalized vector space.

3D human modeling from a single depth image dealing with self-occlusion

This paper presents a 2D to 3D conversion scheme to generate a 3D human model using a single depth image with several color images. In building a complete 3D model, no prior knowledge such as a pre-computed scene structure and photometric and geometric calibrations is required since the depth camera can directly acquire the calibrated geometric and color information in real time. The proposed method deals with a self-occlusion problem which often occurs in images captured by a monocular camera. When an image is obtained from a fixed view, it may not have data for a certain part of an object due to occlusion. The proposed method consists of following steps to resolve this problem. First, the noise in a depth image is reduced by using a series of image processing techniques. Second, a 3D mesh surface is constructed using the proposed depth image-based modeling method. Third, the occlusion problem is resolved by removing the unwanted triangles in the occlusion region and filling the corresponding hole. Finally, textures are extracted and mapped to the 3D surface of the model to provide photo-realistic appearance. Comparison results with the related work demonstrate the efficiency of our method in terms of visual quality and computation time. It can be utilized in creating 3D human models in many 3D applications.

A multi-view vision-based hand motion capturing system

Vision-based hand motion capturing approaches play a critical role in human computer interface owing to its non-invasiveness, cost effectiveness, and user friendliness. This work presents a multi-view vision-based method to capture hand motion. A 3-D hand model with structural and kinematical constraints is developed to ensure that the proposed hand model behaves similar to an ordinary human hand. Human hand motion in a high degree of freedom space is estimated by developing a separable state based particle filtering (SSBPF) method to track the finger motion. By integrating different features, including silhouette, Chamfer distance, and depth map in different view angles, the proposed motion tracking system can capture the hand motion parameter effectively and solve the self-occlusion problem of the finger motion. Experimental results indicate that the hand joint angle estimation generates an average error of 11°.