Silhouette Extraction Algorithm Research Articles

A review of silhouette extraction algorithms for use within visual hull pipelines

ABSTRACT Markerless motion capture would permit the study of human biomechanics in environments where marker-based systems are impractical, e.g. outdoors or underwater. The visual hull tool may enable such data to be recorded, but it requires the accurate detection of the silhouette of the object in multiple camera views. This paper reviews the top-performing algorithms available to date for silhouette extraction, with the visual hull in mind as the downstream application; the rationale is that higher-quality silhouettes would lead to higher-quality visual hulls, and consequently better measurement of movement. This paper is the first attempt in the literature to compare silhouette extraction algorithms that belong to different fields of Computer Vision, namely background subtraction, semantic segmentation, and multi-view segmentation. It was found that several algorithms exist that would be substantial improvements over the silhouette extraction algorithms traditionally used in visual hull pipelines. In particular, FgSegNet v2 (a background subtraction algorithm), DeepLabv3+ JFT (a semantic segmentation algorithm), and Djelouah 2013 (a multi-view segmentation algorithm) are the most accurate and promising methods for the extraction of silhouettes from 2D images to date, and could seamlessly be integrated within a visual hull pipeline for studies of human movement or biomechanics.

FISHnet: Learning to Segment the Silhouettes of Swimmers

We present a novel silhouette extraction algorithm designed for the binary segmentation of swimmers underwater. The intended use of this algorithm is within a 2D-to-3D pipeline for the markerless motion capture of swimmers, a task which has not been achieved satisfactorily, partly due to the absence of silhouette extraction methods that work well on images of swimmers. Our algorithm, FISHnet, was trained on the novel Scylla dataset, which contains 3,100 images (and corresponding hand-traced silhouettes) of swimmers underwater, and achieved a dice score of 0.9712 on its test data. Our algorithm uses a U-Net-like architecture and VGG16 as a backbone. It introduces two novel modules: a modified version of the Semantic Embedding Branch module from ExFuse, which increases the complexity of the features learned by the layers of the encoder; and the Spatial Resolution Enhancer module, which increases the spatial resolution of the features of the decoder before they are skip connected with the features of the encoder. The contribution of these two modules to the performance of our network was marginal, and we attribute this result to the lack of data on which our network was trained. Nevertheless, our model outperformed state-of-the-art silhouette extraction algorithms (namely DeepLabv3+) on Scylla, and it is the first algorithm developed specifically for the task of accurately segmenting the silhouettes of swimmers.

Frontal View Gait Recognition With Fusion of Depth Features From a Time of Flight Camera

Frontal view gait recognition for people identification has been carried out using single RGB, stereo RGB, Kinect 1.0, and Doppler radar. However, existing methods based on these camera technologies suffer from several problems. Therefore, we propose a four-part method for frontal view gait recognition based on the fusion of multiple features acquired from a Time-of-Flight (ToF) camera. We have developed a gait data set captured by a ToF camera. The data set includes two sessions recorded seven months apart, with 46 and 33 subjects, respectively, each with six walks with five covariates. The four-part method includes: a new human silhouette extraction algorithm that reduces the multiple reflection problem experienced by ToF cameras; a frame selection method based on a new gait cycle detection algorithm; four new gait image representations; and a novel fusion classifier. Rigorous experiments are carried out to compare the proposed method with state-of-the-art methods. The results show distinct improvements over recognition rates for all covariates. The proposed method outperforms all major existing approaches for all covariates and results in 66.1% and 81.0% Rank 1 and Rank 5 recognition rates, respectively, in overall covariates, compared with a best state-of-the-art method performance of 35.7% and 57.7%.

Robust human silhouette extraction with Laplacian fitting

Human silhouette extraction has been a primary step to estimate human poses or classify activities from videos. While the accuracy of human silhouettes has great impact on the follow-on human pose/gait estimation, it has been important to guarantee the highly-accurate extraction of human silhouettes. However, traditional methods such as motion segmentation can be fragile due to the complexity of real-world environment. In this paper, we propose an automated human silhouette extraction algorithm to attain this highly-demanded task. In our proposed scheme, the initial motion segmentation of foreground objects was roughly computed by Stauffer’s background subtraction using Gaussian mixtures, and then refined by the proposed Laplacian fitting scheme. In our method, the candidate regions of human objects are taken as the initial input, their Laplacian matrices are constructed, and Eigen mattes are then obtained by minimizing on Laplacian matrices. RANSAC algorithm is then applied to fit the Eigen mattes iteratively with inliers of the initially estimated motion blob. Finally, the foreground human silhouettes are obtained from the optimized matte fitting. Experimental results on a number of test videos validated that the proposed Laplacian fitting scheme enhances the accuracy in automated human silhouette extraction, exhibiting a potential use of our Laplacian fitting algorithm in many silhouette-based applications such as human pose estimation.

Gait Recognition Based on Outermost Contour

Gait recognition aims to identify people by the way they walk. In this paper, a simple but effective gait recognition method based on outermost contour is proposed. For each gait image sequence, an adaptive silhouette extraction algorithm is firstly used to segment the frames of the sequence and a series of post-processing is applied to obtain the normalized silhouette images with less noise. Then a novel feature extraction method based on outermost contour is performed. Principal Component Analysis (PCA) is adopted to reduce the dimensionality of the distance signals derived from the outermost contours of silhouette images. Then Multiple Discriminant Analysis (MDA) is used to optimize the separability of gait features belonging to different classes. Nearest Neighbor (NN) classifier and Nearest Neighbor classifier with respect to class Exemplars (ENN) are used to classify the final feature vectors produced by MDA. In order to verify the effectiveness and robustness of our feature extraction algorithm, we ...

A neural tracking and motor control approach to improve rehabilitation of upper limb movements

BackgroundRestoration of upper limb movements in subjects recovering from stroke is an essential keystone in rehabilitative practices. Rehabilitation of arm movements, in fact, is usually a far more difficult one as compared to that of lower extremities. For these reasons, researchers are developing new methods and technologies so that the rehabilitative process could be more accurate, rapid and easily accepted by the patient. This paper introduces the proof of concept for a new non-invasive FES-assisted rehabilitation system for the upper limb, called smartFES (sFES), where the electrical stimulation is controlled by a biologically inspired neural inverse dynamics model, fed by the kinematic information associated with the execution of a planar goal-oriented movement. More specifically, this work details two steps of the proposed system: an ad hoc markerless motion analysis algorithm for the estimation of kinematics, and a neural controller that drives a synthetic arm. The vision of the entire system is to acquire kinematics from the analysis of video sequences during planar arm movements and to use it together with a neural inverse dynamics model able to provide the patient with the electrical stimulation patterns needed to perform the movement with the assisted limb.MethodsThe markerless motion tracking system aims at localizing and monitoring the arm movement by tracking its silhouette. It uses a specifically designed motion estimation method, that we named Neural Snakes, which predicts the arm contour deformation as a first step for a silhouette extraction algorithm. The starting and ending points of the arm movement feed an Artificial Neural Controller, enclosing the muscular Hill's model, which solves the inverse dynamics to obtain the FES patterns needed to move a simulated arm from the starting point to the desired point. Both position error with respect to the requested arm trajectory and comparison between curvature factors have been calculated in order to determine the accuracy of the system.ResultsThe proposed method has been tested on real data acquired during the execution of planar goal-oriented arm movements. Main results concern the capability of the system to accurately recreate the movement task by providing a synthetic arm model with the stimulation patterns estimated by the inverse dynamics model. In the simulation of movements with a length of ± 20 cm, the model has shown an unbiased angular error, and a mean (absolute) position error of about 1.5 cm, thus confirming the ability of the system to reliably drive the model to the desired targets. Moreover, the curvature factors of the factual human movements and of the reconstructed ones are similar, thus encouraging future developments of the system in terms of reproducibility of the desired movements.ConclusionA novel FES-assisted rehabilitation system for the upper limb is presented and two parts of it have been designed and tested. The system includes a markerless motion estimation algorithm, and a biologically inspired neural controller that drives a biomechanical arm model and provides the stimulation patterns that, in a future development, could be used to drive a smart Functional Electrical Stimulation system (sFES). The system is envisioned to help in the rehabilitation of post stroke hemiparetic patients, by assisting the movement of the paretic upper limb, once trained with a set of movements performed by the therapist or in virtual reality. Future work will include the application and testing of the stimulation patterns in real conditions.