3D Depth Map Research Articles

An efficient depth map filtering based on spatial and texture features for 3D video coding

A depth map is used to synthesize virtual views in the 3D video depth-enhanced format. In contrast to texture video, depth map is characterized by piecewise smooth regions bounded by sharp object boundaries. Conventional video coding standards compress depth map often introduces coding artifacts along the depth map boundary, which severely affects the rendered view quality. To address this problem, we propose an efficient depth map filtering for 3D depth map compression in the high efficiency video coding (HEVC) process. The proposed depth map filtering is designed considering spatial resolution, texture boundary similarity, and coding artifacts features. It consists of a newly designed nonlinear down/up sampling filtering and a depth reconstruction multilateral filtering. Firstly, the depth map is down-sampled and coded with 3D video HEVC. Then in the decoding side, the depth map is up-sampled. Finally, a depth reconstruction multilateral filtering is used to align the object boundaries for the coding artifacts in edges of the decoded depth image. Experimental results demonstrate that the proposed depth map filtering can significantly reduce the bit rate as well as achieving a better quality of the rendered view in comparison with 3D video HEVC test model (3DV-HTM).

3D shape and texture morphing using 2D projection and reconstruction

We present an algorithm for morphing shape and view-dependent texture that utilizes an unstructured lumigraph representation to (1) gracefully handle topological changes, (2) create plausible appearance transformations, and (3) provide user control independent from the underlying mesh structure. It works by reducing the otherwise complex problem of a 3D shape and material morph into multiple simpler 2D morphs. The shape morph operates on 2D depth maps and then reconstructs a conventional mesh, making it insensitive to the input geometry quality. Our morphing algorithm robustly handles topology changes and outputs complete meshes and lumigraphs for later use. We show how 2D image morphs can be computed from 3D correspondences, while eliminating ambiguities that might result in the projected 2D morphs.

Surface roughness inspection in milling operations with photometric stereo and PNN

Offline human-assisted surface quality inspection techniques are time-consuming and need labor intensive setup requirements that are sometimes unacceptable due to their slowing down of manufacturing productivity. This paper proposes an efficient surface roughness inspection method which uses photometric stereo and probabilistic neural network (PNN). A photometric stereo scheme with six images was used to generate a 3D depth map of the machined surface. Probabilistic neural network and statistical features extracted from the depth map were employed for the classification of surface roughness quality. The method was validated on several machined surfaces with different surface roughnesses. The satisfying accuracy of the introduced method demonstrates the feasibility of the proposed system.

Vision-Based Hand Gesture Spotting and Recognition Using CRF and SVM

In this paper, a novel gesture spotting and recognition technique is proposed to handle hand gesture from continuous hand motion based on Conditional Random Fields in conjunction with Support Vector Machine. Firstly, YCbCr color space and 3D depth map are used to detect and segment the hand. The depth map is to neutralize complex background sense. Secondly, 3D spatio-temporal features for hand volume of dynamic affine-invariants like elliptic Fourier and Zernike moments are extracted, in addition to three orientations motion features. Finally, the hand gesture is spotted and recognized by using the discriminative Conditional Random Fields Model. Accordingly, a Support Vector Machine verifies the hand shape at the start and the end point of meaningful gesture, which enforces vigorous view invariant task. Experiments demonstrate that the proposed method can successfully spot and recognize hand gesture from continuous hand motion data with 92.50% recognition rate.

An enhanced method for registration of dental surfaces partially scanned by a 3D dental laser scanning

In this paper, we propose the fast and accurate registration method of partially scanned dental surfaces in a 3D dental laser scanning. To overcome the multiple point correspondence problems of conventional surface registration methods, we propose the novel depth map-based registration method to register 3D surface models. First, we convert a partially scanned 3D dental surface into a 2D image by generating the 2D depth map image of the surface model by applying a 3D rigid transformation into this model. Then, the image-based registration method using 2D depth map images accurately estimates the initial transformation between two consequently acquired surface models. To further increase the computational efficiency, we decompose the 3D rigid transformation into out-of-plane (i.e. x-, y-rotation, and z-translation) and in-plane (i.e. x-, y-translation, and z-rotation) transformations. For the in-plane transformation, we accelerate the transformation process by transforming the 2D depth map image instead of transforming the 3D surface model. For the more accurate registration of 3D surface models, we enhance iterative closest point (ICP) method for the subsequent fine registration. Our initial depth map-based registration well aligns each surface model. Therefore, our subsequent ICP method can accurately register two surface models since it is highly probable that the closest point pairs are the exact corresponding point pairs. The experimental results demonstrated that our method accurately registered partially scanned dental surfaces. Regarding the computational performance, our method delivered about 1.5 times faster registration than the conventional method. Our method can be successfully applied to the accurate reconstruction of 3D dental objects for orthodontic and prosthodontic treatment.

A multi-modal face recognition method using complete local derivative patterns and depth maps.

In this paper, we propose a multi-modal 2D + 3D face recognition method for a smart city application based on a Wireless Sensor Network (WSN) and various kinds of sensors. Depth maps are exploited for the 3D face representation. As for feature extraction, we propose a new feature called Complete Local Derivative Pattern (CLDP). It adopts the idea of layering and has four layers. In the whole system, we apply CLDP separately on Gabor features extracted from a 2D image and depth map. Then, we obtain two features: CLDP-Gabor and CLDP-Depth. The two features weighted by the corresponding coefficients are combined together in the decision level to compute the total classification distance. At last, the probe face is assigned the identity with the smallest classification distance. Extensive experiments are conducted on three different databases. The results demonstrate the robustness and superiority of the new approach. The experimental results also prove that the proposed multi-modal 2D + 3D method is superior to other multi-modal ones and CLDP performs better than other Local Binary Pattern (LBP) based features.

Efficient multiview depth coding optimization based on allowable depth distortion in view synthesis.

Depth video is used as the geometrical information of 3D world scenes in 3D view synthesis. Due to the mismatch between the number of depth levels and disparity levels in the view synthesis, the relationship between depth distortion and rendering position error can be modeled as a many-to-one mapping function, in which different depth distortion values might be projected to the same geometrical distortion in the synthesized virtual view image. Based on this property, we present an allowable depth distortion (ADD) model for 3D depth map coding. Then, an ADD-based rate-distortion model is proposed for mode decision and motion/disparity estimation modules aiming at minimizing view synthesis distortion at a given bit rate constraint. In addition, an ADD-based depth bit reduction algorithm is proposed to further reduce the depth bit rate while maintaining the qualities of the synthesized images. Experimental results in intra depth coding show that the proposed overall algorithm achieves Bjontegaard delta peak signal-to-noise ratio gains of 1.58 and 2.68 dB on average for half and integer-pixel rendering precisions, respectively. In addition, the proposed algorithms are also highly efficient for inter depth coding when evaluated with different metrics.

G.O.16: Reliability and validity of the ACTIVE-mini (Ability Captured Through Interactive Video Evaluation-mini) to quantify infant movement

The success of clinical trials in infants with spinal muscular atrophy (SMA) requires careful trial planning and appropriate outcomes. Traditional tests are fatiguing for SMA infants, use non-linear ordinal scales and require highly trained evaluators to grade movement. Our study provides preliminary reliability and validity data on a video-based assessment, ACTIVE-mini, which uses the Microsoft Kinect to assess infant movement. The infant lies supine and spontaneous movements are recorded using a series of 3D depth map and color images. Algorithms are used to quantify infant movement speed (i.e. velocity, acceleration, and jerk), as well as total movement volume, and the complexity of infant movement within space. Weekly longitudinal and cross-sectional data were collected over 6 months on 6 SMA infants and 30 controls. Movement speed data are plotted as kernel density plots separated in 2-week age-matched bins. Differences are seen in each movement variable in every age bin. On average the SMA infants have an average jerk value of 57.8% of age-matched peers. Tracings of the extremities and the ratio of convex hull’s surface area to volume ratio, where higher numbers indicate decreased ability, illustrate that an SMA infant has a smaller total movement excursion as well as decreased spatial complexity (upper extremity average: 139–170; lower extremity average: 756–891) when compared to age-matched controls (all extremities average range: 43–68). Preliminary results indicate ACTIVE-mini differentiates movement differences related to speed, variability, quantity, and complexity of movement in SMA infants compared to controls in the first 2 weeks of life. ACTIVE-mini has both clinical and research potential to detect movement abnormalities and measure small changes in ability over time. Further research is needed in a larger sample of diverse diagnoses to refine the statistical models of normal movement and identify typical longitudinal changes in infants.

Late Palaeozoic to Early Cenozoic geological evolution of the northwestern German North Sea (Entenschnabel): New results and insights

AbstractThe results of a detailed seismic mapping campaign of 13 horizons in the northwestern German North Sea, covering Late Permian to Palaeogene sedimentary successions, are presented. Based on the interpretation of four 3D and two 2D seismic surveys, thickness and depth maps of prominent stratigraphic units were constructed. These maps provide an overview of key structural elements, the sedimentation and erosion, and give insights into the evolution of the German Central Graben. The base of the Zechstein Group reaches a maximum depth of 7800 m within the German Central Graben. Lateral thickness variations in the Zechstein reflect the extensive mobilisation of Zechstein salt. Complex rift-related structures, with the Central Graben as the main structural element, were found not later than the Early Triassic. Up to 3000-m thick Triassic sediments are preserved in the eastern German Central Graben of which 1800 m consist of Keuper sediments. The Lower Buntsandstein unit shows increasing thicknesses towards the southeastern study area, likely related to distinct lateral subsidence. As a consequence of uplift of the North Sea Dome, Middle Jurassic sediments were eroded in large parts of the northwestern German North Sea and are only preserved in the German Central Graben. The NNW–SSE oriented John Basin is another important structural element, which shows maximum subsidence during the Late Jurassic. In most parts of the study area Lower Cretaceous sediments are absent due to either erosion or non-deposition. Lower Cretaceous deposits are preserved in the Outer Rough Basin in the northwest and within the German Central Graben. Upper Cretaceous sediments are found at depths between 1500 and 3600 m, reaching a maximum thickness of approximately 1600 m on the Schillgrund High. Contraction and inversion of pre-existing Mesozoic faults during the Late Cretaceous is distinct at the Schillgrund Fault, i.e. the eastern border fault of the Central Graben. The Palaeogene is predominantly a period of strong basin subsidence. Within 37 Myrs, up to 1400 m of Palaeogene sediments were deposited in the northwesternmost part of the study area. Detailed mapping of salt structures enables a reconstruction of halokinetic movements over time and a deciphering of the influence of the Zechstein salt on the sedimentary evolution during the Mesozoic and Cenozoic. Increasing sediment thicknesses in rim-synclines indicate that most of the salt structures in the German Central Graben had their main growth phase during the Late Jurassic.

Rapid Implementation of 3D Facial Reconstruction from a Single Image on an Android Mobile Device

In this paper, we propose the rapid implementation of a 3-dimensional (3D) facial reconstruction from a single frontal face image and introduce a design for its application on a mobile device. The proposed system can effectively reconstruct human faces in 3D using an approach robust to lighting conditions, and a fast method based on a Canonical Correlation Analysis (CCA) algorithm to estimate the depth. The reconstruction system is built by first creating 3D facial mapping from a personal identity vector of a face image. This mapping is then applied to real-world images captured with a built-in camera on a mobile device to form the corresponding 3D depth information. Finally, the facial texture from the face image is extracted and added to the reconstruction results. Experiments with an Android phone show that the implementation of this system as an Android application performs well. The advantage of the proposed method is an easy 3D reconstruction of almost all facial images captured in the real world with a fast computation. This has been clearly demonstrated in the Android application, which requires only a short time to reconstruct the 3D depth map.

Evaluating spatiotemporal interest point features for depth-based action recognition

Human action recognition has lots of real-world applications, such as natural user interface, virtual reality, intelligent surveillance, and gaming. However, it is still a very challenging problem. In action recognition using the visible light videos, the spatiotemporal interest point (STIP) based features are widely used with good performance. Recently, with the advance of depth imaging technology, a new modality has appeared for human action recognition. It is important to assess the performance and usefulness of the STIP features for action analysis on the new modality of 3D depth map. In this paper, we evaluate the spatiotemporal interest point (STIP) based features for depth-based action recognition. Different interest point detectors and descriptors are combined to form various STIP features. The bag-of-words representation and the SVM classifiers are used for action learning. Our comprehensive evaluation is conducted on four challenging 3D depth databases. Further, we use two schemes to refine the STIP features, one is to detect the interest points in RGB videos and apply to the aligned depth sequences, and the other is to use the human skeleton to remove irrelevant interest points. These refinements can help us have a deeper understanding of the STIP features on 3D depth data. Finally, we investigate a fusion of the best STIP features with the prevalent skeleton features, to present a complementary use of the STIP features for action recognition on 3D data. The fusion approach gives significantly higher accuracies than many state-of-the-art results.

Rate-distortion based reference viewpoints selection for multi-view video plus depth coding

For multi-view video plus depth (MVD) coding in 3D video (3DV) applications, texture videos and depth maps of a few reference viewpoints should be compressed and transmitted. The other intermediate virtual viewpoints are synthesized by depth-image-based rendering algorithm. The videos displayed for terminal users include the decoded reference viewpoints and the synthesized virtual viewpoints. Under a given bit rate constraint, how to choose the optimal reference viewpoints setting to obtain the best video quality of all display viewpoints is an important research issue. This paper proposes a rate distortion model based algorithm to obtain the optimal reference viewpoints selection. First, the impacts of the reference viewpoints setting on the entire quality of all display viewpoints is investigated, and then a quadratic distortion model for all display viewpoints is derived. Based on this distortion model, the optimal viewpoints selection problem is solved by computing a few model parameters. Experimental results demonstrate that the proposed distortion model estimates the actual distortion of all display viewpoints with high accuracy, and the proposed reference viewpoints selection method achieves almost the same rate distortion performance as compared to the full search method with a very low computational cost.

Collision-Free Multi-Robot Navigation Using Autonomous Object Selection

For successful SLAM, landmarks for pose estimation should be continuously observed. Firstly, we proposed the object selection algorithm from 2D images and 3D depth maps without human’s supervision. We used the SIFT algorithm to obtain descriptors of point features inside the object, and the surface segmentation algorithm to obtain separated objects from point clouds of 3D depth maps. Automatically selected objects were tested by the threshold function using repeatability, distinctiveness and saliency whether the objects were suitable or not for reusing and sharing between the robots. Secondly, we suggested the closed-form solution to estimate the 3D pose of robots from the information of selected objects. Furthermore, we provided the effective way to accomplish the tasks using multi-robot by compensating the accumulated navigating errors and re-planning the collision-free motion of the robots using the extended collision map algorithm.

3D facial depth map recognition based on manifold learning-LOGMAP algorithm

3D facial depth map recognition based on manifold learning-LOGMAP algorithm

3D laser ablation-ICP-mass spectrometry mapping for the study of surface layer phenomena – a case study for weathered glass

In this work, a multi-elemental 3D laser ablation-ICP-mass spectrometry mapping procedure for high-resolution depth information retrieval to investigate surface layer phenomena is presented. The procedure is based on laser drilling on a virtual grid on the surface, followed by extraction of depth maps along the z-axis (for each element monitored). Using a burst of 50 laser pulses at 1 Hz on each point of the grid, a penetration rate of ca. 150 nm per pulse (in glass) was obtained and a lateral resolution in the order of the laser beam diameter. By ultrafast ICP-MS monitoring of individual ablation pulses (58 ms for a set of 19 elements) using a laser ablation cell with fast signal washout (less than 0.5 s for whole laser pulse), the corresponding peak areas could be consistently integrated, resulting in spatial elemental data associated with individual pulses. The usual laser drilling limitations such as pulse mixing and signal tailing are avoided with this approach. After manipulation of the spatial elemental datasets and quantification, stacks of 50 2D depth maps (for each element monitored) were produced which could be visualized as volume images or time-lapse movies. As a proof of concept, this approach was successfully used to investigate the degradation mechanisms of a medieval, weathered glass artifact by colocalization analysis of selected cross-sectional 2D elemental images in arbitrary planes of the volume images. It was shown that degradation must have started as a result of dealkalinization leading to depletion of alkalis/earth alkalis in glass surface layers and enrichment of network formers (Si and Al), and subsequent worsening by cracking and formation of corrosion pits and so-called spatiotemporal Liesegang rings indicative of radial leaching.

휴머노이드 로봇의 움직임 생성을 위한 장애물 인식방법

This paper proposes a method to extract accurate plane of an object in unstructured environment for a humanoid robot by using a laser scanner. By panning and tilting 2D laser scanner installed on the head of a humanoid robot, 3D depth map of unstructured environment is generated. After generating the 3D depth map around a robot, the proposed plane extraction method is applied to the 3D depth map. By using the hierarchical clustering method, points on the same plane are extracted from the point cloud in the 3D depth map. After segmenting the plane from the point cloud, dimensions of the planes are calculated. The accuracy of the extracted plane is evaluated with experimental results, which show the effectiveness of the proposed method to extract planes around a humanoid robot in unstructured environment.

Gender and Ethnicity Specific Generic Elastic Models from a Single 2D Image for Novel 2D Pose Face Synthesis and Recognition

In this paper, we propose a novel method for generating a realistic 3D human face from a single 2D face image for the purpose of synthesizing new 2D face images at arbitrary poses using gender and ethnicity specific models. We employ the Generic Elastic Model (GEM) approach, which elastically deforms a generic 3D depth-map based on the sparse observations of an input face image in order to estimate the depth of the face image. Particularly, we show that Gender and Ethnicity specific GEMs (GE-GEMs) can approximate the 3D shape of the input face image more accurately, achieving a better generalization of 3D face modeling and reconstruction compared to the original GEM approach. We qualitatively validate our method using publicly available databases by showing each reconstructed 3D shape generated from a single image and new synthesized poses of the same person at arbitrary angles. For quantitative comparisons, we compare our synthesized results against 3D scanned data and also perform face recognition using synthesized images generated from a single enrollment frontal image. We obtain promising results for handling pose and expression changes based on the proposed method.

A Verification-Aware Design Methodology for Thread Pipelining Parallelization

SUMMARY This paper proposes a verification-aware design methodology that provides developers with a systematic and reliable approach to performing thread-pipelining parallelization on sequential programs. In contrast to traditional design flow, a behavior-model program is constructed before parallelizing as a bridge to help developers gradually leverage the technique of thread-pipelining parallelization. The proposed methodology integrates verification mechanisms into the design flow. To demonstrate the practicality of the proposed methodology, we applied it to the parallelization of a 3D depth map generator with thread pipelining. The parallel 3D depth map generator was further integrated into a 3D video playing system for evaluation of the verification overheads of the proposed methodology and the system performance. The results show the parallel system can achieve 33.72fps in D1 resolution and 12.22fps in HD720 resolution through a five-stage pipeline. When verifying the parallel program, the proposed verification approach keeps the performance degradation within 23% and 21.1% in D1 and HD720 resolutions, respectively.

Denoising algorithm for the 3D depth map sequences based on multihypothesis motion estimation

This article proposes an efficient wavelet-based depth video denoising approach based on a multihypothesis motion estimation aimed specifically at time-of-flight depth cameras. We first propose a novel bidirectional block matching search strategy, which uses information from the luminance as well as from the depth video sequence. Next, we present a new denoising technique based on weighted averaging and wavelet thresholding. Here we take into account the reliability of the estimated motion and the spatial variability of the noise standard deviation in both imaging modalities. The results demonstrate significantly improved performance over recently proposed depth sequence denoising methods and over state-of-the-art general video denoising methods applied to depth video sequences.

Face recognition with 3D depth map in varied poses

Face recognition with 3D depth map in varied poses