Depth Fusion Research Articles

Two Depth Fusion, from “Manufacture” to “Intellectual Creation” —Based on Qingdao Hong Ling Group Innovative MTM Mode

Promoting the deep integration of informationization and industrialization is an inevitable choice for the transformation and upgrading of traditional industries. Based on information technology and based on smart manufacturing, Hong Ling Group actively develops MTM business targeting high-end consumer groups, achieves deep integration of the two industries, and en-hances the company’s core competitiveness and brand influence. This article combines the current trend of deep integration, using case studies, analyzing the MTM business model of the Hong Ling Group, and exploring its innovations in business models, technologies, production, and operations, and bringing new opportunities to the Chinese textile and apparel industry and other industries. Industrial roads, promote the integration of the two deep integration, refine the theoretical value and reference significance.

Research on Intelligent Analysis and Depth Fusion of Multi-Source Traffic Data

Intelligence transportation system (ITS) and vehicular networks have attracted the research community in the recent years which generate the “big data” in traffic. However, the collection and application of the big traffic data is limited by the privacy of people who generate data. Besides, data-driven-based ITS only needs information that could reflect one or more types of vehicles at specific intersections, sections, and road networks, rather than that of each individual vehicle. Overall, intelligent analysis and data fusion of multi-source traffic data play an important role to reduce the phenomenon of privacy disclosure and ensure the quality of data. As a result, a complete method of multi-source traffic data analyzing and processing is proposed in this paper, including the data analysis method based on the spatio-temporal regression model and the data fusion method using evidence theory based on the confidence tensor. Finally, the practical data is used to conform the ways proposed before. And not only do the results show that the implicit privacy information has been removed but also present a higher accuracy of the proceed data.

Multimodal Trip Hazard Affordance Detection on Construction Sites

Trip hazards are a significant contributor to accidents on construction and manufacturing sites. Current safety inspections are labor intensive and limited by human fallibility, making automation of trip hazard detection appealing from both a safety and economic perspective. Trip hazards present an interesting challenge to modern learning techniques because they are defined as much by affordance as by object type, for example, wires on a table are not a trip hazard, but can be if lying on the ground. To address these challenges, we conduct a comprehensive investigation into the performance characteristics of 11 different colors and depth fusion approaches, including four fusion and one nonfusion approach, using color and two types of depth images. Trained and tested on more than 600 labeled trip hazards over four floors and 2000 m2 in an active construction site, this approach was able to differentiate between identical objects in different physical configurations. Outperforming a color-only detector, our multimodal trip detector fuses color and depth information to achieve a 4% absolute improvement in F1-score. These investigative results and the extensive publicly available dataset move us one step closer to assistive or fully automated safety inspection systems on construction sites.

Efficient and fast multi-modal foreground-background segmentation using RGBD data

This paper addresses the problem of foreground and background segmentation. Multi-modal data specifically RGBD data has gain many tasks in computer vision recently. However, techniques for background subtraction based only on single-modality still report state-of-the-art results in many benchmarks. Succeeding the fusion of depth and color data for this task requires a robust formalization allowing at the same time higher precision and faster processing. To this end, we propose to make use of kernel density estimation technique to adapt multi-modal data. To speed up kernel density estimation, we explore the fast Gauss transform which allows the summation of a mixture of M kernel at N evaluation points in O(M+N) time as opposed to O(MN) time for a direct evaluation. Extensive experiments have been carried out on four publicly available RGBD foreground/background datasets. Results demonstrate that our proposal outperforms state-of-the-art methods for almost all of the sequences acquired in challenging indoor and outdoor contexts with a fast and non-parametric operation.

RGB-D object detection and semantic segmentation for autonomous manipulation in clutter

Autonomous robotic manipulation in clutter is challenging. A large variety of objects must be perceived in complex scenes, where they are partially occluded and embedded among many distractors, often in restricted spaces. To tackle these challenges, we developed a deep-learning approach that combines object detection and semantic segmentation. The manipulation scenes are captured with RGB-D cameras, for which we developed a depth fusion method. Employing pretrained features makes learning from small annotated robotic datasets possible. We evaluate our approach on two challenging datasets: one captured for the Amazon Picking Challenge 2016, where our team NimbRo came in second in the Stowing and third in the Picking task; and one captured in disaster-response scenarios. The experiments show that object detection and semantic segmentation complement each other and can be combined to yield reliable object perception.

Image Defogging by Multiscale Depth Fusion and Hybrid Scattering Model

Image Defogging by Multiscale Depth Fusion and Hybrid Scattering Model

Fast depth extraction from a single image

Predicting depth from a single image is an important problem for understanding the 3-D geometry of a scene. Recently, the nonparametric depth sampling (DepthTransfer) has shown great potential in solving this problem, and its two key components are a Scale Invariant Feature Transform (SIFT) flow–based depth warping between the input image and its retrieved similar images and a pixel-wise depth fusion from all warped depth maps. In addition to the inherent heavy computational load in the SIFT flow computation even under a coarse-to-fine scheme, the fusion reliability is also low due to the low discriminativeness of pixel-wise description nature. This article aims at solving these two problems. First, a novel sparse SIFT flow algorithm is proposed to reduce the complexity from subquadratic to sublinear. Then, a reweighting technique is introduced where the variance of the SIFT flow descriptor is computed at every pixel and used for reweighting the data term in the conditional Markov random fields. Our proposed depth transfer method is tested on the Make3D Range Image Data and NYU Depth Dataset V2. It is shown that, with comparable depth estimation accuracy, our method is 2–3 times faster than the DepthTransfer.

A MEDIAN-BASED DEPTHMAP FUSION STRATEGY FOR THE GENERATION OF ORIENTED POINTS

Abstract. Due to good scalability, systems for image-based dense surface reconstruction often employ stereo or multi-baseline stereo methods. These types of algorithms represent the scene by a set of depth or disparity maps which eventually have to be fused to extract a consistent, non-redundant surface representation. Generally the single depth observations across the maps possess variances in quality. Within the fusion process not only preservation of precision and detail but also density and robustness with respect to outliers are desirable. Being prune to outliers, in this article we propose a local median-based algorithm for the fusion of depth maps eventually representing the scene as a set of oriented points. Paying respect to scalability, points induced by each of the available depth maps are streamed to cubic tiles which then can be filtered in parallel. Arguing that the triangulation uncertainty is larger in the direction of image rays we define these rays as the main filter direction. Within an additional strategy we define the surface normals as the principle direction for median filtering/integration. The presented approach is straight-forward to implement since employing standard oc- and kd-tree structures enhanced by nearest neighbor queries optimized for cylindrical neighborhoods. We show that the presented method in combination with the MVS (Rothermel et al., 2012) produces surfaces comparable to the results of the Middlebury MVS benchmark and favorably compares to an state-of-the-art algorithm employing the Fountain dataset (Strecha et al., 2008). Moreover, we demonstrate its capability of depth map fusion for city scale reconstructions derived from large frame airborne imagery.

A MEDIAN-BASED DEPTHMAP FUSION STRATEGY FOR THE GENERATION OF ORIENTED POINTS

Due to good scalability, systems for image-based dense surface reconstruction often employ stereo or multi-baseline stereo methods. These types of algorithms represent the scene by a set of depth or disparity maps which eventually have to be fused to extract a consistent, non-redundant surface representation. Generally the single depth observations across the maps possess variances in quality. Within the fusion process not only preservation of precision and detail but also density and robustness with respect to outliers are desirable. Being prune to outliers, in this article we propose a local median-based algorithm for the fusion of depth maps eventually representing the scene as a set of oriented points. Paying respect to scalability, points induced by each of the available depth maps are streamed to cubic tiles which then can be filtered in parallel. Arguing that the triangulation uncertainty is larger in the direction of image rays we define these rays as the main filter direction. Within an additional strategy we define the surface normals as the principle direction for median filtering/integration. The presented approach is straight-forward to implement since employing standard oc- and kd-tree structures enhanced by nearest neighbor queries optimized for cylindrical neighborhoods. We show that the presented method in combination with the MVS (Rothermel et al., 2012) produces surfaces comparable to the results of the Middlebury MVS benchmark and favorably compares to an state-of-the-art algorithm employing the Fountain dataset (Strecha et al., 2008). Moreover, we demonstrate its capability of depth map fusion for city scale reconstructions derived from large frame airborne imagery.

Hand Gesture Recognition for Thai Sign Language in Complex Background Using Fusion of Depth and Color Video

Hand detection and gesture recognition are the active research area in the computer vision. The main purpose to develop the sign language recognition and Human Computer Interaction (HCI). This article investigates and develops the technique to recognize hand posture of Thai sign language in a complex background using fusion of depth and color video. The new technology of sensors, such as the Microsoft Kinect, recently provides the depth video which helps researchers to find the hand position in the scene. This advantage is used to segment the hand sign in the color video without the environment interference such as skin color background. The histograms of oriented gradients are used to extract the image features of hand sign. These features are then pass to the artificial neural network for training and recognition. The result showed that the proposed method is robust to detect the hand gestures in the complex background. It provides the accuracy recognition for the Thai fingerspelling of 84.05%

A Simple Outdoor Environment Obstacle Detection Method Based on Information Fusion of Depth and Infrared

In allusion to the existing low recognition rate and robustness problem in obstacle detection; a simple but effective obstacle detection algorithm of information fusion in the depth and infrared is put forward. The scenario is segmented by the mean-shift algorithm and the pixel gradient of foreground is calculated. After pretreatment of edge detection and morphological operation, the depth information and infrared information are fused. The characteristics of depth map and infrared image in edge detection are used for the raised method, the false rate of detection is reduced, and detection precision is improved. Since the depth map and infrared image are not affected by natural sunlight, the influence on obstacle recognition due to the factors such as light intensity and shadow is effectively reduced and the robustness of the algorithm is also improved. Experiments indicate that the detection algorithm of information fusion can accurately identify the small obstacle in the view and the accuracy of obstacle recognition will not be affected by light. Hence, this method has great significance for mobile robot or intelligent vehicles on obstacle detection in outdoor environment.

Vision-based Adaptive Hand Tracking by Fusion of Depth and Skin Color

Vision-based Adaptive Hand Tracking by Fusion of Depth and Skin Color

STEM, STEAM 교육과 우리나라 융합인재교육의 이해와 해결 과제

Since 2011, after beginning of the systematic study on STEAM education, South Korea has developed a number of related programs. At this point we see that this is the time to clarify the challenges. The purpose of this study is to clarify the characteristics of their education through the review of many domestic and foreign papers in order to propose the challenges of STEAM education of Korea. The results are as follows. First, the course of integration cannot be separated by the difference in superiority but should be separated in accordance with the purpose of integration. Second, curriculum integration such as STEAM education is characterized by the emphasis on horizontal linking than vertical depth fusion. Accordingly, the content knowledge and vertical linkages are inevitably weakened. In order to overcome this problem, the key concepts and features that were emphasized in the previous curriculum need to also be emphasized in STEAM training, and the comparative study on the core concept and function of each subjects should be preceded. Third, after looking upon the current situation of our country's fusion research and talent training, the limits and the challenges that need to be overcome has been suggested. Fourth, with the basis on research results, we offered an example of the approach on STEAM education which is applicable to the current situation and proposed the challenges and implications that need to be addressed in the STEAM education of Korea in educational contexts such as curriculum, teaching and learning, and evaluation.

A Unified Probabilistic Framework for Real-Time Depth Map Fusion

This paper proposes a unified probabilistic framework for real-time depth map fusion. By modeling the depth imaging process as a random experiment, the depth map fusion is converted into probability density function (pdf) estimation. The depth fusion problem is decoupled into four parts: the fusion space, the influence term, the visibility term and the confidence term. We combine these four terms in a unified probabilistic framework, and apply the framework in two cases to evaluate the performance. In the first case, multiple stereo vision cameras are used to acquire multiple depth map streams from multiple viewpoints simultaneously in real time. In the second case, two cameras and a Kinect are combined to provide two depth map streams. In both cases, strategies for each part of the framework are presented to perform real-time depth fusion. Experimental results show that the proposed framework is promising for real-time depth map fusion.

Accurate 3D Human Pose Recognition via Fusion of Depth and Motion Sensors

Accurate 3D Human Pose Recognition via Fusion of Depth and Motion Sensors

Real-time large-scale dense RGB-D SLAM with volumetric fusion

We present a new simultaneous localization and mapping (SLAM) system capable of producing high-quality globally consistent surface reconstructions over hundreds of meters in real time with only a low-cost commodity RGB-D sensor. By using a fused volumetric surface reconstruction we achieve a much higher quality map over what would be achieved using raw RGB-D point clouds. In this paper we highlight three key techniques associated with applying a volumetric fusion-based mapping system to the SLAM problem in real time. First, the use of a GPU-based 3D cyclical buffer trick to efficiently extend dense every-frame volumetric fusion of depth maps to function over an unbounded spatial region. Second, overcoming camera pose estimation limitations in a wide variety of environments by combining both dense geometric and photometric camera pose constraints. Third, efficiently updating the dense map according to place recognition and subsequent loop closure constraints by the use of an ‘as-rigid-as-possible’ space deformation. We present results on a wide variety of aspects of the system and show through evaluation on de facto standard RGB-D benchmarks that our system performs strongly in terms of trajectory estimation, map quality and computational performance in comparison to other state-of-the-art systems.

Development of gesture‐based human–computer interaction applications by fusion of depth and colour video streams

Hand detection and gesture recognition are two of the most studied topics in human–computer interaction (HCI). The increasing availability of sensors able to provide real-time depth measurements, such as time-of-flight cameras or the more recent Kinect, has helped researchers to find more and more efficient solutions for these issues. With the main aim to implement effective gesture-based interaction systems, this study presents an approach to hand detection and tracking that exploits two different video streams: the depth one and the colour one. Both hand and gesture recognition are based only on geometrical and colour constraints, and no learning phase is needed. The use of a Kalman filter to track hands guarantees system robustness also in presence of many persons in the scene. The entire procedure is designed to maintain a low computational cost and is optimised to efficiently execute HCI tasks. As use cases two common applications are described: a virtual keyboard and a three-dimensional object manipulation virtual environment. These applications have been tested with a representative sample of non-trained users to assess the usability and flexibility of the system.

Fully automatic 3D facial expression recognition using polytypic multi-block local binary patterns

3D facial expression recognition has been greatly promoted for overcoming the inherent drawbacks of 2D facial expression recognition and has achieved superior recognition accuracy to the 2D. In this paper, a novel holistic, full-automatic approach for 3D facial expression recognition is proposed. First, 3D face models are represented in 2D-image-like structure which makes it possible to take advantage of the wealth of 2D methods to analyze 3D models. Then an enhanced facial representation, namely polytypic multi-block local binary patterns (P-MLBP), is proposed. The P-MLBP involves both the feature-based irregular divisions to depict the facial expressions accurately and the fusion of depth and texture information of 3D models to enhance the facial feature. Based on the BU-3DFE database, three kinds of classifiers are employed to conduct 3D facial expression recognition for evaluation. Their experimental results outperform the state of the art and show the effectiveness of P-MLBP for 3D facial expression recognition. Therefore, the proposed strategy is validated for 3D facial expression recognition; and its simplicity opens a promising direction for fully automatic 3D facial expression recognition.

Single image defogging by multiscale depth fusion.

Restoration of fog images is important for the deweathering issue in computer vision. The problem is ill-posed and can be regularized within a Bayesian context using a probabilistic fusion model. This paper presents a multiscale depth fusion (MDF) method for defog from a single image. A linear model representing the stochastic residual of nonlinear filtering is first proposed. Multiscale filtering results are probabilistically blended into a fused depth map based on the model. The fusion is formulated as an energy minimization problem that incorporates spatial Markov dependence. An inhomogeneous Laplacian-Markov random field for the multiscale fusion regularized with smoothing and edge-preserving constraints is developed. A nonconvex potential, adaptive truncated Laplacian, is devised to account for spatially variant characteristics such as edge and depth discontinuity. Defog is solved by an alternate optimization algorithm searching for solutions of depth map by minimizing the nonconvex potential in the random field. The MDF method is experimentally verified by real-world fog images including cluttered-depth scene that is challenging for defogging at finer details. The fog-free images are restored with improving contrast and vivid colors but without over-saturation. Quantitative assessment of image quality is applied to compare various defog methods. Experimental results demonstrate that the accurate estimation of depth map by the proposed edge-preserved multiscale fusion should recover high-quality images with sharp details.

The Development of 3D Print Technology

Three-dimensional printing technology as a new thing, the technology is a high and new technology of manufacturing of the industrial revolution significance, represents the new trend of world manufacturing industry development, for accelerating the development of advanced manufacturing industry, promote the two depth fusion, promote transformation and upgrading of industries have the important leading role.