Commercial Depth Sensors Research Articles

Monocular Depth Estimation for Glass Walls with Context: A New Dataset and Method.

Traditional monocular depth estimation assumes that all objects are reliably visible in the RGB color domain. However, this is not always the case as more and more buildings are decorated with transparent glass walls. This problem has not been explored due to the difficulties in annotating the depth levels of glass walls, as commercial depth sensors cannot provide correct feedbacks on transparent objects. Furthermore, estimating depths from transparent glass walls requires the aids of surrounding context, which has not been considered in prior works. To cope with this problem, we introduce the first Glass Walls Depth Dataset (GW-Depth dataset). We annotate the depth levels of transparent glass walls by propagating the context depth values within neighboring flat areas, and the glass segmentation mask and instance level line segments of glass edges are also provided. On the other hand, a tailored monocular depth estimation method is proposed to fully activate the glass wall contextual understanding. First, we propose to exploit the glass structure context by incorporating the structural prior knowledge embedded in glass boundary line segment detections. Furthermore, to make our method adaptive to scenes without structure context where the glass boundary is either absent in the image or too narrow to be recognized, we propose to derive a reflection context by utilizing the depth reliable points sampled according to the variance between two depth estimations from different resolutions. High-resolution depth is thus estimated by the weighted summation of depths by those reliable points. Extensive experiments are conducted to evaluate the effectiveness of the proposed dual context design. Superior performances of our method is also demonstrated by comparing with state-of-the-art methods. We present the first feasible solution for monocular depth estimation in the presence of glass walls, which can be widely adopted in autonomous navigation.

ActiveZero++: Mixed Domain Learning Stereo and Confidence-based Depth Completion with Zero Annotation.

Learning-based stereo methods usually require a large scale dataset with depth, however obtaining accurate depth in the real domain is difficult, but groundtruth depth is readily available in the simulation domain. In this paper we propose a new framework, ActiveZero++, which is a mixed domain learning solution for active stereovision systems that requires no real world depth annotation. In the simulation domain, we use a combination of supervised disparity loss and self-supervised loss on a shape primitives dataset. By contrast, in the real domain, we only use self-supervised loss on a dataset that is out-of-distribution from either training simulation data or test real data. To improve the robustness and accuracy of our reprojection loss in hard-to-perceive regions, our method introduces a novel self-supervised loss called temporal IR reprojection. Further, we propose the confidence-based depth completion module, which uses the confidence from the stereo network to identify and improve erroneous areas in depth prediction through depth-normal consistency. Extensive qualitative and quantitative evaluations on real-world data demonstrate state-of-the-art results that can even outperform a commercial depth sensor. Furthermore, our method can significantly narrow the Sim2Real domain gap of depth maps for state-of-the-art learning based 6D pose estimation algorithms.

SensatUrban: Learning Semantics from Urban-Scale Photogrammetric Point Clouds

With the recent availability and affordability of commercial depth sensors and 3D scanners, an increasing number of 3D (i.e., RGBD, point cloud) datasets have been publicized to facilitate research in 3D computer vision. However, existing datasets either cover relatively small areas or have limited semantic annotations. Fine-grained understanding of urban-scale 3D scenes is still in its infancy. In this paper, we introduce SensatUrban, an urban-scale UAV photogrammetry point cloud dataset consisting of nearly three billion points collected from three UK cities, covering 7.6 km\(^2\). Each point in the dataset has been labelled with fine-grained semantic annotations, resulting in a dataset that is three times the size of the previous existing largest photogrammetric point cloud dataset. In addition to the more commonly encountered categories such as road and vegetation, urban-level categories including rail, bridge, and river are also included in our dataset. Based on this dataset, we further build a benchmark to evaluate the performance of state-of-the-art segmentation algorithms. In particular, we provide a comprehensive analysis and identify several key challenges limiting urban-scale point cloud understanding. The dataset is available at http://point-cloud-analysis.cs.ox.ac.uk/.

Automatic Super-Surface Removal in Complex 3D Indoor Environments Using Iterative Region-Based RANSAC.

Removing bounding surfaces such as walls, windows, curtains, and floor (i.e., super-surfaces) from a point cloud is a common task in a wide variety of computer vision applications (e.g., object recognition and human tracking). Popular plane segmentation methods such as Random Sample Consensus (RANSAC), are widely used to segment and remove surfaces from a point cloud. However, these estimators easily result in the incorrect association of foreground points to background bounding surfaces because of the stochasticity of randomly sampling, and the limited scene-specific knowledge used by these approaches. Additionally, identical approaches are generally used to detect bounding surfaces and surfaces that belong to foreground objects. Detecting and removing bounding surfaces in challenging (i.e., cluttered and dynamic) real-world scene can easily result in the erroneous removal of points belonging to desired foreground objects such as human bodies. To address these challenges, we introduce a novel super-surface removal technique for 3D complex indoor environments. Our method was developed to work with unorganized data captured from commercial depth sensors and supports varied sensor perspectives. We begin with preprocessing steps and dividing the input point cloud into four overlapped local regions. Then, we apply an iterative surface removal approach to all four regions to segment and remove the bounding surfaces. We evaluate the performance of our proposed method in terms of four conventional metrics: specificity, precision, recall, and F1 score, on three generated datasets representing different indoor environments. Our experimental results demonstrate that our proposed method is a robust super-surface removal and size reduction approach for complex 3D indoor environments while scoring the four evaluation metrics between 90% and 99%.

Intelligent autonomous vehicles in digital supply chains

PurposeThe purpose of this paper is twofold: first, to discuss key challenges associated with the use of either simulation or real-world application of intelligent autonomous vehicles (IAVs) in supply network operations; and second, to provide a theoretical and empirical evidence-based methodological framework that supports the integrated application of conceptualisation, simulation, emulation and physical application of IAVs for the effective design of digital supply networks.Design/methodology/approachFirst, this study performs a critical review of the extant literature to identify major benefits and shortcomings related to the use of either simulation modelling or real-word application of physical IAVs. Second, commercial and bespoke software applications, along with a three-dimensional validation and verification emulation tool, are developed to evaluate an IAV’s operations in a conceptual warehouse. Third, a commercial depth-sensor is used as a test bed in a physical setting.FindingsThe results demonstrate that conceptual and simulation modelling should be initially used to explore alternative supply chain operations in terms of ideal performance while emulation tools and real-world IAV test beds are eminent in validating preferred digital supply chain design options.Research limitations/implicationsThe provided analysis framework was developed using literature evidence along with experimental work and research experience, without consulting any industry experts. In addition, this study was developed based on the application of a single physical device application as a test bed and, thus, the authors should further progress with the testing of a physical IAV in an industrial warehouse.Practical implicationsThe study provides bespoke simulation modelling and emulation tools that can be useful for supply chain practitioners in effectively designing network operations.Originality/valueThis work contributes in the operations management field by providing both a multi-stage methodological framework and a practical “toolbox” for the proactive assessment and incorporation of IAVs in supply network operations.

A dynamic depth estimation method for towed optical fiber hydrophone array.

A method for estimating the dynamic depth variation along a maneuverable towed array using optical fiber hydrophones is proposed. The method utilizes the low-frequency components of the hydrophone signal which are typically high-pass filtered to suppress the adverse impact of hydrostatic pressure. Experimental depth data using a 32-element towed array in a lake agree well with measurements from commercial depth sensors. The temporal evolutions of the array shape in the vertical plane at towing speeds of 4 and 8 knots are also obtained.

Survey on 3D Hand Gesture Recognition

Three-dimensional hand gesture recognition has attracted increasing research interests in computer vision, pattern recognition, and human-computer interaction. The emerging depth sensors greatly inspired various hand gesture recognition approaches and applications, which were severely limited in the 2D domain with conventional cameras. This paper presents a survey of some recent works on hand gesture recognition using 3D depth sensors. We first review the commercial depth sensors and public data sets that are widely used in this field. Then, we review the state-of-the-art research for 3D hand gesture recognition in four aspects: 1) 3D hand modeling; 2) static hand gesture recognition; 3) hand trajectory gesture recognition; and 4) continuous hand gesture recognition. While the emphasis is on 3D hand gesture recognition approaches, the related applications and typical systems are also briefly summarized for practitioners.

Hollywood 3D: What are the Best 3D Features for Action Recognition?

Action recognition “in the wild” is extremely challenging, particularly when complex 3D actions are projected down to the image plane, losing a great deal of information. The recent growth of 3D data in broadcast content and commercial depth sensors, makes it possible to overcome this. However, there is little work examining the best way to exploit this new modality. In this paper we introduce the Hollywood 3D benchmark, which is the first dataset containing “in the wild” action footage including 3D data. This dataset consists of 650 stereo video clips across 14 action classes, taken from Hollywood movies. We provide stereo calibrations and depth reconstructions for each clip. We also provide an action recognition pipeline, and propose a number of specialised depth-aware techniques including five interest point detectors and three feature descriptors. Extensive tests allow evaluation of different appearance and depth encoding schemes. Our novel techniques exploiting this depth allow us to reach performance levels more than triple those of the best baseline algorithm using only appearance information. The benchmark data, code and calibrations are all made available to the community.

K-motion: visualizing election information for live television broadcasts

We present an interactive information visualization system for live television broadcasts of the ballot opening process. Our proposed system satisfies the general requirements for ballot opening processes in modern democratic political systems, in which a great deal of complex information floods in, but the time frame for gathering and processing information for live broadcasting is very limited. Traditional virtual studio techniques cannot properly handle such urgent situations, because the final presenter must wait for the visual content to be determined by the directors or engineers behind the scenes. As a solution, we implement a gesture-based interactive system based on commercial depth sensors that utilize the infra-red (IR) spectrum. Because commercial depth sensors experience interference and do not work properly under common studio lights that emit IR spectrum, we apply filters and compensation algorithms to implement a robust and stable visualization system for live television broadcasts. We also address usability and information visualization with our system, providing unique features for both the presenter and the television audience. Our system has been named "K-Motion," and it was used to broadcast on 11---12 April 2012 by the Korea Broadcasting System, one of the largest television networks in South Korea, and achieved the highest peak viewing record.

Real-Time Refinement of Kinect Depth Maps using Multi-Resolution Anisotropic Diffusion

In this paper, we present a novel real-time algorithm to refine depth maps generated by low-cost commercial depth sensors like the Microsoft Kinect. The Kinect sensor falls under the category of RGB-D sensors that can generate a high resolution depth map and color image of a scene. They are relatively inexpensive and are commercially available off-the-shelf. However, owing to their low complexity, there are several artifacts that one encounters in the depth map like holes, mis-alignment between the depth map and color image and lack of sharp object boundaries in the depth map. This is a potential problem in applications that require the color image to be projected in 3-D using the depth map. Such applications depend heavily on the depth map and thus the quality of the depth map is of vital importance. In this paper, a novel multi-resolution anisotropic diffusion based algorithm is presented that accepts a Kinect generated depth map and color image and computes a dense depth map in which the holes have been filled and the edges of the objects are sharpened and aligned with the objects in the color image. The proposed algorithm also ensures that regions in the depth map where the depth is properly estimated are not filtered and ensures that the depth values in the final depth map are the same values that existed in the original depth map. Experimental results are provided to demonstrate the improvement in the quality of the depth map and also execution time results are provided to prove that the proposed method can be executed in real-time.

Multi-modal user interaction method based on gaze tracking and gesture recognition

This paper presents a gaze tracking technology which provides a convenient human–centric interface for multimedia consumption without any wearable device. It enables a user to interact with various multimedia on a large display in distance by tracking user movement and acquiring high resolution eye images. This paper also presents a gesture recognition technology which is helpful to interact with scene descriptions in terms of controlling and rendering scene objects. It is based on Hidden Markov Model and CRF using a commercial depth sensor. And then, this paper shows a collaboration method with those new sensors and MPEG standards in order to achieve interoperability among interactive applications, new user interaction devices and users.