Real-time Depth Image Research Articles

Modern Technology for Image processing and Computer vision -A Review


 
 
 
 This survey outlines the use of computer vision in Image and video processing in multidisciplinary applications; either in academia or industry, which are active in this field. The scope of this paper covers the theoretical and practical aspects in image and video processing in addition of computer vision, from essential research to evolution of application. In this work a various subjects of image processing and computer vision will be demonstrated, these subjects are spanned from the evolution of mobile augmented reality (MAR) applications, to augmented reality under 3D modeling and real time depth imaging, video processing algorithms will be discussed to get higher depth video compression, beside that in the field of mobile platform an automatic computer vision system for citrus fruit has been implemented, where the Bayesian classification with Boundary Growing to detect the text in the video scene. Also the paper illustrates the usability of the handed interactive method to the portable projector based on augmented reality.
 © 2018 JASET, International Scholars and Researchers Association
 
 
 

Demonstration of real-time structured-light depth sensing based on a solid-state VCSEL beam scanner.

We demonstrated a real-time scanning structured-light depth sensing system based on a solid-state vertical cavity surface-emitting laser (VCSEL) beam scanner integrated with an electro-thermally tunable VCSEL. Through a swept voltage added to the tunable VCSEL, a field of view of 6°×12° could be scanned with a scanning speed of 100 kHz by the beam scanner. Adopting the beam scanner, the real-time depth image with a lateral resolution of 10,000 (20×500) was obtained by measuring a step target placed at 35cm. The frame rate could be >10Hz even if sunlight shot noise is artificially added to the experimental data. By using a higher-speed camera, a potential lateral resolution could be reached at 50,000 (100×500) with a frame rate of > 20Hz. By using flat optics, a compact scanning module offering line pattern with FoV of >40°×20° was also demonstrated. It could help to realize high-resolution and high-accuracy structured-light sensing with a compact module.

Network algorithm real-time depth image 3D human recognition for augmented reality

This paper studies the application of augmented reality real-time depth image technology to 3D human motion recognition technology. The accuracy and real-time performance of sensor-based 3D human reconstruction are affected by visual characteristics and illumination changes. Features are not easily extracted and cannot be tracked, leading to failures in the 3D reconstruction of the human body. Based on this system, the sensor-based visual inertial initialization algorithm is studied, which is integrated in the two-frame image time interval to provide accurate initial values for vision-based motion estimation, improve the accuracy of the calculated posture, and finally improve the accuracy of the 3D reconstruction system. Based on the relationship between the depth image and the distance and reflectivity, a model for correcting the distance error and reflectivity error of the depth image is established to improve the accuracy of the depth image, and finally the accuracy of the three-dimensional reconstruction of the human body.

Biologically Inspired Intensity and Depth Image Edge Extraction.

In recent years, artificial vision research has moved from focusing on the use of only intensity images to include using depth images, or RGB-D combinations due to the recent development of low-cost depth cameras. However, depth images require a lot of storage and processing requirements. In addition, it is challenging to extract relevant features from depth images in real time. Researchers have sought inspiration from biology in order to overcome these challenges resulting in biologically inspired feature extraction methods. By taking inspiration from nature, it may be possible to reduce redundancy, extract relevant features, and process an image efficiently by emulating biological visual processes. In this paper, we present a depth and intensity image feature extraction approach that has been inspired by biological vision systems. Through the use of biologically inspired spiking neural networks, we emulate functional computational aspects of biological visual systems. The results demonstrate that the proposed bioinspired artificial vision system has increased performance over existing computer vision feature extraction approaches.

Real-Time, Curvature-Sensitive Surface Simplification Using Depth Images

With the rising popularity of handheld virtual reality (VR) devices and depth sensing RGB-D cameras, a variety of VR applications merging these two technologies has been suggested. However, immersive quality of experience in such VR applications is constrained mainly by the large data size and the hardware limitations to handle it. The depth data captured by RGB-D cameras provide a dense sampling of the surface, resulting in a high-poly mesh, which is difficult to be rendered on handheld VR devices due to their limited processing power. To improve the immersive VR experience, a sparse approximation of the depth data is needed. Traditional mesh and point cloud simplification methods are iterative and so are unsuitable for real-time applications. In this paper, we introduce a depth-image-based approach that is capable of generating a good quality sparse mesh for visualization in real time. We propose a curvature-sensitive surface simplification— $\text{CS}^3$ operator that assigns an importance measure to each point in the depth image, based on the local curvature. Further, it applies an importance-order-based restrictive sampling to generate a sparse representation that retains the overall shape as well as the finer features of the object. We also modify the 2-D sweep-line-based constrained Delaunay triangulation to generate 3-D meshes from the sparse point sampling obtained using $\text{CS}^3$ . In addition, the proposed approach preserves key surface properties, such as texture coordinates and materials. We used three different datasets containing dense 3-D models with and without texture, which are scanned using various sensors to validate and compare the robustness, real-time performance, and accuracy of the proposed method over existing approaches. Based on the experimental results, we show that the proposed $\text{CS}^3$ operator and modified 2-D sweep-line-based triangulation generate sparse meshes from depth image in real time, performing significantly faster than current state-of-the-art methods while maintaining similar visual quality.

Histogram of 3D Facets: A depth descriptor for human action and hand gesture recognition

The recent successful commercialization of depth sensors has made it possible to effectively capture depth images in real time, and thus creates a new modality for many computer vision tasks including hand gesture recognition and activity analysis. Most existing depth descriptors simply encode depth information as intensities while ignoring the richer 3D shape information. In this paper, we propose a novel and effective descriptor, the Histogram of 3D Facets (H3DF), to explicitly encode the 3D shape information from depth maps. A 3D Facet associated with a 3D cloud point characterizes the 3D local support surface. By robust coding and circular pooling 3D Facets from a depth map, the proposed H3DF descriptor can effectively represent both 3D shapes and structures of various depth maps. To address the recognition problems of dynamic actions and gestures, we further extend the proposed H3DF by combining it with an N-gram model and dynamic programming. The proposed descriptor is extensively evaluated on two public 3D static hand gesture datasets, one dynamic hand gesture dataset, and one popular 3D action recognition dataset. The recognition results outperform or are comparable with state-of-the-art performances.

계층적 결합형 양방향 필터를 이용한 실시간 깊이 영상 보정 방법

본 논문에서는 결합형 양방향 필터를 이용하여 깊이 영상을 실시간으로 보정하는 방법을 제안한다. 제안한 방법은 Kinect 깊이 카메라로부터 얻은 깊이 영상의 화질을 실시간으로 향상시키기 위해 GPU 내의 상수 메모리와 2차원 영상 처리에 적합한 텍스쳐 메모리를 사용한다. 또한, 단일 화소에 대한 결합형 양방향 필터 연산을 각 GPU 쓰레드(thread)에 할당한 다음 병렬로 처리하여 계산량을 현저히 감소시킨다. 그리고 깊이 영상의 품질을 더욱 높이기 위해 CUDA를 이용해 구현한 결합형 양방향 필터를 계층형 구조로 반복적으로 수행하여 폐색 영역이 채워진 깊이 영상을 얻을 수 있다. 실험 결과를 통해, 제안한 실시간 깊이 영상 보정 방법이 깊이 영상의 주관적 화질을 향상시키고, 초당 55 화면의 속도로 동작하는 것을 확인했다. In this paper, we propose a method for real-time depth image refinement. In order to improve the quality of the depth map acquired from Kinect camera, we employ constant memory and texture memory which are suitable for a 2D image processing in the graphics processing unit (GPU). In addition, we applied the joint bilateral filter (JBF) in parallel to accelerate the overall execution. To enhance the quality of the depth image, we applied the JBF hierarchically using the compute unified device architecture (CUDA). Finally, we obtain the refined depth image. Experimental results showed that the proposed real-time depth image refinement algorithm improved the subjective quality of the depth image and the computational time was 260 frames per second.

Localization and navigation of the CoBots over long-term deployments

For the last three years, we have developed and researched multiple collaborative robots, CoBots, which have been autonomously traversing our multi-floor buildings. We pursue the goal of long-term autonomy for indoor service mobile robots as the ability for them to be deployed indefinitely while they perform tasks in an evolving environment. The CoBots include several levels of autonomy, and in this paper we focus on their localization and navigation algorithms. We present the Corrective Gradient Refinement (CGR) algorithm, which refines the proposal distribution of the particle filter used for localization with sensor observations using analytically computed state space derivatives on a vector map. We also present the Fast Sampling Plane Filtering algorithm that extracts planar regions from depth images in real time. These planar regions are then projected onto the 2D vector map of the building, and along with the laser rangefinder observations, used with CGR for localization. For navigation, we present a hierarchical planner, which computes a topological policy using a graph representation of the environment, computes motion commands based on the topological policy, and then modifies the motion commands to side-step perceived obstacles. We started logging the deployments of the CoBots one and a half years ago, and have since collected logs of the CoBots traversing more than 130 km over 1082 deployments and a total run time of 182 h, which we publish as a dataset consisting of more than 10 million laser scans. The logs show that although there have been continuous changes in the environment, the robots are robust to most of them, and there exist only a few locations where changes in the environment cause increased uncertainty in localization.

Real-Time Depth Image Acquisition and Restoration for Image Based Rendering and Processing Systems

Depth information is an important ingredient in image-based rendering (IBR) systems. Traditional depth acquisition is mainly based on computer vision or depth sensing devices. With the advent of el...

Reducing the gap between Augmented Reality and 3D modeling with real-time depth imaging

Whereas 3D surface models are often used for augmented reality (e.g., for occlusion handling or model-based camera tracking), the creation and the use of such dense 3D models in augmented reality applications usually are two separated processes. The 3D surface models are often created in offline preparation steps, which makes it difficult to detect changes and to adapt the 3D model to these changes. This work presents a 3D change detection and model adjustment framework that combines AR techniques with real-time depth imaging to close the loop between dense 3D modeling and augmented reality. The proposed method detects the differences between a scene and a 3D model of the scene in real time. Then, the detected geometric differences are used to update the 3D model, thus bringing AR and 3D modeling closer together. The accuracy of the geometric difference detection depends on the depth measurement accuracy as well as on the accuracy of the intrinsic and extrinsic parameters. To evaluate the influence of these parameters, several experiments were conducted with simulated ground truth data. Furthermore, the evaluation shows the applicability of AR and depth image–based 3D modeling for model-based camera tracking.

Dynamic 3D human actor generation method using a time-of-flight depth camera

In this paper, we present a novel method to generate a dynamic 3D human actor using a time-of-flight depth camera, which provides synchronized color and depth images in real time. In practice, depth data obtained from the depth camera have critical problems, such as optical noise, absence of depth information for certain regions, and unmatched boundaries. In this work, we significantly enhance the visual quality of raw depth images of the human actor by performing noise reduction, recovery of the lost hair region, and boundary refinement. The 3D surface of the human actor is then created by applying a 3D mesh triangulation technique on the enhanced depth images. Finally, we generate a dynamic 3D human actor by providing the surface with corresponding color images. Experimental results show that the proposed method produces a natural and immersive 3D human actor successfully while minimizing problems inherent in the current depth camera-based images.