RealSense Camera Research Articles

Real-time 2D\u20133D door detection and state classification on a low-power device

In this paper, we propose three methods for door state classification with the goal to improve robot navigation in indoor spaces. These methods were also developed to be used in other areas and applications since they are not limited to door detection as other related works are. Our methods work offline, in low-powered computers as the Jetson Nano, in real-time with the ability to differentiate between open, closed and semi-open doors. We use the 3D object classification, PointNet, real-time semantic segmentation algorithms such as, FastFCN, FC-HarDNet, SegNet and BiSeNet, the object detection algorithm, DetectNet and 2D object classification networks, AlexNet and GoogleNet. We built a 3D and RGB door dataset with images from several indoor environments using a 3D Realsense camera D435. This dataset is freely available online. All methods are analysed taking into account their accuracy and the speed of the algorithm in a low powered computer. We conclude that it is possible to have a door classification algorithm running in real-time on a low-power device.

Dynamic Crosswalk Scene Understanding for the Visually Impaired.

Independent mobility poses a great challenge to the visually impaired individuals. This paper proposes a novel system to understand dynamic crosswalk scenes, which detects the key objects, such as crosswalk, vehicle, and pedestrian, and identifies pedestrian traffic light status. The indication of where and when to cross the road is provided to the visually impaired based on the crosswalk scene understanding. Our proposed system is implemented on a head-mounted mobile device (SensingAI G1) equipped with an Intel RealSense camera and a cellphone, and provides surrounding scene information to visually impaired individuals through audio signal. To validate the performance of the proposed system, we propose a crosswalk scene understanding dataset which contains three sub-datasets: a pedestrian traffic light dataset with 7447 images, a dataset of key objects on the crossroad with 1006 images and a crosswalk dataset with 3336 images. Extensive experiments demonstrated that the proposed system was robust and outperformed the state-of-the-art approaches. The experiment conducted with the visually impaired subjects shows that the system is practical useful.

Dynamic Hybrid Filter for Vision‐Based Pose Estimation of a Hexa Parallel Robot

One of the problems with industrial robots is their ability to accurately locate the pose of the end‐effector. Over the years, many other solutions have been studied including static calibration and dynamic positioning. This paper presents a novel approach for pose estimation of a Hexa parallel robot. The vision system uses three simple color feature points fixed on the surface of the end‐effector to measure the pose of the robot. The Intel RealSense Camera D435i is used as a 3D measurement of feature points, which offers a cheap solution and high accuracy in positioning. Based on the constraint of three color feature points, the pose of the end‐effector, including position and orientation, is determined. A dynamic hybrid filter is designed to correct the vision‐based pose measurement. The complementary filter is used to eliminate the noise of image processing due to environmental light source interference. The unscented Kalman filter is designed to smooth out the pose estimation of the vision system based on robot’s kinematic parameters. The combination of two filters in the same control scheme contributes to increased stability and improved accuracy of robot’s positioning. The simulation, experiment, and comparison demonstrate the effectiveness and feasibility of the proposed method.

Deep Learning-Based Object Detection, Localisation and Tracking for Smart Wheelchair Healthcare Mobility.

This paper deals with the development of an Advanced Driver Assistance System (ADAS) for a smart electric wheelchair in order to improve the autonomy of disabled people. Our use case, built from a formal clinical study, is based on the detection, depth estimation, localization and tracking of objects in wheelchair’s indoor environment, namely: door and door handles. The aim of this work is to provide a perception layer to the wheelchair, enabling this way the detection of these keypoints in its immediate surrounding, and constructing of a short lifespan semantic map. Firstly, we present an adaptation of the YOLOv3 object detection algorithm to our use case. Then, we present our depth estimation approach using an Intel RealSense camera. Finally, as a third and last step of our approach, we present our 3D object tracking approach based on the SORT algorithm. In order to validate all the developments, we have carried out different experiments in a controlled indoor environment. Detection, distance estimation and object tracking are experimented using our own dataset, which includes doors and door handles.

Tomato Fruit Detection and Counting in Greenhouses Using Deep Learning.

Accurately detecting and counting fruits during plant growth using imaging and computer vision is of importance not only from the point of view of reducing labor intensive manual measurements of phenotypic information, but also because it is a critical step toward automating processes such as harvesting. Deep learning based methods have emerged as the state-of-the-art techniques in many problems in image segmentation and classification, and have a lot of promise in challenging domains such as agriculture, where they can deal with the large variability in data better than classical computer vision methods. This paper reports results on the detection of tomatoes in images taken in a greenhouse, using the MaskRCNN algorithm, which detects objects and also the pixels corresponding to each object. Our experimental results on the detection of tomatoes from images taken in greenhouses using a RealSense camera are comparable to or better than the metrics reported by earlier work, even though those were obtained in laboratory conditions or using higher resolution images. Our results also show that MaskRCNN can implicitly learn object depth, which is necessary for background elimination.

3D foot scanning using multiple RealSense cameras

3D scanning of the foot is of great significance for footwear customization, intelligent shoe size recommendation and foot disease diagnosis. In this paper, we propose a 3D foot scanning system that consists of four Intel RealSense cameras and one host PC and scans both feet simultaneously. A novel calibration method that is based on a Tower-type block was proposed for calculating the extrinsic parameters of multiple RGB-Depth cameras. The Tower-type block was designed to realize the automatic execution of the multi-camera calibration process and reduce the operational complexity. This paper introduced the complete procedure of the system, including partial view scanning, point cloud filtering, registration, and non-visible area filling, reconstruction and foot measurement. The presented experimental results demonstrated that the proposed methods were efficient and versatile approaches for 3D foot scanning.

Robotic grasping with obstacle avoidance using octrees

This paper considers the problems of the integration of independent manipulator control systems. Areas of control of the manipulator are: recognition of objects and obstacles, identification of objects to be grasped, determination of reliable positions by the grasping device, planning of movement of the manipulator to certain positions with avoidance of obstacles, and recognition of slipping or determination of reliable grasping. This issue is a current problem primarily in industry, general-purpose robots, and experimental robots. This paper considers current publications that address these issues. Existing algorithms and approaches have been found in the management of both parts of the robot manipulator and solutions that combine several areas, or the integration of several existing approaches. There is a brief review of current literature and publications on the above algorithms and approaches. The advantages and disadvantages of the considered methods and approaches are determined. There are solutions that cover either some areas or only one of them, which does not meet the requirements of the problem. Using existing approaches, integration points of existing implementations are identified to get the best results. In the process, a system was developed that analyzes the environment, finds obstacles, objects for interaction, poses for grasping, plans the movement of the manipulator to a specific position, and ensures reliable grasping of the object. The next step was to test the system, test the performance, and adjust the parameters for the best results. The resulting system was developed by the research team of RT-Lions, Technik University, Reutlingen. The hardware research robot includes an Intel Realsense camera, a Sawyer Arm manipulator from Rethink Robotics, and an internally grabbing device.

Evaluation of low-cost depth cameras for agricultural applications

Low-cost depth-cameras have been used in many agricultural applications with reported advantages of low cost, reliability and speed of measurement. However, some problems were also reported and seem to be technology-related, so understanding the limitations of each type of depth camera technology could provide a basis for technology selection and the development of research involving its use. The cameras use one or a combination of two of the three available technologies: structured light, time-of-flight (ToF), and stereoscopy. The objectives were to evaluate these different technologies for depth sensing, including measuring accuracy and repeatability of distance data and measurements at different positions within the image, and cameras usefulness in indoor and outdoor settings. Then, cameras were tested in a swine facility and in a corn field. Five different cameras were used: (1) Microsoft Kinect v.1, (2) Microsoft Kinect v.2, (3) Intel® RealSense™ Depth Camera D435, (4) ZED Stereo Camera (StereoLabs), and (5) CamBoard Pico Flexx (PMD Technologies). Results indicate that there were significant camera to camera differences for ZED Stereo Camera and Kinect v.1 camera (p < 0.05). All cameras showed an increase in the standard deviation as the distance between camera and object increased; however, the Intel RealSense camera had a larger increase. Time-of-flight cameras had the smallest error between different sizes of objects. Time-of-flight cameras had non-readable zones on the corners of the images. The results indicate that the ToF technology is the best to be used for indoor applications and stereoscopy is the best technology for outdoor applications.

Effects of the post-processing on depth value accuracy of the images captured by RealSense cameras

Effects of the post-processing on depth value accuracy of the images captured by RealSense cameras

Edge-preserving Filtering and Fuzzy Image Enhancement in Depth Images Captured by Realsense Cameras in Robotic Applications

This paper presents both the use of depth cameras in robotic applications and effects of post-processing on the captured depth images. The performance of depth cameras and post-process ...

Dynamic Filtered Path Tracking Control for a 3RRR Robot Using Optimal Recursive Path Planning and Vision-Based Pose Estimation

This study proposes a dynamic filtered path tracking control schema for a 3RRR planar parallel robot to reduce the positioning errors caused by various sources such as backlash in the mechanical system, system nonlinearities, uncertainties, and unknown disturbances. The optimal recursive algorithm is implemented absolutely in path planning based on the optimization problem. Compared to other methods, this algorithm not only helps the robot avoid singularities but also ensures the shortest movement distance of links thanks to the optimization in path planning. The vision system using the 3D Intel RealSense camera D435i is proposed to provide the visual measurement as feedback for the pose estimation of the 3RRR planar parallel robot. The unscented Kalman filter is designed to reduce the errors of the pose estimation due to the camera’s intrinsic parameters, the vibration of the mechanical system, and unknown noises. The establishment of important equations and parameters of the unscented Kalman filter is detailed in this study. The simulation, experiment, and comparison are conducted to verify the effectiveness and feasibility of the proposed approaches.

Hole-Filling of RealSense Depth Images Using a Color Edge Map

The RealSense camera frequently includes hole regions—unfilled regions where depth information are missing in the depth image. The holes cause serious problems in applications that use depth sensors and often straddle the background and object simultaneously. Accordingly, a hole must be filled by considering the boundary of the object. Fortunately, the object boundary inside the hole can be estimated by the RealSense color image. This paper proposes a hole-filling method in which filling converges to the object boundary from both border pixels. However, the color edge elements of the object boundary are often missing. Because the missing edge highly influences the hole-filling procedure, the boundary of the object close to the camera should be filled as completely as possible. Consequently, we introduce a simple method of filling the gap between two edge endpoints. Furthermore, we can address the situation in which there are one or more edge pixels between two border pixels. We repeatedly change the filling direction by alternating between the horizontal and vertical directions until no hole remains. Our experiment reveals that the proposed method outperforms other filling methods when numerically comparing the hole-filled depth image with the ground truth image for four methods: root-mean-square error (RMSE), peak signal-to-noise ratio (PSNR), structural similarity index map (SSIM), and correlation coefficient (CC). The proposed method also produces a superior visual when we observe the 3D stereo image pair constructed using an original color image and a left-view color image generated with depth-image-based rendering (DIBR).

Lightweight Architecture for Real-Time Hand Pose Estimation with Deep Supervision

The high demand for computational resources severely hinders the deployment of deep learning applications in resource-limited devices. In this work, we investigate the under-studied but practically important network efficiency problem and present a new, lightweight architecture for hand pose estimation. Our architecture is essentially a deeply-supervised pruned network in which less important layers and branches are removed to achieve a higher real-time inference target on resource-constrained devices without much accuracy compromise. We further make deployment optimization to facilitate the parallel execution capability of central processing units (CPUs). We conduct experiments on NYU and ICVL datasets and develop a demo1 using the RealSense camera. Experimental results show our lightweight network achieves an average running time of 32 ms (31.3 FPS, the original is 22.7 FPS) before deployment optimization. Meanwhile, the model is only about half parameters size of the original one with 11.9 mm mean joint error. After the further optimization with OpenVINO, the optimized model can run at 56 FPS on CPUs in contrast to 44 FPS running on a graphics processing unit (GPU) (Tensorflow) and it can achieve the real-time goal.

Metrological and Critical Characterization of the Intel D415 Stereo Depth Camera.

Low-cost RGB-D cameras are increasingly being used in several research fields, including human–machine interaction, safety, robotics, biomedical engineering and even reverse engineering applications. Among the plethora of commercial devices, the Intel RealSense cameras have proven to be among the most suitable devices, providing a good compromise between cost, ease of use, compactness and precision. Released on the market in January 2018, the new Intel model RealSense D415 has a wide acquisition range (i.e., ~160–10,000 mm) and a narrow field of view to capture objects in rapid motion. Given the unexplored potential of this new device, especially when used as a 3D scanner, the present work aims to characterize and to provide metrological considerations for the RealSense D415. In particular, tests are carried out to assess the device performance in the near range (i.e., 100–1000 mm). Characterization is performed by integrating the guidelines of the existing standard (i.e., the German VDI/VDE 2634 Part 2) with a number of literature-based strategies. Performance analysis is finally compared against the latest close-range sensors, thus providing a useful guidance for researchers and practitioners aiming to use RGB-D cameras in reverse engineering applications.

Condition Monitoring and Cloud-based Energy Analysis for Autonomous Mobile Manipulation - Smart Factory Concept with LUHbots

In this paper, a smart factory concept for autonomous mobile robots is presented. The main purpose is to increase productivity of the transport in machine-floor. It is based on advanced methods for failure handling and prevention, leading to increased robustness, less downtime and less effort in maintenance [1], [2]. Therefore, condition data and states of the robot are collected by Robot Operation System (ROS) and transferred to a factory hub (server). The collected data, e.g. voltages, currents, set points, velocities and accelerations are used to identify important system parameters, e.g. moving masses and friction parameters to enable the proposed smart factory concept. Further aim is to let the factory hub control a group of mobile robots using a self-organizing algorithm for different tasks.Due to the increasing customization of products causing smaller lot sizes [3], manufacturers of mobile robotic production systems have developed a diversity of flexible robots [4], [5], [6], [7], [8]. Mobile robots inside the production line allow for collecting and evaluation of system-inherent data e.g. handling and transportation time, wheel friction, workpieces mass, center of gravity and energy consumption during trajectory execution.In general, mobile robots are electrically driven. Hence, an estimation of the battery state is essential in order to automatically plan charging cycles and to organize and optimize the cooperation behavior of a group of mobile robots. In this proposed approach, mobile robots are equipped with a measurement system and connected via Bluetooth to a factory hub, providing monitoring, analyzing and planning tools. The battery states of all robots are considered in the process planning.The robots are based on the KUKA youBot, equipped with a soft gripper and a RealSense camera. A condition monitoring system measures the energy consumption of all components and transfers the information to the factory hub. The state of charge limits the number of executable operations. Therefore, in a first step the power consumption of all individual consumers is captured, e.g. EC-Maxxon base motors, PC, gripper, camera and five-axis arm. Experimental results show, that the youBot requires 46 W in standstill plus the drive power depending on the movement. Here, the results for mobile manipulation in industrial scenarios during preparation for the RoboCup@Work 2016 will be presented. The transfer of raw measurement data to the hub is shown, as well as the proposed algorithms allowing for range prediction and optimized set point generation. The concept provides excellent capability in data collection, analysis of existing production and production planning.