Low-cost RGB-D Camera Research Articles

View Planning for Robotic Inspection of Tolerances Through Visual Tracking of Manual Surface Finishing Operations

This article presents an approach for determining suitable camera view poses for inspection of surface tolerances based on visual tracking of the tool movements performed by a skilled worker. Automated surface inspection of a workpiece adjusted by manual operations depends on manual programming of the inspecting robot, or a timeconsuming exhaustive search over the entire surface. The proposed approach is based on the assumption that the tool movements of the skilled worker coincide with the most relevant regions of the underlying surface of the workpiece, namely the parts where a manual process has been performed. The affected region is detected with a visual tracking system, which measures the motion of the tool using a low-cost RGBD-camera, a particle filter, and a CAD model of the tool. The main contribution is a scheme for selecting relevant camera view poses for inspecting the affected region using a robot equipped with a high-accuracy RGBDcamera. A principal component analysis of the tracked tool paths allows for evaluating the view poses by the Hotelling’s T-squared distribution test in order to sort and select suitable camera view poses. The approach is implemented and tested for the case where a large ship propeller blade cast in NiAl bronze is to be inspected by a robot after manual adjustments of its surface.

Automatic Gait Phases Detection in Parkinson Disease: A Comparative Study.

Parkinson's disease (PD) is a chronic condition that can be diagnosed and monitored by evaluating changes in the gait and arm movement parameters. In the gait movement, each cycle consists of two phases: stance and swing. Using gait analysis techniques, it is possible to get spatiotemporal variables derived from both phases. In this paper, we compared two techniques: wavelet and peak detection. Previously, the wavelet technique was assessed for the gait phases detection, and peak detection was evaluated for arm swing analysis. These methods were evaluated using a low-cost RGB-D camera as data input source. This comparison could provide a unified and integrated method to analyze gait and arm swing signals. Twenty-five PD patients and 25 age-matched, healthy subjects were included. Mann-Whitney U test was used to compare the continuous variables between groups. Hamming distances and Spearman rank correlation were used to evaluate the agreement between the signals and the spatiotemporal variables obtained by both methods. PD group showed significant reductions in speed (wavelet p = 0.001, peak detection p <0.001) and significantly greater swing (wavelet p = 0.003, peak detection p =0.005) and stance times (wavelet p = 0.003, peak detection p =0.004). Hamming distances showed small differences between the signals obtained by both methods (16 to 18 signal points). A very strong correlation (Spearman rho > 0.8, p <0.05) was found between the spatiotemporal variables obtained by each signal processing technique. Wavelet and peak detection techniques showed a high agreement in the signal obtained from gait data. The spatiotemporal variables obtained by both methods showed significant differences between the walking patterns of PD patients and healthy subjects. The peak detection technique can be used for integral motion analysis, providing the identification of the phases in the gait cycle, and arm swing parameters.Clinical Relevance- this establishes that peaks and wavelet techniques are comparable and may use it interchangeably to process signals from the gait of Parkinson's disease patients to support diagnosis and follow up made by a clinical expert.

GRNet: Geometric relation network for 3D object detection from point clouds

Rapid detection of 3D objects in indoor environments is essential for indoor mapping and modeling, robotic perception and localization, and building reconstruction. 3D point clouds acquired by a low-cost RGB-D camera have become one of the most commonly used data sources for 3D indoor mapping. However, due to the sparse surface, empty object center, and various scales of point cloud objects, 3D bounding boxes are challenging to be estimated and located accurately. To address this, geometric shape, topological structure, and object relation are commonly employed to extract box reasoning information. In this paper, we describe the geometric feature among object points as an intra-object feature and the relation feature between different objects as an inter-object feature. Based on these two features, we propose an end-to-end point cloud geometric relation network focusing on 3D object detection, which is termed as geometric relation network (GRNet). GRNet first extracts intra-object and inter-object features for each representative point using our proposed backbone network. Then, a centralization module with a scalable loss function is proposed to centralize each representative object point to its center. Next, proposal points are sampled from these shifted points, following a proposal feature pooling operation. Finally, an object-relation learning module is applied to predict bounding box parameters. Such parameters are the additive sum of prediction results from the relation-based inter-object feature and the aggregated intra-object feature. Our model achieves state-of-the-art 3D detection results with 59.1% mAP@0.25 and 39.1% mAP@0.5 on ScanNetV2 dataset, 58.4% mAP@0.25 and 34.9% mAP@0.5 on SUN RGB-D dataset.

In-Field Grape Cluster Size Assessment for Vine Yield Estimation Using a Mobile Robot and a Consumer Level RGB-D Camera

Current practice for vine yield estimation is based on RGB cameras and has limited performance. In this letter we present a method for outdoor vine yield estimation using a consumer grade RGB-D camera mounted on a mobile robotic platform. An algorithm for automatic grape cluster size estimation using depth information is evaluated both in controlled outdoor conditions and in commercial vineyard conditions. Ten video scans (3 camera viewpoints with 2 different backgrounds and 2 natural light conditions), acquired from a controlled outdoor experiment and a commercial vineyard setup, are used for analyses. The collected dataset (GRAPES3D) is released to the public. A total of 4542 regions of 49 grape clusters were manually labeled by a human annotator for comparison. Eight variations of the algorithm are assessed, both for manually labeled and auto-detected regions. The effect of viewpoint, presence of an artificial background, and the human annotator are analyzed using statistical tools. Results show 2.8–3.5 cm average error for all acquired data and reveal the potential of using low-cost commercial RGB-D cameras for improved robotic yield estimation.

Utilising low cost RGB-D cameras to track the real time progress of a manual assembly sequence

Purpose The purpose of this paper is to explore the role that computer vision can play within new industrial paradigms such as Industry 4.0 and in particular to support production line improvements to achieve flexible manufacturing. As Industry 4.0 requires “big data”, it is accepted that computer vision could be one of the tools for its capture and efficient analysis. RGB-D data gathered from real-time machine vision systems such as Kinect ® can be processed using computer vision techniques. Design/methodology/approach This research exploits RGB-D cameras such as Kinect® to investigate the feasibility of using computer vision techniques to track the progress of a manual assembly task on a production line. Several techniques to track the progress of a manual assembly task are presented. The use of CAD model files to track the manufacturing tasks is also outlined. Findings This research has found that RGB-D cameras can be suitable for object recognition within an industrial environment if a number of constraints are considered or different devices/techniques combined. Furthermore, through the use of a HMM inspired state-based workflow, the algorithm presented in this paper is computationally tractable. Originality/value Processing of data from robust and cheap real-time machine vision systems could bring increased understanding of production line features. In addition, new techniques that enable the progress tracking of manual assembly sequences may be defined through the further analysis of such visual data. The approaches explored within this paper make a contribution to the utilisation of visual information “big data” sets for more efficient and automated production.

Improving the Depth Accuracy and Assessment of Microsoft Kinect v2 Towards a Usage for Mechanical Part Modeling

From 2010, the first version of Microsoft Kinect, a low-cost RGB-D camera, was released which used structured light technology to capture depth information. This device has been widely applied in many segments of the industry. In July 2014, the second version of Microsoft Kinect was launched with improved hardware. Obtaining point clouds of an observed scene with high frequency being possible leads to imaging its application to meeting the demand of 3D data acquisition. However, evaluating device capacity for mechanical part modeling has been a challenge needed to be solved. This paper intends to enhance acquired depth maps of the Microsoft Kinect v2 device for mechanical part modeling and receive an assessment about the accuracy of 3D reconstruction. Influence of materials for mechanical part modeling is also evaluated. Additionally, experimental methodology for 3D modeling of the mechanical part is finally reported to ascertain the proposed model in this paper.

Age Matters: Objective Gait Assessment in Early Parkinson's Disease Using an RGB-D Camera.

Background Gait alterations are hallmarks for the diagnosis and follow-up of patients with Parkinson's disease (PD). In normal conditions, age could affect gait dynamics. Although it is known that objective assessment of gait is a valuable tool for diagnosis and follow-up of patients with PD, only few studies evaluate the effect of aging on the gait pattern of patients with PD. Objective The purpose of this study was to assess differences in gait dynamics between PD patients and healthy subjects and to investigate the effects of aging on these differences using a low-cost RGB-D depth-sensing camera. Methods 30 PD patients and 30 age-matched controls were recruited. Descriptive analysis was used for clinical variables, and Spearman's rank correlation was used to correlate age and gait variables. The sample was distributed in age groups; then, Mann–Whitney U test was used for comparison of gait variables between groups. Results PD patients exhibited prolonged swing (p=0.002) and stance times (p < 0.001) and lower speed values (p < 0.001) compared to controls. This was consistent in all age groups, except for the one between 76 and 88 years old, in which the controls were slower and had longer swing and stance times. These results were statically significant for the group from 60 to 66 years. Conclusion Gait speed, swing, and stance times are useful for differentiating PD patients from controls. Quantitative gait parameters measured by an RGB-D camera can complement clinical assessment of PD patients. The analysis of these spatiotemporal variables should consider the age of the subject.

Metrological and Critical Characterization of the Intel D415 Stereo Depth Camera.

Low-cost RGB-D cameras are increasingly being used in several research fields, including human–machine interaction, safety, robotics, biomedical engineering and even reverse engineering applications. Among the plethora of commercial devices, the Intel RealSense cameras have proven to be among the most suitable devices, providing a good compromise between cost, ease of use, compactness and precision. Released on the market in January 2018, the new Intel model RealSense D415 has a wide acquisition range (i.e., ~160–10,000 mm) and a narrow field of view to capture objects in rapid motion. Given the unexplored potential of this new device, especially when used as a 3D scanner, the present work aims to characterize and to provide metrological considerations for the RealSense D415. In particular, tests are carried out to assess the device performance in the near range (i.e., 100–1000 mm). Characterization is performed by integrating the guidelines of the existing standard (i.e., the German VDI/VDE 2634 Part 2) with a number of literature-based strategies. Performance analysis is finally compared against the latest close-range sensors, thus providing a useful guidance for researchers and practitioners aiming to use RGB-D cameras in reverse engineering applications.

Nonrigid reconstruction of 3D breast surfaces with a low-cost RGBD camera for surgical planning and aesthetic evaluation.

Accounting for 26% of all new cancer cases worldwide, breast cancer remains the most common form of cancer in women. Although early breast cancer has a favourable long-term prognosis, roughly a third of patients suffer from a suboptimal aesthetic outcome despite breast conserving cancer treatment. Clinical-quality 3D modelling of the breast surface therefore assumes an increasingly important role in advancing treatment planning, prediction and evaluation of breast cosmesis. Yet, existing 3D torso scanners are expensive and either infrastructure-heavy or subject to motion artefacts. In this paper we employ a single consumer-grade RGBD camera with an ICP-based registration approach to jointly align all points from a sequence of depth images non-rigidly. Subtle body deformation due to postural sway and respiration is successfully mitigated leading to a higher geometric accuracy through regularised locally affine transformations. We present results from 6 clinical cases where our method compares well with the gold standard and outperforms a previous approach. We show that our method produces better reconstructions qualitatively by visual assessment and quantitatively by consistently obtaining lower landmark error scores and yielding more accurate breast volume estimates.

Low Cost Bin-Picking and Placing System for Automotive Parts

In this paper, we propose the automated bin-picking and placing system that is composed of two low cost RGB-D cameras, two robots and the assembly jig for automotive parts manufacturing industry. In the system, the depth image of the target part is captured by the RGB-D camera to estimate its position and orientation. Then, the robot picks one of parts from randomly stacked in a pallet, and places it onto the assembly jig automatically. We describe the method for integration the vision technology and the robot technology into one system, and evaluate its performance by applying on real automotive parts.

FusionMLS: Highly dynamic 3D reconstruction with consumer-grade RGB-D cameras

Multi-view dynamic three-dimensional reconstruction has typically required the use of custom shutter-synchronized camera rigs in order to capture scenes containing rapid movements or complex topology changes. In this paper, we demonstrate that multiple unsynchronized low-cost RGB-D cameras can be used for the same purpose. To alleviate issues caused by unsynchronized shutters, we propose a novel depth frame interpolation technique that allows synchronized data capture from highly dynamic 3D scenes. To manage the resulting huge number of input depth images, we also introduce an efficient moving least squares-based volumetric reconstruction method that generates triangle meshes of the scene. Our approach does not store the reconstruction volume in memory, making it memory-efficient and scalable to large scenes. Our implementation is completely GPU based and works in real time. The results shown herein, obtained with real data, demonstrate the effectiveness of our proposed method and its advantages compared to state-of-the-art approaches.

System for automatic gait analysis based on a single RGB-D camera

Human gait analysis provides valuable information regarding the way of walking of a given subject. Low-cost RGB-D cameras, such as the Microsoft Kinect, are able to estimate the 3-D position of several body joints without requiring the use of markers. This 3-D information can be used to perform objective gait analysis in an affordable, portable, and non-intrusive way. In this contribution, we present a system for fully automatic gait analysis using a single RGB-D camera, namely the second version of the Kinect. Our system does not require any manual intervention (except for starting/stopping the data acquisition), since it firstly recognizes whether the subject is walking or not, and identifies the different gait cycles only when walking is detected. For each gait cycle, it then computes several gait parameters, which can provide useful information in various contexts, such as sports, healthcare, and biometric identification. The activity recognition is performed by a predictive model that distinguishes between three activities (walking, standing and marching), and between two postures of the subject (facing the sensor, and facing away from it). The model was built using a multilayer perceptron algorithm and several measures extracted from 3-D joint data, achieving an overall accuracy and F1 score of 98%. For gait cycle detection, we implemented an algorithm that estimates the instants corresponding to left and right heel strikes, relying on the distance between ankles, and the velocity of left and right ankles. The algorithm achieved errors for heel strike instant and stride duration estimation of 15 ± 25 ms and 1 ± 29 ms (walking towards the sensor), and 12 ± 23 ms and 2 ± 24 ms (walking away from the sensor). Our gait cycle detection solution can be used with any other RGB-D camera that provides the 3-D position of the main body joints.

Detecting humans in the robot workspace

PurposeThis paper aims to provide a technical insight into a selection of robotic people detection technologies and applications.Design/methodology/approachFollowing an introduction, this paper first discusses people-sensing technologies which seek to extend the capabilities of human-robot collaboration by allowing humans to operate alongside conventional, industrial robots. It then provides examples of developments in people detection and tracking in unstructured, dynamic environments. Developments in people sensing and monitoring by assistive robots are then considered and finally, brief concluding comments are drawn.FindingsRobotic people detection technologies are the topic of an extensive research effort and are becoming increasingly important, as growing numbers of robots interact directly with humans. These are being deployed in industry, in public places and in the home. The sensing requirements vary according to the application and range from simple person detection and avoidance to human motion tracking, behaviour and safety monitoring, individual recognition and gesture sensing. Sensing technologies include cameras, lasers and ultrasonics, and low cost RGB-D cameras are having a major impact.Originality/valueThis article provides details of a range of developments involving people sensing in the important and rapidly developing field of human-robot interactions.

Static and dynamic displacement measurements of structural elements using low cost RGB-D cameras

Optical displacement measurements can provide data of the behavior of structural elements without altering key parameters such as damping, stiffness, or mass. Optical techniques can also reduce costs and substantially simplify the equipment required to acquire data effectively and map displacements at specific locations of elements with complex geometries, such as steel connections. In addition, these techniques can provide simultaneous data at multiple points using a single device, hence lowering instrumentation costs. For example, motion capture systems are used in applications that range from medicine to cinematography, and involve different types of imaging techniques. However, their application to civil and mechanical structures is complex and costly. On the other hand, range/depth cameras can provide a 3-D imaging solution to capture motion at an affordable cost. These cameras are widely available and used in the videogame industry. This paper presents the characterization and implementation of a methodology for the measurement of large-displacements using artificial vision techniques, for structural applications such as experimental modal analysis, cyclic loading tests, creep tests, and static displacements tests. Two low cost depth-cameras, one a time-of-flight camera and the other a structured light camera, were calibrated, evaluated and implemented in static and dynamic tests, during a full-scale reinforced concrete wall cyclic load test and shaking table tests. The results show that both devices provide high accuracy measurements in comparison with commonly used sensors for displacement measurements, with the added advantages of being contactless and providing three-dimensional information.

Object Segmentation Ensuring Consistency Across Multi-Viewpoint Images.

We present a hybrid approach that segments an object by using both color and depth information obtained from views captured from a low-cost RGBD camera and sparsely-located color cameras. Our system begins with generating dense depth information of each target image by using Structure from Motion and Joint Bilateral Upsampling. We formulate the multi-view object segmentation as the Markov Random Field energy optimization on the graph constructed from the superpixels. To ensure inter-view consistency of the segmentation results between color images that have too few color features, our local mapping method generates dense inter-view geometric correspondences by using the dense depth images. Finally, the pixel-based optimization step refines the boundaries of the results obtained from the superpixel-based binary segmentation. We evaluate the validity of our method under various capture conditions such as numbers of views, rotations, and distances between cameras. We compared our method with the state-of-the-art methods that use the standard multi-view datasets. The comparison verified that the proposed method works very efficiently especially in a sparse wide-baseline capture environment.

Human Motion Tracking by Multiple RGBD Cameras

The advent of low-cost depth cameras, such as the Microsoft Kinect in the consumer market, has made many indoor applications and games based on motion tracking available to the everyday user. However, it is a large challenge to track human motion via such a camera because of its low-quality images, missing depth values, and noise. In this paper, we propose a novel human motion capture method based on a cooperative structure of multiple low-cost RGBD cameras, which can effectively avoid these problems. This structure can also manage the problem of body occlusions that appears when a single camera is used. Moreover, the whole process does not require training data, which makes this approach easily deployed and reduces operation time. We use the color image, depth image, and point cloud acquired in each view as the data source, and an initial pose is extracted in our optimization framework by aligning multiple point clouds from different cameras. The pose is dynamically updated by combining a filtering approach with a Markov model to estimate new poses in video streams. To verify the efficiency and robustness of our approach, we capture a wide variety of human actions via three cameras in indoor scenes and compare the tracking results of the proposed method to those of the current state-of-the-art methods. Moreover, our system is tested on more complex situations, in which multiple humans move within a scene, possibly occluding each other to some extent. The actions of multiple humans are tracked simultaneously, which would assist group behavior analysis.

Combining Semantic and Geometric Features for Object Class Segmentation of Indoor Scenes

Scene understanding is a necessary prerequisite for robots acting autonomously in complex environments. Low-cost RGB-D cameras such as Microsoft Kinect enabled new methods for analyzing indoor scenes and are now ubiquitously used in indoor robotics. We investigate strategies for efficient pixelwise object class labeling of indoor scenes that combine both pretrained semantic features transferred from a large color image dataset and geometric features, computed relative to the room structures, including a novel distance-from-wall feature, which encodes the proximity of scene points to a detected major wall of the room. We evaluate our approach on the popular NYU v2 dataset. Several deep learning models are tested, which are designed to exploit different characteristics of the data. This includes feature learning with two different pooling sizes. Our results indicate that combining semantic and geometric features yields significantly improved results for the task of object class segmentation.

3D object detection with deep learning

Finding an appropriate environment representation is a crucial problem in robotics. 3D data has been recently used thanks to the advent of low cost RGB-D cameras. We propose a new way to represent a 3D map based on the information provided by an expert. Namely, the expert is the output of a Convolutional Neural Network trained with deep learning techniques. Relying on such information, we propose the generation of 3D maps using individual semantic labels, which are associated with environment objects or semantic labels. So, for each label we are provided with a partial 3D map whose data belong to the 3D perceptions, namely point clouds, which have an associated probability above a given threshold. The final map is obtained by registering and merging all these partial maps. The use of semantic labels provide us a with way to build the map while recognizing objects.

Care respite: taking care of the caregivers

Introduction : With an ageing population, the issue of care provision is becoming increasingly critical. Since the aspiration of the majority of older people is to live safely and well at home, housing monitoring will be part of health & care integration in the next decades. As a consequence, a higher proportion of people will have to rely on family, friends or neighbours as their informal caregivers, providing that this group already count as around 80% of all caregivers in the EU). The varied and cumulative problems that this caregivers face include loss of personal opportunities and self-esteem, financial problems or physical and emotional problems. Several strategies need to be enforced to prevent the burnout and improve the quality of life of these informal caregivers, such as the possibility of providing them with useful and reliable monitoring tools that can help reduce the exhausting and continuous supervision activity they have to perform, reduce their stress, and increase their leisure time while improving the autonomy and wellness of the dependent people. The technological solutions currently in use are invasive (wearable sensors) or require actions difficult to perform by the elder (call or press the button), in critical situations (e.g. falling or fainting situations, where the elder is not able to give any response). As an alternative to these solutions, this paper describes *Care Respite, a novel non-invasive, real-time and privacy-preserving monitoring technology conceived for the first time as a real respite for those taking care of others* and describes the results of the case studies. Short description : The Care Respite technology is composed of an ambient intelligent device (AID) and a remote receiver. The AID device includes a Microsoft Kinect multi-modal sensor and a computer unit. The sensor captures audio and depth maps from around 25 m2 (thanks to its infrared structured light technology) to be processed in a computer unit by an advanced generation of Computer Vision software. This software is able to recognize specific events like: leaving the environment, falling/fainting, high agitation, etc. Monitoring can be done in dark environments thanks to the infrared technology of the sensor, therefore the appearance of the person is not used at all. The alert is transmitted to a receiver, any smartphone with Internet connection. With our smartphone application, different caregivers can share the monitoring task of a single person, and a single caregiver can monitor multiple dependent people at the same time. Key findings : The innovative aspect of Care Respite relies on (i) the benefits of the incorporation and exploitation of novel computer vision technologies now available thanks to the use of low-cost depth cameras, and (ii) on the feedback provided by informal and professional caregivers during these last two years. No other monitoring system in the market addressed to the caregivers is comparable in terms of safety, relief, autonomy, satisfaction, intimacy and cost reduction. The system does not require a fixed installation point, therefore the monitoring camera can be placed anywhere inside a home or a retirement house. Highlights : Low cost RGB-D cameras are completely changing the computer vision world, as they are being successfully used in several applications and research areas. Following this trend, the first users of Care Respite liked the idea of exploiting depth data (not the appearance) recorded at the older person’s homes to send alarms and information to their caregivers, who are connected using smartphone. An additional feature that liked the first users of Care Respite is its personalization aspect: the remote smartphone application allows the professionals or family carers to choose which risky events to monitor. Another lesson learnt during implementation is that the remote visualization and communication via image and audio is an essential and supporting tool for fast responses in case of risk, while improving the autonomy and life quality of both the elderly and their caregivers. Conclusion : This project aims at ensuring that all disabled or chronically ill people can get the help they need without overburdening their families. Improving carers' quality of life and preventing their burnout require new solutions that are affordable and user friendly for families and caregivers. In this context, Care Respite represents a technological solution for the automatic detection of events like falling down, not moving during a period of time, receiving an unexpected visit or being absent from a room. The intelligent software is able to send an alert to the mobile device of the caregivers allowing real time response.

Patches Detection and Fusion for 3D Face Cloning

3D face clones are used in many fields such as video games and Human-Computer Interaction. However, high-resolution sensors generating high quality clones are expensive and not accessible to all. In this paper, we propose to make a fully automated and accurate 3D reconstruction of a face with a low cost RGB-D camera. For each subject, we capture the depth and RGB data of their face in different positions while performing a rotational movement of the head. We fit a 3D Morphable Face Model on each frame to eliminate noise, increase resolution and provide a structured mesh. This type of mesh is a mesh which the semantic and topological structure is known. We propose to only keep the suitable parts of each mesh called Patch. This selection is performed using an error distance and the direction of the normal vectors. To create the 3D face clone, we merge the different patches of each mesh. These patches contain relevant information on the specificity of individuals and lead to the construction of a more accurate clone. We perform quantitative tests by comparing our clone to ground truth and qualitative tests by comparing visual features. These results show that our method outperforms the FaceWarehouse process of Cao et al [2]. This 3D face clone on a structured mesh can be used as pretreatment in applications such as emotion analysis [13] or facial animation.