Intel RealSense Research Articles

228 Hind Leg Angle and Stride Length Measured by 3-D Imaging Account for Variance of Locomotion Score and Growth Performance of Cattle in Slatted Feeding Facilities

Abstract The objective was to determine the variance of locomotion score (LS) and growth performance attributable to hind leg angle and stride length (SL) measured by 3-D image analysis for cattle in slatted feeding facilities. Inherent individual differences in structural conformation may be related to cattle mobility and growth performance in indoor slatted facilities. Angus × Simmental steers (n = 189; BW = 352 ± 43 kg) were blocked by initial BW and assigned to 21 pens. Pens were randomly assigned to 1 of 3 treatments (TRT): concrete slats with no matting, 15-year-old Animat Pebble matting, and new Animat Pebble matting. Steers were fed for 152 d. Individual steers videos were recorded on d 0 using an Intel RealSense depth camera and processed using MATLAB to estimate hind leg angle, SL, and body length (BL). Locomotion scores were assigned using a 0-3 scale (Zinpro Step-Up Locomotion Scoring System) throughout the finishing phase. The MIVQUE0 option of the MIXED procedure of SAS 9.4 was utilized to estimate the proportion of variance in average LS, overall ADG, and final BW. Variance of average LS was attributed to SL and SL × BL × TRT at 64% and 28%, respectively. For overall ADG, variance was attributable to SL × BL, SL × BL × TRT, and TRT at 38%, 35%, and 25%, respectively. Variables of SL, BL, SL × BL × TRT, and TRT accounted for 38%, 23%, 23%, and 15% of the variance in final BW, respectively. Hind leg angle did not account for variance in average LS, ADG, or final BW. Overall, variance of average LS, overall ADG, and final BW were primarily attributed to stride length, body length, flooring type, and their interactions. Individual animal differences in structural conformation are related to cattle mobility and growth performance in slatted indoor facilities.

Method for discovering spatial arm positions with depth sensor data at low-performance devices

A method of arm aiming direction estimation for low performance Internet of Things devices is proposed. It uses Human Pose Estimation (HPE) algorithms for retrieving human skeleton key points. Having these key points, arm aiming directions model is calculated. Two well-known HPE methods (PoseNet and OpenPose) are examined. These algorithms have been tested and compared by the average angle of error. The system includes a Raspberry Pi 4B single-board computer and an Intel RealSense D435i depth sensor. The developed approach may be utilized in “smart home” gesture control systems.

Application of Artificial Intelligence and Computer Vision Technologies in Solving Problems of Automation of Processing and Recognition of Biological Objects

The article considers the application of artificial intelligence and computer vision technologies to solve the automation of processing and analysis of botanical micro and macro objects (images of fern spores). Also, there is a problem of developing software for a digital twin of an agrobot. The first problem is an interdisciplinary research aimed at solving applied and fundamental problems in botanical objects' biosystematics and studying microevolutionary processes using computer vision technologies, methods of intelligent image analysis, machine learning, and artificial intelligence. The article presents the developed software module FAST (Functional Automated System Tool) for solving the direct problem — performing measurements from images obtained by scanning electron microscopy, virtual herbaria image library, entomological collections, or images taken in a natural environment.
 The second problem is software development for the digital twin of the agrorobot, designed for precise mechanical processing of plants and soil. The proposed solution includes several components: the control unit — NVIDIA Jetson NANO computing module; the actuator — 6-axis robotic arm; the machine vision unit based on an Intel RealSense camera; the chassis unit — tracked tracks and software drivers and components for their control. The digital twin of the robot considers the environmental conditions and the landscape of the operation area.
 The use of ROS (Robot Operating System) allows minimal effort to transfer a digital model to a physical one (prototype and serial robot) without changing the source code. Furthermore, consideration of the environmental conditions during the programming stage provides opportunities for further development and testing of real-life mathematical models for device control.

Bin-Picking Solution for Randomly Placed Automotive Connectors Based on Machine Learning Techniques

This paper presents the development of a bin-picking solution based on low-cost vision systems for the manipulation of automotive electrical connectors using machine learning techniques. The automotive sector has always been in a state of constant growth and change, which also implies constant challenges in the wire harnesses sector, and the emerging growth of electric cars is proof of this and represents a challenge for the industry. Traditionally, this sector is based on strong human work manufacturing and the need arises to make the digital transition, supported in the context of Industry 4.0, allowing the automation of processes and freeing operators for other activities with more added value. Depending on the car model and its feature packs, a connector can interface with a different number of wires, but the connector holes are the same. Holes not connected with wires need to be sealed, mainly to guarantee the tightness of the cable. Seals are inserted manually or, more recently, through robotic stations. Due to the huge variety of references and connector configurations, layout errors sometimes occur during seal insertion due to changed references or problems with the seal insertion machine. Consequently, faulty connectors are dumped into boxes, piling up different types of references. These connectors are not trash and need to be reused. This article proposes a bin-picking solution for classification, selection and separation, using a two-finger gripper, of these connectors for reuse in a new operation of removal and insertion of seals. Connectors are identified through a 3D vision system, consisting of an Intel RealSense camera for object depth information and the YOLOv5 algorithm for object classification. The advantage of this approach over other solutions is the ability to accurately detect and grasp small objects through a low-cost 3D camera even when the image resolution is low, benefiting from the power of machine learning algorithms.

A Two-Stage Industrial Defect Detection Framework Based on Improved-YOLOv5 and Optimized-Inception-ResnetV2 Models

Aiming to address the currently low accuracy of domestic industrial defect detection, this paper proposes a Two-Stage Industrial Defect Detection Framework based on Improved-YOLOv5 and Optimized-Inception-ResnetV2, which completes positioning and classification tasks through two specific models. In order to make the first-stage recognition more effective at locating insignificant small defects with high similarity on the steel surface, we improve YOLOv5 from the backbone network, the feature scales of the feature fusion layer, and the multiscale detection layer. In order to enable second-stage recognition to better extract defect features and achieve accurate classification, we embed the convolutional block attention module (CBAM) attention mechanism module into the Inception-ResnetV2 model, then optimize the network architecture and loss function of the accurate model. Based on the Pascal Visual Object Classes 2007 (VOC2007) dataset, the public dataset NEU-DET, and the optimized dataset Enriched-NEU-DET, we conducted multiple sets of comparative experiments on the Improved-YOLOv5 and Inception-ResnetV2. The testing results show that the improvement is obvious. In order to verify the superiority and adaptability of the two-stage framework, we first test based on the Enriched-NEU-DET dataset, and further use AUBO-i5 robot, Intel RealSense D435 camera, and other industrial steel equipment to build actual industrial scenes. In experiments, a two-stage framework achieves the best performance of 83.3% mean average precision (mAP), evaluated on the Enriched-NEU-DET dataset, and 91.0% on our built industrial defect environment.

Intelligent System for Estimation of the Spatial Position of Apples Based on YOLOv3 and Real Sense Depth Camera D415

Despite the great possibilities of modern neural network architectures concerning the problems of object detection and recognition, the output of such models is the local (pixel) coordinates of objects bounding boxes in the image and their predicted classes. However, in several practical tasks, it is necessary to obtain more complete information about the object from the image. In particular, for robotic apple picking, it is necessary to clearly understand where and how much to move the grabber. To determine the real position of the apple relative to the source of image registration, it is proposed to use the Intel Real Sense depth camera and aggregate information from its depth and brightness channels. The apples detection is carried out using the YOLOv3 architecture; then, based on the distance to the object and its localization in the image, the relative distances are calculated for all coordinates. In this case, to determine the coordinates of apples, a transition to a symmetric coordinate system takes place by means of simple linear transformations. Estimating the position in a symmetric coordinate system allows estimating not only the magnitude of the shift but also the location of the object relative to the camera. The proposed approach makes it possible to obtain position estimates with high accuracy. The approximate root mean square error is 7–12 mm, depending on the range and axis. As for precision and recall metrics, the first is 100% and the second is 90%.

Human Position Detection Based on Depth Camera Image Information in Mechanical Safety

The devices used for human position detection in mechanical safety mainly include safety light curtain, safety laser scanner, safety pad, and vision system. However, these devices may be bypassed when used, and human or equipment cannot be distinguished. To solve this problem, a depth camera is proposed as a human position detection device in mechanical safety. The process of human position detection based on depth camera image information is given; it mainly includes image information acquisition, human presence detection, and distance measurement. Meanwhile, a human position detection method based on Intel RealSense depth camera and MobileNet-SSD algorithm is proposed and applied to robot safety protection. The result shows that the image information collected by the depth camera can detect the human position in real time, which can replace the existing mechanical safety human position detection device. At the same time, the depth camera can detect only human but not mobile devices and realize the separation and early warning of people and mobile devices.

A Deep Learning-Based Novel Approach for Weed Growth Estimation

Automation of agricultural food production is growing in popularity in scientific communities and industry. The main goal of automation is to identify and detect weeds in the crop. Weed intervention for the duration of crop establishment is a serious difficulty for wheat in North India. The soil nutrient is important for crop production. Weeds usually compete for light, water and air of nutrients and space from the target crop. This research paper assesses the growth rate of weeds due to macronutrients (nitrogen, phosphorus and potassium) absorbed from various soils (fertile, clay and loamy) in the rabi crop field. The weed image data have been collected from three different places in Madhya Pradesh, India with 10 different rabi crops (Maize, Lucerne, Cumin, Coriander, Wheat, Fenugreek, Gram, Onion, Mustard and Tomato) and 10 different weeds (Corchorus Capsularis, Cynodondactylon, Chloris barbata, Amaranthaceae, Argemone mexicana, Carthamus oxyacantha, Capsella bursa Pastoris, Chenopodium Album, Dactyloctenium aegyptium and Convolvulus Ravens). Intel Real Sense LiDAR digital camera L515 and Canon digital SLR DIGICAM EOS 850 D 18-55IS STM cameras were mounted over the wheat crop in 10 × 10 square feet area of land and 3670 different weed images have been collected. The 2936 weed images were used for training and 734 images for testing and validation. The Efficient Net-B7 and Inception V4 architectures have been used to train the model that has provided accuracy of 97% and 94% respectively. The Image classification using Inspection V4 was unsuccessful with less accurate results as compared to EfficientNet-B7.

Globally Convergent Observer for the Rigid Body System on SE(3) in Presence of Intermittent Measurements

We propose a globally asymptotically convergent hybrid observer for the rigid body rotation and translation system evolving on the special Euclidean group SE(3) in the presence of intermittent measurements of the pose and continuous measurements of the velocities. We embed the system into an ambient Euclidean space and design an observer on the Euclidean space for global convergence.We perform numerical simulation of the proposed observer to show convergence.We also perform simulation using data collected from an Intel Realsense T265 camera.

Automated Quantification of Occupant Posture and Shoulder Belt Fit Using Safety Specific Key Points

Virtual evaluation of automotive safety with variation in occupant posture and shoulder belt fit is gaining importance, and there is a need of methods facilitating analysis of occupant postures in driving studies. This study is aimed to develop an AI-based computer vision method to automatically quantify occupant posture and shoulder belt position over time in a car. Traceable defined key points on the occupant were related with the shoulder belt and quantified over time in real 3D coordinates by predefined key measurements, utilising the underlying spatial information of a Intel RealSense 3D Camera. The key points are defined as traceable key points relevant to relate the occupant to the vehicle environment and to estimate shoulder belt position. Key point prediction results suggest an average deviation of around 1cm per coordinate, which enable a reliable spatial categorization of the respective tracked occupant by analyzing the key measurements. This method providing continuous information of the occupant position and belt fit will be useful to identify common occupant postures as well as more extreme postures, to be used for expanding variations in postures for vehicle safety assessments.

Auto-Refining 3D Mesh Reconstruction Algorithm From Limited Angle Depth Data

Despite 3D object reconstruction using a single perspective being a rapidly developing field, the majority of research is focused around a single static object reconstruction from a synthetically generated dataset. This leaves a major knowledge gap when it comes to morphing 3D object reconstruction from an imperfect real world frame. As a solution to this problem, we introduce a three-staged deep auto-refining adversarial neural network architecture that can denoise and refine real-world depth sensor data captured using Intel Realsense devices for a full human body posture reconstruction. The proposed solution was able to achieve results which are on par with other state-of-the-art approaches in both Earth Mover’s and Chamfer distances, 0.059 and 0.079 respectively, while having the benefit of reconstructing from mask-less real world depth frames.With visual inspection of the reconstructed point-cloud suggesting great adaptation capabilities to the majority of real world depth sensor noise deformities for both LiDAR and structured light depth sensors.

Real-Time Detection of Ripe Oil Palm Fresh Fruit Bunch Based on YOLOv4

Fresh Fruit Bunch (FFB) is the main ingredient in palm oil production. Harvesting FFB from oil palm trees at its peak ripeness stage is crucial to maximise the oil extraction rate (OER) and quality. In current harvesting practices, misclassification of FFB ripeness can occur due to human error, resulting in OER loss. Therefore, a vision-based ripe FFB detection system is proposed as the first step in a robotic FFB harvesting system. In this work, live camera input is fed into a Convolutional Neural Network (CNN) model known as YOLOv4 to detect the presence of ripe FFBs on the oil palm trees in real time. Once a ripe FFB is detected on the tree, a signal is transmitted via ROS to the robotic harvesting mechanism. To train the YOLOv4 model, a large number of ripe FFB images were collected using an Intel Realsense Camera D435 with the resolution of 1920×1080. During data acquisition, a subject matter expert assisted in classifying the FFBs in terms of ripe or unripe. During the testing phase, the result of the mean Average Precision (mAP) and recall are 87.9 % and 82 % as the detection is fulfilled the Intersect over Union (IoU) with more than 0.5 after 2000 iterations and the system operated at the real-time speed of roughly 21 Frame Per Second (FPS).

Разработка Python приложения для распознавания жестов рук из видеопотока RGB и RGBD камер

Gesture recognition systems have changed a lot recently, due to the development of modern data capture devices (sensors) and the development of new recognition algorithms. The article presents the results of a study for recognizing static and dynamic hand gestures from a video stream from RGB and RGBD cameras, namely from the Logitech HD Pro Webcam C920 webcam and from the Intel RealSense D435 depth camera. Software implementation is done using Python 3.6 tools. Open source Python libraries provide robust implementations of image processing and segmentation algorithms. The feature extraction and gesture classification subsystem is based on the VGG-16 neural network architecture implemented using the TensorFlow and Keras deep learning frameworks. The technical characteristics of the cameras are given. The algorithm of the application is described. The research results aimed at comparing data capture devices under various experimental conditions (distance and illumination) are presented. Experimental results show that using the Intel RealSense D435 depth camera provides more accurate gesture recognition under various experimental conditions.

Trellis wire detection for obstacle avoidance in apple orchards

Modern high-density apple orchards utilize trellis support systems to provide robust tree architecture that improves overall orchard health, supports increasing crop load, and enhances the opportunity for orchard automation. However, these trellis wires can create a significant issue when agricultural robots interact with the apple trees to pick fruits or prune branches. These issues can interfere with the robotic field operations due to collision, harming the robot, tree, or trellis wire itself. This study localizes trellis wires to assist robotic apple harvesting using RGB images captured with an Intel RealSense 435D camera. Trellis wires are positioned horizontally, have a very thin diameter, and are only partially visible throughout the image. In this study, a Mask RCNN-based model was trained to detect the visible segments of trellis wires in the orchard environment using images collected for robotic apple harvesting. These segments were then processed using Hough Transform to estimate the wire location even when the wires were occluded by foliage or other canopy structures. As the trellis wires have linear geometry, their slope and intercept were computed and compared with the ground truth reading for performance evaluation. The proposed system detected the trellis wires in 81 of the 100 test images. In more than 88% of the cases when a trellis was detected, the intercept measured was within 9 mm of the ground truth. Similarly, slope error was within 9° for > 88% of the time. In the future, a field test with a robotic end-effector will be conducted further to validate the outcome and usefulness of this study.

Object Detection, Localization and Tracking-Based Deep Learning for Smart Wheelchair

In this article, we present our work regarding the development of advanced driver-assistance systems for an electric-powered wheelchair. Our project aims at improving the autonomy of people with reduced mobility. After conducting a clinical study, we identified several use-cases. In this paper, we introduce the detection, localization and tracking of points of interest in the immediate surroundings of the chair in an indoor environment, i.e.: doors, handles, light switches, etc. The aim is not only to improve perception around the chair but also to enable semi-autonomous driving towards these targets. First, we introduce a repurposing of YOLOv3, the object detection algorithm, to our use case. Then, we show our use of the Intel Realsense camera for depth estimation. Finally, we describe our adaptation of the SORT algorithm to track 3D interest points. To validate our approach, we realized several experiments in a controlled indoor environment. The detection, distance estimation, and tracking pipeline is tested using our custom dataset. This includes corridors, doors, handles, and switches. One of the scenarios studied to validate the proposed platform includes not only the detection and tracking of objects but also the movement of the wheelchair towards one of these points of interest.

Breed recognition and estimation of live weight of cattle based on methods of machine learning and computer vision

A method of measuring cattle parameters using neural network methods of image processing was proposed. To this end, several neural network models were used: a convolutional artificial neural network and a multilayer perceptron. The first is used to recognize a cow in a photograph and identify its breed followed by determining its body dimensions using the stereopsis method. The perceptron was used to estimate the cow's weight based on its breed and size information. Mask RCNN (Mask Regions with CNNs) convolutional network was chosen as an artificial neural network. To clarify information on the physical parameters of animals, a 3D camera (Intel RealSense D435i) was used. Images of cows taken from different angles were used to determine the parameters of their bodies using the photogrammetric method. The cow body dimensions were determined by analyzing animal images taken with synchronized cameras from different angles. First, a cow was identified in the photograph and its breed was determined using the Mask RCNN convolutional neural network. Next, the animal parameters were determined using the stereopsis method. The resulting breed and size data were fed to a predictive model to determine the estimated weight of the animal. When modeling, Ayrshire, Holstein, Jersey, Krasnaya Stepnaya breeds were considered as cow breeds to be recognized. The use of a pre-trained network with its subsequent training applying the SGD algorithm and Nvidia GeForce 2080 video card has made it possible to significantly speed up the learning process compared to training in a CPU. The results obtained confirm the effectiveness of the proposed method in solving practical problems.

OVERVIEW OF EQUIPMENT 
 FOR CAPTURING 3D IMAGES WITH SUBSEQUENT TRANSMISSION THROUGH A COMMUNICATION NETWORK

The relatively recent advent of the publicly available consumer RGB-D sensors has enabled a wide range of functionality to interact with 3D technology. For example, devices such as Microsoft Kinect v1, Microsoft Kinect v2, and Intel Realsense F200 are currently available RGB-D sensors using Structured light and Time of Flight technologies for getting information about pixel depth. The article contains an extensive comparison of RGB-D devices in terms of the resolution of RGB and Depth cameras, latency (the time required to build a depth map and its processing), and also compares viewing angles, required USB interfaces, sensor sizes, etc. The main goal of the article is to provide a complete and visual comparison of devices that can potentially be used to capture three-dimensional images with their subsequent transmission by communication networks.

Sistema de detección de robots humanoides en ambientes semiestructurados basado en visión estereoscópica

El presente trabajo propone el diseño, desarrollo e implementación de un sistema de visión artificial basado en vídeo estéreo, el cual es ejecutado en un sistema embebido, con el fin de identificar a robots humanoides dentro de un área semiestructurada. El sistema embebido utiliza una cámara RealSense de Intel que además de poder obtener distancias hacia los objetos debido a la visión estéreo que posee, es capaz de discriminar información después de un umbral de distancia determinado por el usuario, eliminando los objetos de fondo de la escena y mediante reconocimiento de imágenes basado en una red neuronal convolucional reconoce a robots humanoides dentro de esta. La aplicación del sistema se basa en el concurso Robocup Humanoid League, donde dos equipos de robots juegan fútbol, por lo que además de reconocer a robots humanoides a diferentes ángulos y distancias, el sistema puede clasificar a los robots reconocidos como compañeros o contrincantes (dependiendo del reconocimiento de marcas de color que estos porten), emulando que, a futuro, el sistema propuesto estará montado sobre otro robot humanoide.

Fusion in dissimilarity space for RGB-D person re-identification

Person re-identification (Re-id) is the task of recognizing people across different non-overlapping sensors of a camera network. Despite the recent advances with deep learning (DL) models for multi-modal fusion, state-of-the-art Re-id approaches fail to leverage the depth guided contextual information to dynamically select the most discriminant convolutional filters for better feature embedding and inference. Thanks to low cost modern RGB-D sensors (e.g. Microsoft Kinect and Intel RealSense Depth camera) that avail us with different modalities such as illumination invariant high-quality depth images, RGB images and skeleton information can be obtained simultaneously. State-of-the-art Re-id approaches utilize multi-modal fusion in feature space where the chances of fused noisy features to dominate the final recognition process is high. In this paper, we address this issue by exploiting the advantage of using an effective fusion technique in dissimilarity space. Given a query RGB-D image of an individual, two CNNs are separately trained with 3-channel RGB and 4-channel RGB-D images to produce two different feature embeddings required for pair-wise matching with embeddings for reference images, where dissimilarity scores w.r.t reference images from both modalities are fused together for final ranking. Additionally, lack of a proper RGB-D Re-id dataset prompts us to contribute a new RGB-D Re-id dataset named SUCVL RGBD-ID, including RGB and depth images of 58 identities from three cameras where one camera was installed in poor illumination conditions and the remaining two cameras were installed in two different indoor locations with different indoor lighting environments. Extensive experimental analysis on our dataset and two publicly available datasets show the effectiveness of our proposed method. Moreover, our proposed method is general and can be applied to a multitude of different RGB-D based applications.

Colored 3D Path Extraction Based on Depth-RGB Sensor for Welding Robot Trajectory Generation

The necessity for intelligent welding robots that meet the demand in real industrial production, according to the objectives of Industry 4.0, has been supported owing to the rapid development of computer vision and the use of new technologies. To improve the efficiency in weld location for industrial robots, this work focuses on trajectory extraction based on color features identification on three-dimensional surfaces acquired with a depth-RGB sensor. The system is planned to be used with a low-cost Intel RealSense D435 sensor for the reconstruction of 3D models based on stereo vision and the built-in color sensor to quickly identify the objective trajectory, since the parts to be welded are previously marked with different colors, indicating the locations of the welding trajectories to be followed. This work focuses on 3D color segmentation with which the points of the target trajectory are segmented by color thresholds in HSV color space and a spline cubic interpolation algorithm is implemented to obtain a smooth trajectory. Experimental results have shown that the RMSE error for V-type butt joint path extraction was under 1.1 mm and below 0.6 mm for a straight butt joint; in addition, the system seems to be suitable for welding beads of various shapes.