Depth Camera Research Articles

Research on Indoor Automatic Path Planning Algorithm for Robots

The field of robotics has significantly advanced, especially in indoor autonomous navigation, with applications in households, offices, factories, and tourism. This research focuses on developing and optimizing algorithms for indoor autonomous path planning, aiming to enhance robots' ability to navigate efficiently and safely in various indoor environments. The research adopts a comprehensive methodological approach, beginning with a literature review to understand current state-of-the-art techniques in path planning and obstacle avoidance. Novel algorithms were designed and implemented, focusing on efficiency and real-time performance. These algorithms were tested in simulation environments and validated on actual robotic platforms. Performance metrics such as path efficiency, computational time, and obstacle avoidance success rates were used to assess the algorithms. The study evaluated global path planning algorithms like A* and Dijkstra’s, and local path planning algorithms such as Rapidly-exploring Random Trees (RRT) and Dynamic Window Approach (DWA). A* and Dijkstra's algorithms proved effective for global planning but required optimization for real-time applications. RRT excelled in complex environments but needed improvements for path optimality. DWA provided robust real-time obstacle avoidance. Hybrid approaches combining these algorithms showed enhanced performance, balancing global and local planning strengths. Machine learning techniques further improved adaptability and efficiency. The integration of advanced sensor technologies and accurate map construction methods is crucial for effective indoor navigation. LIDAR, ultrasonic sensors, and depth cameras were key in providing precise environmental perception. Hybrid maps combining grid and topological approaches offered detailed local information and efficient long-distance planning. Optimization techniques like parameter tuning and algorithm fusion, along with machine learning, significantly enhanced algorithm performance. Future research should explore intelligent algorithms, multi-robot collaboration, and human-robot interaction to further advance the field.

Automated body measurement of beef cattle based on keypoint detection and local point cloud clustering

Abstract Body size parameters of beef cattle are crucial for assessing growth status and breeding value. In actual farming environments, the various postures of beef cattle and complex backgrounds can affect the accuracy and stability of non-contact body measurement methods. Therefore, this paper proposes a novel method called the cattle body measurement method (CBMM), which combines keypoint detection with local point cloud clustering. First, a keypoint detection model based on YOLOv8-SimBiFPN is constructed. This model enhances the feature extraction and fusion capabilities of YOLOv8-pose by introducing SimAM and BiFPN into the backbone and neck networks, respectively, and realizes 2D keypoint detection for beef cattle in various postures. Second, a 3D keypoint-locating algorithm based on Density-based spatial clustering of applications with noise (DBSCAN) is proposed. This algorithm utilizes 2D keypoints, depth maps and camera parameters to generate local point clouds, which are then clustered using DBSCAN to segment cattle body point clouds, thereby relocating the 3D keypoints based on their positional features. Finally, body size parameters are calculated based on the 3D keypoints and distance formulae. In our experiment, the mean average precision (mAP@0.5) of YOLOv8-SimBiFPN reached 99.1% on an Angus beef cattle keypoint detection dataset. The mean absolute percentage errors for measuring beef cattle withers height, hip height, body depth, body length, and oblique body length using the CBMM were 4.37%, 4.96%, 6.47%, 4.84%, and 4.14%, respectively. In summary, our method can achieve non-contact body measurement for beef cattle in a free-moving state with high accuracy and stability.

Design and Testing of a Fruit Tree Variable Spray System Based on ExG-AABB

This paper addresses the issue of pesticide waste and low utilization rates resulting from traditional plant protection via spraying operations, which apply equal dosages to different targets or to different parts of the same target. To tackle this problem, we designed a variable fruit tree spraying system based on the ExG-AABB (excess green and axis-aligned bounding box) algorithm. We used a Kinect depth camera to capture information about the fruit tree canopy and constructed a spray flow model using pulse width modulation and variable spray control technology. Variable multi-nozzle spraying was guided by combining this canopy data. We evaluated the accuracy of each model in calculating canopy volume by comparing the coefficient of determination (R2) and root mean square error (RMSE) of the ExG-AABB with the slice convex hull method, voxel method, three-dimensional alpha-shape method, and QuickHull method. The ExG-AABB algorithm had the highest R2 value (0.9334) and the lowest RMSE value (0.0353 m3) among the five models, indicating that it most accurately reflects the true volume of the fruit tree canopy. This validates the effectiveness of the ExG-AABB algorithm in calculating canopy volume. We established a correlation model between canopy volume and spray volume, designed a canopy-adaptive layering method based on point cloud processing, and achieved precise calculation of nozzle flow. Comparative field experiments were conducted to analyze the spray coverage rate and observed flow, thereby evaluating the spraying effect of this variable spraying system. The experimental results showed that compared to conventional continuous spraying, this variable spraying system not only achieves more uniform spray coverage but also significantly reduces pesticide usage by 48.1%. Furthermore, through system optimization, the average coverage rate of the middle layer of the canopy decreased by 17.53%, effectively reducing the phenomenon of overlapping spraying from multiple nozzles and improving spraying efficiency.

Depth-based interactive assistive system for dementia care

Dementia often leads to reduced autonomy and the need for help with activities of daily living, such as the use of the lavatory. In response, we present ToiletHelp, a novel system to provide automated, step-by-step guidance to individuals with dementia when using the lavatory. In such an intimate environment, it is crucial to maintain the user’s privacy, which ToiletHelp achieves by sensing only depth maps. ToiletHelp uses a depth sensor to detect when the user is in need of assistance and tailors the guidance accordingly, using a combination of verbal and visual prompts. Our main contribution is the development of the first assistance system for people with dementia in such a sensitive environment as the lavatory. In addition, what sets ToiletHelp apart is its comprehensive involvement of 60 participants, 33 end-users and 27 healthcare workers, throughout its development process. A user-centered evaluation validates the interaction design, demonstrating enhanced feelings of security and independence among older adults, as well as successful comprehension of instructions by participants with dementia. Caregivers report high satisfaction with the system’s modalities and confirm its effectiveness in reducing their workload while assisting end-users. These findings not only validate the success of ToiletHelp and underscore its potential benefits but also contribute to the field of human-computer interaction design. Our comprehensive analysis of the results provides valuable information for researchers developing solutions for individuals with dementia, which is especially valuable given the limited existing literature on the subject.

Research on Automatic Recharging Technology for Automated Guided Vehicles Based on Multi-Sensor Fusion

Automated guided vehicles (AGVs) play a critical role in indoor environments, where battery endurance and reliable recharging are essential. This study proposes a multi-sensor fusion approach that integrates LiDAR, depth cameras, and infrared sensors to address challenges in autonomous navigation and automatic recharging. The proposed system overcomes the limitations of LiDAR’s blind spots in near-field detection and the restricted range of vision-based navigation. By combining LiDAR for precise long-distance measurements, depth cameras for enhanced close-range visual positioning, and infrared sensors for accurate docking, the AGV’s ability to locate and autonomously connect to charging stations is significantly improved. Experimental results show a 25% increase in docking success rate (from 70% with LiDAR-only to 95%) and a 70% decrease in docking error (from 10 cm to 3 cm). These improvements demonstrate the effectiveness of the proposed sensor fusion method, ensuring more reliable, efficient, and precise operations for AGVs in complex indoor environments.

Method for Bottle Opening with a Dual-Arm Robot.

This paper introduces a novel approach to robotic assistance in bottle opening using the dual-arm robot TIAGo++. The solution enhances accessibility by addressing the needs of individuals with injuries or disabilities who may require help with common manipulation tasks. The aim of this paper is to propose a method involving vision, manipulation, and learning techniques to effectively address the task of bottle opening. The process begins with the acquisition of bottle and cap positions using an RGB-D camera and computer vision. Subsequently, the robot picks the bottle with one gripper and grips the cap with the other, each by planning safe trajectories. Then, the opening procedure is executed via a position and force control scheme that ensures both grippers follow the unscrewing path defined by the cap thread. Within the control loop, force sensor information is employed to control the vertical axis movements, while gripper rotation control is achieved through a Deep Reinforcement Learning (DRL) algorithm trained to determine the optimal angle increments for rotation. The results demonstrate the successful training of the learning agent. The experiments confirm the effectiveness of the proposed method in bottle opening with the TIAGo++ robot, showcasing the practical viability of the approach.

Triple-Camera Rectification for Depth Estimation Sensor.

In this study, we propose a novel rectification method for three cameras using a single image for depth estimation. Stereo rectification serves as a fundamental preprocessing step for disparity estimation in stereoscopic cameras. However, off-the-shelf depth cameras often include an additional RGB camera for creating 3D point clouds. Existing rectification methods only align two cameras, necessitating an additional rectification and remapping process to align the third camera. Moreover, these methods require multiple reference checkerboard images for calibration and aim to minimize alignment errors, but often result in rotated images when there is significant misalignment between two cameras. In contrast, the proposed method simultaneously rectifies three cameras in a single shot without unnecessary rotation. To achieve this, we designed a lab environment with checkerboard settings and obtained multiple sample images from the cameras. The optimization function, designed specifically for rectification in stereo matching, enables the simultaneous alignment of all three cameras while ensuring performance comparable to traditional methods. Experimental results with real camera samples demonstrate the benefits of the proposed method and provide a detailed analysis of unnecessary rotations in the rectified images.

A 3D Vision-Based Weld Seam Extraction Method for Arbitrarily Placed Spatial Planar Weldments

Vision-based weld seam extraction poses a significant challenge for weldments with complex spatial structures in automated welding. Existing research primarily focuses on identifying weld seams from weldments with given positions and postures, while practical weld path planning requires multiple weld seams identified within arbitrarily placed weldments. This paper proposes a methodology that identifies weld seams from arbitrarily placed spatial planar weldments in a single run. First, by introducing a turntable calibrated with respect to a 3D camera, we perform 3D reconstruction on an arbitrarily placed spatial planar weldment. Second, an improved RANSAC algorithm based on Euclidean clustering is proposed to carry out plane segmentation, focusing on segmentation accuracy. Finally, we present a novel weld seam extraction algorithm leveraging the half-edge data structure to efficiently compute weld seams from the segmented planes. The experiments conducted in this study demonstrate that the average segmentation errors (as an indirect indicator of weld seam extraction error) are reduced by 90.3% to 99.8% over conventional segmentation methods, and the standard deviations are reduced by 64.8% to 97.0%.

MonoVisual3DFilter: 3D tomatoes’ localisation with monocular cameras using histogram filters

Abstract Performing tasks in agriculture, such as fruit monitoring or harvesting, requires perceiving the objects’ spatial position. RGB-D cameras are limited under open-field environments due to lightning interferences. So, in this study, we state to answer the research question: “How can we use and control monocular sensors to perceive objects’ position in the 3D task space?” Towards this aim, we approached histogram filters (Bayesian discrete filters) to estimate the position of tomatoes in the tomato plant through the algorithm MonoVisual3DFilter. Two kernel filters were studied: the square kernel and the Gaussian kernel. The implemented algorithm was essayed in simulation, with and without Gaussian noise and random noise, and in a testbed at laboratory conditions. The algorithm reported a mean absolute error lower than 10 mm in simulation and 20 mm in the testbed at laboratory conditions with an assessing distance of about 0.5 m. So, the results are viable for real environments and should be improved at closer distances.

A Fast Multi-Scale of Distributed Batch-Learning Growing Neural Gas for Multi-Camera 3D Environmental Map Building.

Biologically inspired intelligent methods have been applied to various sensing systems in order to extract features from a huge size of raw sensing data. For example, point cloud data can be applied to human activity recognition, multi-person tracking, and suspicious person detection, but a single RGB-D camera is not enough to perform the above tasks. Therefore, this study propose a 3D environmental map-building method integrating point cloud data measured via multiple RGB-D cameras. First, a fast multi-scale of distributed batch-learning growing neural gas (Fast MS-DBL-GNG) is proposed as a topological feature extraction method in order to reduce computational costs because a single RGB-D camera may output 1 million data. Next, random sample consensus (RANSAC) is applied to integrate two sets of point cloud data using topological features. In order to show the effectiveness of the proposed method, Fast MS-DBL-GNG is applied to perform topological mapping from several point cloud data sets measured in different directions with some overlapping areas included in two images. The experimental results show that the proposed method can extract topological features enough to integrate point cloud data sets, and it runs 14 times faster than the previous GNG method with a 23% reduction in the quantization error. Finally, this paper discuss the advantage and disadvantage of the proposed method through numerical comparison with other methods, and explain future works to improve the proposed method.

Research on SLAM Localization Algorithm for Orchard Dynamic Vision Based on YOLOD-SLAM2

With the development of agriculture, the complexity and dynamism of orchard environments pose challenges to the perception and positioning of inter-row environments for agricultural vehicles. This paper proposes a method for extracting navigation lines and measuring pedestrian obstacles. The improved YOLOv5 algorithm is used to detect tree trunks between left and right rows in orchards. The experimental results show that the average angle deviation of the extracted navigation lines was less than 5 degrees, verifying its accuracy. Due to the variable posture of pedestrians and ineffective camera depth, a distance measurement algorithm based on a four-zone depth comparison is proposed for pedestrian obstacle distance measurement. Experimental results showed that within a range of 6 m, the average relative error of distance measurement did not exceed 1%, and within a range of 9 m, the maximum relative error was 2.03%. The average distance measurement time was 30 ms, which could accurately and quickly achieve pedestrian distance measurement in orchard environments. On the publicly available TUM RGB-D dynamic dataset, YOLOD-SLAM2 significantly reduced the RMSE index of absolute trajectory error compared to the ORB-SLAM2 algorithm, which was less than 0.05 m/s. In actual orchard environments, YOLOD-SLAM2 had a higher degree of agreement between the estimated trajectory and the true trajectory when the vehicle was traveling in straight and circular directions. The RMSE index of the absolute trajectory error was less than 0.03 m/s, and the average tracking time was 47 ms, indicating that the YOLOD-SLAM2 algorithm proposed in this paper could meet the accuracy and real-time requirements of agricultural vehicle positioning in orchard environments.

Development of a Real-Time Radiation Exposure Estimation Method Using a Depth Camera for Radiation Protection Education

X-ray fluoroscopy causes relatively high radiation exposure to physicians, radiation professionals, and patients. Understanding the behavior of scattered radiation is crucial for reducing occupational exposure. We developed a system for estimating radiation exposure during fluoroscopy by monitoring the position of the physician using a depth camera for radiation protection education. The dose distribution of scattered radiation in an X-ray room was simulated using Monte Carlo code. The data were displayed using augmented reality markers, and the dose at each joint point location was estimated using body tracking. Additional functions were created, such as displaying arbitrary two-dimensional cross-sections. The system performance ranged from 9.0 to 11.0 FPS with or without motion and a protective apron. The estimated doses were 0.93 to 1.21 times the measured doses for all joint points, except for the chest and pelvis. The estimated doses for the chest and pelvis were lower than the measured dose, with the minimum values being 0.72 and 0.60 times lower for the chest and pelvis, respectively. The system provides valuable insight into the estimation of radiation dose at joint points based on the physician’s position and movements, the physician’s optimal fluoroscopy location, and warning of dangerous exposure doses.

Visual-Inertial RGB-D SLAM with Encoder Integration of ORB Triangulation and Depth Measurement Uncertainties.

In recent years, the accuracy of visual SLAM (Simultaneous Localization and Mapping) technology has seen significant improvements, making it a prominent area of research. However, within the current RGB-D SLAM systems, the estimation of 3D positions of feature points primarily relies on direct measurements from RGB-D depth cameras, which inherently contain measurement errors. Moreover, the potential of triangulation-based estimation for ORB (Oriented FAST and Rotated BRIEF) feature points remains underutilized. To address the singularity of measurement data, this paper proposes the integration of the ORB features, triangulation uncertainty estimation and depth measurements uncertainty estimation, for 3D positions of feature points. This integration is achieved using a CI (Covariance Intersection) filter, referred to as the CI-TEDM (Triangulation Estimates and Depth Measurements) method. Vision-based SLAM systems face significant challenges, particularly in environments, such as long straight corridors, weakly textured scenes, or during rapid motion, where tracking failures are common. To enhance the stability of visual SLAM, this paper introduces an improved CI-TEDM method by incorporating wheel encoder data. The mathematical model of the encoder is proposed, and detailed derivations of the encoder pre-integration model and error model are provided. Building on these improvements, we propose a novel tightly coupled visual-inertial RGB-D SLAM with encoder integration of ORB triangulation and depth measurement uncertainties. Validation on open-source datasets and real-world environments demonstrates that the proposed improvements significantly enhance the robustness of real-time state estimation and localization accuracy for intelligent vehicles in challenging environments.

NeRF-OR: neural radiance fields for operating room scene reconstruction from sparse-view RGB-D videos.

RGB-D cameras in the operating room (OR) provide synchronized views of complex surgical scenes. Assimilation of this multi-view data into a unified representation allows for downstream tasks such as object detection and tracking, pose estimation, and action recognition. Neural radiance fields (NeRFs) can provide continuous representations of complex scenes with limited memory footprint. However, existing NeRF methods perform poorly in real-world OR settings, where a small set of cameras capture the room from entirely different vantage points. In this work, we propose NeRF-OR, a method for 3D reconstruction of dynamic surgical scenes in the OR. Where other methods for sparse-view datasets use either time-of-flight sensor depth or dense depth estimated from color images, NeRF-OR uses a combination of both. The depth estimations mitigate the missing values that occur in sensor depth images due to reflective materials and object boundaries. We propose to supervise with surface normals calculated from the estimated depths, because these are largely scale invariant. We fit NeRF-OR to static surgical scenes in the 4D-OR dataset and show that its representations are geometrically accurate, where state of the art collapses to sub-optimal solutions. Compared to earlier work, NeRF-OR grasps fine scene details while training 30 faster. Additionally, NeRF-OR can capture whole-surgery videos while synthesizing views at intermediate time values with an average PSNR of 24.86dB. Last, we find that our approach has merit in sparse-view settings beyond those in the OR, by benchmarking on the NVS-RGBD dataset that contains as few as three training views. NeRF-OR synthesizes images with a PSNR of 26.72dB, a 1.7% improvement over state of the art. Our results show that NeRF-OR allows for novel view synthesis with videos captured by a small number of cameras with entirely different vantage points, which is the typical camera setting in the OR. Code is available via: github.com/Beerend/NeRF-OR .

Analysis of Factors Influencing the Precision of Body Tracking Outcomes in Industrial Gesture Control.

The body tracking systems on the current market offer a wide range of options for tracking the movements of objects, people, or extremities. The precision of this technology is often limited and determines its field of application. This work aimed to identify relevant technical and environmental factors that influence the performance of body tracking in industrial environments. The influence of light intensity, range of motion, speed of movement and direction of hand movement was analyzed individually and in combination. The hand movement of a test person was recorded with an Azure Kinect at a distance of 1.3 m. The joints in the center of the hand showed the highest accuracy compared to other joints. The best results were achieved at a luminous intensity of 500 lx, and movements in the x-axis direction were more precise than in the other directions. The greatest inaccuracy was found in the z-axis direction. A larger range of motion resulted in higher inaccuracy, with the lowest data scatter at a 100 mm range of motion. No significant difference was found at hand velocity of 370 mm/s, 670 mm/s and 1140 mm/s. This study emphasizes the potential of RGB-D camera technology for gesture control of industrial robots in industrial environments to increase efficiency and ease of use.

3D reconstruction and volume measurement of irregular objects based on RGB-D camera

Abstract To address the challenge of measuring volumes of irregular objects, this paper proposes a volume measurement method based on 3D point cloud reconstruction. The point clouds of the object with multiple angles are obtained from an RGB-D camera mounted on a robotic arm, and then are reconstructed to form a whole complete point cloud to calculate the volume of the object. Firstly, the robotic arm is controlled to move to four angles for capturing the original point clouds of the target. Then, by using the rotation and translation matrices obtained from the calibration block pre-registration, the point clouds data from the four angles are fused and reconstructed. Subsequently, the issue of missing bottom point cloud data is addressed using a bottom-filling algorithm. Following this, the efficiency of the point cloud volume calculation algorithm is enhanced through the application of AABB filtering. Finally, the reconstructed point cloud volume is calculated using a slicing algorithm that integrates 2D point cloud segmentation and point cloud sorting. Experimental results show that this method achieves a volume measurement accuracy of over 95% for irregular objects and exhibits good robustness.

Enhancing surgical navigation: a robust hand-eye calibration method for the Microsoft HoloLens 2.

Optical-see-through head-mounted displays have the ability to seamlessly integrate virtual content with the real world through a transparent lens and an optical combiner. Although their potential for use in surgical settings has been explored, their clinical translation is sparse in the current literature, largely due to their limited tracking capabilities and the need for manual alignment of virtual representations of objects with their real-world counterparts. We propose a simple and robust hand-eye calibration process for the depth camera of the Microsoft HoloLens2, utilizing a tracked surgical stylus fitted with infrared reflective spheres as the calibration tool. Using a Monte Carlo simulation and a paired-fiducial registration algorithm, we show that a calibration accuracy of 1.65mm can be achieved with as little as 6 fiducial points. We also present heuristics for optimizing the accuracy of the calibration. The ability to use our calibration method in a clinical setting is validated through a user study, with users achieving a mean calibration accuracy of 1.67mm in an average time of 42s. This work enables real-time hand-eye calibration for the Microsoft HoloLens2, without any need for a manual alignment process. Using this framework, existing surgical navigation systems employing optical or electromagnetic tracking can easily be incorporated into an augmented reality environment with a high degree of accuracy.

AI-Enabled Sensor Fusion of Time-of-Flight Imaging and mmWave for Concealed Metal Detection.

In the field of detection and ranging, multiple complementary sensing modalities may be used to enrich information obtained from a dynamic scene. One application of this sensor fusion is in public security and surveillance, where efficacy and privacy protection measures must be continually evaluated. We present a novel deployment of sensor fusion for the discrete detection of concealed metal objects on persons whilst preserving their privacy. This is achieved by coupling off-the-shelf mmWave radar and depth camera technology with a novel neural network architecture that processes radar signals using convolutional Long Short-Term Memory (LSTM) blocks and depth signals using convolutional operations. The combined latent features are then magnified using deep feature magnification to reveal cross-modality dependencies in the data. We further propose a decoder, based on the feature extraction and embedding block, to learn an efficient upsampling of the latent space to locate the concealed object in the spatial domain through radar feature guidance. We demonstrate the ability to detect the presence and infer the 3D location of concealed metal objects. We achieve accuracies of up to 95% using a technique that is robust to multiple persons. This work provides a demonstration of the potential for cost-effective and portable sensor fusion with strong opportunities for further development.

Enhancing the starting performance of elite swimmers through eight weeks of lateral entry training

ABSTRACT This study aims to explore the effects of 8 weeks of lateral entry training on the start performance of elite athletes and its impact on ordinary entry performance. Data were collected from 12 elite-level participants before and after the training using Kistler(9691A1) system with 3D force platform and high-speed cameras. 19 parameters of ordinary and lateral entry were assessed and analysed using paired samples t-tests. Compared to ordinary entry, the lateral entry performance was characterised by: decrease of both vertical force (5.77%) and maximum depth (6.09%) at a significance level (p < .01), and increase of flight distance (4.33%), first kick distance (3.90%) and push force (14.06%) at (p < .05). The results show that, compared to an ordinary entry technique, the lateral entry technique provides a 3.93%, 6.14%, and 8.29% reduction in time to reaching a 5 m, 10 m and 15 m distance (p ≤ .01), showing performance improvements at start times. The 8-week training on the lateral entry technique did not have a significant impact on the time performance of the ordinary entry technique (p > .050). The findings presented herein offer pertinent insights for elite-level athletes, coaches and future investigations aimed at enhancing swimming start techniques and performance.

Transparent Depth Completion Using Segmentation Features

Estimating the depth of transparent objects is one of the well-known challenges of RGB-D cameras due to the reflection and refraction effects. Previously, researchers propose to correct the depth of transparent objects by using their estimated segmentation masks, because it is possible to recover the internal depth of an object just from its boundary, as illustrated by those depth-from-silhouette methods. However, these algorithms only use segmentation masks. They ignore the internal structure information from the mask segmentation features, which we argue are more useful for transparent depth estimation. In this work, we demonstrate the effectiveness of segmentation features for transparent object depth estimation. We show that it is even possible to recover the depth map just from segmentation features, without any RGB or depth map as input. Based on this observation, we propose DualTransNet which uses segmentation features for transparent depth completion. In our DualTransNet, we feed segmentation features from an extra module to the main network for better depth completion quality. Extensive experiments have shown the superiority of segmentation features as well as the state-of-the-art performance of our network.