Depth Camera Research Articles

In-field performance evaluation of robotic arm developed for harvesting cotton bolls

Robotic harvesting of cotton bolls combines the advantages of manual picking and mechanical harvesting, including selective picking and reduced labor dependency. Thus, we designed, developed, and tested a cotton-picking robotic arm equipped with a depth camera, a 4-DoF manipulator, and a vacuum-type end-effector. Convolutional neural networks were implemented for cotton boll recognition, while an artificial neural network-assisted particle swarm optimization-based model was utilized for solving inverse kinematics. Linear actuators, controlled by a microcontroller and relays, positioned the end-effector for cotton boll harvesting. Laboratory testing resulted in picking 44–49 cotton bolls per hour, taking approximately 73.0 s per boll. During field testing, the robotic arm picked 40–42 bolls per hour with an average time of 86.0 s per boll. Picking efficiency ranged from 76.19% to 85.71% in the lab and 66.32% to 72.04% in the field. The complex structure of cotton plants sometimes hindered the end-effector’s path, resulting in lower picking efficiency during field testing. As observed in present study, manual pickers achieved a significantly higher picking rate, harvesting 1.381 kg of cotton per hour, nearly 4.7 times more than the robotic arm. Manual pickers achieved 98.04% picking efficiency, significantly higher than the robotic arm’s 68.25%. This significant difference in efficiency is due to the agility, adaptability, and decision-making abilities of human pickers. The developed robotic arm demonstrates competitive efficiency in the domain of harvesting robots, contributing to advancements in robotic agriculture. Future research and development efforts in this field hold the potential to enhance the efficiency and productivity of robotic cotton harvesters.

Estimation of aboveground biomass of Alfalfa using field robotics

Alfalfa is a high-yielding forage crop that is widely grown in the United States for grazing, hay and silage making. A proper maintenance of these grasslands is necessary to ensure optimum productivity and profits. The pre-harvest estimation of biomass yield helps in quantifying the profits and optimizing the forage allocation in advance. Most traditional methods of forage estimation are relatively laborious and time-consuming. Recent developments in contact and remote sensing technologies opened numerous paths for performing aboveground biomass estimation tasks with flexibility and easiness. This study focused on the development of crop height measurement systems for estimating the aboveground biomass yield of Alfalfa (Medicago sativa L.). Five different systems for measuring crop height were evaluated on their ability to estimate aboveground wet and dry biomass. The crop height measurement systems used in this study were Structure-from-Motion, Ultrasound Sensor and Ski, Inertial Measurement Unit and Ski, Inertial Measurement Units and Roller, and a Depth Camera. The results indicated that the system using the Inertial Measurement Unit sensor and ski (IMU-Ski) performed the best among ground-based methods (R2= 0.79; SeY= 3166 kg-wet/ha). The Structure-from-Motion (SfM) method using UAV also provided satisfactory results for biomass predictions (R2= 0.74; SeY= 2543 kg-wet/ha). The models based on IMU-Ski and UAV-based SfM methods were facilitated with vegetation coverage as an additional independent variable to evaluate their effect on biomass predictions. The results indicated that the vegetation coverage did not improve the predictions in any of these systems. Thus, the models based on only the crop height (IMU-Ski and UAV-SfM) were the recommended approaches for Alfalfa biomass estimations. The addition of data points for wide ranges of crop height and vegetation coverage is recommended for future studies to improve the results and ensure the adaptability of these systems in varying environmental conditions.

Robotic arm, 3D vision and geometric matching for an efficient implementation of inverse Patch Transfer Function method

The inverse patch transfer function method was developed to reconstruct the acoustic fields (velocity, pressure, intensity) at the surface of a vibrating source with a complex geometry using the concept of a virtual acoustic domain. This method can be used to manage sources with complex shapes in a non-anechoic acoustic environment, even in the presence of masking objects. However, this method is quite demanding experimentally as it requires a large number of measurement positions with good spatial positioning accuracy. This article presents a dedicated experimental platform developed to simplify and automate the entire measurement process. First, a 3D camera placed on a robotic arm constructs a point cloud characterising the source and its environment. Using a geometric matching algorithm, the CAD of the object under test is detected in the cloud of points, enabling the different coordinate systems (CAD, robot and physical part) to be matched. Secondly, the measurement points required to apply iPTF can be defined directly in the CAD and reproduced in near-real time on the test rig, regardless of the type of antenna used, providing an efficient means of applying iPTF and opening the way to many advanced applications.

Improved yolov5 algorithm combined with depth camera and embedded system for blind indoor visual assistance

To assist the visually impaired in their daily lives and solve the problems associated with poor portability, high hardware costs, and environmental susceptibility of indoor object-finding aids for the visually impaired, an improved YOLOv5 algorithm was proposed. It was combined with a RealSense D435i depth camera and a voice system to realise an indoor object-finding device for the visually impaired using a Raspberry Pi 4 B device as its core. The algorithm uses GhostNet instead of the YOLOv5s backbone network to reduce the number of parameters and computation of the model, incorporates an attention mechanism (coordinate attention), and replaces the YOLOv5 neck network with a bidirectional feature pyramid network to enhance feature extraction. Compared to the YOLOv5 model, the model size was reduced by 42.4%, number of parameters was reduced by 47.9%, and recall rate increased by 1.2% with the same precision. This study applied the improved YOLOv5 algorithm to an indoor object-finding device for the visually impaired, where the searched object was input by voice, and the RealSense D435i was used to acquire RGB and depth images to realize the detection and ranging of the object, broadcast the specific distance of the target object by voice, and assist the visually impaired in finding the object.

Robotic frameless brain biopsy system enhanced by facial mesh registration

AbstractIn this study, a new approach is presented that eliminates stereotactic frameworks and the use of markers, offering an alternative to traditional brain biopsy systems. The classical biopsy operation involves the registration of magnetic resonance (MR) and computed tomography (CT) information taken from the patient at different times. Typically, the surgeon's planning information, which takes an average of 4 h on MR, is transferred to CT, and the surgical operation commences. However, this approach necessitates two separate acquisitions (MR and CT), adversely affecting patient comfort and increasing the workload. In the proposed system, it is recommended to register MR‐Depth camera data instead of MR‐CT registration. To achieve this, a 3D face pattern is obtained from the data received from the depth camera attached to the robot arm and overlapped with the mesh obtained by segmentation of the MR. It was observed that registration with sub‐millimeter precision was achieved using the CMFreg surface registration technique.

Total body weight estimation by 3D camera systems: Potential high-tech solutions for emergency medicine applications? A scoping review.

Weight estimation is required in adult patients when weight-based medication must be administered during emergency care, as measuring weight is often not possible. Inaccurate estimations may lead to inaccurate drug dosing, which may cause patient harm. High-tech 3D camera systems driven by artificial intelligence might be the solution to this problem. The aim of this review was to describe and evaluate the published literature on 3D camera weight estimation methods. A systematic literature search was performed for articles that studied the use of 3D camera systems for weight estimation in adults. Data on the study characteristics, the quality of the studies, the 3D camera methods evaluated, and the accuracy of the systems were extracted and evaluated. A total of 14 studies were included, published from 2012 to 2024. Most studies used Microsoft Kinect cameras, with various analytical approaches to weight estimation. The 3D camera systems often achieved a P10 of 90% (90% of estimates within 10% of actual weight), with all systems exceeding a P10 of 78%. The studies highlighted a significant potential for 3D camera systems to be suitable for use in emergency care. The 3D camera systems offer a promising method for weight estimation in emergency settings, potentially improving drug dosing accuracy and patient safety. Weight estimates were satisfactorily accurate, often exceeding the reported accuracy of existing weight estimation methods. Importantly, 3D camera systems possess characteristics that could make them very appropriate for use during emergency care. Future research should focus on developing and validating this methodology in larger studies with true external and clinical validation.

Topical application of anti-inflammatory agents on burn wounds and their effect on healing

Advancements in the treatment of burns have considerably improved overall survival rates, but they have also highlighted several long-term sequelae related to the injury. Hypertrophic scars can impair function, reduce quality of life, and require multiple procedures as well as physical therapy. The purpose of this study was to investigate the effects of topical application of anti-inflammatory drugs in the treatment of burns. Up to 15 deep-partial thickness burns were created on the dorsum of four anesthetized swine. Subsequently, the burn wounds were randomized to receive amiloride, celecoxib, dexamethasone or minocycline mixed in a hydrogel. Silver sulfadiazine cream and blank hydrogel acted as controls. The animals were followed for 90 days and the wounds were assessed on days 3, 7, 14, 28 and 90 post-burn. Assessments were performed using photographs (macroscopic healing, contraction), laser-speckle imaging (blood perfusion), 3D camera (scarring, pigmentation), and histology (inflammation, burn depth, epidermal maturation). Inflammation was present in all burn wound histology specimens and peaked on day 7 in all groups. Regardless of the treatment the burns progressed and were deeper on day 7 in comparison to day 3. The burns were 50 – 80 % healed by day 14, but no significant differences were observed. No differences in epidermal thickness, rete ridges, contraction, hypopigmentation, or scar elevation were seen on day 90. Topical anti-inflammatories did not significantly decrease inflammation or mitigate burn wound progression in deep partial thickness burns in pigs. Also, no significant differences in wound healing or quality of healing were observed.

A Novel, Autonomous, Module-Based Surgical Lighting System

Optimal illumination of the surgical site is crucial for successful surgeries. Current lighting systems, however, suffer from significant drawbacks, particularly shadows cast by surgeons and operating room personnel. We introduce an innovative, module-based lighting system that actively prevents shadows using an array of swiveling, ceiling-mounted light modules. The intensity and orientation of these modules are autonomously controlled by novel algorithms utilizing multiple depth sensors mounted above the operating table. This paper presents our complete system, detailing the algorithms for autonomous control and the initial optimization of the light module setup. Unlike prior work that was largely conceptual and based on simulations, this study introduces a real prototype featuring 56 light modules and three depth sensors. We evaluate this prototype through measurements, semi-structured interviews (n=4), and an extensive quantitative user study (n=11). The evaluation focuses on illumination quality, shadow elimination, and suitability for open surgeries compared to conventional OR lights. Our results demonstrate that the novel lighting system and optimization algorithms outperform conventional OR lights for abdominal surgeries, according to both objective measures and subjective ratings by surgeons.

Dynamic Scene Representation for Docking in Urban Waters Using a Stereo Camera

Abstract To navigate and dock safely in urban waters, an autonomous ferry must assess its safe navigation paths by identifying navigable free space and potential obstacles. A deep understanding of its surroundings is crucial for situational awareness. Knowing which objects are static and dynamic is an indispensable ingredient for path planning algorithms and collision avoidance systems, and is especially useful for docking operations, as moving obstacles can interfere with the optimal path to the dock. To handle all types of obstacles, we present a novel dynamic scene representation for docking in urban waters using a short baseline stereo camera. Obstacles protruding from the water surface are represented with vertically oriented rectangles, known as stixels. The stixels are tracked over consecutive frames with dense optical flow. Together with stereo correspondences, their 3D motion is estimated. Stixels representing the same object are grouped together into line segments which are classified as either dynamic or static based on the motion of its associated stixels. Our dynamic representation presents objects in the 3D camera coordinate system as a bird’s eye view map. Additionally, stereo depth uncertainty can significantly impact depth estimates and is considered using a Kalman filter, providing smooth position and velocity estimates. We test our dynamic scene representation on a real docking scenario recorded with an autonomous ferry using a stereo camera. We evaluate the accuracy of the reconstructed 3D scene points and the velocity estimates of the dynamic objects using LiDAR and GNSS.

Unsupervised detection and mapping of sparks in the Electrochemical Discharge Machining (ECDM) process

Material removal in electrochemical discharge machining is caused by sparks generated in a tool immersed in an electrolytic solution. Being the primary machining agent in this non-contact machining process, mapping the locations of microscopic sparks is of great interest. The distribution of sparks around the tool surface could give insights into the machined hole properties like the size, surface finish, and depth as compared to the machining parameters such as applied voltage, tool size, rotation speed, and feed rate. This paper is focused on detecting sparks in photographs of the ECDM process captured using a high-speed camera. A novel approach of using a tri-planar reflective surface for capturing the location of sparks in 3D space using a 2D camera output is attempted. Traditional spark detection methods use neural network classifiers that need labeled data for training. This labeled data often comes from human intervention and contains inherent biases that could lead to misclassification. In this paper, an unsupervised spark detection methodology is demonstrated, which eliminates the need for human intervention and relies on the number of neighboring pixels detected in regions of interest (ROIs). The feasibility of using adaptive background modeling to classify thousands of images and identify the ones with sparks is demonstrated in this work. The masking technique combining effects of erosion followed by dilation is used to determine the exact boundaries of the spark contours in every image. Centroids for each of these contours are then transformed from the skewed coordinate system as observed in camera images, to a three-dimensional orthogonal coordinates system centered around the tool. The same procedure is repeated for various voltages to benchmark the distribution of sparks around a tool tip in an ECDM process.

BrightMice: a low-cost do-it-yourself instrument, designed for in vivo fluorescence mouse imaging.

In vivo fluorescent imaging represents a potent means for real-time probe quantification, facilitating insights into disease pathophysiology and therapeutic responses. Nonetheless, accurate signal quantification remains challenging due to inherent factors like light scattering and tissue absorption. Existing imaging systems, though sophisticated, often entail high costs and are typically restricted to well-funded laboratory settings. This study introduces BrightMice, an innovative in vivo fluorescent imaging system that harnesses 3D printing and consumer-grade digital cameras. Tailored for various fluorophores such as EYFP and E2-crimson, the system showcases both adaptability and effectiveness in detecting in vivo fluorescent signals in several reporter mouse strains. Comparative analyses against commercial instruments confirm BrightMice's sensitivity and underscore its potential to democratize in vivo fluorescence imaging. By providing a cost-effective and accessible solution, BrightMice stands to benefit diverse research environments.

Design and Implementation of Gesture Music Composition System based on RealSense and Recurrent Neural Network

Aiming at the problems of slow human-computer interaction and low accuracy of hand gesture recognition (HGR) in music composition systems such as human-computer interaction, gesture recognition, and algorithmic composition, this paper designs an HGR music composition system based on real-sense technology and recurrent neural network. The system uses an Intel RealSense 3D camera to shoot the user's hand and extract the information on the hand joint points with Smart Wearable Biosensors (SWBSs). The joint points are used to construct a three-dimensional model of the hand bone, and the joint points' equivalent distance and joint deflection are calculated by the joint point information to match the corresponding Curwen HGR. The matching music data is input into the recurrent neural network. GRU is combined with the Markov chain algorithm to complete the composition and avoid the phenomenon of gradient disappearance or gradient explosion. The HGR recognition function and composition algorithm are simulated and tested to verify the method's feasibility. Cohort verification investigations and efficacy evaluations of wearable biosensors are essential to support their clinical acceptability. The research results show that the HGR music composition system based on real-sense technology and recurrent neural networks can effectively identify the tester's HGRs and assist the tester in completing the composition using intelligent biosensors. The composition mode is close to the professional composition process.

ROS Gateway: Enhancing ROS Availability across Multiple Network Environments.

As the adoption of large-scale model-based AI grows, the field of robotics is undergoing significant changes. The emergence of cloud robotics, where advanced tasks are offloaded to fog or cloud servers, is gaining attention. However, the widely used Robot Operating System (ROS) does not support communication between robot software across different networks. This paper introduces ROS Gateway, a middleware designed to improve the usability and extend the communication range of ROS in multi-network environments, which is important for processing sensor data in cloud robotics. We detail its structure, protocols, and algorithms, highlighting improvements over traditional ROS configurations. The ROS Gateway efficiently handles high-volume data from advanced sensors such as depth cameras and LiDAR, ensuring reliable transmission. Based on the rosbridge protocol and implemented in Python 3, ROS Gateway is compatible with rosbridge-based tools and runs on both x86 and ARM-based Linux environments. Our experiments show that the ROS Gateway significantly improves performance metrics such as topic rate and delay compared to standard ROS setups. We also provide predictive formulas for topic receive rates to guide the design and deployment of robotic applications using ROS Gateway, supporting performance estimation and system optimization. These enhancements are essential for developing responsive and intelligent robotic systems in dynamic environments.

Zero-shot autonomous robot manipulation via natural language

Smart manufacturing revolutionizes advanced automation by leveraging deep learning and robotics technology. Nevertheless, deep learning implementation in manufacturing faces several challenges in preparing datasets, training a model, and adapting to various applications. Furthermore, programming languages are required to develop deep learning models and communicate with robots, which is not straightforward to quickly and precisely reflect human intentions in manufacturing processes. To alleviate the issue, this study proposes an autonomous zero-shot robot manipulation framework via natural language. Specifically, the Segmentation Anything Model (SAM), a promptable image segmentation model, first segments objects in images captured by an RGB-D camera, which are passed to Generative Pre-trained Transformers (GPTs), a Large Language Model (LLM), to identify and localize a designated object based on human commands given in natural language. Afterwards, the Robot Operating System (ROS) manipulates a robot to a localized position. With no additional programming or specialized skills in object localization and robot manipulation, the framework demonstrates an autonomous robot automation approach through intuitive communication between humans and machines through natural language.

Overview of Structure from Motion

In this paper is given an overview of tools and techniques for obtain information about the geometry of 3D scenes from 2D images. The generating three-dimensional structure from a series of 2D images or video of scenes is known as Structure from Motion (SfM). A central tenet of structure from motion is that, given the position of a feature in one image, it is possible to find the corresponding position of the same feature in successive images. We described methods for the simultaneous recovery of 3D points and camera projection matrices using corresponding image points in multiple views. Structure from Motion (SfM) is a fascinating field within computer vision that seeks to reconstruct a three-dimensional structure of the environment from a sequence of two-dimensional images. At the heart of SfM lies a set of sophisticated algorithms that enable the extraction of spatial information from a series of images. Two public repositories that may be of use are Open SfM and Colmap.

A Transformer-Based Image-Guided Depth-Completion Model with Dual-Attention Fusion Module.

Depth information is crucial for perceiving three-dimensional scenes. However, depth maps captured directly by depth sensors are often incomplete and noisy, our objective in the depth-completion task is to generate dense and accurate depth maps from sparse depth inputs by fusing guidance information from corresponding color images obtained from camera sensors. To address these challenges, we introduce transformer models, which have shown great promise in the field of vision, into the task of image-guided depth completion. By leveraging the self-attention mechanism, we propose a novel network architecture that effectively meets these requirements of high accuracy and resolution in depth data. To be more specific, we design a dual-branch model with a transformer-based encoder that serializes image features into tokens step by step and extracts multi-scale pyramid features suitable for pixel-wise dense prediction tasks. Additionally, we incorporate a dual-attention fusion module to enhance the fusion between the two branches. This module combines convolution-based spatial and channel-attention mechanisms, which are adept at capturing local information, with cross-attention mechanisms that excel at capturing long-distance relationships. Our model achieves state-of-the-art performance on both the NYUv2 depth and SUN-RGBD depth datasets. Additionally, our ablation studies confirm the effectiveness of the designed modules.

Enhanced Visual SLAM for Collision-Free Driving with Lightweight Autonomous Cars.

The paper presents a vision-based obstacle avoidance strategy for lightweight self-driving cars that can be run on a CPU-only device using a single RGB-D camera. The method consists of two steps: visual perception and path planning. The visual perception part uses ORBSLAM3 enhanced with optical flow to estimate the car's poses and extract rich texture information from the scene. In the path planning phase, the proposed method employs a method combining a control Lyapunov function and control barrier function in the form of a quadratic program (CLF-CBF-QP) together with an obstacle shape reconstruction process (SRP) to plan safe and stable trajectories. To validate the performance and robustness of the proposed method, simulation experiments were conducted with a car in various complex indoor environments using the Gazebo simulation environment. The proposed method can effectively avoid obstacles in the scenes. The proposed algorithm outperforms benchmark algorithms in achieving more stable and shorter trajectories across multiple simulated scenes.

A preliminary application and its error estimates of a simple stereo-camera measure system for the far-sea fishery of Pacific saury

Pacific saury is one of the most economically important species in the northwestern Pacific Ocean. The management of this resource relies on precise input of biological data such as body length and is often hindered by a lack of such data on captured fish. This study explores the potential of electronic monitoring (EM) using off-the-shelf stereo cameras to overcome the challenges of collecting and measuring saury body length from the Pacific saury fishery. Using a calibrated WEEVIEW SID WV3000 3D camera, a total of 252 paired images with different shooting angles and distances were obtained for further measurement using Sebastes Stereo Image Analysis Software (SSIAS). The treatments for the measurement distance (MD) were 30 cm, 60 cm, and 100 cm, for the depression angle (DA) were 30°, 60°, and 90°, and for the position angle (PA) were 0°, 45°, and 315° (− 45°). An assistant calibration (AC) in the form of a 16 cm black ruler was also added used. The SSIAS measurement results indicated that the best measurement was obtained with 0° position angle, 90° depression angle, and 30 cm distance from the target fish. The use of AC in the SSIAS + AC measurement was proven to reduce the measurement error from 2.45 to 8.64% to − 1.86 to 0.01%. This study set the baseline for the application of EM on collecting saury body length and the use of AC has been proven to increase the measurement accuracy.

Modified blepharoplasty combined with autologous fat transplantation into the orbital septum for the correction of dermatochalasis and sunken eyelids

BackgroundPeriorbital aging typically manifests with dermatochalasis and depression of the upper eyelid. Blepharoplasty is a popular choice for addressing upper eyelid skin laxity but is not suitable for sunken eyelids. This study introduces an innovative approach for periorbital rejuvenation by addressing dermatochalasis and sunken upper eyelids concurrently in aging Asian women. MethodThe study involved 45 patients with upper eyelid dermatochalasis accompanied by moderate to severe depression. Subbrow blepharoplasty was performed in conjunction with upper eyelid granular fat grafting in the orbital septum for these patients. Preoperative measurements of the depth and volume of the upper eyelid depression were taken and the surgical outcomes were assessed at 3 and 6 months postoperatively using the GAIS score and the Antera 3D camera. ResultThe average depth of the upper eyelid depression before surgery was 6.501±1.473mm, and the average volume was 0.579±0.144 cm3. There were significant improvements in the depth and volume of the depression at 3 and 6 months after surgery compared to the preoperative measurements. The GAIS scores were also improved and postoperative outcomes were deemed satisfactory. ConclusionFor patients with dermatochalasis accompanied by moderate to severe upper eyelid depression, a standard blepharoplasty was unable to improve the sunken eyelid. This novel technique combines autologous granular fat injection into the orbital septum and blepharoplasty to effectively address periorbital aging, thereby promoting facial rejuvenation.

A LiDAR-depth camera information fusion method for human robot collaboration environment

With the evolution of human–robot collaboration in advanced manufacturing, multisensor integration has increasingly become a critical component for ensuring safety during human–robot interactions. Given the disparities in range scales, densities, and arrangement patterns among multisensor data, such as that from depth cameras and LiDAR, accurately fusing information from multiple sources has emerged as a pressing need to safeguard human–robot safety. This paper focuses on LiDAR and depth cameras, addressing the challenges posed by the differences in data collection range, point density, and distribution patterns which complicate information fusion. We propose a heterogeneous sensor information fusion method for human–robot collaborative environments. To solve the problem of substantial differences in point cloud range scales, a moving sphere space coarse localization algorithm is introduced, narrowing down the scale of interest based on similar features. Furthermore, to address the challenge of significant density differences and low overlap rates between point clouds, we present an improved FPFH coarse registration algorithm based on overlap ratio and an enhanced ICP fine registration algorithm based on the generation of corresponding points. The method proposed herein is applied to the fusion of information from a 64-line LiDAR and a depth camera within a human–robot collaboration scene. Experimental results demonstrate an absolute translational accuracy of 4.29 cm and an absolute rotational accuracy of 0.006 rad, meeting the requirements for heterogeneous sensor information fusion in the context of human–robot collaboration.