RGB-D Imaging Research Articles

End-to-end optimization of single-shot monocular metasurface camera for RGBD imaging

RGBD cameras that capture color and depth information have a wide range of applications such as robotics, autonomous driving, and cons umer electronics. Traditional RGBD imaging usually requires multi-cameras or extra active illumination which inevitably leads to bulky or complex imaging systems, hindering the growing demand for compact integrated optical devices. Optical metasurface have emerged as a powerful substitute to the traditional diffractive optical element due to the superior dispersion manipulation and extremely compact size. However, it is still a great challenge to realize a single-shot monocular metasurface camera, due to the strong and unignorable wavelength-dependent aberrations. In this work, we demonstrate an end-to-end compact single-shot monocular metasurface camera for RGBD imaging. By utilizing end-to-end joint optimization framework to learn metasurface physical structure in conjunction with deep neural networks reconstruction algorithm, the multidimensional light field information RGBD of a scene in a single shot can be reconstructed within a depth range of 0.5 m. Compared with traditional lens-based RGBD imaging, our proposed metasurface-based imaging achieves an improvement of about 2 dB in chromatic imaging and a 4 times increase in depth estimation accuracy. Our proposed scheme could facilitate the further development of the intelligent computational meta-optics in diverse fields ranging from machine vision to biomedical imaging.

Revolutionizing Robotic Depalletizing: AI-Enhanced Parcel Detecting with Adaptive 3D Machine Vision and RGB-D Imaging for Automated Unloading.

Detecting parcels accurately and efficiently has always been a challenging task when unloading from trucks onto conveyor belts because of the diverse and complex ways in which parcels are stacked. Conventional methods struggle to quickly and accurately classify the various shapes and surface patterns of unordered parcels. In this paper, we propose a parcel-picking surface detection method based on deep learning and image processing for the efficient unloading of diverse and unordered parcels. Our goal is to develop a systematic image processing algorithm that emphasises the boundaries of parcels regardless of their shape, pattern, or layout. The core of the algorithm is the utilisation of RGB-D technology for detecting the primary boundary lines regardless of obstacles such as adhesive labels, tapes, or parcel surface patterns. For cases where detecting the boundary lines is difficult owing to narrow gaps between parcels, we propose using deep learning-based boundary line detection through the You Only Look at Coefficients (YOLACT) model. Using image segmentation techniques, the algorithm efficiently predicts boundary lines, enabling the accurate detection of irregularly sized parcels with complex surface patterns. Furthermore, even for rotated parcels, we can extract their edges through complex mathematical operations using the depth values of the specified position, enabling the detection of the wider surfaces of the rotated parcels. Finally, we validate the accuracy and real-time performance of our proposed method through various case studies, achieving mAP (50) values of 93.8% and 90.8% for randomly sized and rotationally covered boxes with diverse colours and patterns, respectively.

On-Plant Size and Weight Estimation of Tomato Fruits Using Deep Neural Networks and RGB-D Imaging

Highlights Deep learning-based instance segmentation models were applied and evaluated for tomato fruit detection. Mask R-CNN with vision transformer backbone showed the highest accuracy for tomato instance detection. Size and weight estimation indexes were calculated using tomato region depth data from instance segmentation models. Area-based index has higher accuracy for weight estimation than indexes based on weight and height information. Abstract. The size and weight of fruits are crucial factors in yield prediction and determining harvesting time. Machine vision, including fruit detection, is a key technology in the automated monitoring and harvesting of fruits. In particular, deep learning-based fruit-detection methods have been actively applied. Estimation of fruit size after fruit detection requires depth information, which can be acquired using depth imaging. RGB-D cameras include color and depth information required for fruit detection and size estimation. In this study, the RGB-D imaging technique was used to estimate the size and weight of tomatoes. Furthermore, deep learning-based instance segmentation models, including Mask R-CNN, YOLACT, and RTMDet for tomato fruit detection, were trained and evaluated. The proposed method estimated the fruit width with a root mean square error (RMSE) of 4 mm, a mean absolute percentage error (MAPE) of 4.28%, and a fruit height with an RMSE of 5.12 mm and a MAPE of 6.42%. Furthermore, the weight-prediction model based on the area index estimated the tomato fruit weight with an RMSE of 19.69 g and MAPE of 9.44%. Thus, the method can be used for accurate size and weight estimation and can be applied in growth monitoring and automated tomatoes harvesting. Keywords: Deep learning, Fruit sizing, Instance segmentation, RGB-D, Tomato.

REALITY CAPTURE METHODS FOR REMOTE INSPECTION OF BUILDING WORK

Abstract. Conventional building inspection, which requires in-person visits by a qualified inspector, can be costly, time-consuming, and even pose health and safety risks. The travel restrictions of the global Covid-19 pandemic further highlighted the need for remote inspection methods. Using reality capture techniques to create a digital 3D representation of the site offers promise for remote inspection of building work. This paper aims to assess different reality capture methods and their visualizations for remote inspection of building work through a case study conducted in Melbourne, Australia. The reality capture methods included terrestrial and mobile laser scanning, RGB-D imaging, and aerial photogrammetry using a UAV. Professional building inspectors participated in the evaluation process by conducting remote inspection based on the captured data and comparing it to the on-site experience. The assessment involved different visualizations of the data, including 3D panoramas, point clouds, and triangulated surface meshes. The results indicate that image visualisation and the ability to make measurement on images are desirable by professional inspectors. These findings support the adoption of reality capture techniques for remote inspection of building work, which can enhance the safety and efficiency of the process.

Estimation of Plant Height and Aboveground Biomass of Toona sinensis under Drought Stress Using RGB-D Imaging

Rapid and accurate plant growth and biomass estimation is essential for formulating and implementing targeted forest cultivation measures. In this study, RGB-D imaging technology was used to obtain the RGB and depth imaging data for a Toona sinensis seedling canopy to estimate plant growth and aboveground biomass (AGB). Three hundred T. sinensis seedlings from 20 varieties were planted under five different drought stress treatments. The U-Net model was applied first to achieve highly accurate segmentation of plants from complex backgrounds. Simple linear regression (SLR) was used for plant height prediction, and the other three models, including multivariate linear (ML), random forest (RF) and multilayer perceptron (MLP) regression, were applied to predict the AGB and compared for optimal model selection. The results showed that the SLR model yields promising and reliable results for the prediction of plant height, with R2 and RMSE values of 0.72 and 1.89 cm, respectively. All three regression methods perform well in the prediction of AGB estimation. MLP yields the highest accuracy in predicting dry and fresh aboveground biomass compared to the other two regression models, with R2 values of 0.77 and 0.83, respectively. The combination of Gray, Green minus red (GMR) and Excess green index (ExG) was identified as the key predictor by RReliefF for predicting dry AGB. GMR was the most important in predicting fresh AGB. This study demonstrated that the merits of RGB-D and machine learning models are effective phenotyping techniques for plant height and AGB prediction, and can be used to assist dynamic responses to drought stress for breeding selection.

Augmented reality-based robot teleoperation system using RGB-D imaging and attitude teaching device

Augmented reality (AR)-based programming using the demonstration method has been widely studied. However, studies on AR-based programming for remote robots are lacking because of the limitation of human–computer interaction. This paper proposes an AR-based robot teleoperation system and method using RGB-D imaging and an attitude teaching device. By sending the color and depth images of the remote robot environment to the local side, the operators can complete the teleoperation of the robot at the local side. First, the operators select key positions on the motion path of the robot endpoint from color images via a mouse, and the computer calculates the 3D coordinates of these key points in the robot base coordinate system to complete the position teaching process. In the robot attitude teaching process, the AR technology is used to superimpose the virtual robot model onto the color images of the robot teleoperation environment, so as to make the virtual robot endpoint to move along the teaching path. An operator can use the portable attitude teaching device designed in this study to control the robot movement parameters, such as the attitude and motion speed, during the movement of the virtual robot. After the position and attitude teaching processes, the robot movement trajectory can be generated. To make the base coordinate system of the virtual model consistent with that of the physical robot, we propose an online AR registration method, which does not require manually placing the AR registration marker. The proposed AR-based robot teleoperation system can quickly and easily complete robot teleoperation at the local side.

Pose-aware C-arm for automatic re-initialization of interventional 2D/3D image registration.

In minimally invasive interventions assisted by C-arm imaging, there is a demand to fuse the intra-interventional 2D C-arm image with pre-interventional 3D patient data to enable surgical guidance. The commonly used intensity-based 2D/3D registration has a limited capture range and is sensitive to initialization. We propose to utilize an opto/X-ray C-arm system which allows to maintain the registration during intervention by automating the re-initialization for the 2D/3D image registration. Consequently, the surgical workflow is not disrupted and the interaction time for manual initialization is eliminated. We utilize two distinct vision-based tracking techniques to estimate the relative poses between different C-arm arrangements: (1) global tracking using fused depth information and (2) RGBD SLAM system for surgical scene tracking. A highly accurate multi-view calibration between RGBD and C-arm imaging devices is achieved using a custom-made multimodal calibration target. Several in vitro studies are conducted on pelvic-femur phantom that is encased in gelatin and covered with drapes to simulate a clinically realistic scenario. The mean target registration errors (mTRE) for re-initialization using depth-only and RGB[Formula: see text]depth are 13.23mm and 11.81mm, respectively. 2D/3D registration yielded 75% success rate using this automatic re-initialization, compared to a random initialization which yielded only 23% successful registration. The pose-aware C-arm contributes to the 2D/3D registration process by globally re-initializing the relationship of C-arm image and pre-interventional CT data. This system performs inside-out tracking, is self-contained, and does not require any external tracking devices.

Preclinical usability study of multiple augmented reality concepts for K-wire placement.

In many orthopedic surgeries, there is a demand for correctly placing medical instruments (e.g., K-wire or drill) to perform bone fracture repairs. The main challenge is the mental alignment of X-ray images acquired using a C-arm, the medical instruments, and the patient, which dramatically increases in complexity during pelvic surgeries. Current solutions include the continuous acquisition of many intra-operative X-ray images from various views, which will result in high radiation exposure, long surgical durations, and significant effort and frustration for the surgical staff. This work conducts a preclinical usability study to test and evaluate mixed reality visualization techniques using intra-operative X-ray, optical, and RGBD imaging to augment the surgeon's view to assist accurate placement of tools. We design and perform a usability study to compare the performance of surgeons and their task load using three different mixed reality systems during K-wire placements. The three systems are interventional X-ray imaging, X-ray augmentation on 2D video, and 3D surface reconstruction augmented by digitally reconstructed radiographs and live tool visualization. The evaluation criteria include duration, number of X-ray images acquired, placement accuracy, and the surgical task load, which are observed during 21 clinically relevant interventions performed by surgeons on phantoms. Finally, we test for statistically significant improvements and show that the mixed reality visualization leads to a significantly improved efficiency. The 3D visualization of patient, tool, and DRR shows clear advantages over the conventional X-ray imaging and provides intuitive feedback to place the medical tools correctly and efficiently.

Efficient foreground extraction using RGB-D imaging

Image segmentation is one of the most important topics in the field of computer vision. As a result, many image segmentation approaches have been proposed, and interactive methods based on energy minimization such as GrabCut, have shown successful results. Automating the entire segmentation process is, however, very difficult because virtually all interactive methods require a considerable amount of user interaction. We believe that if additional information is provided to users in order to guide them effectively, the amount of interaction required can be reduced. Consequently, in this paper we propose an efficient foreground extraction algorithm, which utilizes depth information from RGB-D sensors such as Microsoft Kinect and offers users guidance in the foreground extraction process. Our approach can be applied as a pre-processing step for interactive and energy-minimization-based segmentation approaches. Our proposed method is able to segment the foreground from images and give hints that reduce interaction with users. In our method, we make use of the characteristics of depth information captured by RGB-D sensors and describe them using information from the structure tensor. Further, we show experimentally that our proposed method separates foreground from background sufficiently well for real world images.