3D Vision System Research Articles

Analysis of State-of-the-art Software Solutions for Unstructured Outdoor Environment Perception in Goal-oriented Navigation of Off-road Unmanned Ground Vehicles

This paper presents a comprehensive analysis of state-of-the-art software solutions designed for the perception of unstructured outdoor environments, with a focus on their applicability in goal-oriented navigation for off-road unmanned ground vehicles (UGVs). The analysis encompasses the evaluation of various sensors for UGV navigation, comparison of methods for extracting perceptual information from sensor data, and the development of a perception system based on the findings. The proposed system leverages advancements in simultaneous localization and mapping (SLAM), visual odometry, and traversability segmentation using 3D LiDAR and visual systems. The analysis suggests a phased approach for the development of autonomous navigation, where the initial phase relies on complex SLAM and teleoperation assistance, and the subsequent phase introduces an autonomous mode utilizing semantical terrain representation.

Research on the development of three-dimensional visual management system for mines: A case study of Chongqing Shuijiang aluminum mine

In recent years, problems such as unclear mine mining data and illegal mining have brought challenges to mine safety production, ecological environment restoration and mineral resources management. From the perspective of mine production and resource management, this study carried out research on the construction of 3D geological model and the development of visual management system based on multisource data fusion, and selected the Shuijiang aluminum mine in Nanchuan District, Chongqing as the research object to establish a set of 3D visual management system for mine. The visual management system can reflect the production status of the mine in real time, and browse, query, section, reserve and statistics of the three-dimensional model of the mine, which has important practical significance for the supervision of the production mine.

DEVELOPMENT OF INTELLECTUAL SUPPORT’S ALGORITHM AT MANAGE DECISIONS MAKING IN THE FEEDING PROCESS BASED ON A MACHINE VISION SYSTEM’S ON A FARM

Analysis of information obtained from open sources has shown that feed’s abnormal overspending is up to 10% and occurs in 5% of cases. It also means that the farm loses the same share of milk yield, and with it profit. This problem solution can be achieved in various ways due diversity of both hardware (vision system) and software (methods and algorithms). It is possible cows’ feed on dairy farms malnutrition reducing with the help of software and hardware complex (PAC) being developed. This PAC will determine the remaining feed volume on the feed table’s distribution by the 3D vision system of obtaining data use analyzing. Based on this analysis, the PAC will draw conclusions and make recommendations to the farm workers, thereby feedback for the farm feeding system providing. To do this, it is necessary a vision system to choose, an algorithm for obtaining a point cloud to implement an algorithm for finding reference points and gluing images into one, so as an device algorithm for orientation in space to select and implement, an algorithm for calculating feed volume and hardware to develop and laboratorial and operational tests to conduct, as well as a database for development of recommendations to create. Two vision systems are considered for use - a stereo system and a ToF camera, and an algorithm finding for reference points based on 3D data is described.

METROLOGICAL CHARACTERIZATION OF A LASER-CAMERA 3D VISION SYSTEM THROUGH PERSPECTIVE-N-POINT POSE COMPUTATION AND MONTE CARLO SIMULATIONS

Abstract. This study focuses on the metrological characterization of a 3D vision system consisting in the fusion of a CMOS camera sensor with a 2D laser scanner for contactless dimensional measurements. The purpose is to obtain an enhanced measurement information as a result of the combination of two different data sources. On one side, we can estimate the pose of the target measurand by solving the well-known Perspective-n-Point (PnP) problem from the calibrated camera. On the other side, the 2D laser scanner generates a discrete point cloud which describes the profile of the intercepted surface of the same target object. This solution allows to estimate the target’s geometrical parameters through the application of fit-to-purpose algorithms that see the data acquired by the overall system as their input. The measurement uncertainty is evaluated by applying the Monte Carlo Method (MCM) to estimate the uncertainty deriving from the Probability Distribution Functions (PDF) of the input variables. Through a Design of Experiments (DOE) model the effects of different influence factors were evaluated.

MT29 Advantages of 3D Visualization Systems for Teaching in Ophthalmology

MT29 Advantages of 3D Visualization Systems for Teaching in Ophthalmology

INTRAOPERATIVE THREE-DIMENSIONAL FLUORESCEIN ANGIOGRAPHY-GUIDED PARS PLANA VITRECTOMY FOR BRANCH RETINAL VEIN OCCLUSION.

To show surgical results of the intraoperative 3D fluorescein angiography-guided pars plana vitrectomy for branch retinal vein occlusion with vitreous hemorrhage and neovascularization elsewhere. The NGENUITY 3D visualization system was used for the digital-assisted vitrectomy. Three-dimensional fluorescein angiography-guided pars plana vitrectomy was performed in three patients with branch retinal vein occlusion with vitreous hemorrhage and neovascularization elsewhere. In all eyes, the scatter retinal photocoagulations for a nonperfusion area, depicted as hypofluorescein, and the segmentation and delamination of the perivascular fibrovascular proliferative membrane, depicted as hyperfluorescein, could be safely performed on the same screen while implementing intraoperative 3D fluorescein angiography. Three-dimensional fluorescein angiography-guided pars plana vitrectomy, a novel approach that fully uses the advantages of digital-assisted vitrectomy, can be one of the useful techniques for the treatment of branch retinal vein occlusion with vitreous hemorrhage and neovascularization elsewhere.

Autonomous Wristband Placement in a Moving Hand for Victims in Search and Rescue Scenarios With a Mobile Manipulator

In this letter, we present an autonomous method for the placement of a sensorized wristband to victims in a Search-And-Rescue (SAR) scenario. For this purpose, an all-terrain mobile robot includes a mobile manipulator, which End-Effector (EE) is equipped with a detachable sensorized wristband. The wristband consists of two links with a shared shaft and a spring. This configuration allows the wristband to maintain fixed to the EE while moving and get placed around the victim's forearm once the contact is produced. The method has two differentiated phases: i) The visual moving hand tracking phase, where a 3D vision system detects the victim's hand pose. At the same time, the robotic manipulator tracks it with a Model Predictive Controller (MPC). ii) The haptic force-controlled phase, where the wristband gets placed around the victim's forearm controlling the forces exerted. The wristband design is also discussed, considering the magnitude of the force needed for the attachment and the torque the wristband exerts to the forearm. Two experiments are carried out, one in the laboratory to evaluate the performance of the method and the second one in a SAR scenario, with the robotic manipulator integrated with the all-terrain mobile robot. Results show a 97.4% success in the wristband placement procedure and a good performance of the whole system in a large scale disaster exercise.

Robot-Assisted Surgery in Gynecology.

The advancement of robotics-based procedures in the medical industry is the subject of this review article. The purpose of the surgical robot is to increase surgical abilities and address human shortcomings. The robot's success has been predicated on its ability to accurately and consistently repeat tasks. The following are a few objectives and quantifiable benefits of robotic technology improving surgical maneuverability and physical capabilities. In 2005, the FDA granted gynecological surgery approval for the Da Vinci surgical system. It has been quickly embraced and has already taken on a significant role at many of the locations where it is offered. It consists of a cart with robotic branches next to the patient and a high-resolution three-dimensional (3D) vision system. This study covers laparoscopy via robots in benign gynecological surgeries, myomectomy surgery, hysterectomies, endometriosis, tubal anastomosis, and sacrocolpopexy. The appropriate published studies were evaluated after a PubMed search was conducted. Additionally, procedures that may be used in the future are highlighted. In benign gynecological illness, most currently available research does not show a substantial benefit over traditional laparoscopic surgery. Robotics, however, does provide help in more complicated operations. Compared to laparoscopy, robotic assistance has a lower conversion rate to open procedures. Endo wrist movement of an automated machine during myomectomy surgery can perform proper and better suturing than traditional laparoscopy. The automated program is a noticeable improvement over laparoscopic surgery and, if price issues are resolved, can gain popularity among gynecological surgeons around the globe.

Multi-line laser structured light fast visual positioning system with assist of TOF and CAD

As a reliable and high-precision three-dimensional (3D) vision method, line laser structured light vision has been widely used in the engineering field. However, to obtain overall 3D shape of target to be measured, scanning mechanisms need to be added. Multi-line laser structured light vision can obtain multi-line array 3D shape of workpiece without scanning; however, the encoding of multi-line lasers is a difficult task. This paper proposes a composite vision system, which is composed of time-of-flight(TOF) camera, color camera and vertical cross multi-line laser structured light. Color camera and multi-line laser form a multi-line array 3D vision system, while the coded signal of multi-line laser stripes is provided by TOF camera and corresponding coding algorithm is also proposed in this paper. Based on this system, automatic positioning of welding seam is realized by registration of CAD data and multi-line laser 3D data. The registration is divided into two steps, which are coarse registration and precise registration. Coarse registration is performed by using CAD point cloud data and original point cloud data of multi-line laser; while precise registration is performed with CAD welding seam point cloud data and welding seam point cloud data of multi-line laser. This paper also analyzes accuracy of vision system based on constructing structured light vision model. The measurement accuracy of vision system in x, y and z directions is 0.25, 0.06 and 0.29 mm respectively. This composite vision system can realize precise positioning of target within a single exposure time, which provides possibility for positioning under movement.

409 Evaluation of Advanced Bimanual Skills in Novices Using the Wrist-Like FlexDex™ Articulating Laparoscopic Needle Holder in 2D and 3D Vision: A Randomised Trial

Abstract Aim The aim of this study was to evaluate the novice performance of advanced bimanual laparoscopic skills using the articulating FlexDex™ laparoscopic needle holder in two-dimensional (2D) and three-dimensional (3D) visual systems. Method In this prospective randomised trial, novices (n=40) without laparoscopic experience were recruited from a university cohort and randomised into two groups, which used the FlexDex™ and 2D or the FlexDex™ and 3D. Both groups performed 10 repetitions of a validated assessment task. Times taken and error rates were measured, and assessments were made based on completion times, error rates and learning curves. Results The intervention group that used FlexDex™ and 3D visual output completed 10 attempts of the standardised laparoscopic task quicker than the control group that used FlexDex™ with standard 2D visual output (268 seconds vs 415 seconds taken for the first three attempts and 176 seconds vs 283 seconds taken for the last three attempts, respectively). Moreover, each attempt was completed faster by the intervention group compared to the control group. The difference in average time for the first three and last three attempts reached statistical significance (P < 0.001). Conclusion Combination of 3D visual systems and the FlexDex™ laparoscopic needle holder resulted in superior task performance speed, leading to shorter completion times and quicker learning effect. Although the 3D group demonstrated lower mean error rates, it did not reach statistical significance.

3D Visualization System in Descemet Membrane Endothelial Keratoplasty (DMEK): A Six-Month Comparison with Conventional Microscope.

Background: To compare the efficacy and safety of Descemet membrane endothelial keratoplasty (DMEK) surgery using the three-dimensional (3D) display system NGENUITY to DMEK surgery performed with the traditional microscope (TM) in patients affected by Fuchs Endothelial Corneal Disease (FECD). Methods: Retrospective comparative study of 40 pseudophakic eyes of 40 patients affected by FECD who underwent DMEK surgery. Twenty patients (3D group) were operated on using the 3D display system and 20 patients (TM group) were operated on using the traditional microscope. Best spectacle corrected visual acuity (BSCVA), central corneal thickness (CCT), endothelial cell density (ECD) and corneal densitometry (CD) values were documented before and at 1, 3 and 6 months after DMEK. Intra- and postoperative complications were recorded. Results: The baseline assessments did not differ between the two groups (p > 0.05). Global surgical time and time to perform descemetorhexis were significantly lower in the TM group (p = 0.04 and p = 0.02, respectively). BSCVA, CCT, ECD and CD values did not differ significantly in the two groups at all follow-ups (p > 0.05). Complication rate was similar between the two groups. Conclusion: Three-dimensional display systems can be securely employed in DMEK surgery considering the satisfactory clinical outcomes, including Scheimpflug CD. Nevertheless, the slightly longer surgical time of the 3D DMEKs may lead to surgeons’ hesitancy. The main advantages of the heads-up approach may be the improved ergonomic comfort during surgery and the utility of assistants in surgical training.

MEMS mirror based omnidirectional scanning for lidar optical systems

Lidar 3D vision system is currently one of the most effective technical means for high-precision perception of three-dimensional image information of the surrounding environment. Microelectron-mechanical systems (MEMS) micromirror lidar has advantages of miniaturization, fast scanning speed, and relatively low price. The MEMS micromirror scanning method has become the best choice for laser scanning 3D image sensors. However, due to the small size of the MEMS micromirror, it can only scan a small angle range, which limits its industrial applications. In this paper, combined with MEMS micro-mirror spiral scanning form, we design a new type of omnidirectional optical system that can realize MEMS lidar omnidirectional scanning imaging. The overall optical system is composed of the catadioptric prism, the MEMS micromirror, and the compensation prism. The overall system can achieve 360° omnidirectional scanning imaging with a horizontal resolution of 0.1°; a vertical field view with a 0.5° resolution. The design results are analyzed by the optical design software (Zemax). The point diagram shows that the emitted light spots are circular and evenly distributed. The divergence angle is less than 1mrad, which can well meet the actual needs and solve the problems of solid-state, miniaturization, and omnidirectional scanning imaging of lidar.

Extensive internal limiting membrane peeling for proliferative vitreoretinopathy.

The aim of this study was to describe the anatomical outcomes of Brilliant Blue G (BBG)-assisted extensive internal limiting membrane peeling for proliferative vitreoretinopathy (PVR) under three-dimensional (3D) visualization. This study constitutes a retrospective case series conducted in a private retina practice, of 14 consecutive patients (14 eyes) with rhegmatogenous retinal detachment complicated by PVR who underwent pars plana vitrectomy between January 2019 and January 2020. The internal limiting membrane (ILM) was selectively stained with BBG, and perspectives were enhanced with a 3D visualization system. We peeled off the ILM beyond the vascular arcades up to the periphery. The main outcome was anatomical success, defined as persistent retinal reattachment after removal of the silicone oil tamponade. Anatomic success was achieved with a single surgery in 11 of 14 (78.6%) eyes, and eventual success was achieved in all eyes. The mean patient follow-up time was 12.3months (range, 7-16months). The mean preoperative best-corrected visual acuity (BCVA) was 2.93 ± 0.79 logMAR which improved to 1.75 + 0.91 at the last follow-up. Extensive ILM peeling allowed the creation of a cleavage plane underlying the PVR membranes that facilitated its complete removal, thereby achieving anatomically reattached retina and reducing the risk of recurrence of retinal detachment. The long-term effects of this technique need further research.

Comparison of Three-Dimensional Surgical System Versus Binocular Microscope for Clear Corneal Incision in Cataract Surgery.

IntroductionTo compare surgical outcomes of 2.2 mm clear corneal incision (CCI) between a three-dimensional (3D) visualization system and traditional binocular microscope (BM) for phacoemulsification and intraocular lens implantation surgery.MethodsIn this randomized controlled clinical study, 60 eyes with age-related cataracts were divided into two groups receiving cataract surgery using either a 3D vision system (n = 30 eyes) (3D group) or a binocular microscope (n = 30 eyes) (BM group). We recorded and statistically analyzed surgical parameters and pre- and postoperative ocular parameters. Primary outcomes included the change in endothelial cell density (ECD) and CCI architecture, and secondary outcomes comprised other ocular parameters and surgical parameters. All procedures complied with the tenets of the Declaration of Helsinki.ResultsOf the 60 eyes randomly assigned between January 5, 2021, and May 9, 2021, 55 (26 eyes in the 3D group and 29 eyes in the BM group) were analyzed. The ECD loss rate was 8.1% in the 3D group and 12.3% in the BM group, but the difference was not statistically significant. Local detachment of Descemet’s membrane was seen in 50% (13 eyes, 3D group) and 51.6% (15 eyes, BM group), wound gaping at the endothelial side in 15.4% (four eyes, 3D group) and 10.3% (four eyes, BM group), gaping at the epithelial side in 11.5% (three eyes, 3D group) and 6.9% (two eyes, BM group), and misalignment of the incision in 3.4% (one eye, BM group) 1 day after surgery. These abnormalities improved with time. There was no difference between the 3D group and BM group in terms of other ocular parameters or surgical parameters before and after surgery.ConclusionsUsing the 3D surgical system for phacoemulsification and IOL implantation surgery seems to result in similar ECD and CCI conditions as using a conventional binocular microscope.Trial RegistrationThe protocol was registered on ClinicalTrials.gov (NCT04839250).

3D Visual Tracking to Quantify Physical Contact Interactions in Human-to-Human Touch.

Across a plethora of social situations, we touch others in natural and intuitive ways to share thoughts and emotions, such as tapping to get one’s attention or caressing to soothe one’s anxiety. A deeper understanding of these human-to-human interactions will require, in part, the precise measurement of skin-to-skin physical contact. Among prior efforts, each measurement approach exhibits certain constraints, e.g., motion trackers do not capture the precise shape of skin surfaces, while pressure sensors impede skin-to-skin contact. In contrast, this work develops an interference-free 3D visual tracking system using a depth camera to measure the contact attributes between the bare hand of a toucher and the forearm of a receiver. The toucher’s hand is tracked as a posed and positioned mesh by fitting a hand model to detected 3D hand joints, whereas a receiver’s forearm is extracted as a 3D surface updated upon repeated skin contact. Based on a contact model involving point clouds, the spatiotemporal changes of hand-to-forearm contact are decomposed as six, high-resolution, time-series contact attributes, i.e., contact area, indentation depth, absolute velocity, and three orthogonal velocity components, together with contact duration. To examine the system’s capabilities and limitations, two types of experiments were performed. First, to evaluate its ability to discern human touches, one person delivered cued social messages, e.g., happiness, anger, sympathy, to another person using their preferred gestures. The results indicated that messages and gestures, as well as the identities of the touchers, were readily discerned from their contact attributes. Second, the system’s spatiotemporal accuracy was validated against measurements from independent devices, including an electromagnetic motion tracker, sensorized pressure mat, and laser displacement sensor. While validated here in the context of social communication, this system is extendable to human touch interactions such as maternal care of infants and massage therapy.

3D Visualization Analysis of Motion Trajectory of Knee Joint in Sports Training Based on Digital Twin.

With the development of China's social economy, the pace of people's life is also accelerating. Therefore, in the face of the increasing pressure of life and work, people have begun to pay more and more attention to improving their physical fitness and they also pay more and more attention to sports in their spare time. The knee joint is a very important body part that will be involved in sports. Therefore, in sports, it is very important for healthy sports to pay attention to the state of the knee joint at all times and avoid injury to the knee joint during sports. The three-dimensional visual analysis of the knee joint movement trajectory in sports training refers to the trajectory capture and three-dimensional visual analysis of the knee joint movement in sports training based on the computer-based description and understanding of model technology. The three-dimensional visual analysis of the motion trajectory of the knee joint in sports training is conducive to the attention and analysis of the motion state of the knee joint in sports training, so as to help find problems and avoid knee joint damage during sports training as much as possible. This paper aims to study the 3D visualization analysis of knee joint motion trajectory in sports training based on digital twin and establish a 3D visualization system of knee joint motion trajectory based on digital twin technology, and based on this system, the three-dimensional visualization experiment of knee joint movement trajectory in sports training is carried out. The experiment concluded that the signal-to-noise ratio of the three-dimensional visualization image of the knee joint movement trajectory in sports training based on the three-dimensional visualization system established by the digital twin is 68% and the clarity is 89%. It shows that the system has a good 3D visualization effect.

Integrated three-dimensional visualization and soft-sensing system for underground paste backfilling

Cemented paste backfilling (CPB) has attracted worldwide attention because of its advantages in the processing of waste tailings and underground goafs. A problem of the underground CPB procedure that has received little attention is the shortage of devices or techniques for monitoring the real-time backfilling process accurately due to the harsh underground environment. Imprecise information of the backfilling progress will mislead the operators to make improper decisions, such as not pausing the transport system promptly or tuning the ratio of cement to tailings incorrectly. This study proposes a 3D visualization and soft-sensing system for the cemented paste backfilling process. By employing a GeoSLAM ZEB-HORIZON laser scanner to scan the 3D model of the underground goaf, the system estimates the backfilling height in real-time according to the accumulated volume of filled paste and simulates the backfilling of the underground goaf on web pages. In a real Copper Mine, we examined the developed system and soft-sensing technique in goafs and found that the relative error of estimated backfilling height is under 10%, which is acceptable in real production. The results indicate that the developed backfilling visualization and the soft-sensing techniques provide significant guidance for production management in underground backfilling.

Development of a Toolchain for Automated Optical 3D Metrology Tasks

Modern manufacturing processes are characterized by growing product diversities and complexities alike. As a result, the demand for fast and flexible process automation is ever increasing. However, higher individuality and smaller batch sizes hamper the use of standard robotic automation systems, which are well suited for repetitive tasks but struggle in unknown environments. Modern manipulators, such as collaborative industrial robots, provide extended capabilities for flexible automation. In this paper, an adaptive ROS-based end-to-end toolchain for vision-guided robotic process automation is presented. The processing steps comprise several consecutive tasks: CAD-based object registration, pose generation for sensor-guided applications, trajectory generation for the robotic manipulator, the execution of sensor-guided robotic processes, test and the evaluation of the results. The main benefits of the ROS framework are readily applicable tools for digital twin functionalities and established interfaces for various manipulator systems. To prove the validity of this approach, an application example for surface reconstruction was implemented with a 3D vision system. In this example, feature extraction is the basis for viewpoint generation, which, in turn, defines robotic trajectories to perform the inspection task. Two different feature point extraction algorithms using neural networks and Voronoi covariance measures, respectively, are implemented and evaluated to demonstrate the versatility of the proposed toolchain. The results showed that complex geometries can be automatically reconstructed, and they outperformed a standard method used as a reference. Hence, extensions to other vision-controlled applications seem to be feasible.

Novel Surgical Pathway for Controlled Access to the Subretinal Space: A Case Series

PurposeWe report a case series demonstrating a novel technique to access the subretinal space through a trans-scleral approach.MethodsThis was a prospective, interventional case series. All 25 cases underwent three-port vitrectomy using CONSTELLATION Vision System (Alcon. Laboratories, Inc., Fort Worth, TX) and NGENUITY 3D Visualization System. In all cases a fourth 25G valved cannula was inserted posteriorly in the vitreous cavity to secure a controlled access to the subretinal space. This route was used to remove subretinal fibrosis, drain subretinal fluid, and inject vital dye to identify an occult retinal break in some cases.ResultsIn 23 of 25 cases, the retina remained attached postoperatively after a single surgery, and vision improvement was observed. In two cases, the retina was attached after an additional procedure. Although mild subretinal hemorrhage was seen in three cases, no other major complication was observed.ConclusionsAccess to the subretinal space by a posteriorly placed fourth valved cannula is safe and helps in subretinal fibrosis removal and drainage of subretinal fluid and can be useful in complicated retinal detachments.Translational RelevanceThis case series uses knowledge acquired from developing small gauge microincision instruments for pars plana vitrectomy to present a novel clinical application applicable to difficult, complex retinal surgeries.

A Dynamic Task Allocation Strategy to Mitigate the Human Physical Fatigue in Collaborative Robotics

In human-robot collaboration, the mitigation of human physical workload is a crucial factor to avoid musculoskeletal disorders that might jeopardize the operator’s safety and job performance. In this work, we propose a novel, non-invasive method to estimate online the muscle fatigue experienced by the worker during the task execution. The estimation process relies on a sophisticated musculoskeletal model of the human upper body and on a 3D vision system used to track human motions in real-time. Based on this estimate, we develop a strategy that dynamically allocates the task activities to the human and to the robot with the aim of minimizing his/her muscular fatigue, thus improving the quality of the cooperation. The proposed approach has been experimentally validated in a collaborative industrial use case and compared to a static allocation strategy.