3D Vision System Research Articles

Accurate 3D Surface Reconstruction for Smart Farming Application with an Inexpensive Shape from Focus System

In precision agriculture, 3D vision systems are becoming increasingly important. By applying different optical 3D vision techniques, the acquired 3D data can provide information regarding the most important phenotype features in every agricultural scenario. However, most of these 3D vision systems are expensive, except some of the triangulation techniques. In this study, we focus on estimating accurate shapes using shape from focus (SFF), which is a triangulation technique. Typically, the SFF system incurs significant errors from images, including noise. As a solution to this problem, a simple low-pass filter such as the Gaussian filter has generally been used in most studies. However, when a low filter is applied, the noise is depressed but the signals are also blurred, which results in inaccuracies regarding the depth map. In this study, the noise is depressed independently without losing the original signals, and the edge components, which play important roles in finding a focused surface, are enhanced using the independent component analysis (ICA). The edge signals are amplified with a simple basis vector correction in the IC vector space. The experiments are implemented with simulated objects and real objects. The experimental results demonstrate that the obtained accuracy is comparable to that of existing methods.

3D vision system integration on Additive Manufacturing machine for in-line part inspection

Laser based technologies enable major opportunities in performing quite heterogeneous deposition and subtraction processes characterized by very similar process dynamics. Hybrid machines integrating high power and ultra-short laser sources can efficiently exploit such technologies only when they embed a sensing system able to execute in-line inspection of partially manufactured parts. The current work presents an innovative configuration of Direct Energy Deposition hybrid machine equipped with a custom 3D vision system and software that allow: 1) the part shape measurement at every user defined checkpoint; 2) the acquisition of the temperature distribution onto this part surface; 3) the analysis of the data and the defect identification. The system is based on structured light technique and measures the spatial position of thousands of points belonging to the part surface. The measurements performed by a thermal camera are then associated to this point cloud and a dedicated software performs the comparison of the acquired geometry to the CAD model to detect over- or under-deposition areas and analyses the thermal distribution to detect potential defects. Corrective strategies can be generated for the layers that still have to be manufactured in terms of additional deposition or ablation. The benefits of the approach have been evaluated on experimental tests inspired by industrial scenarios.

Technology Driven Mitigation of COVID-19 Infection Risk in Retinal Surgery, by means of 3D Visualization Systems

In cases in which the proximity of the medical personnel and the patient is necessary, such as in ophthalmology, both the medical personnel and the patient have high infection risk from each other’s respiratory system secretions, which can float in a range usually larger than the required proximity of such medical operations. In this work, the authors examine and evaluate the infection risk change between the ordinary retinal operation practice with microscope and the one with utilizing a 3D visualization system, by keeping the other surgery conditions untouched. It was found that the 3D visualization provides a significant reduction of the COVID-19 infection risk for the surgeon, his paramedics’ assistant, and the patient. Many reasons trigger ophthalmologists to use this new way of operating; The ergonomics for the surgeon, the excellent image quality, the better depth of view, the ability to reproduce large differences in bright and dark conditions to retain details and depends on the ability of the human pupil to fluctuate in size, the ability to change lighting conditions, the teaching potential and the ease to adapt.

Joint Investigation and 3D Visual Evaluation of Rock Mass Quality

In order to realize the high efficiency quality classification and three‐dimensional visualization of engineering rock mass and to solve the technical difficulties of the traditional rock mass quality evaluation method such as high labor intensity, long process time consumption, many intervention processes such as scale measurement and manual calculation, and nonintuitive classification results, this paper puts forward a 3D visual rock mass quality evaluation method and system based on close‐range photography, which optimizes the traditional rock mass quality evaluation method, makes the rock mass classification three‐dimensional and visible, and realizes the estimation of unrevealed rock mass quality evaluation index. The research results show the following: (1) The method of storing joint information by close‐range photography and extracting joint information by human‐computer interaction improves the working efficiency and the process is safe and controllable compared with the traditional method of collecting fracture parameters. (2) Based on the statistical analysis of 97 groups of roadway survey data, the comprehensive statistical regression formula between BQ value of Chinese national standard and RMR value is given, and there is a good correlation between BQ value and RMR value of rock mass quality index. (3) Based on the power‐inverse ratio method, the three‐dimensional model of rock mass classification of the mine was established, and the cutting model obtained the current distribution diagram of rock mass quality grade, providing scientific reference for drilling, blasting, support, and other production design optimizations.

3D Vision System for a Robotic Arm Based on Equal Baseline Camera Array

This paper presents a lightweight 3D vision system called Equal Baseline Camera Array (EBCA). EBCA can work in different light conditions and it can be applied for measuring large range of distances. The system is a useful alternative to other known distance measuring devices such as structured-light 3D scanners, time-of-flight cameras, Light Detection and Ranging (LIDAR) devices and structure from motion techniques. EBCA can be mounted on a robotic arm without putting significant load on its construction. EBCA consists of a central camera and a ring of side cameras. The system uses stereo matching algorithms to acquire disparity maps and depth maps similarly as in case of using stereo cameras. This paper introduces methods of adapting stereo matching algorithms designed for stereo cameras to EBCA. The paper also presents the analysis of local, semi-global and global stereo matching algorithms in the context of the EBCA usage. Experiments show that, on average, results obtained from EBCA contain 37.49% less errors than the results acquired from a single stereo camera used in the same conditions.

Measurement of the pick holders position on the side surface of the cutting head of a mining machine with the use of stereoscopic vision

The efficiency of rock cutting with mining machines is largely determined by the arrangement of picks, i.e. the number and their arrangement on the working unit of the mining machine. Not only the correct selection of the pick system for given conditions at the design stage is important, but also ensuring compliance with the design of the finished product. Strives, among others therefore, for robotisation of the process of manufacturing cutting heads/drums. From the point of view of the robotisation of the pick holders welding process, it is necessary to assess in real time the position of the pick holders relative to the side surface of the cutting head body. A convenient way is to use contactless measurement methods based on vision systems. The article presents a method of determining the position of pick holders relative to the side surface of the cutting head body of a roadheader, during their positioning, using a 3D vision system. Data processing was carried out in the Matlab software using the libraries of the Computer Vision Toolbox. A mathematical model describing the transformation of images recorded by cameras has been presented. On the basis of this model, the distribution of distances between the pick holder base points and the side surface of the cutting head was determined for a given pick holder setting. The developed measurement method was tested on an experimental stand built in the Laboratory of robotics of the Department of Mining Mechanization and Robotisation at the Silesian University of Technology.

<p>The Scope Of Three-Dimensional Digital Visualization Systems In Vitreoretinal Surgery</p>

IntroductionSignificant advances in three-dimensional (3D) imaging technology have allowed for the incorporation of 3D digital displays into medical and surgical devices. Despite initial adoption of the NGENUITY® 3D Visualization System in vitreoretinal surgery, there are limited publications regarding its use. The generally accepted main benefits include improved ergonomics, enhanced surgical team communication and education, reduced retinal phototoxicity, increased depth of field, and display image manipulation. Despite these potential benefits, many retina specialists have questioned its universal applicability to a wide variety of vitreoretinal surgeries.ObjectiveTo report on the variety of indications and surgical efficacy of the NGENUITY® 3D Visualization System in vitreoretinal surgery via a review of surgical experience at two vitreoretinal practices in both the academic and community settings.MethodsA retrospective review was conducted of consecutive surgical cases performed on the NGENUITY® 3D Visualization System at Massachusetts Eye and Ear Infirmary and Florida Retina Institute from June 1st, 2017 to November 1st, 2018. Age, presenting diagnosis, surgical procedure, and intraoperative details were recorded.Results272 vitreoretinal surgeries on the Alcon NGENUITY® 3D Visualization System were identified between June 1st, 2017 and November 1st, 2018 at the participating institutions. A detailed breakdown of the indications for surgery and related procedures is reported. During all 272 cases on the 3D digital system, there were no complications attributed to the visualization system.ConclusionThis series illustrates the diversity of vitreoretinal surgeries that can be performed on this system without compromising surgical viewing or increasing surgical complications. The Alcon NGENUITY® 3D Visualization System possesses favorable ergonomics, illumination levels, depth of field, display filters, and trainee experience.

Utility Of Three-Dimensional Heads-Up Surgery In Cataract And Minimally Invasive Glaucoma Surgeries.

PurposeThis study aimed to assess the utility of the three-dimensional (3D) heads-up visualization system for minimal incision cataract surgery (MICS) and minimally invasive glaucoma surgeries (MIGSs).MethodsToric intraocular lens (IOL) implantation with phacoemulsification and trabecular microbypass stent implantation in patients with cataract and open-angle glaucoma were performed using the heads-up 3D visualization system combined with surgical navigation rather than the conventional microscope.ResultsThis procedure was found to have the following advantages: the ability to clearly observe the anterior chamber angle image without requiring frequent focus adjustment owing to the extended depth of field and emphasized stereoscopic effect provided by this system and maintain the surgeon’s posture.ConclusionThe feasibility and comfort of this system are greater than those of the conventional microscopic for performing MICS and MIGS.

Illuminant-invariant stereo matching using cost volume and confidence-based disparity refinement.

In stereo-matching techniques for three-dimensional (3D) vision, illumination change is a major problem that degrades matching accuracy. When large intensity differences are observed between a pair of stereos, it is difficult to find the similarity in the matching process. In addition, inaccurately estimated disparities are obtained in textureless regions, since there are no distinguishable features in the region. To solve these problems, this paper presents a robust stereo-matching method using illuminant-invariant cost volume and confidence-based disparity refinement. In the step of matching a stereo pair, the proposed method combines two cost volumes using an invariant image and Weber local descriptor (WLD), which was originally motivated by human visual characteristics. The invariant image used in the matching step is insensitive to sudden brightness changes by shadow or light sources, and WLD reflects structural features of the invariant image with consideration of a gradual illumination change. After aggregating the cost using a guided filter, we refine the initially estimated disparity map based on the confidence map computed by the combined cost volume. Experimental results verify that the matching computation of the proposed method improves the accuracy of the disparity map under a radiometrically dynamic environment. Since the proposed disparity refinement method can also reduce the error of the initial disparity map in textureless areas, it can be applied to various 3D vision systems such as industrial robots and autonomous vehicles.

Updated chandelier illumination-assisted scleral buckling using 3D visualization system.

PurposeTo assess the updated chandelier illumination-assisted scleral buckle procedures using a 3D visualization system.Subjects and methodsWe report a single-center, retrospective, consecutive surgical case series of 18 eyes of 16 patients that underwent scleral buckle procedures combining the use of chandelier endoillumination, a noncontact wide-angle viewing system, an anti-drying contact lens, and a recently developed 3D visualization system. The mean age of the patients was 30.2 years, and the mean follow-up time was 11.3 months.ResultsRetinal reattachment was achieved after the first surgery in all cases. There were no cases that required conversion of the 3D visualization to a conventional microscopic view. This updated chandelier-assisted scleral buckle procedure was successfully performed without any intraoperative or postoperative complications, except for the occurrence of postoperative diplopia in one case in which a supplementary radial buckle was used for a deep tear.ConclusionThe updated chandelier illumination-assisted scleral buckling surgery using a 3D visualization system appears to be a valuable and promising method for managing rhegmatogenous retinal detachment due to atrophic holes in lattice degeneration or ciliary tears in younger cases.

RESOLUTION, DEPTH OF FIELD, AND PHYSICIAN SATISFACTION DURING DIGITALLY ASSISTED VITREORETINAL SURGERY.

To evaluate depth of field, lateral resolution, and image quality of a heads-up 3D visualization system for vitreoretinal surgery using physician survey and optical measurement outcomes. Depth of field and lateral resolution were compared between the standard ocular viewing system and the digital 3D system at ×5, ×13, and ×18 magnification by 6 retinal surgeons. Optical techniques were used as well as a survey of surgeon impression. Surgeon impression surveys were performed after 6 weeks of surgical use of the device. Physician questionnaire survey scores for depth of field at high magnification were better for the digital 3D system and equivalent for all other categories. Measured lateral resolution was 36.7 mm and 16.6 mm at ×5 magnification (P < 0.001), 14.3 mm and 6.4 mm at ×13 magnification (P < 0.001), and 9.8 mm and 4.2 mm (P < 0.001) at ×18 magnification for the digital 3D and oculars, respectively. Measured depth of field was 4.00 mm and 6.78 mm at ×5 magnification (P = 0.027), 0.72 mm and 0.86 mm at ×13 (P = 0.311), and 0.28 mm and 0.40 mm at ×18 magnification (P = 0.235) for the oculars and digital 3D, respectively. Lateral resolution of the digital 3D system was half that of the ocular viewing system and there was some improvement in depth of field with the digital system. Surgeon impression suggested that the digital system was superior when evaluating depth of field at high magnification.

GEOGRAPHICAL INFORMATION SYSTEMS AS A TOOL FOR 3D VISUALIZATION OF LOST ARCHITECTURAL HERITAGE

Abstract. Existing architectural heritage provides information about the architectural and urban characteristics of the civilizations that built them. Their surviving façade compositions, plan organizations, structural and material elements, give clues about various architectural and urban characteristics of these buildings and the periods that they belong to. In the case of lost architectural heritage, it is challenging to know more about these attributes due to the fact that the accessible information about these buildings could be limited. At this point, cartographic and photographic resources provide valuable information about lost architectural heritage. In addition to their geographic locations, information about their dimensions, structural systems, material finishes, colours could be some of the information that can be collected through cartographic and photographic historic documents. This paper presents a summary of the methodological approach of using Geographical Information System (GIS) for 3D visualization of lost architectural heritage in the case of Izmir – Turkey, by using 1905 fire insurance plans of Charles E. Goad.

Hot topic: Automated assessment of reticulo-ruminal motility in dairy cows using 3-dimensional vision

Reticulo-ruminal motility is a well-established indicator of gastrointestinal health in dairy cows. The currently available methods for assessing motility are labor-intensive, costly, and impractical to use regularly for all cows on a farm. We hypothesized that the reticulo-ruminal motility of dairy cows could be assessed automatically and remotely using a low-cost 3-dimensional (3D) camera. In this study, a 3D vision system was constructed and mounted on the frame of an automatic milking robot to capture the left paralumbar fossa of 20 primiparous cows. For each cow, the system recorded 3D images at 30 frames per second during milking. Each image was automatically processed to locate the left paralumbar fossa region and quantify its average concavity. Then, the average concavity values from all images of 1 cow during 1 milking process were chronologically assembled to form an undulation signal. By applying fast Fourier transformation to the signal, we identified cyclic oscillations that occurred in the same frequency range as reticulo-ruminal contractions. To validate the oscillation identification, 2 trained assessors visually identified reticulo-ruminal contractions from the same 3D image recordings on screen. The matching sensitivity between the automatically identified oscillations and the manually identified reticulo-ruminal contractions was 0.97. This 3D vision system can automate the assessment of reticulo-ruminal motility in dairy cows. It is noninvasive and can be implemented on farms without distressing the cows. It is a promising tool for farmers, giving them regular information about the gastrointestinal health of individual cows and helping them in daily farm management.

Photoacoustic lymphangiography.

Photoacoustic lymphangiography, which is based on photoacoustic technology, is an optical imaging that visualizes the distribution of light absorbing tissue components like hemoglobin or melanin, as well as optical absorption contrast imaging agents like indocyanine green (ICG) in the lymphatic channels, with high spatial resolution. In this report, we introduce the three-dimensional (3D) images of human lymphatic vessels obtained with photoacoustic lymphangiography. We used the 3D photoacoustic visualization system (PAI-05). Some healthy subjects and lymphedema patients were recruited. To image the lymphatic structures of the limbs ICG was administered subcutaneously as in fluorescence lymphangiography. Photoacoustic images were acquired by irradiating the tissue using a laser at wavelengths of near-infrared region. On the same occasion, fluorescence images were also recorded. The lymphatic vessels up to the diameter of 0.2 mm could be observed three-dimensionally with the venules around them. In the patient-group, dermal backflow patterns were often observed as dense interconnecting 3D structures of lymphatic vessels. Collecting vessels passing below the dermis were also observed, which were not observed by fluorescence lymphography. Photoacoustic lymphangiography provided the detailed observation of each lymphatic vessel, leading to deeper understanding of 3D structures and physiological state of the vessel.

Weed-removal system based on artificial vision and movement planning by A* and RRT techniques

The recent exploration of autonomous robotics tasks in agro-industry has permitted the integration of theories of artificial vision and mobile robotics with tasks in precision agriculture. Artificial vision allows for the classification of weeds and crops from images of plantations. With 3D-image processing systems, the location of the weeds is determined, and then the movement of the tool responsible for eradication is proposed. This article presents the solution for finding weeds within a crop field using classifiers and the integration of a 3D-vision system that builds a point cloud featuring the plants to safeguard, the weeds and the free space using Zed technology. With this information, search techniques such as A* (A star) and RRT (Rapidly exploring Random Tree) are used to determine the trajectory that the weed-removal tool must follow. The last feature is an integral part of an XYZ-positioning system, and this is part of a mobile robot dedicated to precision agriculture tasks.

Determining the position of pick holders on the side surface of the working unit of the cutting machine in the robotic technology of their assembly

One of the important issues related to the robotization of the manufacturing process of cutting machine working units is to determine the degree of movement of pick holders positioned by the industrial robot to the side surface of the cutting drum/head. This article examines the possibility of using vision system for the purpose of measuring the distance between the pick holder base and the side surface of the working unit of the cutting machine. Based on the boom-type roadheader cutting head, a mathematical measurement model has been formulated and a measurement procedure algorithm has been developed. Tests of the proposed measurement method with the use of a single camera were carried out at a test station. The station is equipped with the KUKA KR 16-2 industrial robot working together with the VisionTech vision system. This work contains examples of measurement results for three pick holders positioned differently in relation to the side surface of the cutting head. Based on distance measurement between the respective elements with the use of the 2D vision system along one line of known direction and position, a prediction of the distance distribution along the whole surface of the pick holder base has been made.

New technologies applied to the study and application of digital anatomy. HDM project (human dissection models). 5 years of experience

Introduction: Students learn and process information in different ways. In macroscopic anatomy, certainly the visual part plays an important role [1]. The increasing prevalence of technology means that medical educational software applications have an increasingly important role in medical education. We come up with our experience of 5 years in the creation of models and three-dimensional (3D) materials applied to the teaching and understanding of macroscopic anatomy. Presented at the 3rd World Congress of the UNESCO Chair for Teaching and Research in Digital Anatomy Paris Descartes at Egas Moniz University Institute - Almada, Portugal. Materials and methods: Surface scanning technologies such as 3D scanners and photometry software were used to obtain 3D models made from anatomical dissection models in cadaveric material. The models were loaded in the free Access online platform (www.proyectohdm.com) applied to online 3D visualization systems, anaglyph, augmented reality, virtual reality, as well as advanced prototypes of mixed reality and 3D printing in solid and flexible materials to obtain surgical educational simulators. Results: An online platform, easily accessible and multiplatform, was created with 50 three-dimensional topographic models of different anatomical structures, which allow users to view all angles of the anatomical piece. 3D anatomical models can be visualized in technologies such as anaglyph, augmented reality and virtual reality. The 3D printing allowed obtaining replicas of the cadaveric model of dissection, in various types of materials at low cost for its anatomical study. Printing on flexible materials enabled the possibility of creating educational neurosurgical simulators. The platform currently has more than 100 thousand monthly visitors from around the world. Discussion and conclusions: The use of 3D technologies applied to the field of anatomical study has been previously demonstrated [2]. The anatomical models presented in this work to be generated from the cadaverous structure allow a high level of detail and precision. Through ts incorporation into visualization systems in anaglyph technologies, virtual reality and increased reintegration of the virtual model to a tangible plane, using mixed reality and 3D printing, students and academics have a better understanding of the three-dimensional disposition of the anatomical structure manifested by the academic acceptance of the online platform, based on the growing number of visitors. Figure 1. A) Brain photography B) Reconstruction 3D brain C) Virtual reality D) 3D printing flexible material E) Augmented reality.

Three-dimensional visual assessment and VR study of hilar cholangiocarcinoma with portal vein as the axis

Objective To explore the application value of three-dimensional visual (3DV) assessment and virtual reality (VR) study of type Ⅲ and Ⅳ hilar cholangiocarcinoma with portal vein as the axis. Methods The CT image data of 10 patients who were diagnosed as Bismuth type Ⅲ and Ⅳ hilar cholangiocarcinoma by B-ultrasound and CT in Zhujiang Hospital, Southern Medical University were imported into 3D visualization system (3DVS) for 3D image reconstruction, and the portal vein was used as the axis for 3DV analysis and evaluation. The 3D data were then imported into the VR development engine to perform VR research, so as to help the operation planning. Both of the image data were taken into the operation room for intraoperative navigation. Results 10 patients completed 3D visualization reconstruction and VR model transformation. According to the 3D visualization of hepatic hilar hepatocarcinoma, there were 1 case of type Ⅱ, 2 cases of type Ⅲa, 5 cases of type Ⅲb, 1 case of type Ⅳa and 1 case of type Ⅳb. The portal vein was classified according to Cheng type: 6 cases of type Ⅰ, 3 cases of type Ⅱ, and 1 case of type Ⅲ. The hepatic artery was classified according to Michels type: 5 cases of type Ⅰ, 1 case of type Ⅱ, 3 cases of type Ⅲ, and 1 case of type Ⅸ. The 3DV assessment and VR study were almost consistent with the intraoperative findings. No liver failure or death occurred during the perioperative period. Conclusion The 3DV assessment and VR study with the portal vein as the axis have potential application for the surgical treatment of type Ⅲ and Ⅳ hilar cholangiocarcinoma. Key words: Bile duct neoplasms; Biliary tract surgical procedures; Imaging, three-dimensional; Portal vein; Visualization; Virtual reality

Validation and Verification of Three-dimensional Systems in Laparoscopic Distal Pancreatectomy.

The three-dimensional (3D) visualization system has been introduced for the technical improvement of laparoscopic surgery, but clinical evidence for its usefulness is lacking. This study aimed to evaluate the usefulness of a 3D system in laparoscopic surgery. Between 2015 and 2016, a randomized controlled trial was performed on 61 patients who underwent laparoscopic distal pancreatectomy. The primary outcome was the shortening of operative time. The hospital course, number of technical errors, and clinicians' subjective scores were compared as secondary outcome. There were no significant differences in operative times, hospital course and technical errors. However, depth perception was significantly improved and physical demand was reduced in the 3D group. These advantages became clearer in the later period of the study, with longer operative times, and in spleen-preserving procedures. Subjective scoring through 3D systems was significantly superior, which might lead to better surgical performance in difficult cases.

3D data visualization system of immersive underground laboratory

The construction of underground laboratories is a key step in the deep geological disposal of high- level radioactive waste. Due to the complex construction structure of underground laboratory engineering and the difficulty of project implementation, how to effectively plan, design and evaluate underground laboratory construction projects is an important issue in the construction of underground laboratories. Based on geometric structure, topological relationship and geological semantics, this paper integrates multi-source data from underground laboratories and constructs a 3D digital model of underground laboratory. The 3D data visualization system of underground laboratory of immersed high-level waste geological disposal was constructed by using terrain automatic generation algorithm, perspective and section in 3D model visualization, and immersive interactive means of viewpoint positioning. The experimental results show that the system can provide an intuitive interactive means for the comprehensive analysis of underground laboratory data, and provide reference for the demonstration of underground laboratory engineering construction planning and design.