3D Pipeline Research Articles

Real-time, dense UAV mapping by leveraging monocular depth prediction with monocular-inertial SLAM

We present a dense and metric 3D mapping pipeline designed for embedded operation on-board UAVs, by loosely coupling deep neural networks trained to infer dense depth single images with a SLAM system that restores metric scale from sparse depth. In contrast to computationally restrictive approaches that leverage multiple views, we propose a highly efficient, single-view approach without sacrificing 3D mapping performance. This enables real-time construction of a global 3D voxel map by iterative fusion of the rescaled dense depth maps obtained via raycasting from the estimated camera poses. Quantitative and qualitative experimentations of our framework in challenging environmental conditions show comparable or superior performance with respect to state-of-the-art approaches via a better effectiveness-efficiency trade-off.

Intelligent recognition and automatic localization of pipeline welds based on multi-vision system

Abstract Currently, the leakage detection of spacecraft pipeline welds relies on manual point-by-point inspection using a detection gun, which is inefficient and inadequate for the automation needs of spacecraft production. However, the accurate recognition and precise localization of widely distributed and small pipeline welds are crucial for automated detection. Therefore, this paper proposes a multi-vision detection and localization system that integrates global and local information, considering both comprehensive global 3D search and high-precision local 3D measurement. The improved YOLOv8 model is employed for pipeline weld recognition, which improves the recognition rate of welds. Based on the deep learning recognized and segmented welds, this paper proposes stereo matching and segmentation extraction methods for 3D localization and pipeline orientation determination. Additionally, the system integrates a robot to perform automated point-by-point inspection of welds within the area without collisions. The experimental results demonstrate the effectiveness of the improved YOLOv8 and the proposed methods for 3D weld localization and pipeline orientation determination. The maximum deviation of the spatial distance of fine weld positioning is 0.20 mm, and the repeatability of the 3D coordinates is around 0.1 mm. The system can perform precise localization and detection, meeting the requirements for automatic weld recognition and localization.

Explicit 3D reconstruction from images with dynamic graph learning and rendering-guided diffusion

High-quality 3D reconstruction is becoming increasingly important in a variety of fields. Recently, implicit representation methods have made significant progress in image-based 3D reconstruction. However, these methods tend to yield entangled neural representations which lack support for standard 3D pipelines, and their reconstruction results usually experience a sharp drop in quality when input views are reduced. To obtain high-quality 3D content, we propose an explicit 3D reconstruction method that directly extracts textured meshes from images and remains robust using reduced input views. Our central components include a dynamic graph convolutional network (GCN) and a rendering-guided diffusion model. The dynamic GCN aims to improve mesh reconstruction quality by effectively aggregating features from vertex neighborhoods. The aggregation is accelerated through sampling geometric-related neighbors with different SDF signs, which gradually converges in quantity during training. The rendering-guided diffusion model learns prior distributions for unseen regions to improve reconstruction performance using sparse-view inputs. It uses the rendered image under an interpolated camera pose as conditioned input and its diffusion strength can be controlled with the rendering loss of explicit reconstruction. In addition, the rendering-guided diffusion model can be jointly trained to generate plausible novel views with 3D consistency. Experiments demonstrate that our method can produce high-quality explicit reconstruction results and maintain realistic reconstruction using sparse-view inputs.

Extended nnU-Net for Brain Metastasis Detection and Segmentation in Contrast-Enhanced Magnetic Resonance Imaging With a Large Multi-Institutional Data Set

PurposeTo investigate an extended self-adapting nnU-Net framework for detecting and segmenting brain metastases (BM) on MRI. ApproachSix different nnU-Net systems with adaptive data sampling, adaptive Dice loss (ADL) or different patch/batch sizes were trained and tested for detecting and segmenting intraparenchymal BM with a size ≥ 2 mm on 3D post-Gd T1-weighted MRI volumes using 2092 patients from seven institutions (1712, 195 and 185 patients for training, validation and testing, respectively). Gross tumor volumes (GTVs) of BM delineated by physicians for stereotactic radiosurgery (SRS) were collected retrospectively and curated at each institute. Additional centralized data curation was carried out to create GTVs of uncontoured BM by two radiologists to improve accuracy of ground truth. The training dataset was augmented with synthetic BMs of 1025 MRI volumes using a 3D generative pipeline. BM detection was evaluated by lesion-level sensitivity and false-positive (FP) rate. BM segmentation was assessed by lesion-level Dice similarity coefficient (DSC), 95-percentile Hausdorff distance (HD95) and average HD. The performances were assessed across different BM sizes. Additional testing was performed using a second dataset of 206 patients. ResultsOf the six nnU-Net systems, the nnU-Net with ADL achieved the best detection and segmentation performance on the first testing dataset. At an FP rate of 0.65±1.17, overall sensitivity was 0.904 for all sizes of BM, 0.966 for BM ≥ 0.1 cm3 and 0.824 for BM < 0.1 cm3. Mean values of DSC, HD95 and average HD of all detected BMs were 0.758, 1.45 mm and 0.23 mm, respectively. Performances on the second testing dataset achieved sensitivity of 0.907 at an FP rate of 0.57±0.85 for all BM sizes, and average HD of 0.33 mm for all detected BM. ConclusionsOur proposed extension of the self-configuring nnU-Net framework substantially improved small BM detection sensitivity while maintaining a controlled FP rate. Clinical utility of the extended nnU-Net model for assisting early BM detection and SRS planning will be investigated.

Spark-Discharge-Activated 3D-Printed Electrochemical Sensors.

3D printing technology is a tremendously powerful technology to fabricate electrochemical sensing devices. However, current conductive filaments are not aimed at electrochemical applications and therefore require intense activation protocols to unleash a suitable electrochemical performance. Current activation methods based on (electro)chemical activation (using strong alkaline solutions and organic solvents and/or electrochemical treatments) or combined approaches are time-consuming and require hazardous chemicals and dedicated operator intervention. Here, pioneering spark-discharge-activated 3D-printed electrodes were developed and characterized, and it was demonstrated that their electrochemical performance was greatly improved by the effective removal of the thermoplastic support polylactic acid (PLA) as well as the formation of sponge-like and low-dimensional carbon nanostructures. This reagent-free approach consists of a direct, fast, and automatized spark discharge between the 3D-electrode and the respective graphite pencil electrode tip using a high-voltage power supply. Activated electrodes were challenged toward the simultaneous voltammetric determination of dopamine (DP) and serotonin (5-HT) in cell culture media. Spark discharge has been demonstrated as a promising approach for conductive filament activation as it is a fast, green (0.94 GREEnness Metric Approach), and automatized procedure that can be integrated into the 3D printing pipeline.

Automatic generation of pipelines from 2D CAD drawings to 3D IFC models using a parametric method

Creating 3D models of pipelines is essential for effective digital building management. However, generating 3D pipeline models from 2D CAD drawings still heavily depends on human-computer interaction at the design stage. This paper addresses this challenge by proposing a method to generate parametric 3D pipeline models from CAD drawings automatically. The framework includes three parts. Firstly, automatic layer recognition algorithms are used to extract data from CAD drawings, and algorithms are designed to match pipeline coordinates and diameters. Then, data on straight pipes is processed, including judging types of pipe fittings and calculating relevant parameters to realize the automatic generation of 3D BIM models. Finally, IFC is utilized to express the geometry and topology of models for BIM model generation. The process is implemented by writing a C++ program in Visio Studio 2019. Experimental results on a building water supply and drainage system demonstrate that the proposed method can significantly improve the generation efficiency and automation level of pipeline models.

Illuminating the complete ß-cell mass of the human pancreas- signifying a new view on the islets of Langerhans

Pancreatic islets of Langerhans play a pivotal role in regulating blood glucose homeostasis, but critical information regarding their mass, distribution and composition is lacking within a whole organ context. Here, we apply a 3D imaging pipeline to generate a complete account of the insulin-producing islets throughout the human pancreas at a microscopic resolution and within a maintained spatial 3D context. These data show that human islets are far more heterogenous than previously accounted for with regards to their size distribution and cellular make up. By deep tissue 3D imaging, this in-depth study demonstrates that 50% of the human insulin-expressing islets are virtually devoid of glucagon-producing α-cells, an observation with significant implications for both experimental and clinical research.

A novel 3D imaging pipeline for analyzing efficacy of compounds on amyloid-beta plaque dynamics in preclinical alzheimer’s disease animal models

A novel 3D imaging pipeline for analyzing efficacy of compounds on amyloid-beta plaque dynamics in preclinical alzheimer’s disease animal models

IDet3D: Towards Efficient Interactive Object Detection for LiDAR Point Clouds

Accurately annotating multiple 3D objects in LiDAR scenes is laborious and challenging. While a few previous studies have attempted to leverage semi-automatic methods for cost-effective bounding box annotation, such methods have limitations in efficiently handling numerous multi-class objects. To effectively accelerate 3D annotation pipelines, we propose iDet3D, an efficient interactive 3D object detector. Supporting a user-friendly 2D interface, which can ease the cognitive burden of exploring 3D space to provide click interactions, iDet3D enables users to annotate the entire objects in each scene with minimal interactions. Taking the sparse nature of 3D point clouds into account, we design a negative click simulation (NCS) to improve accuracy by reducing false-positive predictions. In addition, iDet3D incorporates two click propagation techniques to take full advantage of user interactions: (1) dense click guidance (DCG) for keeping user-provided information throughout the network and (2) spatial click propagation (SCP) for detecting other instances of the same class based on the user-specified objects. Through our extensive experiments, we present that our method can construct precise annotations in a few clicks, which shows the practicality as an efficient annotation tool for 3D object detection.

Abstract 6767: Multivariate analysis of ex vivo kidney 3D immune-microtumors for functional precision medicine

Abstract Immuno-oncology (IO) represents a significant paradigm shift in cancer treatment, focused on leveraging and mobilizing a patient’s own immune response and anti-tumor capabilities. Despite accounting for the majority of available treatment options, there are currently no diagnostic tests capable of predicting combination immunotherapy response in kidney cancer. We have developed a fully human 3D IO model for clinical drug efficacy prediction through multivariate analysis of the tumor-immune microenvironment (TiME) combining functional assays, artificial intelligence and multi-omics. These immune-microtumors recapitulate a patient’s unique pathophysiology and allow 3D spatio-temporal analyses of functional metrics at both bulk and single-cell resolutions. Renal cell carcinoma (RCC) resections and matched blood were collected and processed for single cell isolation of tumor and PBMCs (N=20, ISRCTN10001405). Flow cytometry (FC) and immunofluorescence (IF) were used to characterize TiME cell subpopulations (cancer, immune, stromal etc.) in pellets and 3D cultures, and to deconvolute bulk RNAseq data. Target (tumor) and effector cells (PBMCs and CD8+) were stained with fluorescent probes, activated using ɑCD3 and encapsulated in TiME-mimicking 3D matrices. 3D immune-microtumors were treated with FDA-approved regimens including immunotherapies (ipilimumab, nivolumab, pembrolizumab) and receptor-tyrosine kinase inhibitors (TKIs)(cabozantinib, lenvatinib, axitinib) as monotherapies or combination therapies. Differential drug efficacies were functionally quantified over 4 days using 3D confocal microscopy and computer vision pipelines to extract cell behavior metrics such as immune cell infiltration, immune/tumor cell migration speed, and tumor killing/viability. We devised a specialized RCC hydrogel formulation with superior viability compared to MatrigelTM alone (N=7). FC/IF/RNAseq were used to quantify relative cell subpopulations after isolation and show protein/transcriptional signature retention after 3D culture (N=7). The platform was conducive to the testing of various drugs with varying MoAs including immunotherapies, TKIs, targeted therapies and cell therapies. Treatment with pembrolizumab led to 26% higher PBMC infiltration, 14% higher CD8+ infiltration and 15% increased tumor cell death compared to control. When in combination, pembrolizumab reduced cell viability by 10% and 19% with lenvatinib and axitinib respectively (N=1). Population analyses revealed intra-patient response heterogeneity (N=7). In conclusion, we showed MoA-agnostic, patient-dependent responses to FDA-approved clinical treatment regimens via molecular and functional spatio-temporal analyses. Clinical outcome correlation is currently underway in our advanced RCC study PEAR-TREE2 as well as in breast NCT05435352 and brain NCT06038760 tumors. Citation Format: Eleonora Peerani, Keqian Nan, Demi A. Wiskerke, Thomas D. Richardson, Polina Maximchick, Jay Kearney, George R. de Fraine, Kerrie L. Loughrey, Jonathan Ient, Jessica A. Hudson, Andreas Kaffa, Aston M. Crawley, Nourdine K. Bah, Edgar G. Lopez Molina, Elli Tham, Francesco Iori, Matthew H. Williams, Maxine G. Tran, Duleek N. Ranatunga. Multivariate analysis of ex vivo kidney 3D immune-microtumors for functional precision medicine [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2024; Part 1 (Regular Abstracts); 2024 Apr 5-10; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2024;84(6_Suppl):Abstract nr 6767.

Machine learning-assisted high-content imaging analysis of 3D MCF7 microtissues for estrogenic effect prediction

Endocrine-disrupting chemicals (EDCs) pose a significant threat to human well-being and the ecosystem. However, in managing the many thousands of uncharacterized chemical entities, the high-throughput screening of EDCs using relevant biological endpoints remains challenging. Three-dimensional (3D) culture technology enables the development of more physiologically relevant systems in more realistic biochemical microenvironments. The high-content and quantitative imaging techniques enable quantifying endpoints associated with cell morphology, cell–cell interaction, and microtissue organization. In the present study, 3D microtissues formed by MCF-7 breast cancer cells were exposed to the model EDCs estradiol (E2) and propyl pyrazole triol (PPT). A 3D imaging and image analysis pipeline was established to extract quantitative image features from estrogen-exposed microtissues. Moreover, a machine-learning classification model was built using estrogenic-associated differential imaging features. Based on 140 common differential image features found between the E2 and PPT group, the classification model predicted E2 and PPT exposure with AUC-ROC at 0.9528 and 0.9513, respectively. Deep learning-assisted analysis software was developed to characterize microtissue gland lumen formation. The fully automated tool can accurately characterize the number of identified lumens and the total luminal volume of each microtissue. Overall, the current study established an integrated approach by combining non-supervised image feature profiling and supervised luminal volume characterization, which reflected the complexity of functional ER signaling and highlighted a promising conceptual framework for estrogenic EDC risk assessment.

Unraveling gender-specific structural brain differences in drug-resistant epilepsy using advanced deep learning techniques

Factors of age, gender, and psychiatric comorbidities in epileptic patients, particularly those with drug-resistant epilepsy (DRE), have not received sufficient attention in clinical practice and research, current research in this domain focus primarily on seizure management. Consequently, a detailed investigation of these differences to understand how each gender's brain changes as a prognosis for epilepsy remains lacking. Furthermore, no previous studies delved into the use of 3D structural MRI (sMRI)-based deep neural networks to predict the biological gender (BG) of epileptic patients. To address this gap and gain insights into the structural aspects of epileptic patients' brains, this study proposed various approaches employing sMRI-based deep neural networks for predicting the BG of epileptic patients. Additionally, it will introduce an innovative preprocessing pipeline, the 3D brain pipeline, and compare it with the standard voxel-based morphometry (VBM) pipeline. the results concluded that there are obvious structural brain differences between genders in epileptic patients, which can be effectively predicted by deep learning approaches, despite the variations that could be raised from age and development of epilepsy. The results also showed that the standard VBM pipeline performs novel 3D brain pipeline, achieving higher metrics, including accuracy (0.961) and AUC (0.97). These findings underscore the significance of considering gender-specific brain changes in epilepsy research and clinical practices, where patients should be treated separately based on their gender.

AppleQSM: Geometry-Based 3D Characterization of Apple Tree Architecture in Orchards.

The architecture of apple trees plays a pivotal role in shaping their growth and fruit-bearing potential, forming the foundation for precision apple management. Traditionally, 2D imaging technologies were employed to delineate the architectural traits of apple trees, but their accuracy was hampered by occlusion and perspective ambiguities. This study aimed to surmount these constraints by devising a 3D geometry-based processing pipeline for apple tree structure segmentation and architectural trait characterization, utilizing point clouds collected by a terrestrial laser scanner (TLS). The pipeline consisted of four modules: (a) data preprocessing module, (b) tree instance segmentation module, (c) tree structure segmentation module, and (d) architectural trait extraction module. The developed pipeline was used to analyze 84 trees of two representative apple cultivars, characterizing architectural traits such as tree height, trunk diameter, branch count, branch diameter, and branch angle. Experimental results indicated that the established pipeline attained an R2 of 0.92 and 0.83, and a mean absolute error (MAE) of 6.1cm and 4.71mm for tree height and trunk diameter at the tree level, respectively. Additionally, at the branch level, it achieved an R2 of 0.77 and 0.69, and a MAE of 6.86mm and 7.48° for branch diameter and angle, respectively. The accurate measurement of these architectural traits can enable precision management in high-density apple orchards and bolster phenotyping endeavors in breeding programs. Moreover, bottlenecks of 3D tree characterization in general were comprehensively analyzed to reveal future development.

Modular 3D printed platform for fluidically connected human brain organoid culture

Brain organoid technology has transformed both basic and applied biomedical research and paved the way for novel insights into developmental processes and disease states of the human brain. While the use of brain organoids has been rapidly growing in the past decade, the accompanying bioengineering and biofabrication solutions have remained scarce. As a result, most brain organoid protocols still rely on commercially available tools and culturing platforms that had previously been established for different purposes, thus entailing suboptimal culturing conditions and excessive use of plasticware. To address these issues, we developed a 3D printing pipeline for the fabrication of tailor-made culturing platforms for fluidically connected but spatially separated brain organoid array culture. This all-in-one platform allows all culturing steps—from cellular aggregation, spheroid growth, hydrogel embedding, and organoid maturation—to be performed in a single well plate without the need for organoid manipulation or transfer. Importantly, the approach relies on accessible materials and widely available 3D printing equipment. Furthermore, the developed design principles are modular and highly customizable. As such, we believe that the presented technology can be easily adapted by other research groups and fuel further development of culturing tools and platforms for brain organoids and other 3D cellular systems.

Abstract 14773: Automated Cardiovascular Calcium Burden: Unlocking the Future of TAVR Prognosis With Artificial Intelligence

Background: Calcific valve disease is now understood as a dynamic inflammatory process that alters the tissue’s adaptive response to stress stimuli. Transcatheter aortic valve replacement (TAVR) is a minimally invasive treatment for severe aortic stenosis. Manual analysis of calcifications in pre-TAVR CT scans is time-consuming and subjective. Hypothesis: Deep learning (DL) models can accurately quantify cardiovascular calcification and predict post-procedural outcomes in TAVR patients. We aimed to develop such a model and investigate its correlation with long-term all-cause mortality in TAVR patients. Methods: This post-hoc analysis examined patients from the FRAILTY-AVR trial and the Royal Victoria Hospital TAVR registry. A 3D UNet-based DL pipeline was developed and trained to detect and quantify coronary artery calcification (CAC), aortic valve calcification (AVC), mitral annular calcification (MAC), and thoracic aorta calcification (TAC) based on pre-procedural CT images. Quantification involved segmentation and CT image analysis techniques. Cox regression analysis, adjusting for various covariates, evaluated the primary endpoint of all-cause mortality one year after TAVR. Results: The study included 585 TAVR patients (mean age: 82.5 years; 44% females) with a median follow-up of 506 days. At one year, 27.5%(161) reached the primary endpoint of all-cause mortality. Hazard ratios for MAC, AVC, TAC, and CAC were 1.26 (95% CI: 1.06-1.51, P=0.010), 0.99 (95% CI: 0.83-1.18, P=0.91), 1.07 (95% CI: 0.89-1.29, P=0.46), and 0.89 (95% CI: 0.72-1.09, P=0.26), respectively. Adjustments for covariates did not significantly change these hazard ratios. Mean calcification volumes were 770 mm3 (AVC), 737 mm3 (CAC), 3076 mm3 (TAC), and 629 mm3 (MAC). No substantial correlations were observed between the four calcifications or cardiovascular co-morbidities however, each score may yield meaningful information. Conclusions: High mitral annular calcification, quantified by our AI pipeline, was independently associated with increased one-year post-TAVR mortality. AI enables efficiency, objectivity, and improved insights, enhancing clinical decision-making and personalized care.

Research on the 3D Reverse Time Migration Technique for Internal Defects Imaging and Sensor Settings of Pressure Pipelines.

Although pressure pipelines serve as a secure and energy-efficient means of transporting oil, gas, and chemicals, they are susceptible to fatigue cracks over extended periods of cyclic loading due to the challenging operational conditions. Their quality and efficiency directly affect the safe operation of the project. Therefore, a thorough and precise characterization approach towards pressure pipelines can proactively mitigate safety risks and yield substantial economic and societal benefits. At present, the current mainstream 2D ultrasound imaging technology faces challenges in fully visualizing the internal defects and topography of pressure pipelines. Reverse time migration (RTM), widely employed in geophysical exploration, has the capability to visualize intricate geological structures. In this paper, we introduced the RTM into the realm of ultrasonic non-destructive testing, and proposed a 3D ultrasonic RTM imaging method for internal defects and sensor settings of pressure pipelines. To accurately simulate the extrapolation of wave field in 3D pressure pipelines, we set the absorbing boundary and double free boundary in cylindrical coordinates. Subsequently, using the 3D ultrasonic RTM approach, we attained higher-precision 3D imaging of internal defects in the pressure pipelines through suppressing imaging artifacts. By comparing and analyzing the imaging results of different sensor settings, the design of the observation system is optimized to provide a basis for the imaging and interpretation of actual data. Both simulations and actual field data demonstrate that our approach delivers top-notch 3D imaging of pipeline defects (with an imaging range accuracy up to 97.85%). This method takes into consideration the complexities of multiple scattering and mode conversions occurring at the base of the defects as well as the optimal sensor settings.

An Automated Brain Metastasis Detection and Segmentation System from MRI with a Large Multi-Institutional Dataset

Developments of automated systems for brain metastasis (BM) detection and segmentation from MRI for assisting early detection and stereotactic radiosurgery (SRS) have been reported but most based upon relatively small datasets from single institutes. This work aims to develop and evaluate a system using a large multi-institutional dataset, and to improve both identification of small/subtle BMs and segmentation accuracy of large BMs. A 3D U-Net system was trained and evaluated to detect and segment intraparenchymal BMs with a size > 2mm using 1856 MRI volumes from 1791 patients treated with SRS from seven institutions (1539 volumes for training, 183 for validation, and 134 for testing). All patients had 3D post-Gd T1w MRI scans pre-SRS. Gross tumor volumes (GTVs) of BMs for SRS were curated by each institute first. Then, additional efforts were spent to create GTVs for the untreated and/or uncontoured BMs, including central reviews by two radiologists, to improve accuracy of ground truth. The training dataset was augmented with synthetic BMs of 3773 MRIs using a 3D generative pipeline. Our system consists of two U-Nets with one using small 3D patches dedicated for detecting small BMs and another using large 3D patches for segmenting large BMs, and a random-forest based fusion module for combining the two network outputs. The first U-Net was trained with 3D patches containing at least one BM < 0.1 cm3. For detection performance, we measured BM-level sensitivity and case-level false-positive (FP) rate. For segmentation performance, we measured BM-level Dice similarity coefficient (DSC) and 95-percentile Hausdorff distance (HD95). We also stratified performances based upon BM sizes. For 739 BMs in the 134 testing cases, the overall lesion-level sensitivity was 0.870 with an average case-level FP of 1.34±1.92 (95% CI: 1.02-1.67). The sensitivity was >0.969 for the BMs >0.1 cm3, but dropped to 0.755 for the BMs < 0.1 cm3 (Table 1). The average DSC and HD95 for all detected BMs were 0.786 and 1.35mm. The worse performance for BMs > 20 cm3 was caused by a case with 83 cm3 GTV and artifacts in the MRI volume. We achieved excellent detection sensitivity and segmentation accuracy for BMs > 0.1 cm3, and promising performance for small BMs (<0.1cm3) with a controlled FP rate using a large multi-institutional dataset. Clinical utility for assisting early detection and SRS planning will be investigated. Table 1: Per-lesion detection and segmentation performance stratified by individual BM size. N is the number of BMs in each category.

GPU Rasterization-Based 3D LiDAR Simulation for Deep Learning.

High-quality data are of utmost importance for any deep-learning application. However, acquiring such data and their annotation is challenging. This paper presents a GPU-accelerated simulator that enables the generation of high-quality, perfectly labelled data for any Time-of-Flight sensor, including LiDAR. Our approach optimally exploits the 3D graphics pipeline of the GPU, significantly decreasing data generation time while preserving compatibility with all real-time rendering engines. The presented algorithms are generic and allow users to perfectly mimic the unique sampling pattern of any such sensor. To validate our simulator, two neural networks are trained for denoising and semantic segmentation. To bridge the gap between reality and simulation, a novel loss function is introduced that requires only a small set of partially annotated real data. It enables the learning of classes for which no labels are provided in the real data, hence dramatically reducing annotation efforts. With this work, we hope to provide means for alleviating the data acquisition problem that is pertinent to deep-learning applications.

Real-Time 3D Reconstruction Pipeline for Room-Scale, Immersive, Medical Teleconsultation

Medical teleconsultation was among the initial use cases for early telepresence research projects since medical treatment often requires timely intervention by highly specialized experts. When remote medical experts support interventions, a holistic view of the surgical site can increase situation awareness and improve team communication. A possible solution is the concept of immersive telepresence, where remote users virtually join the operating theater that is transmitted based on a real-time reconstruction of the local site. Enabled by the availability of RGB-D sensors and sufficient computing capability, it becomes possible to capture such a site in real time using multiple stationary sensors. The 3D reconstruction and simplification of textured surface meshes from the point clouds of a dynamic scene in real time is challenging and becomes infeasible for increasing capture volumes. This work presents a tightly integrated, stateless 3D reconstruction pipeline for dynamic, room-scale environments that generates simplified surface meshes from multiple RGB-D sensors in real time. Our algorithm operates directly on the fused, voxelized point cloud instead of populating signed-distance volumes per frame and using a marching cube variant for surface reconstruction. We extend the formulation of the dual contouring algorithm to work for point cloud data stored in an octree and interleave a vertex-clustering-based simplification before extracting the surface geometry. Our 3D reconstruction pipeline can perform a live reconstruction of six incoming depth videos at their native frame rate of 30 frames per second, enabling the reconstruction of smooth movement. Arbitrarily complex scene changes are possible since we do not store persistent information between frames. In terms of mesh quality and hole filling, our method falls between the direct mesh reconstruction and expensive global fitting of implicit functions.

3D Hyperspectral Light-Field Imaging: a first intraoperative implementation

Abstract Hyperspectral imaging is an emerging technology that has gained significant attention in the medical field due to its ability to provide precise and accurate imaging of biological tissues. The current methods of hyperspectral imaging, such as filter-wheel, snapshot, line-scanning, and push-broom cameras have limitations such as low spatial and spectral resolution, slow acquisition time. New developments on the field of light field cameras show the potential to overcome these limitations. In this paper, we use a novel hyperspectral lightfield camera and try to combine the capability of hyperspectral and 3D analysis. For this purpose we calibrate our system and test it during two ENT-surgeries to show its potential for improving surgical outcomes. The micro-lenses of the camera map 66 spectral sub-images onto the sensor allowing to reconstruct the spectral behavior of the captured scene in the spectral range of 350-1000nm. In addition, we use the sensor data to apply a 3D camera calibration pipeline to allow 3D surface reconstruction. We captured 26 calibration images and achieved calibration results in accordance to stated company data. The best calibration showed a re-projection error of 0.55 px. Further, we tested the camera during a parotidectomy and a neck-dissection. The extracted reflectance spectra of the selected venal and arterial regions correspond perfectly to the spectrum of oxygenated and deoxygenated hemoglobin. For the first time, up to our knowledge, a hyperspectral lightfield camera has been used during a surgery. We were able to continuously capture images and analyze the reconstructed spectra of specific tissue types. Further, we are able to use the sensor data of the micro-lens projections to calibrate the multilens camera system for later intraoperative measurement tasks.