Real-time Reconstruction Research Articles

Application of SLAM in endoscopic imaging

This paper presents the utilization of Simultaneous Localization and Mapping (SLAM) technology in medical endoscopic imaging. The fundamental components, hardware configuration, and process of SLAM are introduced in detail, with a focus on sensor acquisition, data preprocessing, feature extraction and matching, state estimation and update, map construction, and optimization modules. The application of SLAM in the medical field is then discussed, specifically highlighting real-time localization and reconstruction of endoscopic imaging. The integration of SLAM technology can assist doctors in accurately identifying the site of lesions, thereby enhancing surgical precision and safety. Furthermore, the paper introduces various commonly used SLAM algorithms, including the Kalman filter, extended Kalman filter, particle filter, optimization algorithms, among others. It emphasizes the significance of algorithm selection and optimization tailored to different scenarios and requirements. In conclusion, the application of SLAM technology in medical endoscopy imaging holds immense potential, offering improved accuracy and visualization during surgical procedures. This technology provides valuable support for doctors in diagnosis and treatment, ultimately enhancing patient outcomes.

A frequency-domain model-based reconstruction method for transcranial photoacoustic imaging: A 2D numerical investigation

Phase aberration caused by the skull is a major barrier to achieving high quality photoacoustic images of human and non-human primates’ brains. To address this issue, time-reversal methods have been used but they are computationally demanding and slow due to relying on solving the full-wave equation. The proposed approach is based on model-based image reconstruction in the frequency-domain to achieve near real-time image reconstruction. The relationship between an imaging region and transducer array elements can be mathematically described as a model matrix and the image reconstruction can be performed by pseudo-inverse of the model matrix. The model matrix is numerically calculated due to the lack of analytical solutions for transcranial ultrasound. However, this calculation only needs to be performed once for a given experimental setup and the same acoustic medium, and is an offline process not affecting the actual image reconstruction time. This non-iterative mode-based method demonstrates a substantial improvement in image reconstruction time, being approximately 18 times faster than the time-reversal method, all while maintaining comparable image quality.

Real-time localization and 3D semantic map reconstruction for unstructured citrus orchards

Semantic maps play a crucial role in smart agriculture, providing practical three-dimensional fruit tree data for orchard management and aiding the optimization of management strategies and improvement of economic benefits. However, previous map studies have mainly focused on geometric features and have lacked semantic information, limiting robots' ability to reason about useful information in complex tasks and achieve human–machine interaction. Furthermore, most existing map reconstructions have been offline or nonreal-time, making it difficult to satisfy the needs of real-time decision-making and planning in agricultural scenarios. Therefore, this paper proposes a real-time localization and semantic map reconstruction method for unstructured citrus orchards, integrating the visual-inertial SLAM VINS-RGBD framework with the semantic segmentation algorithm BiSeNetV1. By conducting semantic segmentation of 2D RGB images and mapping them to point clouds, a 3D semantic point cloud map is reconstructed. The statistical outlier removal filter and OctoMap are introduced for postprocessing to remove outliers and estimate obstacles in 3D space, constructing a more accurate, efficient and flexible map. The experimental results show that the proposed method achieved a semantic segmentation accuracy mIoU of 79.31% on a self-built citrus dataset, a citrus recall relative error of 11.29% and a localization accuracy mean translational error of 1.917 m with the map constructed under an unstructured orchard scenario. Additionally, the average memory saving rate of the statistical outlier removal filter was 10.36%, and the average memory saving rate of OctoMap was 97.39%. The processing time for each frame of real-time front-end feature detection and tracking was 11.14 ms. Moreover, the deployed semantic segmentation network BiSeNetV1 achieved a processing time of 7.35 ms per frame. These results indicate that the proposed method can achieve both high accuracy and real-time performance in semantic map reconstruction. This exploratory work provides theoretical and technical references for future research on more precise localization and more complete semantic mapping and has extensive application potential, providing essential technical support for intelligent agriculture.

Hybrid spiral scanning in a double-clad fiber-based handheld confocal scanning light ophthalmoscope.

High-speed, accessible, and robust in vivo imaging of the human retina is critical for screening of retinal pathologies, such as diabetic retinopathy, age-related macular degeneration, and others. Scanning light ophthalmoscopy (SLO) is a retinal imaging modality that produces digital, en face images of the human retina with superior image gradability rates when compared to the current standard of care in screening for these diseases, namely the flood-illumination handheld fundus camera (HFC). However, current-generation commercial SLO systems are mostly tabletop devices, limiting their accessibility and utility in screening applications. Moreover, most existing SLO systems use raster scan patterns, which are both inefficient and lead to undesired subject gaze drift when used with visible or pseudo-visible illumination. Non-raster scan patterns, especially spiral scanning as described herein, promise advantages in both scan efficiency and reduced subject eye motion. In this work, we introduce a novel "hybrid spiral" scan pattern and the associated hardware design and real-time image reconstruction techniques necessary for its implementation in an SLO system. Building upon this core hybrid spiral scanning SLO (HSS-SLO) technology, we go on to present a complete handheld HSS-SLO system, featuring a fiber-coupled portable patient interface which leverages a dual-clad fiber (DCF) to form a single-path optical topology, thus ensuring mechanically robust co-alignment of illumination and collection apertures, a necessity for a handheld system. The feasibility of HSS-SLO for handheld, in vivo imaging is demonstrated by imaging eight human volunteers.

Real-Time 3D Reconstruction Pipeline for Room-Scale, Immersive, Medical Teleconsultation

Medical teleconsultation was among the initial use cases for early telepresence research projects since medical treatment often requires timely intervention by highly specialized experts. When remote medical experts support interventions, a holistic view of the surgical site can increase situation awareness and improve team communication. A possible solution is the concept of immersive telepresence, where remote users virtually join the operating theater that is transmitted based on a real-time reconstruction of the local site. Enabled by the availability of RGB-D sensors and sufficient computing capability, it becomes possible to capture such a site in real time using multiple stationary sensors. The 3D reconstruction and simplification of textured surface meshes from the point clouds of a dynamic scene in real time is challenging and becomes infeasible for increasing capture volumes. This work presents a tightly integrated, stateless 3D reconstruction pipeline for dynamic, room-scale environments that generates simplified surface meshes from multiple RGB-D sensors in real time. Our algorithm operates directly on the fused, voxelized point cloud instead of populating signed-distance volumes per frame and using a marching cube variant for surface reconstruction. We extend the formulation of the dual contouring algorithm to work for point cloud data stored in an octree and interleave a vertex-clustering-based simplification before extracting the surface geometry. Our 3D reconstruction pipeline can perform a live reconstruction of six incoming depth videos at their native frame rate of 30 frames per second, enabling the reconstruction of smooth movement. Arbitrarily complex scene changes are possible since we do not store persistent information between frames. In terms of mesh quality and hole filling, our method falls between the direct mesh reconstruction and expensive global fitting of implicit functions.

Benefits of software beamforming for ultrasonic inspections

Nondestructive testing (NDT) is used in more and more industries and inspection techniques must constantly adapt to complex materials (coarse grained materials, composites, etc.), defects (corrosion, fatigue crack, HTHA, etc.) or new inspection standards. Ultrasonic imaging methods have long been limited by low data throughput and a small number of transducers, but this is no longer the case with nowadays electronic devices and computers which enable large amounts of raw data to be acquired and processed in real time. The Delay-and-Sum (DAS) algorithm, which has been used almost automatically in many applications over the past few years, is now outperformed by advanced processing methods. In this context, this article presents few real-time advanced reconstruction techniques that enhance the quality of ultrasonic images on typical industrial applications. The first example concerns coherence beamformers such as Phase Coherence Imaging (PCI) which exploit the phase information of the signals. This type of method proves to be highly effective in detecting diffraction echoes from crack-like defects or in improving contrast and resolution of images from corroded samples inspections. The second example uses cross-correlation techniques that have been known for years to improve detection in various fields. Correlating the data with the transmitted signal reduces electronic noise and improves signal-to-noise ratio. This application also benefits from recent progress of ultrasonic devices that are able to modulate complex excitation signals in emission. This type of method is very promising for the inspection of attenuative materials and coarse-grained steels.

2D-supervised fast neural fluid reconstruction technique for time-resolved volumetric flame reconstruction

The volumetric tomography reconstruction technique (VTRT) can calculate and restore the three-dimensional physical field based on the multi-view projection information of the flame and similar flow fields, which plays an important role in the research of volumetric targets. However, the reconstruction of three-dimensional scalar/vector fields representing the physical information of the flow field (temperature, density, chemical composition, etc.) requires a significant amount of computational resources and time, especially in the turbulent flame flow field, which requires high spatial and temporal resolutions. Based on the said background, in this work, we proposed a Fast Neural Fluid Reconstruction Technique (Fast-NFRT) based on deep learning. In this investigation, we first test the reconstruction accuracy, speed, and noise robustness of Fast-NFRT using numerically simulated flames. Then Fast-NFRT is used to reconstruct the experimental turbulent jet flame under two different conditions. Finally, with the camera settings preserved, the Fast-NFRT model was tested for transfer learning between the numerical simulation flame and the two experimental jet flames to examine the generalization performance of the reconstruction. It is found that the proposed Fast-NFRT model can achieve a temporal resolution of 50-500 fps with a similar reconstruction fidelity to traditional algebraic reconstruction methods, which demonstrates the capacity of the Fast-NFRT model and its potential in real-time reconstruction applications and dynamic analysis for complex flow dynamics.

Dynamic 3D phase-shifting profilometry based ona corner optical flow algorithm.

Real-time 3D reconstruction has been applied in many fields, calling for many ongoing efforts to improve the speed and accuracy of the used algorithms. Phase shifting profilometry based on the Lucas-Kanade optical flow method is a fast and highly precise method to construct and display the three-dimensional shape of objects. However, in this method, a dense optical flow calculation is required for the modulation image corresponding to the acquired deformed fringe pattern, which consumes a lot of time and affects the real-time performance of 3D reconstruction and display. Therefore, this paper proposes a dynamic 3D phase shifting profilometry based on a corner optical flow algorithm to mitigate this issue. Therein, the Harris corner algorithm is utilized to locate the feature points of the measured object, so that the optical flow needs to calculate for only the feature points which, greatly reduces the amount of calculation time. Both our experiments and simulations show that our method improves the efficiency of pixel matching by four times and 3D reconstruction by two times.

Automatic Real-Time Creation of Three-Dimensional (3D) Representations of Objects, Buildings, or Scenarios Using Drones and Artificial Intelligence Techniques

This work presents the development and evaluation of a real-time 3D reconstruction system using drones. The system leverages innovative artificial intelligence techniques in photogrammetry and computer vision (CDS-MVSNet and DROID-SLAM) to achieve the accurate and efficient reconstruction of 3D environments. By integrating vision, navigation, and 3D reconstruction subsystems, the proposed system addresses the limitations of existing applications and software in terms of speed and accuracy. The project encountered challenges related to scheduling, resource availability, and algorithmic complexity. The obtained results validate the applicability of the system in real-world scenarios and open avenues for further research in diverse areas. One of the tests consisted of a one-minute-and-three-second flight around a small figure, while the reconstruction was performed in real time. The reference Meshroom software completed the 3D reconstruction in 136 min and 12 s, while the proposed system finished the process in just 1 min and 13 s. This work contributes to the advancement in the field of 3D reconstruction using drones, benefiting from advancements in technology and machine learning algorithms.

Attack scenario reconstruction via fusing heterogeneous threat intelligence

Nowadays, new-generation threats often use multiple means or perform several steps to intrude into networks and ultimately reach their objective. These new threats have multi-staged, and we can understand their intrusion pattern from the kill-chain defensive model. This paper focuses on fusing heterogeneous threat intelligence collected from security information and event management systems to reconstruct multi-step attack scenarios and discover critical attack paths. However, the need for an agreed-upon vocabulary to represent the heterogeneous threat intelligence makes it difficult to model the attack scenarios accurately and efficiently. Therefore, we devise a heterogeneous threat intelligence fusing approach for real-time reconstruction of the attack scenarios. Firstly, we use structured threat information expression (STIX) to format heterogeneous threat intelligence (TI). We analyze the causal relationship of each heterogeneous threat intelligence and piece them together. Then, we model the multi-stage attack scenario reconstruction as a community discovery problem. We mine the attack scenarios with semantic correlation weight and a community detection algorithm. We finally use the open-source benchmark datasets (DARPA 2000, CICIDS 2017) and the real Internet traffic captured from the China Education Research Network backbone (CERNET) to evaluate our work. Extensive results demonstrate that our proposal can accurately reconstruct multi-step attack scenarios and discover covert C&C channels.

An Efficient Confidence Interval-Based Dual-Key Fuzzy Vault Scheme for Operator Authentication of Autonomous Unmanned Aerial Vehicles

The fuzzy vault is an innovative way to share secret keys, combining traditional cryptography with biometrics and biometric template protection. This method forms the basis for the reliable operation of unmanned aerial vehicles (UAVs) through anonymizing drone operators and safely using their data and onboard information. However, due to the inherent instability of biometrics, traditional fuzzy vault schemes face challenges, such as reduced recognition rates with increased chaff points, impractical runtimes due to high-order polynomial reconstruction, and susceptibility to correlation attacks. This paper proposes an efficient fuzzy vault scheme to address these challenges. We generate two secret keys based on biometrics: the first key is produced from the operator’s unique features like the face and iris, using a confidence interval; the second key, used to construct a polynomial, is based on what the operator remembers. These dual-key fuzzy vaults enable the stable generation of genuine points during encoding, easy extraction during decoding, and effective operator authentication while maintaining anonymity. Our experimental results demonstrate improved security and secret acquisition accuracy using the AR face database. These results are achieved regardless of increased false vaults, enabling real-time polynomial reconstruction and resilience against correlation attacks. Importantly, our enhanced fuzzy vault scheme allows the application of this secure, real-time authentication process, safeguarding the anonymity of drone operators.

Localization Free Super-Resolution Microbubble Velocimetry Using a Long Short-Term Memory Neural Network.

Ultrasound localization microscopy is a super-resolution imaging technique that exploits the unique characteristics of contrast microbubbles to side-step the fundamental trade-off between imaging resolution and penetration depth. However, the conventional reconstruction technique is confined to low microbubble concentrations to avoid localization and tracking errors. Several research groups have introduced sparsity- and deep learning-based approaches to overcome this constraint to extract useful vascular structural information from overlapping microbubble signals, but these solutions have not been demonstrated to produce blood flow velocity maps of the microcirculation. Here, we introduce Deep-SMV, a localization free super-resolution microbubble velocimetry technique, based on a long short-term memory neural network, that provides high imaging speed and robustness to high microbubble concentrations, and directly outputs blood velocity measurements at a super-resolution. Deep-SMV is trained efficiently using microbubble flow simulation on real in vivo vascular data and demonstrates real-time velocity map reconstruction suitable for functional vascular imaging and pulsatility mapping at super-resolution. The technique is successfully applied to a wide variety of imaging scenarios, include flow channel phantoms, chicken embryo chorioallantoic membranes, and mouse brain imaging. An implementation of Deep-SMV is openly available at https://github.com/chenxiptz/SR_microvessel_velocimetry, with two pre-trained models available at https://doi.org/10.7910/DVN/SECUFD.

Real-time reconstruction of unsteady rotating forces acting by rotor blades in moving medium

A time-domain inverse aeroacoustic method based on the convective Ffowcs Williams–Hawkings equation is presented. The method allows to determine, in real-time, the unsteady forces exerted on rotating blades in the presence of a moving medium. The inversion procedure is based on a space-time regularization with a mixed l1,2-norm, which guarantees accuracy and smoothness of the solution. The method is initially verified through synthetic acoustic signals emitted by rotating sources in a constant flow, up to a convective Mach number of about 0.88. Then the method is validated through signals generated by a propeller immersed in a wind-tunnel jet flow, up to a Mach number of 0.06. Due to the reduced convective Mach number, the leading aeroacoustic effect is derived from a variation of the blade loading. It is argued that the onset of flow separation at high values of the rotor advance ratio is responsible for the onset of force fluctuations that the inverse method is able to retrieve both qualitatively and quantitatively.

Robust 3D breast reconstruction based on monocular images and artificial intelligence for robotic guided oncological interventions.

Breast cancer is a global public health concern. For women with suspicious breast lesions, the current diagnosis requires a biopsy, which is usually guided by ultrasound (US). However, this process is challenging due to the low quality of the US image and the complexity of dealing with the US probe and the surgical needle simultaneously, making it largely reliant on the surgeon's expertise. Some previous works employing collaborative robots emerged to improve the precision of biopsy interventions, providing an easier, safer, and more ergonomic procedure. However, for these equipment to be able to navigate around the breast autonomously, 3D breast reconstruction needs to be available. The accuracy of these systems still needs to improve, with the 3D reconstruction of the breast being one of the biggest focuses of errors. The main objective of this work is to develop a method to obtain a robust 3D reconstruction of the patient's breast, based on RGB monocular images, which later can be used to compute the robot's trajectories for the biopsy. To this end, depth estimation techniques will be developed, based on a deep learning architecture constituted by a CNN, LSTM, and MLP, to generate depth maps capable of being converted into point clouds. After merging several from multiple points of view, it is possible to generate a real-time reconstruction of the breast as a mesh. The development and validation of our method was performed using a previously described synthetic dataset. Hence, this procedure takes RGB images and the cameras' position and outputs the breasts' meshes. It has a mean error of 3.9 mm and a standard deviation of 1.2 mm. The final results attest to the ability of this methodology to predict the breast's shape and size using monocular images.Clinical Relevance- This work proposes a method based on artificial intelligence and monocular RGB images to obtain the breast's volume during robotic guided breast biopsies, improving their execution and safety.

Dual-wavelength off-axis digital holography inImageJ: toward real-time phase retrieval usingCUDA streams.

An ImageJ plug-in is developed to realize automatic real-time phase reconstruction for dual-wavelength digital holography (DH). This plug-in assembles the algorithms, including automatic phase reconstruction based on the division algorithm and post-processing. These algorithms are implemented and analyzed using a CPU and GPU, respectively. To hide the CPU-to-GPU data transfer latency, an optimization scheme using Compute Unified Device Architecture (CUDA) streams is proposed in ImageJ. Experimental results show that the proposed plug-in can perform faster reconstruction for dual-wavelength DH, resulting in frame rates up to 48fps even for one-megapixel digital holograms on a normal PC. In other words, the proposed plug-in can realize real-time phase reconstruction for dual-wavelength digital holographic videos.

Structured illumination microscopy with a phase-modulated spinning disk for optical sectioning.

Among various super-resolution microscopic techniques, structured illumination microscopy (SIM) stands out for live-cell imaging because of its higher imaging speed. However, conventional SIM lacks optical sectioning capability. Here we demonstrate a new, to the best of our knowledge, approach using a phase-modulated spinning disk (PMSD) that enhances the optical sectioning capability of SIM. The PMSD consists of a pinhole array for confocal imaging and a transparent polymer layer for light phase modulation. The light phase modulation was designed to cancel the zeroth-order diffracted beam and create a sharp lattice illumination pattern using the interference of four first-order diffracted beams. In the detection optical path, the PMSD serves as a spatial filter to physically reject about 80% of the out-of-focus signals, an approach that allows for real-time optical reconstruction of super-resolved images with enhanced contrast. Furthermore, the simplicity of the design makes it easy to upgrade a conventional fluorescence microscope to a PMSD SIM system.

SpeckleNN: a unified embedding for real-time speckle pattern classification in X-ray single-particle imaging with limited labeled examples.

With X-ray free-electron lasers (XFELs), it is possible to determine the three-dimensional structure of noncrystalline nanoscale particles using X-ray single-particle imaging (SPI) techniques at room temperature. Classifying SPI scattering patterns, or `speckles', to extract single-hits that are needed for real-time vetoing and three-dimensional reconstruction poses a challenge for high-data-rate facilities like the European XFEL and LCLS-II-HE. Here, we introduce SpeckleNN, a unified embedding model for real-time speckle pattern classification with limited labeled examples that can scale linearly with dataset size. Trained with twin neural networks, SpeckleNN maps speckle patterns to a unified embedding vector space, where similarity is measured by Euclidean distance. We highlight its few-shot classification capability on new never-seen samples and its robust performance despite having only tens of labels per classification category even in the presence of substantial missing detector areas. Without the need for excessive manual labeling or even a full detector image, our classification method offers a great solution for real-time high-throughput SPI experiments.

Real-time 3D reconstruction of guidewires and stents using two update X-ray projections in a rotating imaging setup.

Vascular diseases are often treated minimally invasively. The interventional material (stents, guidewires, etc.) used during such percutaneous interventions are visualized by some form of image guidance. Today, this image guidance is usually provided by 2D X-ray fluoroscopy, that is, a live 2D image. 3D X-ray fluoroscopy, that is, a live 3D image, could accelerate existing and enable new interventions. However, existing algorithms for the 3D reconstruction of interventional material require either too many X-ray projections and therefore dose, or are only capable of reconstructing single, curvilinearstructures. Using only two new X-ray projections per 3D reconstruction, we aim to reconstruct more complex arrangements of interventional material than was previously possible. This is achieved by improving a previously presented deep learning-based reconstruction pipeline, which assumes that the X-ray images are acquired by a continuously rotating biplane system, in two ways: (a) separation of the reconstruction of different object types, (b) motion compensation using spatial transformernetworks. Our pipeline achieves submillimeter accuracy on measured data of a stent and two guidewires inside an anthropomorphic phantom with respiratory motion. In an ablation study, we find that the aforementioned algorithmic changes improve our two figuresof merit by 75% (1.76mm → 0.44mm) and 59% (1.15mm → 0.47mm) respectively. A comparison of our measured dose area product (DAP) rate to DAP rates of 2D fluoroscopy indicates a roughly similar doseburden. This dose efficiency combined with the ability to reconstruct complex arrangements of interventional material makes the presented algorithm a promising candidate to enable 3Dfluoroscopy.

Numerical and experimental investigation of many-objective optimization for efficient temperature and velocity fields reconstruction via acoustic tomography

Accurate and high-quality measurement of temperature and velocity fields is important in combustion and flow diagnosis, especially for the control of boiler operations and optimization of combustion processes. Nonlinear acoustic tomography (NAT) is an appealing approach to simultaneously monitor temperature and velocity fields by observing the changes in sound speed they induce. However, the inherent ill-posedness of the tomographic inverse problem and the lack of a priori information may lead to poor robustness, low precision, and the presence of artifacts. On the other hand, the inclusion of a priori information, such as smoothness and box constraints, may require the use of hyperparameters that hinder quasi real-time reconstruction and may degrade performance in actual applications. In this study, we aim to ensure quasi real-time reconstruction and alleviate the ill-posedness of the NAT problem by eliminating the regularization parameters and transforming the inverse problem into a many-objective optimization problem with four objectives. The knee point-driven evolutionary algorithm with improved environmental selection strategy (KEA-IES) is proposed to simultaneously reconstruct the arbitrary inhomogeneous temperature and aerodynamic fields in a furnace. Apart from the investigation of the performance of KEA-IES and the influence of various factors, including measurement noise, function evaluation number, population size, and functional norms, on the quality of the reconstruction, an experimental AT system with independent 16 T/R channels is developed to evaluate the proposed method. The results show that combustion flame temperature and velocity fields can be monitored simultaneously by using the proposed KEA-IES with more accuracy, less consumed time, and better noise immunity compared with the state-of-the-art algorithms. The proposed method can provide valuable guidance in the development of a non-intrusive real-time pyrometry and velocimetry system, particularly for applications with large velocity fields and temperature gradients. Furthermore, its capability in complex combustion and flow diagnosis will offer benefits in terms of energy conservation, emission reduction, boiler operations supervision, and combustion process optimization.

Medical volume data real-time optical reconstruction on light field display with a directional diffuser

In this paper, we proposed an optical reconstruction approach by use of a new type of light field display device(LFDD) with a directional diffuser. The directional diffuser can improve imaging quality of the traditional light field display device. Special ray casting algorithm is applied to render the medical volume data in real time for LFDD. Four different kinds of medical data have been applied for testing the validity of the algorithm and better results have been achieved. The experimental results shows that the number of LFDD’s viewpoints can reach 90 × 90 with 40 degree× 40 degree viewing angle, and the rendering efficiency can reach 30–40 fps for common medial volume data.