Real-time 3D Research Articles

Role of 3D Transesophageal Echocardiography for Transcatheter Mitral Valve Repair-A Mini Review.

Edge-to-edge transcatheter mitral valve repair (TMVr) using MitraClip has been evolving rapidly in patients with severe mitral regurgitation (MR) at high surgical risk or having contraindications for surgery. Three-dimensional (3D) echocardiography plays an important role in the management of severe MR. In particular, 3D transesophageal echocardiography (TEE) imaging allows the evaluation of MV geometry and quantification of MR severity with dedicated software. Real-time 3D TEE is also commonly used to guide TMVr and facilitate the procedure. Further development of 3D echocardiography may help achieve safer and more beneficial results. The following article summarizes the current knowledge and the future perspectives of 3D TEE in TMVr.

Initial observation of contrast profiles for 3D and 2D MRI sequences in MR-guided radiation therapy for locally advanced pancreatic cancer.

541 Background: MR-guided stereotactic body radiation therapy (MR-SBRT) is a novel method of treating mobile tumors with soft-tissue gating and on-table adaptive planning. In our experience using the ViewRay MRIdian system (VR) for treating locally advanced pancreatic cancer (PA) with MR-SBRT, the true-fast imaging with steady-state free precession (TRUFI) sequences on the VR impart differing intensities for relevant structures seen on the pre-treatment high resolution 3D MRI (3D MRI) versus the real-time 2D cine MRI (2D cine) used for target tracking. Since these variations can confound target tracking selection, we propose that an understanding of the differing contrast profiles could improve selection of tracking structures and optimize treatment delivery. Methods: We retrospectively reviewed both 3D MRI and 2D cine images for patients (pts) with PA (n =20) treated on the VR. At simulation, an appropriate tracking target was identified and contoured on a single 3mm sagittal slice of the 3D MRI. This sagittal slice was directly compared to the registered 7mm 2D cine to identify structures with notable discrepancies in signal intensity. The 3D MRI was then explored in additional planes to confirm structure identities. For quantitative verification of the clinically observed differences, the pixel intensity distributions of 3D MRI and 2D cine DICOM image datasets were statistically compared. Results: In all pts reviewed, arteries (aorta, celiac, SMA) appeared with similar contrast profiles on both images. However, veins (portal vein, SMV) appeared hypointense on 3D MRI but hyperintense on 2D cine. Biliary structures appeared hyperintense on 3D MRI but only mildly hyperintense on 2D cine. The pixel intensity distributions extracted from 3D MRI and 2D cine images were confirmed to differ significantly (two sample Kolmogorov-Smirnov test; test statistic =0.40; p < 0.001). Conclusions: There are significant variations in image intensity between the initial treatment planning 3D MRI and the immediate pre-treatment 2D cine obtained with the VR. Understanding these discrepancies can guide radiation oncologists in choosing optimal tracking targets. Future work will focus on identifying the particular causes and frequencies of target tracking failures and exploring alternative tracking algorithms using artificial intelligence which could ultimately allow for VMAT on the ViewRay system.

Towards real-time 3D visualization with multiview RGB camera array.

A real-time 3D visualization (RT3DV) system using a multiview RGB camera array is presented. RT3DV can process multiple synchronized video streams to produce a stereo video of a dynamic scene from a chosen view angle. Its design objective is to facilitate 3D visualization at the video frame rate with good viewing quality. To facilitate 3D vision, RT3DV estimates and updates a surface mesh model formed directly from a set of sparse key points. The 3D coordinates of these key points are estimated from matching 2D key points across multiview video streams with the aid of epipolar geometry and trifocal tensor. To capture the scene dynamics, 2D key points in individual video streams are tracked between successive frames. We implemented a proof of concept RT3DV system tasked to process five synchronous video streams acquired by an RGB camera array. It achieves a processing speed of 44 milliseconds per frame and a peak signal to noise ratio (PSNR) of 15.9 dB from a viewpoint coinciding with a reference view. As a comparison, an image-based MVS algorithm utilizing a dense point cloud model and frame by frame feature detection and matching will require 7 seconds to render a frame and yield a reference view PSNR of 16.3 dB.

Cardiac X-ray image-based haptic guidance for robot-assisted coronary intervention: a feasibility study.

Effective and efficient haptic guidance is desirable for tele-operated robotic surgery because it has a potential to enhance surgeon's skills, especially in coronary interventions where surgeon loses both an eye-hand coordination and a direct sight to the organ. This paper proposes a novel haptic guidance procedure-both kinesthetic and cutaneous, which solely depends upon X-ray images, for tele-robotic system that assists an efficient navigation of the guidewire towards the target location during a coronary intervention. Proposed methodology requires cardiologists to draw virtual fixtures (VFs) on angiograms as a preoperative procedure. During an operation, these VFs direct the guidewire to the desired coronary vessel. For this, the position and orientation of guidewire tip are calculated with respect to VFs' anatomy, using image processing on the real-time 2D fluoroscopic images. The haptic feedbacks are then rendered on to the master device depending on the interaction with attractive and repulsive, guidance and forbidden region VFs. A feasibility study in the laboratory environment is performed by using a webcam as an image acquisition device and a phantom-based coronary vessel model. The subsequent statistical analysis shows that, on an average, a decrease of approx. 37% in task completion time is observed with haptic feedback. Moreover, haptic guidance is found effective for most difficult branch, whereas there is a minimal significance of such haptics for the easiest branch. The proposed haptic guidance procedure may assist cardiologists for an efficient and effective guidewire navigation during a surgical procedure. The cutaneous haptics (vibration feedback) is found more helpful in coronary interventions compared with kinesthetic haptics (force feedback).

A real-time 3D electromagnetic navigation system for percutaneous transforaminal endoscopic discectomy in patients with lumbar disc herniation: a retrospective study

BackgroundIn this study, we present a novel electromagnetic navigation (EMN) system for percutaneous transforaminal endoscopic discectomy (PTED) procedure. The objective of this study was to investigate the safety and effectiveness of the PTED with the assistance of the EMN system and compare it with the conventional PTED with the assistance of fluoroscopic guidance (C-arm).MethodsThe clinical data of 79 patients (32 in EMN group and 47 in C-arm group) undergoing PTED for lumbar disc herniation (LDH) from January to September of 2019 were analyzed retrospectively. The radiation time, puncture time, operation time, visual analog scale (VAS), Oswestry disability index (ODI), modified MacNab criteria, and radiological parameters were recorded in both groups.ResultsRadiation time, puncture time, and operation time were significantly reduced in the EMN group compared with the C-arm group (P < 0.05). Compared with the C-arm group, a steeper learning curve was observed in the EMN group. There were no significant differences between the two groups regarding VAS and ODI scores at different time points (P > 0.05). The satisfaction rates of the EMN and C-arm groups were 90.63 and 87.23%, respectively, but no significant difference was found between the two groups (P > 0.05). There was no significant difference regarding translation and angular motion between the two groups at preoperation and postoperation (P > 0.05).ConclusionsThe EMN system can be applied to facilitate the PETD procedure. It can significantly reduce the intraoperative radiation time, puncture time, and operation time, and reshape the learning curve of PTED. Due to limitations of a retrospective study, results may need validation with larger prospective randomized clinical trials.

State of the Art: Contrast Enhanced 4D Ultrasound to Monitor or Assess Locoregional Therapies

AbstractLocoregional therapies (LRTs) are an essential management tool in the treatment of primary liver cancers or metastatic liver disease. LRTs include curative and palliative modalities. Monitoring treatment response of LRTs is crucial for maximizing benefit and improving clinical outcomes. Clinical use of contrast-enhanced ultrasound (CEUS) was introduced more than two decades ago. Its portability, cost effectiveness, lack of contraindications and safety make it an ideal tool for treatment monitoring in numerous situations. Two-dimensional dynamic CEUS has been proved to be equivalent to the current imaging standard in the guidance of LRTs, assessment of their adequacy, and detection of early tumor recurrence. Recent technical advances in ultrasound transducers and image processing have made 3D CEUS scanning widely available on most commercial ultrasound systems. 3D scanning offers a broad multiplanar view of anatomic structures, overcoming many limitations of two-dimensional scanning. Furthermore, many ultrasound systems provide real-time dynamic 3D CEUS, also known as 4D CEUS. Volumetric CEUS has shown to perform better than 2D CEUS in the assessment and monitoring of some LRTs. CEUS presents a valid alternative to the current imaging standards with reduced cost and decreased risk of complications. Future efforts will be directed toward refining the utility of 4D CEUS through approaches such as multi-parametric quantitative analysis and machine learning algorithms.

360^{circ } real-time and power-efficient 3D DAMOT for autonomous driving applications

Autonomous Driving (AD) promises an efficient, comfortable and safe driving experience. Nevertheless, fatalities involving vehicles equipped with Automated Driving Systems (ADSs) are on the rise, especially those related to the perception module of the vehicle. This paper presents a real-time and power-efficient 3D Multi-Object Detection and Tracking (DAMOT) method proposed for Intelligent Vehicles (IV) applications, allowing the vehicle to track 360^{circ } surrounding objects as a preliminary stage to perform trajectory forecasting to prevent collisions and anticipate the ego-vehicle to future traffic scenarios. First, we present our DAMOT pipeline based on Fast Encoders for object detection and a combination of a 3D Kalman Filter and Hungarian Algorithm, used for state estimation and data association respectively. We extend our previous work ellaborating a preliminary version of sensor fusion based DAMOT, merging the extracted features by a Convolutional Neural Network (CNN) using camera information for long-term re-identification and obstacles retrieved by the 3D object detector. Both pipelines exploit the concepts of lightweight Linux containers using the Docker approach to provide the system with isolation, flexibility and portability, and standard communication in robotics using the Robot Operating System (ROS). Second, both pipelines are validated using the recently proposed KITTI-3DMOT evaluation tool that demonstrates the full strength of 3D localization and tracking of a MOT system. Finally, the most efficient architecture is validated in some interesting traffic scenarios implemented in the CARLA (Car Learning to Act) open-source driving simulator and in our real-world autonomous electric car using the NVIDIA AGX Xavier, an AI embedded system for autonomous machines, studying its performance in a controlled but realistic urban environment with real-time execution (results).

Real-time panoramic-deformation measurement of cabin structure of launch vehicle based on camera network

<p indent="0mm">To satisfy the requirement of real-time panoramic-deformation measurement of a cabin structure in a large-diameter launch vehicle, a camera-network deformation-measurement system composed of 24 cameras and real-time three-dimensional (3D) deformation-measurement software based on CPU parallel computing was developed. The 360° full-circle deformation measurement of the inner surface of a cylindrical coordinate system and real-time measurement of the local deformation of a local Cartesian coordinate system were successfully realized. The camera-network measurement system consisted of 12 sets of binocular systems. The field of view of a single set of binocular systems was 3.6 m×3.6 m, and a certain overlap area was set between two adjacent binocular systems to ensure continuity of measurements. Photogrammetry and binocular stereo vision were used to reconstruct the 3D coordinates of the coded points, and coordinate transformation between the local and global coordinate systems was realized using the coded points. Finally, coordinate unification of the 12 sets of binocular systems was performed. A single set of binocular systems could be connected to a computer. By implementing the seed-point-based parallel algorithm and initial guess-estimation-algorithm-based deformation-transfer scheme, real-time 3D displacement and surface-strain measurements at 70000 points per second were successfully realized. On the basis of the 360° full-circle deformation-measurement results in the cylindrical coordinate system, high-precision calculation of the diffusion angle was successfully realized. The standard deviation of the static noise in the three directions was less than 0.01 mm, and the standard deviation of the static noise in the strain was less than 50 microstrains.

Electrical Tomography Hardware Systems for Real-Time Applications: a Review

This paper presents a review of two-dimensional (2D) and three-dimensional (3D) electrical tomography (ET) hardware accelerators for real-time applications. While many recent review papers have discussed various algorithms for image reconstruction or acquisition systems, none of them has considered state-of-the-art hardware implementations of the associated image reconstruction algorithms to achieve real-time performance, especially for 3D ET where the computation requirement is excessively high. A 3D ET is useful in various applications such as robotics, autonomous vehicles, and process control, but it is computationally very expensive with respect to its 2D counterpart. Most implementations are based on single or multi-core CPUs and, to a lesser extent, on either graphics processing units (GPUs) or field programmable gate arrays (FPGAs). However, there is a clear gap between the currently available processors, whose computation power exceeds hundreds of teraflops per second (TOPS) at a reasonable low power consumption, and the ones recently used in ET systems. This gives great potential for next-generation ET systems to achieve real-time 2D and 3D ET reconstruction within a small form factor. The paper summarizes the most recent ET hardware systems with respect to their performance in terms of quality and processing frame rate, reconstruction methods, along with optimization and future directions.

Real-Time and Accurate Monocular 3d Sensor Using the Reference Plane Calibration and an Optimized Sgm Based on Opencl Acceleration

Real-Time and Accurate Monocular 3d Sensor Using the Reference Plane Calibration and an Optimized Sgm Based on Opencl Acceleration

Characterization of a Low-Profile, Flexible, and Acoustically Transparent Receive-Only MRI Coil Array for High Sensitivity MR-Guided Focused Ultrasound.

Magnetic resonance guided focused ultrasound (MRgFUS) is a non-invasive therapeutic modality for neurodegenerative diseases that employs real-time imaging and thermometry monitoring of targeted regions. MRI is used in guidance of ultrasound treatment; however, the MR image quality in current clinical applications is poor when using the vendor built-in body coil. We present an 8-channel, ultra-thin, flexible, and acoustically transparent receive-only head coil design (FUS-Flex) to improve the signal-to-noise ratio (SNR) and thus the quality of MR images during MRgFUS procedures. Acoustic simulations/experiments exhibit transparency of the FUS-Flex coil as high as 97% at 650 kHz. Electromagnetic simulations show a SNR increase of 13× over the body coil. In vivo results show an increase of the SNR over the body coil by a factor of 7.3 with 2× acceleration (equivalent to 11× without acceleration) in the brain of a healthy volunteer, which agrees well with simulation. These preliminary results show that the use of a FUS-Flex coil in MRgFUS surgery can increase MR image quality, which could yield improved focal precision, real-time intraprocedural anatomical imaging, and real-time 3D thermometry mapping.

Automated real-time 3D visual servoing control of single cell surgery with application to microinjection processes

Automated real-time 3D visual servoing control of single cell surgery with application to microinjection processes

Automated real-time 3D visual servoing control of single cell surgery with application to microinjection processes

Automated real-time 3D visual servoing control of single cell surgery with application to microinjection processes

Real-time Volume Preserving Constraints for Volumetric Model on GPU

This paper presents a parallel method for simulating real-time 3D deformable objects using the volume preservation mass-spring system method on tetrahedron meshes. In general, the conventional mass-spring system is manipulated as a force-driven method because it is fast, simple to implement, and the parameters can be controlled. However, the springs in traditional mass-spring system can be excessively elongated which cause severe stability and robustness issues that lead to shape restoring, simulation blow-up, and huge volume loss of the deformable object. In addition, traditional method that uses a serial process of the central processing unit (CPU) to solve the system in every frame cannot handle the complex structure of deformable object in real-time. Therefore, the first order implicit constraint enforcement for a mass-spring model is utilized to achieve accurate visual realism of deformable objects with tough constraint error. In this paper, we applied the distance constraint and volume conservation constraints for each tetrahedron element to improve the stability of deformable object simulation using the mass-spring system and behave the same as its real-world counterparts. To reduce the computational complexity while ensuring stable simulation, we applied a method that utilizes OpenGL compute shader, a part of OpenGL Shading Language (GLSL) that executes on the graphic processing unit (GPU) to solve the numerical problems effectively. We applied the proposed methods to experimental volumetric models, and volume percentages of all objects are compared. The average volume percentages of all models during the simulation using the mass-spring system, distance constraint, and the volume constraint method were 68.21%, 89.64%, and 98.70%, respectively. The proposed approaches are successfully applied to improve the stability of mass-spring system and the performance comparison from our experimental tests also shows that the GPU-based method is faster than CPU-based implementation for all cases.

Benchmarking of Motion Planning Algorithms with Real-time 3D Occupancy Grid Map for an Agricultural Robotic Manipulator

Benchmarking of Motion Planning Algorithms with Real-time 3D Occupancy Grid Map for an Agricultural Robotic Manipulator

A Data-Driven Waveform Adaptation Method for Mm-Wave Gait Classification at the Edge

The ever-increasing need for real-time 3D perception for autonomy has resulted in development of a new generation of high-resolution perception sensors paired with powerful edge processors and advanced perception algorithms. However, processing of the large data volumes generated by such high-resolution sensors still imposes a great challenge for resource-limited edge devices and systems. In this work, we visit this problem in the context of pedestrian gait analysis using high-resolution mm-Wave sensors with application in autonomous driving. We propose reducing the total computational load of a resource-limited perception system by dynamically controlling the mm-Wave sensor's resolution on the fly. We provide results on three dynamic sensor adaptation approaches including an end-to-end Deep Q-Learning approach that can overcome the resource and performance limitations inherent to conventional static methods.

Real time 4D Radiation Gel Dosimetry on the Australian MRI-Linac

4D radiation dosimetry using a highly radiation-sensitive polymer gel dosimeter with real-time quantitative MRI readout is presented as a technique to acquire the accumulated radiation dose distribution during image guided radiotherapy (IGRT) on an MRI-Linac. Optimized T2 weighted TSE scans are converted into quantitative ΔR2 maps and subsequently to radiation dose maps. The potential of real-time 4D radiation dosimetry in a theragnostic MRI-Linac is demonstrated in test tubes, for a square beam in a cylindrical gel phantom, for a simple step-and-shoot irradiation in a head phantom and a dynamic arc treatment on a cylindrical gel phantom using a rotating couch. The optimal sequence parameters for maximal dose resolution in the dynamic MRI acquisition will be presented and the trade off between MRI scanning speed and dose resolution will be discussed. A further improvement in temporal resolution using a keyhole imaging approach is the focus of future research.

Single-shot 3D profilometry using a color imaging polarimeter

We demonstrate a single-shot uniaxial three-dimensional (3D) profilometry system that illuminates a sample with a color structured polarization pattern with an axial chromatic aberration. We have previously shown that real-time 3D shape measurement can be performed without the occlusion problem using a polarization camera and a monochromatic polarized pattern. Here, we show that it is possible to double the depth range of the system, with no tradeoff in depth resolution, measurement speed, or accuracy, using color pattern projection and an RGB polarization camera. We present the measurement principles of the system and show the results of a real-time 3D profilometry experiment for a sample with a deep hole.

Modern Technology for Image processing and Computer vision -A Review

&#x0D; &#x0D; &#x0D; &#x0D; This survey outlines the use of computer vision in Image and video processing in multidisciplinary applications; either in academia or industry, which are active in this field. The scope of this paper covers the theoretical and practical aspects in image and video processing in addition of computer vision, from essential research to evolution of application. In this work a various subjects of image processing and computer vision will be demonstrated, these subjects are spanned from the evolution of mobile augmented reality (MAR) applications, to augmented reality under 3D modeling and real time depth imaging, video processing algorithms will be discussed to get higher depth video compression, beside that in the field of mobile platform an automatic computer vision system for citrus fruit has been implemented, where the Bayesian classification with Boundary Growing to detect the text in the video scene. Also the paper illustrates the usability of the handed interactive method to the portable projector based on augmented reality.&#x0D; © 2018 JASET, International Scholars and Researchers Association&#x0D; &#x0D; &#x0D; &#x0D;

Real-Time Detection and Location of Potted Flowers Based on a ZED Camera and a YOLO V4-Tiny Deep Learning Algorithm

In order to realize the real-time and accurate detection of potted flowers on benches, in this paper we propose a method based on the ZED 2 stereo camera and the YOLO V4-Tiny deep learning algorithm for potted flower detection and location. First, an automatic detection model of flowers was established based on the YOLO V4-Tiny convolutional neural network (CNN) model, and the center points on the pixel plane of the flowers were obtained according to the prediction box. Then, the real-time 3D point cloud information obtained by the ZED 2 camera was used to calculate the actual position of the flowers. The test results showed that the mean average precision (MAP) and recall rate of the training model was 89.72% and 80%, respectively, and the real-time average detection frame rate of the model deployed under Jetson TX2 was 16 FPS. The results of the occlusion experiment showed that when the canopy overlap ratio between the two flowers is more than 10%, the recognition accuracy will be affected. The mean absolute error of the flower center location based on 3D point cloud information of the ZED 2 camera was 18.1 mm, and the maximum locating error of the flower center was 25.8 mm under different light radiation conditions. The method in this paper establishes the relationship between the detection target of flowers and the actual spatial location, which has reference significance for the machinery and automatic management of potted flowers on benches.