External Tracking Devices Research Articles

On-patient see-through augmented reality based on visual SLAM.

An augmented reality system to visualize a 3D preoperative anatomical model on intra-operative patient is proposed. The hardware requirement is commercial tablet-PC equipped with a camera. Thus, no external tracking device nor artificial landmarks on the patient are required. We resort to visual SLAM to provide markerless real-time tablet-PC camera location with respect to the patient. The preoperative model is registered with respect to the patient through 4-6 anchor points. The anchors correspond to anatomical references selected on the tablet-PC screen at the beginning of the procedure. Accurate and real-time preoperative model alignment (approximately 5-mm mean FRE and TRE) was achieved, even when anchors were not visible in the current field of view. The system has been experimentally validated on human volunteers, in vivo pigs and a phantom. The proposed system can be smoothly integrated into the surgical workflow because it: (1) operates in real time, (2) requires minimal additional hardware only a tablet-PC with camera, (3) is robust to occlusion, (4) requires minimal interaction from the medical staff.

Simultaneous Electromagnetic Tracking and Calibration for Dynamic Field Distortion Compensation.

Electromagnetic (EM) tracking systems are highly susceptible to field distortion. The interference can cause measurement errors up to a few centimeters in clinical environments, which limits the reliability of these systems. Unless corrected for, this measurement error imperils the success of clinical procedures. It is therefore fundamental to dynamically calibrate EM tracking systems and compensate for measurement error caused by field distorting objects commonly present in clinical environments. We propose to combine a motion model with observations of redundant EM sensors and compensate for field distortions in real time. We employ a simultaneous localization and mapping technique to accurately estimate the pose of the tracked instrument while creating the field distortion map. We conducted experiments with six degrees-of-freedom motions in the presence of field distorting objects in research and clinical environments. We applied our approach to improve the EM tracking accuracy and compared our results to a conventional sensor fusion technique. Using our approach, the maximum tracking error was reduced by 67% for position measurements and by 64% for orientation measurements. Currently, clinical applications of EM trackers are hampered by the adverse distortion effects. Our approach introduces a novel method for dynamic field distortion compensation, independent from preoperative calibrations or external tracking devices, and enables reliable EM navigation for potential applications.

Simultaneous localization and calibration for electromagnetic tracking systems.

In clinical environments, field distortion can cause significant electromagnetic tracking errors. Therefore, dynamic calibration of electromagnetic tracking systems is essential to compensate for measurement errors. It is proposed to integrate the motion model of the tracked instrument with redundant EM sensor observations and to apply a simultaneous localization and mapping algorithm in order to accurately estimate the pose of the instrument and create a map of the field distortion in real-time. Experiments were conducted in the presence of ferromagnetic and electrically-conductive field distorting objects and results compared with those of a conventional sensor fusion approach. The proposed method reduced the tracking error from 3.94±1.61 mm to 1.82±0.62 mm in the presence of steel, and from 0.31±0.22 mm to 0.11±0.14 mm in the presence of aluminum. With reduced tracking error and independence from external tracking devices or pre-operative calibrations, the approach is promising for reliable EM navigation in various clinical procedures. Copyright © 2015 John Wiley & Sons, Ltd.

Homogeneous coordinates in motion correction.

Prospective motion correction for MRI and other imaging modalities are commonly based on the assumption of affine motion, i.e., rotations, shearing, scaling and translations. In addition it often involves transformations between different reference frames, especially for applications with an external tracking device. The goal of this work is to develop a computational framework for motion correction based on homogeneous transforms. The homogeneous representation of affine transformations uses 4 × 4 transformation matrices applied to four-dimensional augmented vectors. It is demonstrated how homogenous transforms can be used to describe the motion of slice objects during an MRI scan. Furthermore, we extend the concept of homogeneous transforms to gradient and k-space vectors, and show that the fourth dimension of an augmented k-space vector encodes the complex phase of the corresponding signal sample due to translations. The validity of describing motion tracking in real space and k-space using homogeneous transformations only is demonstrated on phantom experiments. Homogeneous transformations allows for a conceptually simple, consistent and computationally efficient theoretical framework for motion correction applications.

Vision-based endoscope tracking for 3D ultrasound image-guided surgical navigation

This work introduces a self-contained framework for endoscopic camera tracking by combining 3D ultrasonography with endoscopy. The approach can be readily incorporated into surgical workflows without installing external tracking devices. By fusing the ultrasound-constructed scene geometry with endoscopic vision, this integrated approach addresses issues related to initialization, scale ambiguity, and interest point inadequacy that may be faced by conventional vision-based approaches when applied to fetoscopic procedures. Vision-based pose estimations were demonstrated by phantom and ex vivo monkey placenta imaging. The potential contribution of this method may extend beyond fetoscopic procedures to include general augmented reality applications in minimally invasive procedures.

Introductory review on ‘Flying Triangulation’: a motion-robust optical 3D measurement principle

‘Flying Triangulation’ (FlyTri) is a recently developed principle which allows for a motion-robust optical 3D measurement of rough surfaces. It combines a simple sensor with sophisticated algorithms: a single-shot sensor acquires 2D camera images. From each camera image, a 3D profile is generated. The series of 3D profiles generated are aligned to one another by algorithms, without relying on any external tracking device. It delivers real-time feedback of the measurement process which enables an all-around measurement of objects. The principle has great potential for small-space acquisition environments, such as the measurement of the interior of a car, and motion-sensitive measurement tasks, such as the intraoral measurement of teeth. This article gives an overview of the basic ideas and applications of FlyTri. The main challenges and their solutions are discussed. Measurement examples are also given to demonstrate the potential of the measurement principle.

Real-time bronchoscope three-dimensional motion estimation using multiple sensor-driven alignment of CT images and electromagnetic measurements

Bronchoscope three-dimensional motion estimation plays a key role in developing bronchoscopic navigation systems. Currently external tracking devices, particularly electromagnetic trackers with electromagnetic sensors, are increasingly introduced to navigate surgical tools in pre-clinical images. An unavoidable problem, which is to align the electromagnetic tracker to pre-clinical images, must be solved before navigation. This paper proposes a multiple sensor-driven registration method to establish this alignment without using any anatomical fiducials. Although current fiducially free registration methods work well, they limit to the initialization of optimization and manipulating the bronchoscope along the bronchial centerlines, which could be failed easily during clinical interventions. To address these limitations, we utilize measurements of multiple electromagnetic sensors to calculate bronchoscope geometric center positions that are usually closer to the bronchial centerlines than the sensor itself measured positions. We validated our method on a bronchial phantom. The experimental results demonstrate that our idea of using multiple sensors to determine bronchoscope geometric center positions for fiducial-free registration was very effective. Compared to currently available methods in bronchoscope three-dimensional motion estimation, our method reduced fiducial alignment error from at least 6.79 to 4.68–5.26mm and significantly improved motion estimation or tracking accuracy from at least 5.42 to 3.78–4.53mm.

SU‐D‐9A‐01: Listmode‐Driven Optimal Gating (OG) Respiratory Motion Management: Potential Impact On Quantitative PET Imaging

Purpose:To evaluate the potential impact of listmode‐driven amplitude based optimal gating (OG) respiratory motion management technique on quantitative PET imaging.Methods:During the PET acquisitions, an optical camera tracked and recorded the motion of a tool placed on top of patients' torso. PET event data were utilized to detect and derive a motion signal that is directly coupled with a specific internal organ. A radioactivity‐trace was generated from listmode data by accumulating all prompt counts in temporal bins matching the sampling rate of the external tracking device. Decay correction for 18F was performed. The image reconstructions using OG respiratory motion management technique that uses 35% of total radioactivity counts within limited motion amplitudes were performed with external motion and radioactivity traces separately with ordered subset expectation maximization (OSEM) with 2 iterations and 21 subsets. Standard uptake values (SUVs) in a tumor region were calculated to measure the effect of using radioactivity trace for motion compensation. Motion‐blurred 3D static PET image was also reconstructed with all counts and the SUVs derived from OG images were compared with SUVs from 3D images.Results:A 5.7 % increase of the maximum SUV in the lesion was found for optimal gating image reconstruction with radioactivity trace when compared to a static 3D image. The mean and maximum SUVs on the image that was reconstructed with radioactivity trace were found comparable (0.4 % and 4.5 % increase, respectively) to the values derived from the image that was reconstructed with external trace.Conclusion:The image reconstructed using radioactivity trace showed that the blurring due to the motion was reduced with impact on derived SUVs. The resolution and contrast of the images reconstructed with radioactivity trace were comparable to the resolution and contrast of the images reconstructed with external respiratory traces.Research supported by Siemens

Using surface markers for MRI guided breast conserving surgery: a feasibility survey

Breast MRI is frequently performed prior to breast conserving surgery in order to assess the location and extent of the lesion. Ideally, the surgeon should also be able to use the image information during surgery to guide the excision and this requires that the MR image is co-registered to conform to the patient’s position on the operating table. Recent progress in MR imaging techniques has made it possible to obtain high quality images of the patient in the supine position which significantly reduces the complexity of the registration task. Surface markers placed on the breast during imaging can be located during surgery using an external tracking device and this information can be used to co-register the images to the patient. There remains the problem that in most clinical MR scanners the arm of the patient has to be placed parallel to the body whereas the arm is placed perpendicular to the patient during surgery. The aim of this study is to determine the accuracy of co-registration based on a surface marker approach and, in particular, to determine what effect the difference in a patient’s arm position makes on the accuracy of tumour localization. Obtaining a second MRI of the patient where the patient’s arm is perpendicular to body axes (operating room position) is not possible. Instead we obtain a secondary MRI scan where the patient’s arm is above the patient’s head to validate the registration. Five patients with enhancing lesions ranging from 1.5 to 80 cm3 in size were imaged using contrast enhanced MRI with their arms in two positions. A thin-plate spline registration scheme was used to match these two configurations. The registration algorithm uses the surface markers only and does not employ the image intensities. Tumour outlines were segmented and centre of mass (COM) displacement and Dice measures of lesion overlap were calculated. The relationship between the number of markers used and the COM-displacement was also studied. The lesion COM-displacements ranged from 0.9 to 9.3 mm and the Dice overlap score ranged from 20% to 80%. The registration procedure took less than 1 min to run on a standard PC. Alignment of pre-surgical supine MR images to the patient using surface markers on the breast for co-registration therefore appears to be feasible.

Propagation of calibration errors in prospective motion correction using external tracking

Prospective motion correction of MRI scans using an external tracking device (such as a camera) is becoming increasingly popular, especially for imaging of the head. In order for external tracking data to be transformed into the MR scanner reference frame, the pose (i.e., position and orientation) of the camera relative to the scanner--or cross-calibration--must be accurate. In this study, we investigated how errors in cross-calibration affect the accuracy of motion correction feedback in MRI. An operator equation is derived describing how calibration errors relate to errors in applied motion compensation. By taking advantage of spherical symmetry and performing a Taylor approximation for small rotation angles, a closed form expression and upper limit for the residual tracking error is provided. Experiments confirmed theoretical predictions of a bilinear dependence of the residual rotational component on the calibration error and the motion performed, modulated by a sinusoidal dependence on the angle between the calibration error axis and motion axis. The residual translation error is bounded by the sum of the rotation angle multiplied by the translational calibration error plus the linear head displacement multiplied by the calibration error angle. The results make it possible to calculate the required cross-calibration accuracy for external tracking devices for a range of motions. Scans with smaller expected movements require less accuracy in cross-calibration than scans involving larger movements. Typical clinical applications require that the calibration accuracy is substantially below 1 mm and 1°.

Fast noniterative calibration of an external motion tracking device

Prospective motion correction of magnetic resonance (MR) scans commonly uses an external device, such as a camera, to track the pose of the organ of interest. However, in order for external tracking data to be translated into the MR scanner reference frame, the pose of the camera relative to the MR scanner must be known accurately. Here, we describe a fast, accurate, non-iterative technique to determine the position of an external tracking device de novo relative to the MR reference frame. The method relies on imaging a sparse object that allows simultaneous tracking of arbitrary rigid body transformations in the reference frame of the magnetic resonance imaging (MRI) machine and that of the external tracking device. Large motions in the MRI reference frame can be measured using a sparse phantom with an accuracy of 0.2 mm, or approximately 1/10 of the voxel size. By using a dual quaternion algorithm to solve the calibration problem, a good camera calibration can be achieved with fewer than six measurements. Further refinements can be achieved by applying the method iteratively and using motion correction feedback. Independent tracking of a series of movements in two reference frames allows for an analytical solution to the hand-eye-calibration problem for various motion tracking setups in MRI.

Externally Navigated Bronchoscopy Using 2-D Motion Sensors: Dynamic Phantom Validation

The paper presents a new endoscope motion tracking method that is based on a novel external endoscope tracking device and our modified stochastic optimization method for boosting endoscopy navigation. We designed a novel tracking prototype where a 2-D motion sensor was introduced to directly measure the insertion-retreat linear motion and also the rotation of the endoscope. With our developed stochastic optimization method, which embeds traceable particle swarm optimization in the Condensation algorithm, a full six degrees-of-freedom endoscope pose (position and orientation) can be recovered from 2-D motion sensor measurements. Experiments were performed on a dynamic bronchial phantom with maximal simulated respiratory motion around 24.0 mm. The experimental results demonstrate that our proposed method provides a promising endoscope motion tracking approach with more effective and robust performance than several current available tracking techniques. The average tracking accuracy of the position improved from 6.5 to 3.3 mm, which further approaches the clinical requirement of 2.0 mm in practice.

Dynamic sequence respiratory gated perfusion pulmonary SPECT without external tracking device

The purpose of this study was to develop and evaluate a new method for respiratory gated pulmonary perfusion SPECT (RGPS) based on dynamic acquisition without using an external tracking device (ETD) or list-mode data acquisition. In the phantom study, our method used a dynamic sequence technique, which was specified by sequences of 50-ms acquisition during 30 s per view of SPECT instead of using an ETD. For this purpose, we created a computer program that identified respiratory phases by calculating the center of activity (COA) in each dynamic frame image. We compared RGPS using the dynamic sequence acquisition (RGPS-DS) and RGPS using ETD (RGPS-ETD) in phantom studies employing a cylinder phantom filled with technetium-99m solution attached to an instrument providing a simple harmonic motion. In the patient study, RGPS-DS was applied to data collected from 3 patients during a routine study of Tc-MAA pulmonary perfusion SPECT. In the phantom study, the calculation of COA indicated a good agreement between RGPS-DS and RGPS-ETD. With an oscillatory phantom movement amplitude of 30 mm, the amplitudes determined by RGPS-DS and RGPS-ETD (28.36 and 27.58 mm, respectively) were identical on considering a pixel size of 4.66 mm for reconstructed SPECT images. In the patient study, applicability of our method to patient data was demonstrated. We have showed the feasibility of our method to obtain RGPS without ETD, and conclude that RGPS-DS may be an innovative and efficient technique in respiratory gated pulmonary perfusion SPECT. Further studies with a larger number of patients should demonstrate the accuracy of our method.

An EPID based method for performing high accuracy calibration between an optical external marker tracking device and the LINAC reference frame

With the increasing use of external 3D optical tracking cameras to guide modern radiation therapy procedures, it has become vitally important to have an accurate camera to linear accelerator (LINAC) reference frame calibration. To eliminate errors present in current calibration procedures based on the manual hand alignment of a device using the light field crosshairs and in room guidance lasers, a semiautomated quantitative calibration approach requiring only use of an electronic portal imaging device (EPID) was developed. A phantom comprised of seven highly IR reflective plastic BBs was placed on the LINAC treatment couch and imaged with both a 3D stereoscopic IR imager and the on board megavoltage (MV) EPID imager. Having knowledge of the optically determined 3D positions and projected EPID images of the BBs, simulated annealing was used to optimize the location of the BBs in the LINAC frame using four different optimization functions. Singular value decomposition was then used to calculate the transformation matrix between the camera and LINAC reference frames. Results were then compared to a traditional camera calibration method for overall accuracy. Using modeled data, the simulated annealing process was able to determine the actual locations of the BBs with a RMSE of 0.23 mm. Using projection images acquired with an MV imager, the process was able to determine locations of BBs within .26 mm. The results depend on the choice of optimization function. Results show that the method can be used to provide highly accurate spatial registration between an external 3D imaging reference frame and the LINAC frame. The experimental MV imager results, while not as precise as the simulated results, exceed 1 mm accuracy and the current accepted AAPM TG-142 standard of ≤2 mm positioning accuracy.

Passive Markers for Tracking Surgical Instruments in Real-Time 3-D Ultrasound Imaging

A family of passive echogenic markers is presented by which the position and orientation of a surgical instrument can be determined in a 3-D ultrasound volume, using simple image processing. Markers are attached near the distal end of the instrument so that they appear in the ultrasound volume along with the instrument tip. They are detected and measured within the ultrasound image, thus requiring no external tracking device. This approach facilitates imaging instruments and tissue simultaneously in ultrasound-guided interventions. Marker-based estimates of instrument pose can be used in augmented reality displays or for image-based servoing. Design principles for marker shapes are presented that ensure imaging system and measurement uniqueness constraints are met. An error analysis is included that can be used to guide marker design and which also establishes a lower bound on measurement uncertainty. Finally, examples of marker measurement and tracking algorithms are presented along with experimental validation of the concepts.

Hybrid prospective and retrospective head motion correction to mitigate cross‐calibration errors

Utilization of external motion tracking devices is an emerging technology in head motion correction for MRI. However, cross-calibration between the reference frames of the external tracking device and the MRI scanner can be tedious and remains a challenge in practical applications. In this study, we present two hybrid methods, both of which combine prospective, optical-based motion correction with retrospective entropy-based autofocusing to remove residual motion artifacts. Our results revealed that in the presence of cross-calibration errors between the optical tracking device and the MR scanner, application of retrospective correction on prospectively corrected data significantly improves image quality. As a result of this hybrid prospective and retrospective motion correction approach, the requirement for a high-quality calibration scan can be significantly relaxed, even to the extent that it is possible to perform external prospective motion tracking without any prior cross-calibration step if a crude approximation of cross-calibration matrix exists. Moreover, the motion tracking system, which is used to reduce the dimensionality of the autofocusing problem, benefits the retrospective approach at the same time.

WE‐E‐BRB‐06: An EPID Based Method for High Accuracy Calibration of an Optical External Marker Tracking Device with the LINAC Reference Frame

Purpose: The accuracy of an optical camera system used for patient motion monitoring is typically limited to 1–2 mm due to use of the light field crosshair and in‐room lasers for initial camera calibration setup making it unsuitable for highly accurate patient monitoring procedures such as frameless SRS. To provide submillimeter accuracy in calibrating a 3D stereoscopic infrared optical camera with the treatment frame of a LINAC, a semi‐automated calibration method was developed based on simultaneous viewing of external markers with both the MV treatment beam and optical camera system. Methods: A phantom comprised of 9 steel BBs covered in highly IR reflective tape was placed on the LINAC treatment couch and imaged with both a 3D stereoscopic IR imager, and the on board MV EPID imager at a variety of gantry angles. Using the relative 3D BB positions given by the camera, simulated annealing was used to optimize 9 parameters including source‐to‐imager distance, detector tilt, and deviations from nominal isocenter. Singular value decomposition was then used to calculate the transformation between different reference frames. Additionally, a traditional method of calibration was used for comparison. Results: Using modeled data, the simulated annealing process was able to determine the locations of the BBs with .2 mm accuracy. Using projection images acquired with an MV imager, initial preliminary results show the ability to determine locations of BBs with a .2 mm planar standard deviation and a 1 mm axial standard deviation. Conclusions: The method can be used to provide a highly accurate spatial registration between an external 3D camera frame and the LINAC frame. The real MV imager results, while not as precise as the simulated results, exceed current standards and are expected to improve with continued refinement.

Real-time Image-based Localization for Hand-held 3D-modeling

We present a self-referencing hand-held scanning device for vision-based close-range 3D-modeling. Our approach replaces external global tracking devices with ego-motion estimation directly from the camera used for reconstruction. The system is capable of online estimation of the 6DoF pose on hand-held devices with high motion dynamics especially in rotational components. Inertial information supports directly the tracking process to allow for robust tracking and feature management in highly dynamic environments. We introduce a weighting function for landmarks that contribute to the pose estimation increasing the accuracy of the localization and filtering outliers in the tracking process. We validate our approach with experimental results showing the robustness and accuracy of the algorithm. We compare the results to external global referencing solutions used in current modeling systems.

Localizing Broca's area for transcranial magnetic stimulation: Comparison of surface distance measurements and stereotaxic positioning

Precise placement of transcranial magnetic stimulation (TMS) coils over target regions is crucial for correct interpretation of TMS effects. Modern frameless stereotaxic systems yield high accuracy, but require extensive equipment and cannot be used in every setting, for example, during functional imaging sessions. The aim of this study was the development of a method for TMS-coil placement based on individual imaging data without the need for external tracking devices. We compared coil positioning over Broca's area using an advanced stereotaxic navigation system with placement according to the surface distance measurements (SDM) method. By using the SDM-method, 3-dimensional renderings adapted from individual T1-weighted magnetic resonance imaging (MRI) data were created to identify Broca's area and Broca's homologue, respectively, and to define anatomic landmarks on the skin's surface. Distances between these landmarks were used to localize the real target on the individual's head. The mean Euclidean distance between surface positions as determined with the two methods was 8.31 mm and the mean difference of estimated virtual electric field intensity at the target point was 7.37 V/m corresponding to 4.01% of maximum field strength. Our findings suggest that, compared with a state-of-the-art frameless stereotaxy system, the SDM-method yields a reasonable accuracy for positioning of a TMS-coil over Broca's area in terms of spatial coordinates.

Endoscopic navigation for minimally invasive suturing

Manipulating small objects such as needles, screws or plates inside the human body during minimally invasive surgery can be very difficult for less experienced surgeons due to the loss of 3D depth perception. Classical navigation techniques are often incapable of providing support in such situations, as the augmentation of the scene with the necessary artificial markers - if possible at all - is usually cumbersome and leads to increased invasiveness. We present an approach relying solely on a standard endoscope as a tracking device for determining the pose of such objects, using the example of a suturing needle. The resulting pose information is then used to generate artificial 3D cues on the 2D screen to provide optimal support for surgeons during tissue suturing. In addition, if an external tracking device is provided to report the endoscope's position, the suturing needle can be directly tracked in the world coordinate system. Furthermore, a visual navigation aid can be incorporated if a 3D surface is intraoperatively reconstructed from the endoscopic video stream or registered from preoperative imaging.