3D Optical Flow Research Articles

3D optical flow computation using a parallel variational multigrid scheme with application to cardiac C-arm CT motion

Motivated by recent applications to 3D medical motion estimation, we consider the problem of 3D optical flow computation in real time. The 3D optical flow model is derived from a straightforward extension of the 2D Horn–Schunck model and discretized using standard finite differences. We compare memory costs and convergence rates of four numerical schemes: Gauss–Seidel and multigrid with three different strategies of coarse grid operators discretization: direct coarsening, lumping and Galerkin approaches. Experimental results to compute 3D motion from cardiac C-arm CT images demonstrate that our variational multi-grid based on Galerkin discretization outperforms significantly the Gauss–Seidel method. The parallel implementation of the proposed scheme using domain partitioning shows that the algorithm scales well up to 32 processors on a cluster of AMD Opteron CPUs which consists of four-way nodes connected by an Infiniband network.

On Single-Sequence and Multi-Sequence Factorizations

Subspace based factorization methods are commonly used for a variety of applications, such as 3D reconstruction, multi-body segmentation and optical flow estimation. These are usually applied to a single video sequence. In this paper we present an analysis of the multi-sequence case and place it under a single framework with the single sequence case. In particular, we start by analyzing the characteristics of subspace based spatial and temporal segmentation. We show that in many cases objects moving with different 3D motions will be captured as a single object using multi-body (spatial) factorization approaches. Similarly, frames viewing different shapes might be grouped as displaying the same shape in the temporal factorization framework. Temporal factorization provides temporal grouping of frames by employing a subspace based approach to capture non-rigid shape changes (Zelnik-Manor and Irani, 2004). We analyze what causes these degeneracies and show that in the case of multiple sequences these can be made useful and provide information for both temporal synchronization of sequences and spatial matching of points across sequences.

Inverse perspective mapping and optic flow: A calibration method and a quantitative analysis

In most imaging devices (including the human eye), the mechanisms that govern the projection of a 3D scene onto a 2D surface render the extraction of useful 3D information difficult. We investigate the effects of perspective on optic flow in a driver assistance application in which a camera is mounted on the wing mirror in order to observe a driver's so called ‘blind spot’. A car travelling toward the camera appears to increase in speed and size on the projected image although its real speed and size are constant. We show that the inverse perspective mapping, previously used for obstacle detection, can also help in the problem of extracting real world speed from 2D optic flow data. We provide a quantitative analysis that shows precisely to what degree speed uniformity in the 3D world can be recovered by the mapping. To determine some mapping parameters, we devised a calibration method adapted to our specific situation that can be performed on-line and unsupervised. Its simplicity lends itself to fast software or hardware implementation.

TU‐EE‐A3‐06: Sub‐Millimeter Three Dimensional Ventilation Imaging of Rodent Lungs

Purpose: To develop a technique capable of generating a high resolution 3D ventilation image in a rat model from a series of CT images. This technique will be invaluable in developing strategies for radiation treatment optimization and characterization of pulmonary disease states. Method and Materials: A series of three 150gram Fischer 344 rats were euthanized and intubated. The lungs were equilibrated to ambient pressure, then serially inflated and imaged at approximately 1ml increments. A GE flat panel CT scanner capable of 0.15mm resolution was utilized for image acquisition. A deformable image registration algorithm, utilizing 3D optical flow, was applied to each pair of CT images to map, on a voxel by voxel basis, corresponding tissue elements. The difference in average CT number of a nine voxel region surrounding each pair of mapped voxels was then utilized to compute the change in fraction air per voxel (i.e. regional ventilation). This local ventilation measurement was then superimposed onto each voxel of the CT image to generate a 3D ventilation image. The sum of each voxel's ventilation (i.e. total lung ventilation) was then compared to the change in manually segmented lung volumes between each of a series of twenty‐two pair of CT images. Results: Visualization of the 3D ventilation images revealed regional heterogeneity of ventilation throughout the lung fields. The images required smoothing by averaging over a 0.75mm cube surrounding each voxel to minimize artifacts. The calculated total lung ventilation compared favorably with the change in manually segmented lung volumes, with a slope of unity, as expected, and a correlation coefficient of R=0.99. Conclusion: We have demonstrated a unique method of quantifying regional lung ventilation with sub‐millimeter accuracy in a rat model. This model has been developed in order to study lung function and radiation injury.

3D velocity from 3D Doppler radial velocity

We present local least squares and regularization frameworks for computing 3D velocity (3D optical flow) from 3D radial velocity measured by a Doppler radar. We demonstrate the performance of our algorithms quantitatively on synthetic radial velocity data and qualitatively on real radial velocity data, obtained from the Doppler radar at Kurnell Radar station, Botany Bay, New South Wales, Australia. Radial velocity can be used to predict the future positions of storms in sequences of Doppler radar datasets.© 2005 Wiley Periodicals, Inc. Int J Imaging Syst Technol, 15, 189–198, 2005

Validation of the path integration of 4D dose distributions with 4D phantom measurements

4D treatment planning generates a series of 3D dose distributions to represent the motion properties of the patient and/or radiation delivery system. The purpose of this study is to validate with 4D phantom measurements the use of the 3D optical flow method algorithm for the numerical path integration of 4D dose distribution data into a single 3D dose distribution. The Radiological Physics Center (RPC) thoracic phantom was placed on a programmable motion table to simulate respiratory motion. 4D CT imaging was performed on a multi-slice CT scanner (Philips Medical Systems, Anhover, MA). The CT data was binned into eight phases. The CT images were sent to a Pinnacle workstation (version 6.2b, Phillips Medical Systems, Anhover, MA) where a treatment plan was designed on the end expiration image set. The treatment plan was transposed onto the other respiratory phases and a 4D dose data set generated. The plan was delivered using a Varian 2100 linear accelerator (Varian Medical System, Palo Alto, CA). The dose delivered to the RPC thoracic phantom was recorded using radiographic film and thermoluminescent dosimeters for calibration. The 3D optical flow method (3D OFM) algorithm is a deformable image registration algorithm we have implemented. 3D OFM was used to calculate the displacement fields between the respiratory phases. The displacement fields were used to map the dose distributions from the multiple respiratory phases onto the end expiration phase. The resulting dose distributions were summed with equal weighting, performing numerical path integration. The calculated and measured dose distributions were compared with the static plan. The attached figure illustrates the dose distributions. A coronal section through the static treatment plan is shown in figure A.u1/2pick;3197f1;0;11 The 3D OFM algorithm was applied and numerical path integration performed on the 4D dose distribution data resulting in a single 3D dose distribution. The corresponding coronal section is shown in figure B. A coronal section through the measured dose distribution is shown in figure C. The delivered dose distribution had a peak width at 90% maximum of 38.5 mm. The corresponding value for the path integrated 4D plan and the end expiration static plan were 42.6 mm and 47.0 mm. The 50% isodose line was displaced 0.5 mm when compared with the path integrated 4D plan and 8.6 mm when compared with the end expiration static plan. The 3D OFM algorithm provides the displacement field necessary for numerical path integration to map a 4D dose distribution data set into a single 3D dose distribution accurately. The resulting dose distribution was found to more closely represent the delivered dose versus the static plan for a moving phantom.

Quantitative measurement of regional tumor response to therapy

We have validated the use of a 3D deformable image registration algorithm in mapping the motion of a phantom lung tumor. We then applied the method to measure regional intra-tumor response in lung cancer patients treated with induction chemotherapy followed by concurrent chemoradiation. Numerous methods have been developed to quantify global tumor response to therapy. We describe for the first time a method to quantify response within radiographically defined intra-tumor sub-populations. A deformable image registration algorithm using a 3D optical flow method was developed to obtain a displacement vector between each voxel in a pair of related CT scans. The algorithm was validated by mapping the displacement of each voxel in a phantom lung tumor CT scan to a corresponding CT scan with a known 1.2cm and 2.4cm displacement. We then applied this method to a series of four recent lung cancer patients treated on a phase III protocol with two cycles of carboplatin-taxol based chemotherapy followed by concurrent chemoradiation with weekly carboplatin-taxol. Each patient obtained a high resolution spiral CT scan prior to treatment, after induction chemotherapy and after chemoradiation. A total of 32 tumor sub-populations were identified by dividing each pre-treatment tumor volume into octants. Each voxel of the pre-treatment scan was then independently mapped to the post-chemotherapy and post-radiation CT scans. The volume of the mapped tumor sub-populations were then compared to the pre-treatment CT scan as a measure of regional tumor response to each therapy. The measured displacement of the phantom lung tumor voxels was 1.20 and 2.40 cm with a full-width at tenth maximum of 0.008 and 0.006 cm respectively. The maximum error in voxel displacement was 1 mm. In the patient CT data sets, the mapped voxels of the intra-tumor regions were visualized on each of the post-treatment scans as represented in the color overlays of the CT scans in the figure. From this representative course it is evident that significant differences in response to therapy can be measured within tumor sub-populations. Although one tumor responded uniformly with less than 10% difference between octants, the other tumors responded with greater than 20% variation between the maximally and minimally responding octants. In one case, the residual volume in the maximally and minimally responding octants were 6.7% and 75.7% respectively. The residual tumor volume in each octant ranged from 26.7% to 86.9% with a mean of 59.2% after induction chemotherapy compared with a range of 6.7% to 75.7% with a mean of 39.8% after concurrent chemoradiation. A pair-wise comparison of the response to chemotherapy and concurrent chemoradiation in each of the 32 octants measured revealed no correlation between response to chemotherapy and response to concurrent chemoradiation with a correlation coefficient of r = 0.40. We have validated a new method of mapping voxel motion between related CT scans. We then successfully applied this technique to measure regional intra-tumor response to therapy. One tumor appeared to respond uniformly, while others responded with significant regional heterogeneity suggesting the existence of tumor sub-populations with a range of therapeutic sensitivity. Finally, despite the prevailing belief that response to chemotherapy predicts for response to radiation, the magnitude of this response was not correlated within sub-populations of the tumor in our study.

Validation of the path integration of 4D dose distributions with 4D phantom measurements

4D treatment planning generates a series of 3D dose distributions to represent the motion properties of the patient and/or radiation delivery system. The purpose of this study is to validate with 4D phantom measurements the use of the 3D optical flow method algorithm for the numerical path integration of 4D dose distribution data into a single 3D dose distribution.

Intrathoracic tumour motion estimation from CT imaging using the 3D optical flow method**Presented at The IASTED Second International Conference on Biomedical Engineering (BioMED 2004), Innsbruck, Austria, 16–18 February 2004.

The purpose of this work was to develop and validate an automated method for intrathoracic tumour motion estimation from breath-hold computed tomography (BH CT) imaging using the three-dimensional optical flow method (3D OFM). A modified 3D OFM algorithm provided 3D displacement vectors for each voxel which were used to map tumour voxels on expiration BH CT onto inspiration BH CT images. A thoracic phantom and simulated expiration/inspiration BH CT pairs were used for validation. The 3D OFM was applied to the measured inspiration and expiration BH CT images from one lung cancer and one oesophageal cancer patient. The resulting displacements were plotted in histogram format and analysed to provide insight regarding the tumour motion. The phantom tumour displacement was measured as 1.20 and 2.40 cm with full-width at tenth maximum (FWTM) for the distribution of displacement estimates of 0.008 and 0.006 cm, respectively. The maximum error of any single voxel's motion estimate was 1.1 mm along the z-dimension or approximately one-third of the z-dimension voxel size. The simulated BH CT pairs revealed an rms error of less than 0.25 mm. The displacement of the oesophageal tumours was nonuniform and up to 1.4 cm, this was a new finding. A lung tumour maximum displacement of 2.4 cm was found in the case evaluated. In conclusion, 3D OFM provided an accurate estimation of intrathoracic tumour motion, with estimated errors less than the voxel dimension in a simulated motion phantom study. Surprisingly, oesophageal tumour motion was large and nonuniform, with greatest motion occurring at the gastro-oesophageal junction.

3D deformable image processing and integration

Medical images from different modalities, such as X-ray computerized tomography (CT), magnetic resonance imaging (MRI), single photon emission computed tomography (SPECT) and positron emission tomography (PET) are simultaneously used for clinical diagnosis and research. The intention to integrate medical images includes (1) complementing functional information from functional images with anatomical information from CT and MRI, (2) finding changes over time or under different conditions, or (3) integrating information taken from different imaging modalities. In this paper, we present a three-dimensional (3D) image deformation technique to integrate multi-modality medical images. The 3D optical flow (OF) estimation and thin-plate spline (TPS) transformation are applied on landmarks tracing and geometric deformation, respectively, for inter- and intra-subject medical image integration. The proposed technique is separated into two parts—coarse and fine registration. In the coarse registration, the 3D OF method iteratively estimates moving vectors between corresponding pixels in two image datasets. The desired landmarks are obtained according to the estimated vectors, which relate to the corresponding image dataset, estimated by 3D OF method. As the pairs of landmarks are obtained, these reference landmarks are input into the 3D TPS transformation matrix to generate desired deformation results of coarse image registration. The coarse registration result is applied for fine adjustment by 3D OF estimation by iterative multi-resolution procedures at different levels. To validate the integration performance, the normal brain templates, including MRI and PET volume images, are applied as standard datasets in this study, and the clinical physician selects the standard landmarks from the standard images used for 3D TPS deformation. The results show that RMSE between standard templates and deformation results of MRI and PET studies are less than 3% on average, respectively. In conclusion, the automatic landmarks tracing technique (using 3D OF) and image deformation (using TPS transformation) can minimize human interaction, and integrate medical images effectively.

A Modular Multi-Chip Neuromorphic Architecture for Real-Time Visual Motion Processing

The extent of pixel-parallel focal plane image processing is limited by pixel area and imager fill factor. In this paper, we describe a novel multi-chip neuromorphic VLSI visual motion processing system which combines analog circuitry with an asynchronous digital interchip communications protocol to allow more complex pixel-parallel motion processing than is possible in the focal plane. This multi-chip system retains the primary advantages of focal plane neuromorphic image processors: low-power consumption, continuous-time operation, and small size. The two basic VLSI building blocks are a photosensitive sender chip which incorporates a 2D imager array and transmits the position of moving spatial edges, and a receiver chip which computes a 2D optical flow vector field from the edge information. The elementary two-chip motion processing system consisting of a single sender and receiver is first characterized. Subsequently, two three-chip motion processing systems are described. The first three-chip system uses two sender chips to compute the presence of motion only at a particular stereoscopic depth from the imagers. The second three-chip system uses two receivers to simultaneously compute a linear and polar topographic mapping of the image plane, resulting in information about image translation, rotation, and expansion. These three-chip systems demonstrate the modularity and flexibility of the multi-chip neuromorphic approach.

Understanding coronary artery movement: a knowledge-based approach

The aim of this paper is to describe a knowledge-based system that interprets three-dimensional (3D) coronary artery movement, using data from digital subtraction angiography image sequences. Dynamic information obtained from artery centerline 3D reconstruction and optical flow estimation, is classified according to experimental evidence indicating that artery displacements are quasi-homogeneous by a segment analysis. Characteristic motion features like displacement direction, perpendicular/radial components, rotation direction, curvature and torsion are qualitatively described from an image sequence using symbolic labels. These facts are then related and interpreted using anatomical-functional knowledge provided by a specialist, as well as spatial and temporal knowledge, applying spatio-temporal reasoning schemes. Facts, knowledge and reasoning rules are stated in a declarative form. Detailed examples of local and global interpretation results, using a real reconstructed angiographic biplane image sequence are presented in order to illustrate how our system suitably interprets coronary artery dynamic behavior.

Derivation of optical flow using a spatiotemporal-Frequency approach

We advance in this paper the spatiotemporal-frequency (STF) approach for computing the optical flow of a time-varying image. STF flow derivation provides an attractive alternative to earlier approaches based on (1) feature correspondence, (2) spatiotemporal gradients, and (3) Fourier-phase changes. After briefly surveying these three earlier approaches to flow computation, we provide an historical overview of the development of the STF approach. Then an improved STF method for flow derivation that has recently been developed by the authors is presented along with experimental results that demonstrate its use. The STF yields a dense time and spatially varying 2D optical flow field for each image frame. We conclude by showing that STF derivation (a) promises substantially improved performance over the flow computation methods, and (b) provides a partial explanation of motion coherence as observed in human vision.