Image Motion Estimation Research Articles

Ultrafast Cardiac Imaging Using Deep Learning for Speckle-Tracking Echocardiography.

High-quality ultrafast ultrasound imaging is based on coherent compounding from multiple transmissions of plane waves (PW) or diverging waves (DW). However, compounding results in reduced frame rate, as well as destructive interferences from high-velocity tissue motion if motion compensation (MoCo) is not considered. While many studies have recently shown the interest of deep learning for the reconstruction of high-quality static images from PW or DW, its ability to achieve such performance while maintaining the capability of tracking cardiac motion has yet to be assessed. In this article, we addressed such issue by deploying a complex-weighted convolutional neural network (CNN) for image reconstruction and a state-of-the-art speckle-tracking method. The evaluation of this approach was first performed by designing an adapted simulation framework, which provides specific reference data, i.e., high-quality, motion artifact-free cardiac images. The obtained results showed that, while using only three DWs as input, the CNN-based approach yielded an image quality and a motion accuracy equivalent to those obtained by compounding 31 DWs free of motion artifacts. The performance was then further evaluated on nonsimulated, experimental in vitro data, using a spinning disk phantom. This experiment demonstrated that our approach yielded high-quality image reconstruction and motion estimation, under a large range of velocities and outperforms a state-of-the-art MoCo-based approach at high velocities. Our method was finally assessed on in vivo datasets and showed consistent improvement in image quality and motion estimation compared to standard compounding. This demonstrates the feasibility and effectiveness of deep learning reconstruction for ultrafast speckle-tracking echocardiography.

Ultra-fast multi-parametric 4D-MRI image reconstruction for real-time applications using a downsampling-invariant deformable registration (D2R) model

Background and purposeMotion estimation from severely downsampled 4D-MRI is essential for real-time imaging and tumor tracking. This simulation study developed a novel deep learning model for simultaneous MR image reconstruction and motion estimation, named the Downsampling-Invariant Deformable Registration (D2R) model. Materials and methodsForty-three patients undergoing radiotherapy for liver tumors were recruited for model training and internal validation. Five prospective patients from another center were recruited for external validation. Patients received 4D-MRI scans and 3D MRI scans. The 4D-MRI was retrospectively down-sampled to simulate real-time acquisition. Motion estimation was performed using the proposed D2R model. The accuracy and robustness of the proposed D2R model and baseline methods, including Demons, Elastix, the parametric total variation (pTV) algorithm, and VoxelMorph, were compared. High-quality (HQ) 4D-MR images were also constructed using the D2R model for real-time imaging feasibility verification. The image quality and motion accuracy of the constructed HQ 4D-MRI were evaluated. ResultsThe D2R model showed significantly superior and robust registration performance than all the baseline methods at downsampling factors up to 500. HQ T1-weighted and T2-weighted 4D-MR images were also successfully constructed with significantly improved image quality, sub-voxel level motion error, and real-time efficiency. External validation demonstrated the robustness and generalizability of the technique. ConclusionIn this study, we developed a novel D2R model for deformation estimation of downsampled 4D-MR images. HQ 4D-MR images were successfully constructed using the D2R model. This model may expand the clinical implementation of 4D-MRI for real-time motion management during liver cancer treatment.

Mathematics of biomedical imaging today—a perspective

Biomedical imaging is a fascinating, rich and dynamic research area, which has huge importance in biomedical research and clinical practice alike. The key technology behind the processing, and automated analysis and quantification of imaging data is mathematics. Starting with the optimisation of the image acquisition and the reconstruction of an image from indirect tomographic measurement data, all the way to the automated segmentation of tumours in medical images and the design of optimal treatment plans based on image biomarkers, mathematics appears in all of these in different flavours. Non-smooth optimisation in the context of sparsity-promoting image priors, partial differential equations for image registration and motion estimation, and deep neural networks for image segmentation, to name just a few. In this article, we present and review mathematical topics that arise within the whole biomedical imaging pipeline, from tomographic measurements to clinical support tools, and highlight some modern topics and open problems. The article is addressed to both biomedical researchers who want to get a taste of where mathematics arises in biomedical imaging as well as mathematicians who are interested in what mathematical challenges biomedical imaging research entails.

Deep Image Registration With Depth-Aware Homography Estimation

Image registration is a basic task in computer vision, for its wide potential applications in image stitching, stereo vision, motion estimation, and etc. Most current methods achieve image registration by estimating a global homography matrix between candidate images with point-feature-based matching or direct prediction. However, as real-world 3D scenes have point-variant photograph distances (depth), a unified homography matrix is not sufficient to depict the specific pixel-wise relations between two images. Some researchers try to alleviate this problem by predicting multiple homography matrixes for different patches or segmentation areas in images; in this letter, we tackle this problem with further refinement, i.e. matching images with pixel-wise, depth-aware homography estimation. Firstly, we construct an efficient convolutional network, the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">DPH-Net</i> , to predict the essential parameters causing image deviation, the rotation ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$R$</tex-math></inline-formula> ) and translation ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$T$</tex-math></inline-formula> ) of cameras. Then, we feed-in an image depth map for the calculation of initial pixel-wise homography matrixes, which are refined with an online optimization scheme. Finally, with the estimated pixel-specific homography parameters, pixel correspondences between candidate images can be easily computed for registration. Compared with state-of-the-art image registration algorithms, the proposed <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">DPH-Net</i> has the highest performance of 0.912 EPE and 0.977 SSIM, demonstrating the effectiveness of adding depth information and estimating pixel-wise homography into the image registration process.

Variational image motion estimation by preconditioned dual optimization

&lt;p style='text-indent:20px;'&gt;Estimating optical flows is one of the most interesting problems in computer vision, which estimates the essential information about pixel-wise displacements between two consecutive images. This work introduces an efficient dual optimization framework with accelerated preconditioners to the challenging nonsmooth optimization problem of total-variation regularized optical-flow estimation. In theory, the proposed dual optimization framework brings an elegant variational analysis to the given difficult optimization problem, while presenting an efficient algorithmic scheme without directly tackling the corresponding nonsmoothness in numeric. By introducing efficient preconditioners with a multi-scale implementation, the proposed preconditioned dual optimization approaches achieve competitive estimation results of image motion, compared to the state-of-the-art methods. Moreover, we show that the proposed preconditioners can guarantee convergence of the implemented numerical schemes with high efficiency.&lt;/p&gt;

Spatiotemporal imaging with diffeomorphic optimal transportation

Motivated by the image reconstruction in spatiotemporal imaging, we introduce a concept named diffeomorphic optimal transportation (DOT), which combines the Wasserstein distance with Benamou–Brenier formula in optimal transportation and the flow of diffeomorphisms in large deformation diffeomorphic metric mapping. Using DOT, we propose a new variational model for joint image reconstruction and motion estimation, which is suitable for spatiotemporal imaging involving mass-preserving large diffeomorphic deformations. We further get its equivalent PDE-constrained optimal control formulation. The proposed model is easy-to-implement and solved by an alternating gradient descent algorithm, which is compared against existing alternatives theoretically and numerically. The performance is validated by several numerical experiments in spatiotemporal tomography, where the projection data is time-dependent sparse and/or high-noise. Moreover, we present several extensions based on DOT. Under appropriate conditions, the proposed algorithm can be adapted as a new algorithm to solve the models using quadratic Wasserstein distance.

Deep learning-based image reconstruction and motion estimation from undersampled radial k-space for real-time MRI-guided radiotherapy

To enable magnetic resonance imaging (MRI)-guided radiotherapy with real-time adaptation, motion must be quickly estimated with low latency. The motion estimate is used to adapt the radiation beam to the current anatomy, yielding a more conformal dose distribution. As the MR acquisition is the largest component of latency, deep learning (DL) may reduce the total latency by enabling much higher undersampling factors compared to conventional reconstruction and motion estimation methods. The benefit of DL on image reconstruction and motion estimation was investigated for obtaining accurate deformation vector fields (DVFs) with high temporal resolution and minimal latency.2D cine MRI acquired at 1.5 T from 135 abdominal cancer patients were retrospectively included in this study. Undersampled radial golden angle acquisitions were retrospectively simulated. DVFs were computed using different combinations of conventional- and DL-based methods for image reconstruction and motion estimation, allowing a comparison of four approaches to achieve real-time motion estimation. The four approaches were evaluated based on the end-point-error and root-mean-square error compared to a ground-truth optical flow estimate on fully-sampled images, the structural similarity (SSIM) after registration and time necessary to acquire k-space, reconstruct an image and estimate motion.The lowest DVF error and highest SSIM were obtained using conventional methods up to . For undersampling factors , the lowest DVF error and highest SSIM were obtained using conventional image reconstruction and DL-based motion estimation. We have found that, with this combination, accurate DVFs can be obtained up to with an average root-mean-square error up to 1 millimeter and an SSIM greater than 0.8 after registration, taking 60 milliseconds.High-quality 2D DVFs from highly undersampled k-space can be obtained with a high temporal resolution with conventional image reconstruction and a deep learning-based motion estimation approach for real-time adaptive MRI-guided radiotherapy.

A robust optical flow motion estimation and correction method for IRT imaging in brain surgery

ABSTRACT In brain surgery, respiration motion, outliers, and occlusions create artefacts in Infrared Thermography (IRT) imaging. In this paper, we propose a robust method to handle multiple motion, outliers, and occlusions in IRT images which consists of two phases: preprocessing and image motion estimation. In the preprocessing phase, the Region of Interest (RoI) segmentation is employed to extract the brain cortex only. Thereafter, the Phase Correlation method is employed to compensate for large motion followed by occlusion masking based on an approach applying Cellular Nonlinear Networks (CNN). Next, intensity adjustment is made with respect to the reference image. Then, a Gaussian filter is applied. In the following phase, the image motion is estimated by employing Combined Local-Global (CLG) optical flow method. In order to find the best regularization coefficient for the spatial coherency term and the number of iterations for recursive optical flow estimation, we generated ground truth and evaluated the accuracy of the estimated motion vectors based on Average Angular Error (AAE) and Average Magnitude Error (AME). The efficiency improvement of the proposed method was tested on 1024 IRT images based on different comparisons. Thereby, the proposed method shows promising results for motion estimation and correction application in brain surgery.

Improved PET/CT Respiratory Motion Compensation by Incorporating Changes in Lung Density

Positron emission tomography/computed tomography (PET/CT) lung imaging is highly sensitive to motion. Although several techniques exist to diminish motion artifacts, a few accounts for both tissue displacement and changes in density due to the compression and dilation of the lungs, which cause quantification errors. This article presents an experimental framework for joint activity image reconstruction and motion estimation in PET/CT, where the PET image and the motion are directly estimated from the raw data. Direct motion estimation methods for motion-compensated PET/CT are preferable as they require a single attenuation map only and result in optimal signal-to-noise ratio (SNR). Previous implementations, however, failed to address changes in density during respiration. We propose to account for such changes using the Jacobian determinant of the deformation fields. In a feasibility study, we demonstrate on a modified extended cardiac-torso (XCAT) phantom with breathing motion—where the lung density and activity vary—that our approach achieved better quantification in the lungs than conventional PET/CT joint activity image reconstruction and motion estimation that does not account for density changes. The proposed method resulted in lower bias and variance in the activity images, reduced mean relative activity error in the lung at the reference gate (−4.84% to −3.22%) and more realistic Jacobian determinant values.

Image Similarity Metrics Suitable for Infrared Video Stabilization during Active Wildfire Monitoring: A Comparative Analysis

Aerial Thermal Infrared (TIR) imagery has demonstrated tremendous potential to monitor active forest fires and acquire detailed information about fire behavior. However, aerial video is usually unstable and requires inter-frame registration before further processing. Measurement of image misalignment is an essential operation for video stabilization. Misalignment can usually be estimated through image similarity, although image similarity metrics are also sensitive to other factors such as changes in the scene and lighting conditions. Therefore, this article presents a thorough analysis of image similarity measurement techniques useful for inter-frame registration in wildfire thermal video. Image similarity metrics most commonly and successfully employed in other fields were surveyed, adapted, benchmarked and compared. We investigated their response to different camera movement components as well as recording frequency and natural variations in fire, background and ambient conditions. The study was conducted in real video from six fire experimental scenarios, ranging from laboratory tests to large-scale controlled burns. Both Global and Local Sensitivity Analyses (GSA and LSA, respectively) were performed using state-of-the-art techniques. Based on the obtained results, two different similarity metrics are proposed to satisfy two different needs. A normalized version of Mutual Information is recommended as cost function during registration, whereas 2D correlation performed the best as quality control metric after registration. These results provide a sound basis for image alignment measurement and open the door to further developments in image registration, motion estimation and video stabilization for aerial monitoring of active wildland fires.

Heuristic Algorithms of Coincidence for the Estimation of Movements in Compression of Images

The NP-Completeness theory states that exact and efficient algorithms are unlikely to exist for the class of NP-difficult problems. One way to deal with NP hardness is to relax the optimality requirement and look for solutions instead that are close to the optimum. This is the main idea behind the approximation algorithms, which are called heuristic or metaheuristic. The problem of motion estimation is a process with a high degree of computational complexity, it requires sufficient memory space and execution time. It represents the cost of static, dynamic and video image sequence coding. The main task is to minimize the distortion rate and improve visual quality. This makes research in the field of coding, image compression and video focus on finding efficient algorithms to carry out the estimation of movement in a reasonable time. If a list of images of n elements is analyzed, there are feasible solutions. So, an exhaustive search is too slow, even for small values of the solution space. Therefore, from a practical point of view, it is crucial to have efficient and fast heuristic algorithms that avoid thorough search. In this investigation we design and implement heuristic algorithms, based on the frequency domain, which are applied on the coefficients of the discrete transform of the cosine and wavelets. Also, we propose temporal domain algorithms such as block-matching algorithms, which focus your search on the maximum coincidence of the current image with the reference one. The algorithms used during the implementation of this research work were written with the mathematical programming language MATLAB. In addition, we review the basic concepts of image processing, video, compression algorithms and motion estimation frequently used. The evaluation of the algorithms was carried out with a set of images provided by a previous acquisition system. We show the improvement of visual quality, the amount of compressed or reconstructed information and the behavior of the methods in the search for similarities between pixels or images. Finally, we contribute to the dissemination of new lines of scientific research that lead to the expansion and improvement of the study, the generation of new knowledge, since it is a young area within the Education discipline of Nicaragua.

Comparison Between Multiline Transmission and Diverging Wave Imaging: Assessment of Image Quality and Motion Estimation Accuracy.

High frame rate imaging is particularly important in echocardiography for better assessment of the cardiac function. Several studies showed that diverging wave imaging (DWI) and multiline transmission (MLT) are promising methods for achieving a high temporal resolution. The aim of this study was to compare MLT and compounded motion compensation (MoCo) DWI for the same transmitted power, same frame rates [image quality and speckle tracking echocardiography (STE) assessment], and same packet size [tissue Doppler imaging (TDI) assessment]. Our results on static images showed that MLT outperforms DW in terms of resolution (by 30% on average). However, in terms of contrast, MLT outperforms DW only for the depth of 11 cm (by 40% on average), the result being reversed at a depth of 4 cm (by 27% on average). In vitro results on a spinning phantom at nine different velocities showed that similar STE axial errors (up to 2.3% difference in median errors and up to 2.1% difference in the interquartile ranges) are obtained with both ultrafast methods. On the other hand, the median lateral STE estimates were up to 13% more accurate with DW than with MLT. On the contrary, the accuracy of TDI was only up to ~3% better with MLT, but the achievable DW Doppler frame rate was up to 20 times higher. However, our overall results showed that the choice of one method relative to the other is therefore dependent on the application. More precisely, in terms of image quality, DW is more suitable for imaging structures at low depths, while MLT can provide an improved image quality at the focal point that can be placed at higher depths. In terms of motion estimation, DW is more suitable for color Doppler-related applications, while MLT could be used to estimate velocities along selected lines of the image.

Motion estimation and correction in cardiac CT angiography images using convolutional neural networks

Cardiac motion artifacts frequently reduce the interpretability of coronary computed tomography angiography (CCTA) images and potentially lead to misinterpretations or preclude the diagnosis of coronary artery disease (CAD). In this paper, a novel motion compensation approach dealing with Coronary Motion estimation by Patch Analysis in CT data (CoMPACT) is presented. First, the required data for supervised learning is generated by the Coronary Motion Forward Artifact model for CT data (CoMoFACT) which introduces simulated motion to 19 artifact-free clinical CT cases with step-and-shoot acquisition protocol. Second, convolutional neural networks (CNNs) are trained to estimate underlying 2D motion vectors from 2.5D image patches based on the coronary artifact appearance. In a phantom study with computer-simulated vessels, CNNs predict the motion direction and the motion magnitude with average test accuracies of 13.37°±1.21° and 0.77 ± 0.09 mm, respectively. On clinical data with simulated motion, average test accuracies of 34.85°±2.09° and 1.86 ± 0.11 mm are achieved, whereby the precision of the motion direction prediction increases with the motion magnitude. The trained CNNs are integrated into an iterative motion compensation pipeline which includes distance-weighted motion vector extrapolation. Alternating motion estimation and compensation in twelve clinical cases with real cardiac motion artifacts leads to significantly reduced artifact levels, especially in image data with severe artifacts. In four observer studies, mean artifact levels of 3.08 ± 0.24 without MC and 2.28 ± 0.29 with CoMPACT MC are rated in a five point Likert scale.

Papers] Automatic Self-Improvement Scheme in Optical Flow-Based Motion Estimation for Sequential Fisheye Images

Papers] Automatic Self-Improvement Scheme in Optical Flow-Based Motion Estimation for Sequential Fisheye Images

A Numerical Framework for Efficient Motion Estimation on Evolving Sphere-Like Surfaces Based on Brightness and Mass Conservation Laws

In this work we consider brightness and mass conservation laws for motion estimation on evolving Riemannian 2-manifolds that allow for a radial parametrization from the 2-sphere. While conservation of brightness constitutes the foundation for optical flow methods and has been generalized to said scenario, we formulate the principle of mass conservation for time-varying surfaces that are embedded in Euclidean 3-space and derive a generalized continuity equation. The main motivation for this work is efficient cell motion estimation in volumetric fluorescence microscopy images of a living zebrafish embryo. The increasing spatial and temporal resolution of modern microscopes requires efficient analysis of such data. With this application in mind we address this need and follow an emerging paradigm in this field: dimensional reduction. In light of the ill-posedness of considered conservation laws, we employ Tikhonov regularization and propose the use of spatially varying regularization functionals that recover motion only in regions with cells. For the efficient numerical solution, we devise a mesh-free Galerkin method based on compactly supported (tangent) vectorial basis functions. Furthermore, for the fast and accurate estimation of the evolving sphere-like surface from scattered data, we utilize surface interpolation with spatio-temporal regularization. We present numerical results based on the aforementioned data featuring fluorescently labeled cells.

A New Variational Model for Joint Image Reconstruction and Motion Estimation in Spatiotemporal Imaging

We propose a new variational model for joint image reconstruction and motion estimation in spatiotemporal imaging, which is investigated along a general framework that we present with shape theory. This model consists of two components, one for conducting modified static image reconstruction, and the other performs sequentially indirect image registration. For the latter, we generalize the large deformation diffeomorphic metric mapping framework into the sequentially indirect registration setting. The proposed model is compared theoretically against alternative approaches (optical flow based model and diffeomorphic motion models), and we demonstrate that the proposed model has desirable properties in terms of the optimal solution. The theoretical derivations and efficient algorithms are also presented for a time-discretized scenario of the proposed model, which show that the optimal solution of the time-discretized version is consistent with that of the time-continuous one, and most of the computational components is the easy-implemented linearized deformation. The complexity of the algorithm is analyzed as well. This work is concluded by some numerical examples in 2D space + time tomography with very sparse and/or highly noisy data.

A New Rapid-Precision Position Measurement Method for a Linear Motor Mover Based on a 1-D EPCA

This paper is dedicated to studying a new position measurement method of a linear motor mover based on machine vision, which provides a novel idea for the detection of mover position. A rapid-precise measurement system based on a fence image and a phase correlation algorithm (PCA) is established. Multiline sampling and parallel computing methods are used to simplify the fence image directly; these methods can convert the 2-D image motion estimation into a simpler, 1-D signal process. To rapidly detect the mover position with high precision, a 1-D extended phase correlation algorithm (EPCA) is proposed. During position measurement, three key steps for the implementation are applied as follows. 1) Capture objective fence stripes using a high-speed camera installed beside the mover; 2) suppress the spectral leakage of discrete Fourier transform by a Hanning window function; and 3) calculate the displacement vector of the linear motor mover using a 1-D PCA and the least square method. Finally, the actual displacement of the mover can be obtained according to the system calibration. Experimental results demonstrate that the proposed 1-D EPCA implemented at measuring mover position is strongly robust and speedy with a detecting accuracy of 0.02 mm.

Digital video stabilization based on multilayer gray projection

Video stabilization is an important video optimization technology which is aimed at removing unconscious shakiness from videos. In this paper, we present a multilayer gray projection method to estimate image motion. While classical gray projection works only on the videos which just have translation. To achieve this, a multilayer gray projection algorithm is proposed to estimate translation, rotation and scaling motion between target and reference images. First, differential gray projection is applied to the test and target images to estimate relative image translation, the best projection results can help us find corresponding projection center of ring-projection and circular projection in the next two steps. We then calculate the scaling and rotation between test and target images by doing ring-projection and circular projection. The performance of our method is tested by robust experiments, estimation accuracy analysis and experiments over several videos.

Local All-Pass Geometric Deformations.

This paper deals with the estimation of a deformation that describes the geometric transformation between two images. To solve this problem, we propose a novel framework that relies upon the brightness consistency hypothesis-a pixel's intensity is maintained throughout the transformation. Instead of assuming small distortion and linearizing the problem (e.g. via Taylor Series expansion), we propose to interpret the brightness hypothesis as an all-pass filtering relation between the two images. The key advantages of this new interpretation are that no restrictions are placed on the amplitude of the deformation or on the spatial variations of the images. Moreover, by converting the all-pass filtering to a linear forward-backward filtering relation, our solution to the estimation problem equates to solving a linear system of equations, which leads to a highly efficient implementation. Using this framework, we develop a fast algorithm that relates one image to another, on a local level, using an all-pass filter and then extracts the deformation from the filter-hence the name "Local All-Pass" (LAP) algorithm. The effectiveness of this algorithm is demonstrated on a variety of synthetic and real deformations that are found in applications, such as image registration and motion estimation. In particular, when compared with a selection of image registration algorithms, the LAP obtains very accurate results for significantly reduced computation time and is very robust to noise corruption.

A Variational Model for Joint Motion Estimation and Image Reconstruction

The aim of this paper is to derive and analyze a variational model for the joint estimation of motion and reconstruction of image sequences, which is based on a time-continuous Eulerian motion model. The model can be set up in terms of the continuity equation or the brightness constancy equation. The analysis in this paper focuses on the latter for robust motion estimation on sequences of two-dimensional images. We rigorously prove the existence of a minimizer in a suitable function space setting. Moreover, we discuss the numerical solution of the model based on primal-dual algorithms and investigate several examples. Finally, the benefits of our model compared to existing techniques, such as sequential image reconstruction and motion estimation, are shown.