Image Warping Research Articles

Skuteczność ranibizumabu w idiopatycznej neowaskularyzacji naczyniówkowej – opis serii przypadków

Objective: To evaluate the efficacy of the ranibizumab intravitreal injection in idiopathic choroidal neovascularization. Material and methods: The study included 3 patients (4 eyes), aged 17, 37 and 46, with visual acuity deterioration, warping and distortion of the central image lasting for 2–5 weeks. Inflammatory background of the disease was excluded. The baseline examination and follow-up included the best corrected visual acuity (BCVA), fluorescein angiography and central retinal thickness (CRT) analysis using optical coherence tomography (OCT). When ICNV was diagnosed, intravitreal injections of ranibizumab were administered. Results: During the first follow-up test, 3 eyes showed improvement of BCVA, and in the fourth eye BCVA remained unchanged. In funduscopy, all patients showed reduced haemorrhage and oedema of the central retina. The OCT showed a reduction of the CRT and improvement in the morphology of the macula. The observation period ranged from 4 to 55 months, and the number of repeated injections during the observation ranged from 2 to 5. Conclusions: Injections of ranibizumab into the vitreous brought a significant improvement in the function of the treated eye, and in the morphology of the retina in OCT.

Variational Bayesian Approach to Multiframe Image Restoration.

Image restoration is a fundamental problem in the field of image processing. The key objective of image restoration is to recover clean images from images degraded by noise and blur. Recently, a family of new statistical techniques called variational Bayes (VB) has been introduced to image restoration, which enables us to automatically tune parameters that control restoration. While information from one image is often insufficient for high-quality restoration, however, current state-of-the-art methods of image restoration via VB approaches use only a single-degraded image to recover a clean image. In this paper, we propose a novel method of multiframe image restoration via a VB approach, which can achieve higher image quality while tuning parameters automatically. Given multiple degraded images, this method jointly estimates a clean image and other parameters, including an image warping parameter introduced for the use of multiple images, through Bayesian inference that we enable by making full use of VB techniques. Through various experiments, we demonstrate the effectiveness of our multiframe method by comparing it with single-frame one, and also show the advantages of our VB approach over non-VB approaches.

Unsupervised Deep Learning for Optical Flow Estimation

Recent work has shown that optical flow estimation can be formulated as a supervised learning problem. Moreover, convolutional networks have been successfully applied to this task. However, supervised flow learning is obfuscated by the shortage of labeled training data. As a consequence, existing methods have to turn to large synthetic datasets for easily computer generated ground truth. In this work, we explore if a deep network for flow estimation can be trained without supervision. Using image warping by the estimated flow, we devise a simple yet effective unsupervised method for learning optical flow, by directly minimizing photometric consistency. We demonstrate that a flow network can be trained from end-to-end using our unsupervised scheme. In some cases, our results come tantalizingly close to the performance of methods trained with full supervision.

A Global Approach to Fast Video Stabilization

This paper presents a novel formulation of video stabilization by directly solving for optimal image warps toward stabilized sequence. With the estimated shaky motion via long or short feature trajectories, our approach encodes another two steps, motion compensation and image warping, into a single global optimization process, rather than operating as two individual steps. This process is done only with positions of embedded mesh vertices as common variables. Spatial and temporal coherence is therein reformulated with similarity-invariant representation of motion trajectories and intra- (and inter-) frame consistency of similar transformations with respect to mesh vertices. Such a one-shot formulation converts video stabilization into a quadratic energy minimization problem defined for image warps, and thus can be efficiently resolved by using a robust solver for sparse linear systems. Experimental results demonstrate the flexibility and efficiency of our approach in producing visually plausible stabilization effects on a variety of videos.

Efficient Hybrid Image Warping for High Frame-Rate Stereoscopic Rendering.

Modern virtual reality simulations require a constant high-frame rate from the rendering engine. They may also require very low latency and stereo images. Previous rendering engines for virtual reality applications have exploited spatial and temporal coherence by using image-warping to re-use previous frames or to render a stereo pair at lower cost than running the full render pipeline twice. However these previous approaches have shown artifacts or have not scaled well with image size. We present a new image-warping algorithm that has several novel contributions: an adaptive grid generation algorithm for proxy geometry for image warping; a low-pass hole-filling algorithm to address un-occlusion; and support for transparent surfaces by efficiently ray casting transparent fragments stored in per-pixel linked lists of an A-Buffer. We evaluate our algorithm with a variety of challenging test cases. The results show that it achieves better quality image-warping than state-of-the-art techniques and that it can support transparent surfaces effectively. Finally, we show that our algorithm can achieve image warping at rates suitable for practical use in a variety of applications on modern virtual reality equipment.

Multiple-image super-resolution on mobile devices: an image warping approach

This paper discusses a super-resolution (SR) system implemented on a mobile device. We utilized an Android device’s camera to take successive shots and applied a classical multiple-image super-resolution (SR) technique that utilized a set of low-resolution (LR) images. Images taken from the mobile device are subjected to our proposed filtering scheme wherein images that have noticeable presence of blur are discarded to avoid outliers from affecting the produced high-resolution (HR) image. The remaining subset of images are subjected to non-local means denoising, then feature-matched against the first reference LR image. Successive images are then aligned with respect to the first image via affine and perspective warping transformations. The LR images are then upsampled using bicubic interpolation. An L2-norm minimization approach, which is essentially taking the pixel-wise mean of the aligned images, is performed to produce the final HR image.Our study shows that our proposed method performs better than the bicubic interpolation, which makes its implementation in a mobile device quite feasible. We have also proven in our experiments that there are substantial differences from images captured using burst mode that can be utilized by an SR algorithm to create an HR image.

Transformations Based on Continuous Piecewise-Affine Velocity Fields.

We propose novel finite-dimensional spaces of well-behaved transformations. The latter are obtained by (fast and highly-accurate) integration of continuous piecewise-affine velocity fields. The proposed method is simple yet highly expressive, effortlessly handles optional constraints (e.g., volume preservation and/or boundary conditions), and supports convenient modeling choices such as smoothing priors and coarse-to-fine analysis. Importantly, the proposed approach, partly due to its rapid likelihood evaluations and partly due to its other properties, facilitates tractable inference over rich transformation spaces, including using Markov-Chain Monte-Carlo methods. Its applications include, but are not limited to: monotonic regression (more generally, optimization over monotonic functions); modeling cumulative distribution functions or histograms; time-warping; image warping; image registration; real-time diffeomorphic image editing; data augmentation for image classifiers. Our GPU-based code is publicly available.

Improving Retinal Image Quality Using Registration with an SIFT Algorithm in Quasi-Confocal Line Scanning Ophthalmoscope.

When high-magnification images are taken with a quasi-confocal line scanning ophthalmoscope (LSO), the quality of images always suffers from Gaussian noise, and the signal to noise ratio (SNR) is very low for a safer laser illumination. In addition, motions of the retina severely affect the stabilization of the real-time video resulting in significant distortions or warped images. We describe a scale-invariant feature transform (SIFT) algorithm to automatically abstract corner points with subpixel resolution and match these points in sequential images using an affine transformation. Once n images are aligned and averaged, the noise level drops by a factor of [Formula: see text] and the image quality is improved. The improvement of image quality is independent of the acquisition method as long as the image is not warped, particularly severely during confocal scanning. Consequently, even better results can be expected by implementing this image processing technique on higher resolution images.

실시간 교통상황 모니터링 시스템을 위한 유저 친화적인 영상 변형 방법

실시간 교통상황 모니터링 시스템을 위한 유저 친화적인 영상 변형 방법

A Unified Framework for Street-View Panorama Stitching.

In this paper, we propose a unified framework to generate a pleasant and high-quality street-view panorama by stitching multiple panoramic images captured from the cameras mounted on the mobile platform. Our proposed framework is comprised of four major steps: image warping, color correction, optimal seam line detection and image blending. Since the input images are captured without a precisely common projection center from the scenes with the depth differences with respect to the cameras to different extents, such images cannot be precisely aligned in geometry. Therefore, an efficient image warping method based on the dense optical flow field is proposed to greatly suppress the influence of large geometric misalignment at first. Then, to lessen the influence of photometric inconsistencies caused by the illumination variations and different exposure settings, we propose an efficient color correction algorithm via matching extreme points of histograms to greatly decrease color differences between warped images. After that, the optimal seam lines between adjacent input images are detected via the graph cut energy minimization framework. At last, the Laplacian pyramid blending algorithm is applied to further eliminate the stitching artifacts along the optimal seam lines. Experimental results on a large set of challenging street-view panoramic images captured form the real world illustrate that the proposed system is capable of creating high-quality panoramas.

Frequency domain based super-resolution method for mixed-resolution multi-view images

Super-resolution (SR) techniques, which are based on single or multi-frame low-resolution (LR) images, have been extensively investigated in the last two decades. Mixed-resolution multiview video format plays an important role in three-dimensional television (3DTV) coding scheme. Previous work considers multiview or multi-camera images and videos at the same resolution, which performs well under the planar model without or with little projection error among the videos captured by different cameras. In recent years, several researchers have discussed the SR problem in mixed-resolution multi-view video format, where the superresolved image is created using the up-sampled version of the LR image and the high frequency components extracted from the warped image in the adjacent high-resolution (HR) views. Unfortunately, the output HR images suffer from artifacts caused by depth error. To obtain the detailed texture and edge information from the HR image as much as possible, while preserving the structure of the LR image, a novel SR reconstruction algorithm is proposed. The algorithm is composed of three components: the structure term, the detail information term, and the regularization term. The first term preserves the structure similarity of the LR image; the second term extracts detailed information from the adjacent HR image; and the last term ensures the uniqueness of the solution. Experimental results show the effectiveness and robustness of the proposed algorithm, which achieves high performance both subjectively and objectively.

InterFace: A software package for face image warping, averaging, and principal components analysis.

We describe InterFace, a software package for research in face recognition. The package supports image warping, reshaping, averaging of multiple face images, and morphing between faces. It also supports principal components analysis (PCA) of face images, along with tools for exploring the "face space" produced by PCA. The package uses a simple graphical user interface, allowing users to perform these sophisticated image manipulations without any need for programming knowledge. The program is available for download in the form of an app, which requires that users also have access to the (freely available) MATLAB Runtime environment.

Comparison of different QA methods for deformable image registration to the known errors for prostate and head-and-neck virtual phantoms

Contour comparison, landmark tracking, and image similarity are methods that can be used to perform a patient-specific evaluation for the accuracy of deformable image registration (DIR), but they are not necessarily a true measurement of DIR accuracy. The purpose of this work is to evaluate how the calculated errors from these methods compare to the known errors for prostate and head-and-neck virtual phantoms. These DIR accuracy methods were utilized to evaluate the performance of two DIR software packages, MIM and Velocity. For Velocity, three different settings of the algorithm were utilized: deformable (DEF), deformable multi-pass (DMP), and extended deformable multi-pass (XMP). For contour comparison, organs were contoured on both the non-deformed and deformed images. Then, the DIR algorithms were used to transfer contours from the non-deformed to the deformed image and compared to that drawn directly on the deformed image. In the landmark tracking method, visible landmark motion was measured and compared to that predicted by the algorithms. For image similarity, the root-mean-square and the mean-absolute differences were calculated between the warped and non-deformed images. In addition, the actual spatial registration error was calculated for each DIR algorithm from the known ground-truth deformation of the virtual phantoms. The MIM algorithm produced the lowest average errors for the landmark analysis, the closest image similarity, and overall the most accurate contour transfers. When compared to the known deformations, MIM produced the lowest average error, but also the largest errors. None of the QA methods gave any indication of this large maximum error. Beyond this, the sizes of errors from the QA methods were not always the same as the actual errors from the virtual phantom. For this reason, there is a benefit to supplementing the QA methods with virtual phantom studies.

Efficient Image Warping in Parallel for Multiview Three-Dimensional Displays

Three-dimensional (3D) display technologies make great process in recent years. View synthesis for 3D content requires the hole-filling, which is a challenging task. The increase of resolution and the number of views for view synthesis brings new challenges on memory and processing speed. A predicted hole mapping (PHM) algorithm is presented, which requires no filling priority and smoothing operation, allowing parallel computation that facilitates a real-time 3D conversion system. In experiments, the proposed PHM is evaluated and compared with some other methods in terms of peak signal to noise ratio and structural similarity index measurement, and the result shows the advantages in the numbers. The method can operate on the 32-view display with 4K × 2K resolution in real time on GPU.

Investigating deformable image registration and scatter correction for CBCT‐based dose calculation in adaptive IMPT

This work aims at investigating intensity corrected cone-beam x-ray computed tomography (CBCT) images for accurate dose calculation in adaptive intensity modulated proton therapy (IMPT) for prostate and head and neck (H&N) cancer. A deformable image registration (DIR)-based method and a scatter correction approach using the image data obtained from DIR as prior are characterized and compared on the basis of the same clinical patient cohort for the first time. Planning CT (pCT) and daily CBCT data (reconstructed images and measured projections) of four H&N and four prostate cancer patients have been considered in this study. A previously validated Morphons algorithm was used for DIR of the planning CT to the current CBCT image, yielding a so-called virtual CT (vCT). For the first time, this approach was translated from H&N to prostate cancer cases in the scope of proton therapy. The warped pCT images were also used as prior for scatter correction of the CBCT projections for both tumor sites. Single field uniform dose and IMPT (only for H&N cases) treatment plans have been generated with a research version of a commercial planning system. Dose calculations on vCT and scatter corrected CBCT (CBCTcor) were compared by means of the proton range and a gamma-index analysis. For the H&N cases, an additional diagnostic replanning CT (rpCT) acquired within three days of the CBCT served as additional reference. For the prostate patients, a comprehensive contour comparison of CBCT and vCT, using a trained physician's delineation, was performed. A high agreement of vCT and CBCTcor was found in terms of the proton range and gamma-index analysis. For all patients and indications between 95% and 100% of the proton dose profiles in beam's eye view showed a range agreement of better than 3 mm. The pass rate in a (2%,2 mm) gamma-comparison was between 96% and 100%. For H&N patients, an equivalent agreement of vCT and CBCTcor to the reference rpCT was observed. However, for the prostate cases, an insufficient accuracy of the vCT contours retrieved from DIR was found, while the CBCTcor contours showed very high agreement to the contours delineated on the raw CBCT. For H&N patients, no considerable differences of vCT and CBCTcor were found. For prostate cases, despite the high dosimetric agreement, the DIR yields incorrect contours, probably due to the more pronounced anatomical changes in the abdomen and the reduced soft-tissue contrast in the CBCT. Using the vCT as prior, these inaccuracies can be overcome and images suitable for accurate delineation and dose calculation in CBCT-based adaptive IMPT can be retrieved from scatter correction of the CBCT projections.

Generalizing the Prediction Sum of Squares Statistic and Formula, Application to Linear Fractional Image Warp and Surface Fitting

The prediction sum of squares statistic uses the principle of leave-one-out cross-validation in linear least squares regression. It is computationally attractive, as it can be computed non-iteratively. However, it has limitations: it does not handle coupled measurements, which should be held out simultaneously, and is specific to the principle of leave-one-out, which is known to overfit when used for selecting a model’s complexity. We propose multiple-exclusion PRESS (MEXPRESS), which generalizes PRESS to coupled measurements and other types of cross-validation, while retaining computational efficiency with the non-iterative MEXPRESS formula. Using MEXPRESS, various strategies to resolve overfitting can be efficiently implemented. The core principle is to exclude training data too ‘close’ or too ‘similar’ to the validation data. We show that this allows one to select the number of control points automatically in three cases: (i) the estimation of linear fractional warps for dense image registration from point correspondences, (ii) surface reconstruction from a dense depth-map obtained by a depth sensor and (iii) surface reconstruction from a sparse point cloud obtained by shape-from-template.

Real-time virtual view synthesis using light field

Virtual view synthesis technique renders a virtual view image from several pre-collected viewpoint images. The hotspot on virtual view synthesis area is depth image-based rendering (DIBR), which has low one-time imaging quality. To achieve high imaging quality artifacts, the holes must be inpainted after image warping which means high computational complexity. This paper proposed a real-time virtual view synthesis method from light field. Then the light field is transformed into frequency domain. The light field is parameterized and reconstructed from image array. The virtual view is rendered by resampling the light field in frequency domain. After resampling the image by performing Fourier slice, the virtual view image is obtained by inverse Fourier transform. Experiments show that our method can get high one-time imaging quality in real time.

EyeOpener

Closed eyes and look-aways can ruin precious moments captured in photographs. In this article, we present a new framework for automatically editing eyes in photographs. We leverage a user’s personal photo collection to find a “good” set of reference eyes and transfer them onto a target image. Our example-based editing approach is robust and effective for realistic image editing. A fully automatic pipeline for realistic eye editing is challenging due to the unconstrained conditions under which the face appears in a typical photo collection. We use crowd-sourced human evaluations to understand the aspects of the target-reference image pair that will produce the most realistic results. We subsequently train a model that automatically selects the top-ranked reference candidate(s) by narrowing the gap in terms of pose, local contrast, lighting conditions, and even expressions. Finally, we develop a comprehensive pipeline of three-dimensional face estimation, image warping, relighting, image harmonization, automatic segmentation, and image compositing in order to achieve highly believable results. We evaluate the performance of our method via quantitative and crowd-sourced experiments.

Iterative Closest Conformal Maps between Planar Domains

AbstractConformal maps between planar domains are an important tool in geometry processing, used for shape deformation and image warping. The Riemann mapping theorem guarantees that there exists a conformal map between any two simply connected planar domains, yet computing this map efficiently remains challenging. In practice, one of the main algorithmic questions is the correspondence between the boundaries of the domains. On the one hand, there exist a number of conformal maps between any two domains, thus many potential boundary correspondences, yet on the other, given full boundary prescription a conformal map might not exist. Furthermore, an approximate boundary fitting can be enough for many applications. We therefore propose an alternating minimization algorithm for finding a boundary‐approximating conformal map given only an initial global alignment of the two input domains. We utilize the Cauchy‐Green complex barycentric coordinates to parameterize the space of conformal maps from the source domain, and thus compute a continuous map without requiring the discretization of the domain, and without mapping to intermediate domains. This yields a very efficient method which allows to interactively modify additional user‐provided constraints, such as point‐to‐point and stroke‐to‐stroke correspondences. Furthermore, we show how to easily generalize this setup to quasi‐conformal maps, thus enriching the space of mappings and reducing the area distortion. We compare our algorithm to state‐of‐the‐art methods for mapping between planar domains, and demonstrate that we achieve less distorted maps on the same inputs. Finally, we show applications of our approach to stroke based deformation and constrained texture mapping.

고해상도 컬러 영상 워핑의 실시간 구현을 위한 영상 캐시 알고리즘

This paper presents a new image cache algorithm for real-time implementation of high-resolution color image warping. The cache memory is divided into four cache memory modules for simultaneous readout of four input image pixels in consideration of the color filter array (CFA) pattern of an image sensor and CFA image warping. In addition, a pipeline structure from the cache memory to an interpolator is shown to guarantee the generation of an output image pixel at each system clock cycle. The proposed image cache algorithm is applied to an FPGA-based real-time color image warping, and experimental results are presented to show the validity of the proposed method.