Example-based Image Research Articles

Leveraging Feature Extraction and Context Information for Image Relighting

Example-based image relighting aims to relight an input image to follow the lighting settings of another target example image. Deep learning-based methods for such tasks have become highly popular. However, they are often limited by the geometric priors or suffer from shadow reconstruction and lack of texture details. In this paper, we propose an image-to-image translation network called DGATRN to tackle this problem by enhancing feature extraction and unveiling context information to achieve visually plausible example-based image relighting. Specifically, the proposed DGATRN consists of a scene extraction, a shadow calibration, and a rendering network, and our key contribution lies in the first two networks. We propose an up- and downsampling approach to improve the feature extraction capability to capture scene and texture details better. We also introduce a feature attention downsampling block and a knowledge transfer to utilize the attention impact and underlying knowledge connection between scene and shadow. Experiments were conducted to evaluate the usefulness and effectiveness of the proposed method.

Example-based image super-resolution via blur kernel estimation and variational reconstruction

Single image super-resolution aims at generating clear high-resolution image from one low-resolution image. Due to the limited low-resolution information, it is a challenging task to restore clear, artifacts-free image, meanwhile preserving finer structures and textures. This paper proposes an effective example-based image super-resolution method while making clear image and no compromise on quality. Firstly, the image prior is imposed on the anchor neighborhood regression model to optimize mapping coefficient for interim latent image construction. In order to remove its blur, kernel estimation iteration optimization algorithm is proposed based on the salient edges which are extracted through texture-structure discriminate minimum energy function and fractional order mask enhancement. Finally, an accurate reconstruction constraint combined with a simple gradient regularization is applied to reconstruct the super-resolution image. The proposed method is able to produce clear high-frequency texture details and maintain clean edges even under large scaling factors. Experimental results show that the proposed method performs well in visual effects and similarities. Furthermore, we test our algorithm in multi-texture images for robust evaluation. It is demonstrated that our algorithm is robust under complicated textures condition.

Example-based turbulence style transfer

Generating realistic fluid simulations remains computationally expensive, and animators can expend enormous effort trying to achieve a desired motion. To reduce such costs, several methods have been developed in which high-resolution turbulence is synthesized as a post process. Since global motion can then be obtained using a fast, low-resolution simulation, less effort is needed to create a realistic animation with the desired behavior. While much research has focused on accelerating the low-resolution simulation, the problem controlling the behavior of the turbulent, high-resolution motion has received little attention. In this paper, we show that style transfer methods from image editing can be adapted to transfer the turbulent style of an existing fluid simulation onto a new one. We do this by extending example-based image synthesis methods to handle velocity fields using a combination of patch-based and optimization-based texture synthesis. This approach allows us to take into account the incompressibility condition, which we have found to be a important factor during synthesis. Using our method, a user can easily and intuitively create high-resolution fluid animations that have a desired turbulent motion.

Image super-resolution via two stage coupled dictionary learning

Example-based image super-resolution aims to establish a learning model for generating the high resolution image from the coupled training pairs, which is an active study area of mobile media in the recent modern communication. In this paper, we proposed a novel example-based method to address the single image super-resolution problem, where the training pairs are selected from a large amount of natural images. The main idea of our method is to reconstruct the high resolution by a two stage-based scheme. In the first stage, one dictionary is learned to represent the coarse high resolution image from its low version, and the other is trained within the same coding as the coarse image to recover texture details. And then, to further enhance the fine edges in image, a similar dictionary learning scenario are done about the synthesis high resolution image and its fine structure in residual component. Extensive experimental results on some benchmark test images show the advantage of our method compare with other excellent ones.

Example-Based Image Synthesis via Randomized Patch-Matching.

Image and texture synthesis is a challenging task that has long been drawing attention in the fields of image processing, graphics, and machine learning. This problem consists of modeling the desired type of images, either through training examples or via a parametric modeling, and then generating images that belong to the same statistical origin. This paper addresses the image synthesis task, focusing on two specific families of images-handwritten digits and face images. This paper offers two main contributions. First, we suggest a simple and intuitive algorithm capable of generating such images in a unified way. The proposed approach taken is pyramidal, consisting of upscaling and refining the estimated image several times. For each upscaling stage, the algorithm randomly draws small patches from a patch database and merges these to form a coherent and novel image with high visual quality. The second contribution is a general framework for the evaluation of the generation performance, which combines three aspects: the likelihood, the originality, and the spread of the synthesized images. We assess the proposed synthesis scheme and show that the results are similar in nature, and yet different from the ones found in the training set, suggesting that true synthesis effect has been obtained.

Example-Based Image Colorization Using Locality Consistent Sparse Representation.

Image colorization aims to produce a natural looking color image from a given gray-scale image, which remains a challenging problem. In this paper, we propose a novel example-based image colorization method exploiting a new locality consistent sparse representation. Given a single reference color image, our method automatically colorizes the target gray-scale image by sparse pursuit. For efficiency and robustness, our method operates at the superpixel level. We extract low-level intensity features, mid-level texture features, and high-level semantic features for each superpixel, which are then concatenated to form its descriptor. The collection of feature vectors for all the superpixels from the reference image composes the dictionary. We formulate colorization of target superpixels as a dictionary-based sparse reconstruction problem. Inspired by the observation that superpixels with similar spatial location and/or feature representation are likely to match spatially close regions from the reference image, we further introduce a locality promoting regularization term into the energy formulation, which substantially improves the matching consistency and subsequent colorization results. Target superpixels are colorized based on the chrominance information from the dominant reference superpixels. Finally, to further improve coherence while preserving sharpness, we develop a new edge-preserving filter for chrominance channels with the guidance from the target gray-scale image. To the best of our knowledge, this is the first work on sparse pursuit image colorization from single reference images. Experimental results demonstrate that our colorization method outperforms the state-of-the-art methods, both visually and quantitatively using a user study.

Example-based image colorization via automatic feature selection and fusion

Image colorization is an important and difficult problem in image processing with various applications including image stylization and heritage restoration. Most existing image colorization methods utilize feature matching between the reference color image and the target grayscale image. The effectiveness of features is often significantly affected by the characteristics of the local image region. Traditional methods usually combine multiple features to improve the matching performance. However, the same set of features is still applied to the whole images. In this paper, based on the observation that local regions have different characteristics and hence different features may work more effectively, we propose a novel image colorization method using automatic feature selection with the results fused via a Markov Random Field (MRF) model for improved consistency. More specifically, the proposed algorithm automatically classifies image regions as either uniform or non-uniform, and selects a suitable feature vector for each local patch of the target image to determine the colorization results. For this purpose, a descriptor based on luminance deviation is used to estimate the probability of each patch being uniform or non-uniform, and the same descriptor is also used for calculating the label cost of the MRF model to determine which feature vector should be selected for each patch. In addition, the similarity between the luminance of the neighborhood is used as the smoothness cost for the MRF model which enhances the local consistency of the colorization results. Experimental results on a variety of images show that our method outperforms several state-of-the-art algorithms, both visually and quantitatively using standard measures and a user study.

Iterative projection reconstruction for fast and efficient image upsampling

With the development of ultra-high-resolution display devices, the visual perception of fine texture details is becoming increasingly important. Traditional image upsampling methods suffer from either loss of high-frequency texture details or very high time cost. In this paper, we propose an iterative projection reconstruction (IPR) method for fast and efficient image upsampling. The proposed method refines high-frequency texture details with an iterative projection process, and utilizes the pre-computed projection matrix to accelerate the example-based image reconstruction. As a result, the proposed method can reproduce fine texture details with low time cost. Experimental results demonstrate that the proposed method outperforms some state-of-the-art methods.

Non-dyadic fisheye lens correction model for image enhancement.

This paper presents a non-dyadic framework to improve example-based enhancement of radially distorted images acquired by a very wide-angle lens. In order to remove both jagging and blurring artifacts in the correction process of the fisheye lens' barrel distortion, the proposed method first performs non-dyadic or multiple-step geometric correction based on the parabolic equation-based lens distortion model. At each correction step, an example-based image enhancement method removes undesired geometric correction artifacts such as jagging and blurring. Experimental results demonstrate that the proposed method outperforms existing fisheye lens image enhancement methods in the sense of both subjective and objective measures. Based on both theoretical advancement and experimental results, the proposed method can be used for various wide-view imaging applications including vehicle front- and rear-view cameras and wide-angle video surveillance systems.

Image super-resolution by textural context constrained visual vocabulary

Example-based super-resolution (SR) approach hallucinates the missing high-resolution (HR) details by learning the example image patches. This approach implicitly assumes that the similarity of the low-resolution (LR) patches can infer the similarity of the corresponding HR patches. However, this similarity preserving assumption may not be held in practice. Thus the example-based super-resolved image inevitably contains artifacts not close to the ground truth. In this paper, we propose a novel single-image SR method by integrating an enforced similarity preserving process by using visual vocabulary into example-based SR approach. By jointly learning the HR and LR visual vocabularies, we can obtain a geometric co-occurrence prior to make the geometric similarity preserved within each visual word. We further propose a two-step framework for SR. The first step estimates the optimum visual word using textural context cue while the second step enforces the visual word subspace constraint and reconstruction constraint for estimating the final result. Experiments demonstrate the effectiveness of our method for recovering the missing HR details, especially texture.

Example-Based Image Manipulation

Example-Based Image Manipulation

Example-based image color and tone style enhancement

Color and tone adjustments are among the most frequent image enhancement operations. We define a color and tone style as a set of explicit or implicit rules governing color and tone adjustments. Our goal in this paper is to learn implicit color and tone adjustment rules from examples. That is, given a set of examples, each of which is a pair of corresponding images before and after adjustments, we would like to discover the underlying mathematical relationships optimally connecting the color and tone of corresponding pixels in all image pairs. We formally define tone and color adjustment rules as mappings, and propose to approximate complicated spatially varying nonlinear mappings in a piecewise manner. The reason behind this is that a very complicated mapping can still be locally approximated with a low-order polynomial model. Parameters within such low-order models are trained using data extracted from example image pairs. We successfully apply our framework in two scenarios, low-quality photo enhancement by transferring the style of a high-end camera, and photo enhancement using styles learned from photographers and designers.

Partially Supervised Neighbor Embedding for Example-Based Image Super-Resolution

Neighbor embedding algorithm has been widely used in example-based super-resolution reconstruction from a single frame, which makes the assumption that neighbor patches embedded are contained in a single manifold. However, it is not always true for complicated texture structure. In this paper, we believe that textures may be contained in multiple manifolds, corresponding to classes. Under this assumption, we present a novel example-based image super-resolution reconstruction algorithm with clustering and supervised neighbor embedding (CSNE). First, a class predictor for low-resolution (LR) patches is learnt by an unsupervised Gaussian mixture model. Then by utilizing class label information of each patch, a supervised neighbor embedding is used to estimate high-resolution (HR) patches corresponding to LR patches. The experimental results show that the proposed method can achieve a better recovery of LR comparing with other simple schemes using neighbor embedding.

Building and using a database of one trillion natural-image patches.

Many example-based image processing algorithms operate on image patches (texture synthesis, resolution enhancement, image denoising, and so on). However, inaccessibility to a large, varied collection of image patches has hindered widespread adoption of these methods. The authors describe the construction of a database of one trillion image patches and demonstrate its research utility.

Feature Dimensionality Reduction for Example-Based Image Super-Resolution

Close The Journal of Pattern Recognition Research (JPRR) provides an international forum for the electronic publication of high-quality research and industrial experience articles in all areas of pattern recognition, machine learning, and artificial intelligence. JPRR is committed to rigorous yet rapid reviewing. Final versions are published electronically (ISSN 1558-884X) immediately upon acceptance. Submit Manuscript Feature Dimensionality Reduction for Example-Based Image Super-Resolution Liangjun Xie, Dalong Li, Steven J. Simske JPRR Vol 6, No 2 (2011); doi:10.13176/11.323

統計的リーチ特徴法に基づくサンプル学習型画像照合

統計的リーチ特徴法に基づくサンプル学習型画像照合

Resolution enhancement based on learning the sparse association of image patches

Example-based image super-resolution techniques model the co-occurrence patterns between the middle and high frequency layers of example images to estimate the missing high frequency component for low resolution input. However, many existing approaches seek to estimate the optimal solution within a small set of candidates by using empirical criteria. Hence their representational performance is limited by the quality of the candidate set, and the generated super-resolution image is unstable, with noticeable artifacts. In this paper, we propose a novel image super-resolution method based on learning the sparse association between input image patches and the example image patches. We improve an existing sparse-coding algorithm to find sparse association between image patches. We also propose an iterative training strategy to learn a redundancy reduced basis set to speed up the super-resolution process. Comparing to existing example-based approaches, the proposed method significantly improves image quality, and the produced super-resolution images are sharp and natural, with no obvious artifact.

Example-based image super-resolution with class-specific predictors

Example-based super-resolution is a promising approach to solving the image super-resolution problem. However, the learning process can be slow and prediction can be inaccurate. In this paper, we present a novel learning-based algorithm for image super-resolution to improve the computational speed and prediction accuracy. Our new method classifies image patches into several classes, for each class, a class-specific predictor is designed. A class-specific predictor takes a low-resolution image patch as input and predicts a corresponding high-resolution patch as output. The performances of the class-specific predictors are evaluated using different datasets formed by face images and natural-scene images. We present experimental results which demonstrate that the new method provides improved performances over existing methods.

Superresolution imaging: Theory, Algorithms and Applications

The International Conference on Superresolution Imaging was held from 29 August to 31 August, 2005 at the University of Hong Kong. The conference had two major objectives: (1) to improve the dialogue and collaboration between mathematicians, engineers and computer scientists working in superresolution imaging, and (2) to stimulate new theoretical research and emerging applications in superresolution imaging. This proceeding contain six papers from invited speakers of the conference. The contributions cover different aspects in the superresolution imaging research. The first article, “Region-based Super-resolution for Compression”, addresses the problem of synthesizing a high-resolution image from a compressed low-resolution video sequence using region-based information. The second article, “Super-resolution Reconstruction in a Computational Compound-eye Imaging System”, discusses a novel and effective multiple-design parameter approach for high-resolution image reconstruction. The third article, “Example-based Single Image Super-resolution: A Global MAP Approach with Outlier Rejection” proposes an efficient scheme for using image examples as driving a powerful regularization scheme for superresolution imaging. The fourth article, “Super-resolution: Should We Process Locally or Globally?” studies the usefulness of different local and global learning-based, single-frame image super-resolution reconstruction techniques in handling deblurring, denoising and alias removal. The fifth article, “SuperResolutionReconstruction based onWavelet Estimation”, uses linear interpolation to build up an image reconstruction algorithm to obtain the relationship between the detail coefficients in wavelet subbands and the set of low-resolution images. The sixth article, “An Efficient Algorithm for Superresolution in Medium Field Imaging”, applies the preconditioned conjugate

Completion-based texture design using deformation

In this paper, we present a novel approach for designing a variety of large textures from a single small sample texture. Firstly, the original small texture is segmented into layers, each of which contains one particular texture element. Secondly, each layer is deformed using a set of chaotic-based transformation operations. Thirdly, all the deformed layers are added together to form a new texture, which is a natural variation of the original sample texture. Since each layer is deformed independently, adding the deformed layers together usually results in a texture with overlapping regions and holes. We employ the graphcut algorithm and an example-based image inpainting technique to seamlessly patch the overlapping regions and to fill the holes. Moreover, an optimized graphcut synthesis algorithm and a new cyclic texture synthesis technique are also developed for efficiently creating large seamless textures. As a result, our approach shows particular strength in generating a large variety of textures from a single sample texture while avoiding highly repetitive patterns. Our experiments demonstrate that the proposed technique can also be used for other texture synthesis applications, such as texture synthesis from multiple samples.