Image Generation Model Research Articles

Semi-Analytical Model of Charge Image Formation in Electrophotography

High resolution latent image formation is an important step in electrophotography to achieve high quality prints. Presently, analytical models exist for the other important steps: photoconductor charging, development (both single and dual component), and transfer. We present here an analytical model of charge image generation on the surface of an organic photoconductor due to laser exposure. The inputs to the model include laser intensity profile as a function of time, quantum efficiency of the charge generation layer, hole mobility of the charge transport layer, and initial charge density on the surface of the PC. The output of the model is the PC surface charge map as it evolves in time, which is a key input to the toner development process in electrophotography.

Super-Resolution Reconstruction of Diffusion-Weighted Images from Distortion Compensated Orthogonal Anisotropic Acquisitions.

Increasing the spatial resolution in diffusion-weighted imaging (DWI) is challenging with a single-shot EPI acquisition because of the decreased SNR and T2* relaxation. Recently, acquisition of orthogonal anisotropic acquisitions and super-resolution reconstruction (SRR) of the underlying high-resolution image has been proposed to achieve higher resolution. Promising results have been shown with simulations. However, practical evidence that SRR enables resolution enhancement remains unclear. Particularly, real DWI scans acquired in orthogonal directions are subject to very different distortion. This makes the precise alignment of the images impossible and strongly perturbs the reconstruction. In this work we propose to combine distortion compensation and SRR. Distortion compensation is achieved by acquisition of a dual echo field map, providing an estimate of the field inhomogeneity. The SRR is formulated as a maximum a posteriori problem and relies on a realistic image generation model. We evaluate our approach with real anisotropic acquisitions. Importantly, we demonstrate that combining distortion compensation and SRR provides better results than acquisition of a single isotropic scan for the same acquisition duration time. The SRR provides more detailed structures and better tractography results. This work provides the first evidence that SRR, which employs conventional SS-EPI techniques, may enable resolution enhancement in DWI, and may dramatically impact the way to achieve DW imaging in both neuroscience and clinical applications.

A Sparse Coding Model with Synaptically Local Plasticity and Spiking Neurons Can Account for the Diverse Shapes of V1 Simple Cell Receptive Fields

Sparse coding algorithms trained on natural images can accurately predict the features that excite visual cortical neurons, but it is not known whether such codes can be learned using biologically realistic plasticity rules. We have developed a biophysically motivated spiking network, relying solely on synaptically local information, that can predict the full diversity of V1 simple cell receptive field shapes when trained on natural images. This represents the first demonstration that sparse coding principles, operating within the constraints imposed by cortical architecture, can successfully reproduce these receptive fields. We further prove, mathematically, that sparseness and decorrelation are the key ingredients that allow for synaptically local plasticity rules to optimize a cooperative, linear generative image model formed by the neural representation. Finally, we discuss several interesting emergent properties of our network, with the intent of bridging the gap between theoretical and experimental studies of visual cortex.

The statistics of contour fragments in natural scenes

AbstractSummary: Recent electrophysiology recordings in macaque V4/IT suggest that single neuron response to synthetic closed contours can be largely captured by models which only consider a small number of contour fragments (Brincat and Connor 2004). Motivated by this experimental work, we sought firstly to characterize the statistics of contour fragments in natural scenes, and secondly to generate synthetic images which reflect the measured contour-fragment statistics.To detect contour fragments, we defined a set of feature detectors which respond only in the presence of two edges co-occurring at a fixed relative angle – implemented as a logical ‘AND’ of two Gabor-like, laplacian-of-gaussian linear filters. We then determined the pairwise correlations of these contour fragments in a natural image ensemble. If efficient coding extends to higher cortical centers and processing in the ventral visual stream can be modeled as a sequence of logical operations on linear shape features, then the pairwise statistics we measure should be informative about neural shape coding.Using these statistics directly, it is possible to produce a generative model of simple images which contain the measured statistics. We implemented a modified Ising model and solved the inverse problem of determining the optimal model parameters which satisfy the measured correlations. The resulting Ising-like model of the pairwise statistics can generate the probability of any arrangement of contour fragments as measured in the natural image ensemble.As a complementary approach to producing images with naturalistic contour fragment statistics, it is possible to start with a natural scene and isolate the target features. This is achieved by applying our contour fragment detection processing to the single scene and then separately visualizing the fragments detected. This second procedure lends itself to parametric randomization of the generated image.Narrative Elaboration: The central question guiding our study is how shapes are represented in inferotemporal cortex. To that end, we have investigated natural images in order to motivate experiments capable of targeting the extent to which neural processing of shapes involves representing shapes as combinations of key contour features. To simplify, we are focusing on black-and-white images and prioritizing contour features. This project suggests it is possible to generate synthetic images containing only a select set of contour statistics. Our subsequent goals include conducting collaborative macaque electrophysiology experiments with our generated images as visual stimuli.

A Semitransparency-Based Optical-Flow Method With a Point Trajectory Model for Particle-Like Video

This paper proposes a new semitransparency-based optical-flow model with a point trajectory (PT) model for particle-like video. Previous optical-flow models have used ranging from image brightness constancy to image brightness change models as constraints. However, two important issues remain unsolved. The first is how to track/match a semitransparent object with a very large displacement between frames. Such moving objects with different shapes and sizes in an outdoor scene move against a complicated background. Second, due to semitransparency, the image intensity between frames can also violate a previous image brightness-based optical-flow model. Thus, we propose a two-step optimization for the optical-flow estimation model for a moving semitransparent object, i.e., particle. In the first step, a rough optical flow between particles is estimated by a new alpha constancy constraint that is based on an image generation model of semitransparency. In the second step, the optical flow of a particle with a continuous trajectory in a definite temporal interval based on a PT model can be refined. Many experiments using various falling-snow and foggy scenes with multiple moving vehicles show the significant improvement of the optical flow compared with a previous optical-flow model.

A Helmholtz Principle Approach to Parameter Free Change Detection and Coherent Motion Using Exchangeable Random Variables

A parameter free technique for finding changes between images is discussed. The technique uses the Helmholtz principle to find changes in an input image in situations where only one or two images are available to build a background model. The Helmholtz principle locates image regions that are unlikely to occur due to an a priori random image generation model, and it assigns a confidence level to each changed region. All previous algorithms based on the Helmholtz principle assumed that the image was generated using independent and identically distributed (IID) random variables. The IID assumption is replaced with the weaker exchangeable assumption, and a simple exchangeable random variable model based on the hypergeometric distribution is investigated. Furthermore, practical calculation methods are discussed. The calculation methods require new nonasymptotic bounds to the hypergeometric distribution, and a major contribution of this paper is the novel proof of this bound and its relationship to large deviation theory. The calculations also incorporate the fast level set transform tree structure of the image to create shape boundaries of changed regions. The algorithm is applied to the problems of change detection and coherent motion. We also illustrate that the exchangeable random variable model yields more consistent results than an equivalent IID random variable model.

Learning a Generative Model of Images by Factoring Appearance and Shape

Computer vision has grown tremendously in the past two decades. Despite all efforts, existing attempts at matching parts of the human visual system's extraordinary ability to understand visual scenes lack either scope or power. By combining the advantages of general low-level generative models and powerful layer-based and hierarchical models, this work aims at being a first step toward richer, more flexible models of images. After comparing various types of restricted Boltzmann machines (RBMs) able to model continuous-valued data, we introduce our basic model, the masked RBM, which explicitly models occlusion boundaries in image patches by factoring the appearance of any patch region from its shape. We then propose a generative model of larger images using a field of such RBMs. Finally, we discuss how masked RBMs could be stacked to form a deep model able to generate more complicated structures and suitable for various tasks such as segmentation or object recognition.

Assessing the impact of spectral and polarimetric data fusion via simulation to support multimodal sensor system design requirements

A series of trade studies was carried out using the Digital Imaging and Remote Sensing Image Generation (DIRSIG) model to assess how varying the spectral signal-to-noise ratio (SNR), spectral ground sample distance (GSD), or target spectrum affected the impact of spectral and polarimetric data fusion via the spectral polarimetric integration (SPI) algorithm for a notional multimodal sensor. When varying the SNR, the impact depended on the constraints placed on the sensor's tasking. When spectral GSD was varied, the benefit of incorporating polarimetric information increased as the GSD increased. However, a threshold GSD was identified beyond which no benefit was observed. Reducing the target/background spectral contrast by changing the target spectrum from a red vehicle to a green vehicle produced variations in the impact due to fusion, although the SPI algorithm produced a general increase in performance in both cases. The trade studies demonstrated that incorporating additional polarimetric information may enable suitable performance with a less capable multispectral sensor. Finally, the SPI decision fusion algorithm was shown to be robust across a range of scenarios possibly encountered in the multimodal sensor design process.

Defining a process to fuse polarimetric and spectral data for target detection and explore the trade space via simulation

A process is developed to assess the effect of fusing polarimetric and spectral sensing modalities for an urban target detection scenario through simulation with the Digital Imaging and Remote Sensing Image Generation (DIRSIG) model. Two novel multimodal fusion algorithms are proposed--one for the pixel level, and another for the decision level. A synthetic urban scene is validated to ensure the presence of enough background clutter. The signal-to-clutter ratios (SCR) of both the simulated spectral and polarimetric data are calculated, and the synthetic spectral SCR is compared to data collected with the Compact Airborne Spectral Sensor (COMPASS). A qualitative examination of the polarimetric background clutter level is also described. The fusion algorithms' performances are evaluated at 355 different sun-target-sensor viewing geometries, and a method to quantify the increase in performance is described. Tasking conditions where target detection performance is enhanced are identified and the decision fusion algorithm is shown to outperform the pixel fusion algorithm. The utility of polarimetric information is shown to vary with the sun-target-sensor geometry, but data fusion consistently enhances spectral target detection performance when the sensor is located in the sun's specular reflection lobe.

An image formation model for Secondary Ion Mass Spectrometry imaging of biological tissue samples

Secondary Ion Mass Spectrometry (SIMS) can provide distribution images of elements and molecular fragments with high sensitivity and spatial resolution. This study aims to exploit the potential of this modality as an imaging technique for biomedical applications. A model of image generation was developed and validated on experimental SIMS images. The model allowed for the selection of standard distance deviation (SDD) and nearest neighbor index (NNI) as suitable indices for the characterization of SIMS images, as they have been associated with sample morphology. Two regression models were proposed to correlate the SDD index and NNI with an index of effectiveness and acquisition parameters. The SDD index, due to its linear relationship with the image noise parameter, was less sensitive to noise. The model was then applied to study the effect of instrumental and analytical parameters, such as pre-sputtering time, on image generation.

Learning to represent visual input

One of the central problems in computational neuroscience is to understand how the object-recognition pathway of the cortex learns a deep hierarchy of nonlinear feature detectors. Recent progress in machine learning shows that it is possible to learn deep hierarchies without requiring any labelled data. The feature detectors are learned one layer at a time and the goal of the learning procedure is to form a good generative model of images, not to predict the class of each image. The learning procedure only requires the pairwise correlations between the activations of neuron-like processing units in adjacent layers. The original version of the learning procedure is derived from a quadratic ‘energy’ function but it can be extended to allow third-order, multiplicative interactions in which neurons gate the pairwise interactions between other neurons. A technique for factoring the third-order interactions leads to a learning module that again has a simple learning rule based on pairwise correlations. This module looks remarkably like modules that have been proposed by both biologists trying to explain the responses of neurons and engineers trying to create systems that can recognize objects.

Fusion of multiple image types for the creation of radiometrically-accurate synthetic scenes

The Digital Imaging and Remote Sensing Image Generation (DIRSIG) model is an established, first-principles based scene simulation tool that produces synthetic multi-spectral and hyperspectral images from the visible to long wave infrared (0.4 to 20 microns). Over the last few years, significant enhancements such as spectral polarimetric and active Light Detection and Ranging (lidar) models have also been incorporated into the software, providing an extremely powerful tool for algorithm testing and sensor evaluation. However, the extensive time required to create large-scale scenes has limited DIRSIG's ability to generate scenes "on demand." To date, scene generation has been a laborious, time-intensive process, as the terrain model, CAD objects and background maps have to be created and attributed manually. To shorten the time required for this process, we have developed a comprehensive workflow aimed at reducing the man-in-the-loop requirements for many aspects of synthetic hyperspectral scene construction. Through a fusion of 3D lidar data with passive imagery, we have been able to partially-automate many of the required tasks in the creation of high-resolution urban DIRSIG scenes. This paper presents a description of these techniques.

Encoding Probabilistic Brain Atlases Using Bayesian Inference

This paper addresses the problem of creating probabilistic brain atlases from manually labeled training data. Probabilistic atlases are typically constructed by counting the relative frequency of occurrence of labels in corresponding locations across the training images. However, such an "averaging" approach generalizes poorly to unseen cases when the number of training images is limited, and provides no principled way of aligning the training datasets using deformable registration. In this paper, we generalize the generative image model implicitly underlying standard "average" atlases, using mesh-based representations endowed with an explicit deformation model. Bayesian inference is used to infer the optimal model parameters from the training data, leading to a simultaneous group-wise registration and atlas estimation scheme that encompasses standard averaging as a special case. We also use Bayesian inference to compare alternative atlas models in light of the training data, and show how this leads to a data compression problem that is intuitive to interpret and computationally feasible. Using this technique, we automatically determine the optimal amount of spatial blurring, the best deformation field flexibility, and the most compact mesh representation. We demonstrate, using 2-D training datasets, that the resulting models are better at capturing the structure in the training data than conventional probabilistic atlases. We also present experiments of the proposed atlas construction technique in 3-D, and show the resulting atlases' potential in fully-automated, pulse sequence-adaptive segmentation of 36 neuroanatomical structures in brain MRI scans.

Learning the Lie Groups of Visual Invariance

A fundamental problem in biological and machine vision is visual invariance: How are objects perceived to be the same despite transformations such as translations, rotations, and scaling? In this letter, we describe a new, unsupervised approach to learning invariances based on Lie group theory. Unlike traditional approaches that sacrifice information about transformations to achieve invariance, the Lie group approach explicitly models the effects of transformations in images. As a result, estimates of transformations are available for other purposes, such as pose estimation and visuomotor control. Previous approaches based on first-order Taylor series expansions of images can be regarded as special cases of the Lie group approach, which utilizes a matrix-exponential-based generative model of images and can handle arbitrarily large transformations. We present an unsupervised expectation-maximization algorithm for learning Lie transformation operators directly from image data containing examples of transformations. Our experimental results show that the Lie operators learned by the algorithm from an artificial data set containing six types of affine transformations closely match the analytically predicted affine operators. We then demonstrate that the algorithm can also recover novel transformation operators from natural image sequences. We conclude by showing that the learned operators can be used to both generate and estimate transformations in images, thereby providing a basis for achieving visual invariance.

Primal sketch: Integrating structure and texture

This article proposes a generative image model, which is called “primal sketch,” following Marr’s insight and terminology. This model combines two prominent classes of generative models, namely, sparse coding model and Markov random field model, for representing geometric structures and stochastic textures, respectively. Specifically, the image lattice is divided into structure domain and texture domain. The sparse coding model is used to represent image intensities on the structure domain, where edge and ridge segments are modeled by image coding functions with explicit geometric and photometric parameters. The edge and ridge segments form a sketch graph whose nodes are corners and junctions. The sketch graph is governed by a simple spatial prior model. The Markov random field model is used to summarize image intensities on the texture domain, where the texture patterns are characterized by feature statistics in the form of marginal histograms of responses from a set of linear filters. The Markov random fields in-paint the texture domain while interpolating the structure domain seamlessly. A sketch pursuit algorithm is proposed for model fitting. A number of experiments on real images are shown to demonstrate the model and the algorithm.

Data driven image models through continuous joint alignment

This paper presents a family of techniques that we call congealing for modeling image classes from data. The idea is to start with a set of images and make them appear as similar as possible by removing variability along the known axes of variation. This technique can be used to eliminate "nuisance" variables such as affine deformations from handwritten digits or unwanted bias fields from magnetic resonance images. In addition to separating and modeling the latent images-i.e., the images without the nuisance variables-we can model the nuisance variables themselves, leading to factorized generative image models. When nuisance variable distributions are shared between classes, one can share the knowledge learned in one task with another task, leading to efficient learning. We demonstrate this process by building a handwritten digit classifier from just a single example of each class. In addition to applications in handwritten character recognition, we describe in detail the application of bias removal from magnetic resonance images. Unlike previous methods, we use a separate, nonparametric model for the intensity values at each pixel. This allows us to leverage the data from the MR images of different patients to remove bias from each other. Only very weak assumptions are made about the distributions of intensity values in the images. In addition to the digit and MR applications, we discuss a number of other uses of congealing and describe experiments about the robustness and consistency of the method.

What are Textons?

Textons refer to fundamental micro-structures in natural images (and videos) and are considered as the atoms of pre-attentive human visual perception (Julesz, 1981). Unfortunately, the word "texton" remains a vague concept in the literature for lack of a good mathematical model. In this article, we first present a three-level generative image model for learning textons from texture images. In this model, an image is a superposition of a number of image bases selected from an over-complete dictionary including various Gabor and Laplacian of Gaussian functions at various locations, scales, and orientations. These image bases are, in turn, generated by a smaller number of texton elements, selected from a dictionary of textons. By analogy to the waveform-phoneme-word hierarchy in speech, the pixel-base-texton hierarchy presents an increasingly abstract visual description and leads to dimension reduction and variable decoupling. By fitting the generative model to observed images, we can learn the texton dictionary as parameters of the generative model. Then the paper proceeds to study the geometric, dynamic, and photometric structures of the texton representation by further extending the generative model to account for motion and illumination variations. (1) For the geometric structures, a texton consists of a number of image bases with deformable spatial configurations. The geometric structures are learned from static texture images. (2) For the dynamic structures, the motion of a texton is characterized by a Markov chain model in time which sometimes can switch geometric configurations during the movement. We call the moving textons as "motons". The dynamic models are learned using the trajectories of the textons inferred from video sequence. (3) For photometric structures, a texton represents the set of images of a 3D surface element under varying illuminations and is called a "lighton" in this paper. We adopt an illumination-cone representation where a lighton is a texton triplet. For a given light source, a lighton image is generated as a linear sum of the three texton bases. We present a sequence of experiments for learning the geometric, dynamic, and photometric structures from images and videos, and we also present some comparison studies with K-mean clustering, sparse coding, independent component analysis, and transformed component analysis. We shall discuss how general textons can be learned from generic natural images.

Image Parsing: Unifying Segmentation, Detection, and Recognition

In this paper we present a Bayesian framework for parsing images into their constituent visual patterns. The parsing algorithm optimizes the posterior probability and outputs a scene representation as a “parsing graph”, in a spirit similar to parsing sentences in speech and natural language. The algorithm constructs the parsing graph and re-configures it dynamically using a set of moves, which are mostly reversible Markov chain jumps. This computational framework integrates two popular inference approaches—generative (top-down) methods and discriminative (bottom-up) methods. The former formulates the posterior probability in terms of generative models for images defined by likelihood functions and priors. The latter computes discriminative probabilities based on a sequence (cascade) of bottom-up tests/filters. In our Markov chain algorithm design, the posterior probability, defined by the generative models, is the invariant (target) probability for the Markov chain, and the discriminative probabilities are used to construct proposal probabilities to drive the Markov chain. Intuitively, the bottom-up discriminative probabilities activate top-down generative models. In this paper, we focus on two types of visual patterns—generic visual patterns, such as texture and shading, and object patterns including human faces and text. These types of patterns compete and cooperate to explain the image and so image parsing unifies image segmentation, object detection, and recognition (if we use generic visual patterns only then image parsing will correspond to image segmentation (Tu and Zhu, 2002. IEEE Trans. PAMI, 24(5):657–673). We illustrate our algorithm on natural images of complex city scenes and show examples where image segmentation can be improved by allowing object specific knowledge to disambiguate low-level segmentation cues, and conversely where object detection can be improved by using generic visual patterns to explain away shadows and occlusions.

A generative framework for real time object detection and classification

We formulate a probabilistic model of image generation and derive optimal inference algorithms for finding objects and object features within this framework. The approach models images as a collage of patches of arbitrary size, some of which contain the object of interest and some of which are background. The approach requires development of likelihood-ratio models for object versus background generated patches. These models are learned using boosting methods. One advantage of the generative approach proposed here is that it makes explicit the conditions under which it is optimal. We applied the approach to the problem of finding faces and eyes on arbitrary images. Optimal inference under the proposed model works in real time and is robust to changes in lighting, illumination, and differences in facial structure, including facial expressions and eyeglasses. Furthermore, the system can simultaneously track the eyes and blinks of multiple individuals. Finally we reflect on how the development of perceptive systems like this may help advance our understanding of the human brain.

Modeling SAR images with a generalization of the Rayleigh distribution.

Synthetic aperture radar (SAR) imagery has found important applications due to its clear advantages over optical satellite imagery one of them being able to operate in various weather conditions. However, due to the physics of the radar imaging process, SAR images contain unwanted artifacts in the form of a granular look which is called speckle. The assumptions of the classical SAR image generation model lead to a Rayleigh distribution model for the histogram of the SAR image. However, some experimental data such as images of urban areas show impulsive characteristics that correspond to underlying heavy-tailed distributions, which are clearly non-Rayleigh. Some alternative distributions have been suggested such as the Weibull, log-normal, and the k-distribution which had success in varying degrees depending on the application. Recently, an alternative model namely the alpha-stable distribution has been suggested for modeling radar clutter. In this paper, we show that the amplitude distribution of the complex wave, the real and the imaginery components of which are assumed to be distributed by the alpha-stable distribution, is a generalization of the Rayleigh distribution. We demonstrate that the amplitude distribution is a mixture of Rayleighs as is the k-distribution in accordance with earlier work on modeling SAR images which showed that almost all successful SAR image models could be expressed as mixtures of Rayleighs. We also present parameter estimation techniques based on negative order moments for the new model. Finally, we test the performance of the model on urban images and compare with other models such as Rayleigh, Weibull, and the k-distribution.