Simple Image Statistics Research Articles

A Convolutional Framework for Color Constancy.

We introduce a convolutional framework (CF) for computational color constancy, building upon the established low-level image feature-based framework, which utilized simple image statistics for illuminant estimation. Our framework expands upon this through an end-to-end learnable neural architecture. This adaptation enables the learning and usage of advanced filters that are not restricted to Gaussian kernels operating on individual color channels, thus generalizing the capabilities of the original framework. Additionally, our general framework supports deeper convolutional architectures, thus increasing its computational power. It can also be efficiently applied to estimate multiple spatially varying illuminants within a single scene. Our experimental results on standard datasets demonstrate that the CF outperforms the best methods in the low-level framework, improving the illuminant estimation accuracy by up to 34% for single illuminant estimation and 30% for multiple illuminants estimation. Additionally, our framework exhibits superior performance even when the number of training images is reduced. Finally, we document the inference speedup of our implementation reaching up to 30 × , making the CF especially suitable for applications where efficiency is critical. Source code and trained models available at: https://github.com/MarcoBauzz/convolutional-color-constancy.

Color and gloss constancy under diverse lighting environments.

When we look at an object, we simultaneously see how glossy or matte it is, how light or dark, and what color. Yet, at each point on the object's surface, both diffuse and specular reflections are mixed in different proportions, resulting in substantial spatial chromatic and luminance variations. To further complicate matters, this pattern changes radically when the object is viewed under different lighting conditions. The purpose of this study was to simultaneously measure our ability to judge color and gloss using an image set capturing diverse object and illuminant properties. Participants adjusted the hue, lightness, chroma, and specular reflectance of a reference object so that it appeared to be made of the same material as a test object. Critically, the two objects were presented under different lighting environments. We found that hue matches were highly accurate, except for under a chromatically atypical illuminant. Chroma and lightness constancy were generally poor, but these failures correlated well with simple image statistics. Gloss constancy was particularly poor, and these failures were only partially explained by reflection contrast. Importantly, across all measures, participants were highly consistent with one another in their deviations from constancy. Although color and gloss constancy hold well in simple conditions, the variety of lighting and shape in the real world presents significant challenges to our visual system's ability to judge intrinsic material properties.

Direct Image Feature Extraction and Multivariate Analysis for Crystallization Process Characterization

Small-scale crystallization experiments (1–8 mL) are widely used during early-stage crystallization process development to obtain initial information on solubility, metastable zone width, as well as attainable nucleation and/or growth kinetics in a material-efficient manner. Digital imaging is used to monitor these experiments either providing qualitative information or for object detection coupled with size and shape characterization. In this study, a novel approach for the routine characterization of image data from such crystallization experiments is presented employing methodologies for direct image feature extraction. A total of 80 image features were extracted based on simple image statistics, histogram parametrization, and a series of targeted image transformations to assess local grayscale characteristics. These features were utilized for applications of clear/cloud point detection and crystal suspension density prediction. Compared to commonly used transmission-based methods (mean absolute error 8.99 mg/mL), the image-based detection method is significantly more accurate for clear and cloud point detection with a mean absolute error of 0.42 mg/mL against a manually assessed ground truth. Extracted image features were further used as part of a partial least-squares regression (PLSR) model to successfully predict crystal suspension densities up to 40 mg/mL (R2 > 0.81, Q2 > 0.83). These quantitative measurements reliably provide crucial information on composition and kinetics for early parameter estimation and process modeling. The image analysis methodologies have a great potential to be translated to other imaging techniques for process monitoring of key physical parameters to accelerate the development and control of particle/crystallization processes.

The joint role of geometry and illumination on material recognition.

Observing and recognizing materials is a fundamental part of our daily life. Under typical viewing conditions, we are capable of effortlessly identifying the objects that surround us and recognizing the materials they are made of. Nevertheless, understanding the underlying perceptual processes that take place to accurately discern the visual properties of an object is a long-standing problem. In this work, we perform a comprehensive and systematic analysis of how the interplay of geometry, illumination, and their spatial frequencies affects human performance on material recognition tasks. We carry out large-scale behavioral experiments where participants are asked to recognize different reference materials among a pool of candidate samples. In the different experiments, we carefully sample the information in the frequency domain of the stimuli. From our analysis, we find significant first-order interactions between the geometry and the illumination, of both the reference and the candidates. In addition, we observe that simple image statistics and higher-order image histograms do not correlate with human performance. Therefore, we perform a high-level comparison of highly nonlinear statistics by training a deep neural network on material recognition tasks. Our results show that such models can accurately classify materials, which suggests that they are capable of defining a meaningful representation of material appearance from labeled proximal image data. Last, we find preliminary evidence that these highly nonlinear models and humans may use similar high-level factors for material recognition tasks.

Apoptosis detection via automated algorithms to analyze biomarker translocation in reporter cells.

Rapid, efficient, and robust quantitative analyses of dynamic apoptotic events are essential in a high-throughput screening workflow. Currently used methods have several bottlenecks, specifically, limitations in available fluorophores for downstream assays and misinterpretation of statistical image data analysis. In this study, we developed cytochrome-C (Cyt-C) and caspase-3/-8 reporter cell lines using lung (PC9) and breast (T47D) cancer cells, and characterized them from the response to apoptotic stimuli. In these two reporter cell lines, the spatial fluorescent signal translocation patterns served as reporters of activations of apoptotic events, such as Cyt-C release and caspase-3/-8 activation. We also developed a vision-based, tunable, automated algorithm in MATLAB to implement the robust and accurate analysis of signal translocation in single or multiple cells. Construction of the reporter cell lines allows live monitoring of apoptotic events without the need for any other dyes or fixatives. Our algorithmic implementation forgoes the use of simple image statistics for more robust analytics. Our optimized algorithm can achieve a precision greater than 90% and a sensitivity higher than 85%. Combining our automated algorithm with reporter cells bearing a single-color dye/fluorophore, we expect our approach to become an integral component in the high-throughput drug screening workflow.

Perception of human skin conditions and imagestatistics.

The color and texture of human skin provide useful information in evaluating a person's health, emotion, attractiveness, and so on. Whereas the appearance of skin is a product of optical properties originating from its own biophysical substrates, recent psychophysical studies of surface material perception indicate a possibility that humans can perceive skin conditions from relatively simple image features such as texture statistics. The present study investigated the structure of multidimensional perceptual rating data for 289 female skin images and analyzed diagnostic low-level image statistics. We found that various skin appearances can be summarized into two dimensions, pleasantness and glossiness, and either dimension is well correlated with a small set of subband luminance and color statistics. We confirmed that manipulations of these critical image statistics significantly alter the appearance of skin in the expected direction. The results support a notion that humans perceive skin properties essentially based on two cardinal dimensions as predicted by simple image statistics.

A simple and effective image-statistics-based approach to detecting recaptured images from LCD screens

It is now extremely easy to recapture high-resolution and high-quality images from LCD (Liquid Crystal Display) screens. Recaptured image detection is an important digital forensic problem, as image recapture is often involved in the creation of a fake image in an attempt to increase its visual plausibility. State-of-the-art image recapture forensic methods make use of strong prior knowledge about the recapturing process and are based on either the combination of a group of ad-hoc features or a specific and somehow complicated dictionary learning procedure. By contrast, we propose a conceptually simple yet effective method for recaptured image detection which is built upon simple image statistics and a very loose assumption about the recapturing process. The adopted features are pixel-wise correlation coefficients in image differential domains. Experimental results on two large databases of high-resolution, high-quality recaptured images and comparisons with existing methods demonstrate the forensic accuracy and the computational efficiency of the proposed method.

Sound-Event Classification Using Robust Texture Features for Robot Hearing

Sound-event classification often utilizes time–frequency analysis, which produces an image-like spectrogram. Recent approaches such as spectrogram image features and subband power distribution image features extract the image local statistics such as mean and variance from the spectrogram. They have demonstrated good performance. However, we argue that such simple image statistics cannot well capture the complex texture details of the spectrogram. Thus, we propose to extract the local binary pattern (LBP) from the logarithm of the Gammatone-like spectrogram. However, the LBP feature is sensitive to noise. After analyzing the spectrograms of sound events and the audio noise, we find that the magnitude of pixel differences, which is discarded by the LBP feature, carries important information for sound-event classification. We thus propose a multichannel LBP feature via pixel difference quantization to improve the robustness to the audio noise. In view of the differences between spectrograms and natural images, and the reliability issues of LBP features, we propose two projection-based LBP features to better capture the texture information of the spectrogram. To validate the proposed multichannel projection-based LBP features for robot hearing, we have built a new sound-event classification database, the NTU-SEC database, in the context of social interaction between human and robot. It is publicly available to promote research on sound-event classification in a social context. The proposed approaches are compared with the state of the art on the RWCP database and the NTU-SEC database. They consistently demonstrate superior performance under various noise conditions.

Qualitative and quantitative interpretation of SEM image using digital image processing.

The aim of the this study is improvement of qualitative and quantitative analysis of scanning electron microscope micrographs by development of computer program, which enables automatic crack analysis of scanning electron microscopy (SEM) micrographs. Micromechanical tests of pneumatic ventricular assist devices result in a large number of micrographs. Therefore, the analysis must be automatic. Tests for athrombogenic titanium nitride/gold coatings deposited on polymeric substrates (Bionate II) are performed. These tests include microshear, microtension and fatigue analysis. Anisotropic surface defects observed in the SEM micrographs require support for qualitative and quantitative interpretation. Improvement of qualitative analysis of scanning electron microscope images was achieved by a set of computational tools that includes binarization, simplified expanding, expanding, simple image statistic thresholding, the filters Laplacian 1, and Laplacian 2, Otsu and reverse binarization. Several modifications of the known image processing techniques and combinations of the selected image processing techniques were applied. The introduced quantitative analysis of digital scanning electron microscope images enables computation of stereological parameters such as area, crack angle, crack length, and total crack length per unit area. This study also compares the functionality of the developed computer program of digital image processing with existing applications. The described pre- and postprocessing may be helpful in scanning electron microscopy and transmission electron microscopy surface investigations.

Image statistics and the perception of surface gloss and lightness

Despite previous data demonstrating the critical importance of 3D surface geometry in the perception of gloss and lightness, I. Motoyoshi, S. Nishida, L. Sharan, and E. H. Adelson (2007) recently proposed that a simple image statistic--histogram or sub-band skew--is computed by the visual system to infer the gloss and albedo of surfaces. One key source of evidence used to support this claim was an experiment in which adaptation to skewed image statistics resulted in opponent aftereffects in observers' judgments of gloss and lightness. We report a series of adaptation experiments that were designed to assess the cause of these aftereffects. We replicated their original aftereffects in gloss but found no consistent aftereffect in lightness. We report that adaptation to zero-skew adaptors produced similar aftereffects as positively skewed adaptors, and that negatively skewed adaptors induced no reliable aftereffects. We further find that the adaptation effect observed with positively skewed adaptors is not robust to changes in mean luminance that diminish the intensity of the luminance extrema. Finally, we show that adaptation to positive skew reduces (rather than increases) the apparent lightness of light pigmentation on non-uniform albedo surfaces. These results challenge the view that the adaptation results reported by Motoyoshi et al. (2007) provide evidence that skew is explicitly computed by the visual system.

Rapid animal detection in natural scenes: Critical features are local

Thorpe et al (Nature 381, 1996) first showed how rapidly human observers are able to classify natural images as to whether they contain an animal or not. Whilst the basic result has been replicated using different response paradigms (yes-no versus forced-choice), modalities (eye movements versus button presses) as well as while measuring neurophysiological correlates (ERPs), it is still unclear which image features support this rapid categorisation. Recently Torralba and Oliva (Network: Computation in Neural Systems, 14, 2003) suggested that simple global image statistics can be used to predict seemingly complex decisions about the absence and/or presence of objects in natural scences. They show that the information contained in a small number (N=16) of spectral principal components (SPC)—principal component analysis (PCA) applied to the normalised power spectra of the images—is sufficient to achieve approximately 80% correct animal detection in natural scenes.

Animal detection in natural scenes: Critical features revisited

S. J. Thorpe, D. Fize, and C. Marlot (1996) showed how rapidly observers can detect animals in images of natural scenes, but it is still unclear which image features support this rapid detection. A. B. Torralba and A. Oliva (2003) suggested that a simple image statistic based on the power spectrum allows the absence or presence of objects in natural scenes to be predicted. We tested whether human observers make use of power spectral differences between image categories when detecting animals in natural scenes. In Experiments 1 and 2 we found performance to be essentially independent of the power spectrum. Computational analysis revealed that the ease of classification correlates with the proposed spectral cue without being caused by it. This result is consistent with the hypothesis that in commercial stock photo databases a majority of animal images are pre-segmented from the background by the photographers and this pre-segmentation causes the power spectral differences between image categories and may, furthermore, help rapid animal detection. Data from a third experiment are consistent with this hypothesis. Together, our results make it exceedingly unlikely that human observers make use of power spectral differences between animal- and no-animal images during rapid animal detection. In addition, our results point to potential confounds in the commercially available "natural image" databases whose statistics may be less natural than commonly presumed.

Image statistics do not explain the perception of gloss and lightness

A fundamental problem in image analysis is to understand the nature of the computations and mechanisms that provide information about the material properties of surfaces. Information about a surface's 3D shape, optics, illumination field, and atmospheric conditions are conflated in the image, which must somehow be disentangled to derive the properties of surfaces. It was recently suggested that the visual system exploits some simple image statistics-histogram or sub-band skew-to infer the lightness and gloss of surfaces (I. Motoyoshi, S. Nishida, L. Sharan, & E. H. Adelson, 2007). Here, we show that the correlations Motoyoshi et al. (2007) observed between skew, lightness, and gloss only arose because of the limited space of surface geometries, reflectance properties, and illumination fields they evaluated. We argue that the lightness effects they reported were a statistical artifact of equating the means of images with skewed histograms, and that the perception of gloss requires an analysis of the consistency between the estimate of a surface's 3D shape and the positions and orientations of highlights on a surface. We argue that the derivation of surface and material properties requires a photo-geometric analysis, and that purely photometric statistics such as skew fail to capture any diagnostic information about surfaces because they are devoid of the structural information needed to distinguish different types of surface attributes.

Two Principles of High-Level Human Visual Processing Potentially Useful for Image and Video Quality Assessment

Objective assessment of image and video quality should be based on a correct understanding of subjective assessment by human observers. Previous models have incorporated the mechanisms of early visual processing in image quality metrics, enabling us to evaluate the visibility of errors from the original images. However, to understand how human observers perceive image quality, one should also consider higher stages of visual processing where perception is established. In higher stages, the visual system presumably represents a visual scene as a collection of meaningful components such as objects and events. Our recent psychophysical studies suggest two principles related to this level of processing. First, the human visual system integrates shape and color signals along perceived motion trajectories in order to improve visibility of the shape and color of moving objects. Second, the human visual system estimates surface reflectance properties like glossiness using simple image statistics rather than by inverse computation of image formation optics. Although the underlying neural mechanisms are still under investigation, these computational principles are potentially useful for the development of effective image processing technologies and for quality assessment. Ideally, if a model can specify how a given image is transformed into high-level scene representations in the human brain, it would predict many aspects of subjective image quality, including fidelity and naturalness.

Color image based sorter for separating red and white wheat

A simple imaging system was developed to inspect and sort wheat samples and other grains at mod- erate feed-rates (30 kernels/s or 3.5 kg wheat/h). A single camera captured color images of three sides of each kernel by using mirrors, and the images were processed using a personal computer (PC). Real time image acquisition and processing was enabled on an ordinary PC under Windows XP operating system using the IEEE 1394 data transfer protocol, DirectX application software, and dual-core computer processor. Image acquisition and transfer to the PC required approximately 17 ms per kernel, and an additional 1.5 ms was required for image processing. After classification, the computer could output a signal from the parallel port to activate an air valve to divert (sort) kernels into a secondary container. Hard red and hard white wheat kernels were used in this study to test and demonstrate sorter capability. Simple image statistics and histograms were used as features. Discriminant analysis was per- formed with one, two, or three features to demonstrate classification improvements with increased numbers of features. The sorter was able to separate hard red kernels from hard white kernels with 95 to 99% accuracy, depending on the wheat varieties, feed-rate, and number of classification features. The system is an economical and useful instrument for sorting wheat and other grains with high accuracy.

Image statistics for surface reflectance perception

Human observers can distinguish the albedo of real-world surfaces even when the surfaces are viewed in isolation, contrary to the Gelb effect. We sought to measure this ability and to understand the cues that might underlie it. We took photographs of complex surfaces such as stucco and asked observers to judge their diffuse reflectance by comparing them to a physical Munsell scale. Their judgments, while imperfect, were highly correlated with the true reflectance. The judgments were also highly correlated with certain image statistics, such as moment and percentile statistics of the luminance and subband histograms. When we digitally manipulated these statistics in an image, human judgments were correspondingly altered. Moreover, linear combinations of such statistics allow a machine vision system (operating within the constrained world of single surfaces) to estimate albedo with an accuracy similar to that of human observers. Taken together, these results indicate that some simple image statistics have a strong influence on the judgment of surface reflectance.

Image statistics and the perception of surface qualities

The world is full of surfaces, and by looking at them we can judge their material qualities. Properties such as colour or glossiness can help us decide whether a pancake is cooked, or a patch of pavement is icy. Most studies of surface appearance have emphasized textureless matte surfaces, but real-world surfaces, which may have gloss and complex mesostructure, are now receiving increased attention. Their appearance results from a complex interplay of illumination, reflectance and surface geometry, which are difficult to tease apart given an image. If there were simple image statistics that were diagnostic of surface properties it would be sensible to use them. Here we show that the skewness of the luminance histogram and the skewness of sub-band filter outputs are correlated with surface gloss and inversely correlated with surface albedo (diffuse reflectance). We find evidence that human observers use skewness, or a similar measure of histogram asymmetry, in making judgements about surfaces. When the image of a surface has positively skewed statistics, it tends to appear darker and glossier than a similar surface with lower skewness, and this is true whether the skewness is inherent to the original image or is introduced by digital manipulation. We also find a visual after-effect based on skewness: adaptation to patterns with skewed statistics can alter the apparent lightness and glossiness of surfaces that are subsequently viewed. We suggest that there are neural mechanisms sensitive to skewed statistics, and that their outputs can be used in estimating surface properties.

Color balancing of digital photos using simple image statistics

The great diffusion of digital cameras and the widespread use of the internet have produced a mass of digital images depicting a huge variety of subjects, generally acquired by unknown imaging systems under unknown lighting conditions. This makes color balancing, recovery of the color characteristics of the original scene, increasingly difficult. In this paper, we describe a method for detecting and removing a color cast (i.e. a superimposed color due to lighting conditions, or to the characteristics of the capturing device), from a digital photo without any a priori knowledge of its semantic content. First a cast detector, using simple image statistics, classifies the input images as presenting no cast, evident cast, ambiguous cast, a predominant color that must be preserved (such as in underwater images or single color close-ups) or as unclassifiable. A cast remover, a modified version of the white balance algorithm, is then applied in cases of evident or ambiguous cast. The method we propose has been tested with positive results on a data set of some 750 photos.

Statistics of natural image categories

In this paper we study the statistical properties of natural images belonging to different categories and their relevance for scene and object categorization tasks. We discuss how second-order statistics are correlated with image categories, scene scale and objects. We propose how scene categorization could be computed in a feedforward manner in order to provide top-down and contextual information very early in the visual processing chain. Results show how visual categorization based directly on low-level features, without grouping or segmentation stages, can benefit object localization and identification. We show how simple image statistics can be used to predict the presence and absence of objects in the scene before exploring the image.

Statistics of natural image categories

In this paper we study the statistical properties of natural images belonging to different categories and their relevance for scene and object categorization tasks. We discuss how second-order statistics are correlated with image categories, scene scale and objects. We propose how scene categorization could be computed in a feedforward manner in order to provide top-down and contextual information very early in the visual processing chain. Results show how visual categorization based directly on low-level features, without grouping or segmentation stages, can benefit object localization and identification. We show how simple image statistics can be used to predict the presence and absence of objects in the scene before exploring the image.