Second-order Vision Research Articles

Fast Differentiable Matrix Square Root and Inverse Square Root.

Computing the matrix square root and its inverse in a differentiable manner is important in a variety of computer vision tasks. Previous methods either adopt the Singular Value Decomposition (SVD) to explicitly factorize the matrix or use the Newton-Schulz iteration (NS iteration) to derive the approximate solution. However, both methods are not computationally efficient enough in either the forward pass or the backward pass. In this paper, we propose two more efficient variants to compute the differentiable matrix square root and the inverse square root. For the forward propagation, one method is to use Matrix Taylor Polynomial (MTP), and the other method is to use Matrix Padé Approximants (MPA). The backward gradient is computed by iteratively solving the continuous-time Lyapunov equationusing the matrix sign function. A series of numerical tests show that both methods yield considerable speed-up compared with the SVD or the NS iteration. Moreover, we validate the effectiveness of our methods in several real-world applications, including de-correlated batch normalization, second-order vision transformer, global covariance pooling for large-scale and fine-grained recognition, attentive covariance pooling for video recognition, and neural style transfer. The experiments demonstrate that our methods can also achieve competitive and even slightly better performances. Code is available at https://github.com/KingJamesSong/FastDifferentiableMatSqrt.

Linear binocular combination of responses to contrast modulation: contrast-weighted summation in first- and second-order vision.

Binocular combination for first-order (luminance-defined) stimuli has been widely studied, but we know rather little about this binocular process for spatial modulations of contrast (second-order stimuli). We used phase-matching and amplitude-matching tasks to assess binocular combination of second-order phase and modulation depth simultaneously. With fixed modulation in one eye, we found that binocularly perceived phase was shifted, and perceived amplitude increased almost linearly as modulation depth in the other eye increased. At larger disparities, the phase shift was larger and the amplitude change was smaller. The degree of interocular correlation of the carriers had no influence. These results can be explained by an initial extraction of the contrast envelopes before binocular combination (consistent with the lack of dependence on carrier correlation) followed by a weighted linear summation of second-order modulations in which the weights (gains) for each eye are driven by the first-order carrier contrasts as previously found for first-order binocular combination. Perceived modulation depth fell markedly with increasing phase disparity unlike previous findings that perceived first-order contrast was almost independent of phase disparity. We present a simple revision to a widely used interocular gain-control theory that unifies first- and second-order binocular summation with a single principle--contrast--weighted summation-and we further elaborate the model for first-order combination. Second-order combination is controlled by first-order contrast.

A normative framework for the study of second-order sensitivity in vision.

While the contrast sensitivity approach has been successful in evaluating the processing of first-order stimuli, there is a need to develop comparable ways of assessing second-order vision. Our purpose here is to establish normative data on second-order contrast-, orientation-, and motion-modulation sensitivity in humans. We propose a unified framework, applying the quick contrast sensitivity function (qCSF) method, which was recently developed for the rapid measurement of contrast sensitivity across the full spatial-frequency range (Lesmes, Lu, Baek, & Albright, 2010), to measure both first- and second-order sensitivity functions. We first show that the qCSF methodology can be successfully adapted to different kinds of first- and second-order measurements. We provide a normative dataset for both first- and second-order sensitivity, and we show that the sensitivity to all these stimuli is equal in the two eyes. Our results confirm some strong differences between first- and second-order processing, in accordance with the classical filter-rectify-filter model. They suggest a common contrast detection mechanism but different second-order mechanisms.

Pooling of first-order inputs in second-order vision

The processing of texture patterns has been characterized by a model that first filters the image to isolate one texture component, then applies a rectifying nonlinearity that converts texture variation into intensity variation, and finally processes the resulting pattern with mechanisms similar to those used in processing luminance-defined images (spatial-frequency- and orientation-tuned filters). This model, known as FRF for filter rectify filter, has the appeal of explaining sensitivity to second-order patterns in terms of mechanisms known to exist for processing first-order patterns. This model implies an unexpected interaction between the first and second stages of filtering; if the first-stage filter consists of narrowband mechanisms tuned to detect the carrier texture, then sensitivity to high-frequency texture modulations should be much lower than is observed in humans. We propose that the human visual system must pool over first-order channels tuned to a wide range of spatial frequencies and orientations to achieve texture demodulation, and provide psychophysical evidence for pooling in a cross-carrier adaptation experiment and in an experiment that measures modulation contrast sensitivity at very low first-order contrast.

Inconsistent channel bandwidth estimates suggest winner-take-all nonlinearity in second-order vision

The processing of texture patterns has been characterized by a model that postulates a first-stage linear filter to highlight a component texture, a pointwise rectification stage to convert contrast for the highlighted texture into mean response strength, followed by a second-stage linear filter to detect the texture-defined pattern. We estimated the spatial-frequency bandwidth of the second-stage filter mediating orientation discrimination of orientation-modulated second-order gratings by measuring threshold elevation in the presence of filtered noise added to the modulation signal. This experiment yielded no evidence for frequency tuning. A second experiment, in which subjects had to detect similar second-order gratings while judging their modulation frequency, produced bandwidth estimates of 1–1.5 octaves, similar to estimated bandwidths of first-order channels. We propose that an additional dominant-response-selection nonlinearity can account for these apparently contradictory results.

The efficacy of local luminance amplitude in disambiguating the origin of luminance signals depends on carrier frequency: Further evidence for the active role of second-order vision in layer decomposition

When an undulating surface bearing a painted texture is illuminated the resulting shading pattern produces in-phase modulations of the mean luminance (LM) and luminance amplitude (AM) of the texture. Experimentally, in-phase combinations of LM and AM (LM + AM) are seen as undulating surfaces whereas anti-phase combinations (LM − AM) are more ambiguous; being seen as undulating when presented alone but as flat when presented in a plaid with LM + AM. AM is a second-order cue and its influence on shape-from-shading can be explained with a bottom-up layer decomposition model containing second-order mechanisms. However, the role of second-order vision in layer decomposition has not been established. If second-order vision is involved in layer decomposition then the perceptual differences between LM + AM and LM − AM should depend on the properties of the carrier texture in a way that is consistent with the known properties of second-order vision. Here we find a preference for carrier frequencies 3 octaves above the modulation frequency and take this as an indication that second-order (filter-rectify-filter) mechanisms are involved in processing our LM/AM mixes. We introduce a modified model which takes into account the selectivity of second-order vision for carrier frequency.

What is second-order vision for? Discriminating illumination versus material changes

The human visual system is sensitive to second-order modulations of the local contrast (CM) or amplitude (AM) of a carrier signal. Second-order cues are detected independently of first-order luminance signals; however, it is not clear why vision should benefit from second-order sensitivity. Analysis of the first- and second-order contents of natural images suggests that these cues tend to occur together, but their phase relationship varies. We have shown that in-phase combinations of LM and AM are perceived as a shaded corrugated surface whereas the anti-phase combination can be seen as corrugated when presented alone or as a flat material change when presented in a plaid containing the in-phase cue. We now extend these findings using new stimulus types and a novel haptic matching task. We also introduce a computational model based on initially separate first- and second-order channels that are combined within orientation and subsequently across orientation to produce a shading signal. Contrast gain control allows the LM + AM cue to suppress responses to the LM - AM when presented in a plaid. Thus, the model sees LM - AM as flat in these circumstances. We conclude that second-order vision plays a key role in disambiguating the origin of luminance changes within an image.

The role of second-order vision in discriminating shading versus material changes

The role of second-order vision in discriminating shading versus material changes

Extrafoveal viewing reveals the nature of second-order human vision

For the majority of visual tasks, performance in extrafoveal vision can be equated with that at the fovea simply by a change in spatial scale of the stimuli (magnification). We sought to exploit this association to examine the nature of second-order vision. More specifically, we investigate the relationship between the scale of second-order vision and the scale of its first-order input. We find that sensitivity to second-order stimuli can be equated across visual space, but only for stimuli that are magnified in every respect (identical scaling of both first- and second-order characteristics). In other words, sensitivity to stimuli which poses a fixed ratio between the scale of first-order input and second-order spatial scale can be equated across the visual field using a single magnification factor. Moreover, stimuli which poses quite different ratios of first- and second-order scale can be equated across eccentricity using the same magnification factor. This argues for a strict relationship between second-order vision and the scale of its first-order input and reflects a parallel arrangement of dedicated second-order mechanisms having a common eccentricity dependence.

First-order color vision is slow; Second-order color vision is fast

First-order color vision is slow; Second-order color vision is fast

Why is second-order vision less efficient than first-order vision?

Research has shown that the sensitivity to second-order modulations of carrier contrast is lower than that to first-order luminance modulations stimuli. We sought to compare the efficiency of processing first- and second-order information. Employing a phase-discrimination paradigm we found that when humans were given sufficient a priori information of signal parameters they detected both luminance and contrast modulations of 0.6 and 2 c/deg by a phase-sensitive algorithm. The overall detection efficiency for second-order patterns, however, was lower that that for first-order stimuli. To study the factors which limit the efficiency of first- and second-order vision, we measured detection performance for luminance and contrast modulations of 0.6 and 2 c/deg embedded in Gaussian noise. The results showed that the detection of second-order patterns had lower sampling efficiency and higher additive internal noise as compared to the detection of first-order stimuli. Classification images for detecting contrast modulations of 2 c/deg resembled the side-band component of the contrast modulations which suggests that human observers may detect contrast modulations of a sinusoidal carrier using first-order luminance channels. The lower sensitivity of the mechanism detecting second-order patterns might be due to higher levels of additive internal noise and lower sampling efficiency than those of the mechanism analysing first-order patterns.

Interactions between Orientation and Contrast Modulations Suggest Limited Cross-Cue Linkage

Recent studies of texture segmentation and second-order vision have proposed very similar models for the detection of orientation modulation and contrast modulation (OM and CM). From the similarity of the models it is tempting to assume that the two cues might be processed by a single generalised texture mechanism; however, recent results (Kingdom et al, 2003 Visual Neuroscience 2 65-76) have suggested that these cues are detected independently, or at least in a mechanism that is able to maintain an apparent independence between the cues. We tested new combinations of OM and CM and found that CM at 0.4 cycle deg(-1) facilitates the detection of OM at 0.2 cycle deg(-1) when the peaks of contrast align with the extremes of orientation. There is also some evidence of weak facilitation of CM by OM under the same conditions. Further, this facilitation can be predicted by filter--rectify--filter channels optimised for the detection of each cue, adding weight to the argument that texture cues are processed in a single generalised mechanism that nonetheless achieves cue independence or near-independence in many circumstances. We also found that the amount of suprathreshold masking produced by an orientation cue depends on the overall percept formed by that cue.

Temporal properties of the visual responses to luminance and contrast modulated noise

Vision is sensitive to first-order luminance modulations and second-order modulations of carrier contrast. Our knowledge of the temporal properties of second-order vision is insufficient and contradictory. Using temporal summation and reaction time paradigms, we found that the type of visual noise (static or dynamic) determines the temporal properties of the responses to luminance and contrast modulations. In the presence of static noise, the temporal responses to both types of modulation of low and higher spatial frequencies were transient. When dynamic noise was used, the temporal responses to luminance and contrast modulations of higher spatial frequencies were sustained. At low spatial frequency, however, luminance modulations elicited transient responses, while contrast modulated dynamic noise produced sustained responses. The reaction times to near-threshold contrast modulations of low spatial frequency were slower than those to first-order patterns and they did not significantly differ at modulations of higher spatial frequency. The results suggest that the temporal characteristics of first-stage linear filters which feed the second-order pathway may determine the temporal responses to contrast modulated noise.

Sensitivity to contrast modulation: the spatial frequency dependence of second-order vision

We consider the overall shape of the second-order modulation sensitivity function (MSF). Because second-order modulations of local contrast or orientation require a carrier signal, it is necessary to evaluate modulation sensitivity against a variety of carriers before reaching a general conclusion about second-order sensitivity. Here we present second-order sensitivity functions for new carrier types (low pass (1/ f) noise, and high pass noise) and demonstrate that, when first-order artefacts have been accounted for, the shape of the resulting MSFs are similar to one another and to those for white and broad band noise. They are all low pass with a likely upper frequency limit in the range 10–20 c/deg, suggesting that detection of second-order stimuli is relatively insensitive to the structure of the carrier signal. This result contrasts strongly with that found for (first-order) luminance modulations of the same noise types. Here the noise acts as mask and each noise type masks most those frequencies that are dominant in its spectrum. Thus the shape of second-order MSFs are largely independent of the spectrum of their noise carrier, but first-order CSFs depend on the spectrum of an additive noise mask. This provides further evidence for the separation of first- and second-order vision and characterises second-order vision as a low pass mechanism.

What does second-order vision see in an image?

The human visual system is sensitive to both first-order variations in luminance and second-order variations in local contrast and texture. Although there is some debate about the nature of second-order vision and its relationship to first-order processing, there is now a body of results showing that they are processed separately. However, the amount, and nature, of second-order structure present in the natural environment is unclear. This is an important question because, if natural scenes contain little second-order structure in addition to first-order signals, the notion of a separate second-order system would lack ecological validity. Two models of second-order vision were applied to a number of well-calibrated natural images. Both models consisted of a first stage of oriented spatial filters followed by a rectifying nonlinearity and then a second set of filters. The models differed in terms of the connectivity between first-stage and second-stage filters. Output images taken from the models indicate that natural images do contain useful second-order structure. Specifically, the models reveal variations in texture and features defined by such variations. Areas of high contrast (but not necessarily high luminance) are also highlighted by the models. Second-order structure--as revealed by the models--did not correlate with the first-order profile of the images, suggesting that the two types of image 'content' may be statistically independent.

The temporal properties of first- and second-order vision

Vision is sensitive to first-order modulations of luminance and second-order modulations of image contrast. There is now a body of evidence that the two types of modulation are detected by separate mechanisms. Some previous experiments on motion detection have suggested that the second-order system is quite sluggish compared to the first-order system. Here we derive temporal properties of first- and second-order vision at threshold from studies of temporal integration and two-pulse summation. Three types of modulation were tested: luminance gratings alone, luminance modulations added to dynamic visual noise, and contrast modulations of dynamic noise. Data from the two-pulse summation experiment were used to derive impulse response functions for the three types of stimulus. These were then used to predict performance in the temporal integration experiment. Temporal frequency response functions were obtained as the Fourier transform of impulse responses derived from data averaged across two observers. The response to noise-free luminance gratings of 2 c/deg was bi-phasic and transient in the time domain, and bandpass in the frequency domain. The addition of dynamic noise caused the response to become mono-phasic, sustained and low-pass. The response to contrast modulated noise (second-order) was also mono-phasic, sustained and low-pass, with only a slightly longer integration time than in the first-order case. The ultimate roll-off at high frequencies was about the same as for the first-order case. We conclude that second-order vision may not be as sluggish as previously thought.