Natural Movies Research Articles

Deep recurrent neural network reveals a hierarchy of process memory during dynamic natural vision.

The human visual cortex extracts both spatial and temporal visual features to support perception and guide behavior. Deep convolutional neural networks (CNNs) provide a computational framework to model cortical representation and organization for spatial visual processing, but unable to explain how the brain processes temporal information. To overcome this limitation, we extended a CNN by adding recurrent connections to different layers of the CNN to allow spatial representations to be remembered and accumulated over time. The extended model, or the recurrent neural network (RNN), embodied a hierarchical and distributed model of process memory as an integral part of visual processing. Unlike the CNN, the RNN learned spatiotemporal features from videos to enable action recognition. The RNN better predicted cortical responses to natural movie stimuli than the CNN, at all visual areas, especially those along the dorsal stream. As a fully observable model of visual processing, the RNN also revealed a cortical hierarchy of temporal receptive window, dynamics of process memory, and spatiotemporal representations. These results support the hypothesis of process memory, and demonstrate the potential of using the RNN for in-depth computational understanding of dynamic natural vision.

Similar neural responses predict friendship

Human social networks are overwhelmingly homophilous: individuals tend to befriend others who are similar to them in terms of a range of physical attributes (e.g., age, gender). Do similarities among friends reflect deeper similarities in how we perceive, interpret, and respond to the world? To test whether friendship, and more generally, social network proximity, is associated with increased similarity of real-time mental responding, we used functional magnetic resonance imaging to scan subjects’ brains during free viewing of naturalistic movies. Here we show evidence for neural homophily: neural responses when viewing audiovisual movies are exceptionally similar among friends, and that similarity decreases with increasing distance in a real-world social network. These results suggest that we are exceptionally similar to our friends in how we perceive and respond to the world around us, which has implications for interpersonal influence and attraction.

Mapping stimulus feature selectivity in macaque V1 by two-photon Ca2+ imaging: Encoding-model analysis of fluorescence responses to natural movies

In vivo calcium (Ca2+) imaging using two-photon microscopy allows activity to be monitored simultaneously from hundreds of individual neurons within a local population. While this allows us to gain important insights into how cortical neurons represent sensory information, factors such as photo-bleaching of the Ca2+ indicator limit imaging duration (and thus the numbers of stimuli that can be tested), which in turn hampers the full characterization of neuronal response properties. Here, we demonstrate that using an encoding model combined with presentation of natural movies results in detailed characterization of receptive field (RF) properties despite the relatively short time for data collection. During presentation of natural movie clips to macaque monkeys, we recorded fluorescence signals from primary visual cortex (V1) neurons that had been loaded with a Ca2+ indicator. For each recorded neuron, we constructed an encoding model that comprised an array of motion-energy filters that tiled over the RFs. We optimized the weight of each filter's output so that the linear sum of the outputs across the filters mimicked the neuron's Ca2+-signal responses. These models were able to predict the neural responses to a different set of natural movies with a significant degree of accuracy. Moreover, the orientation tunings of neurons simulated by the model were highly correlated with those experimentally obtained when grating stimuli were presented to the monkeys. The model predictions were also consistent with what is known about spatial frequency tunings, the structure of excitatory subfields of RFs (i.e., classical RFs), and functional maps for these RF properties in V1. Further analysis revealed a new aspect of V1 functional architecture; the extent and distribution of suppressive RF subfields varied among nearby neurons, while those for excitatory subfields were shared. Thus, applying our encoding-model analysis to two-photon Ca2+ imaging of neuronal responses to natural movies provides a reliable and efficient means of analyzing a wide range of RF properties in multiple neurons imaged in a local region.

Toward a unified theory of efficient, predictive, and sparse coding

A central goal in theoretical neuroscience is to predict the response properties of sensory neurons from first principles. To this end, "efficient coding" posits that sensory neurons encode maximal information about their inputs given internal constraints. There exist, however, many variants of efficient coding (e.g., redundancy reduction, different formulations of predictive coding, robust coding, sparse coding, etc.), differing in their regimes of applicability, in the relevance of signals to be encoded, and in the choice of constraints. It is unclear how these types of efficient coding relate or what is expected when different coding objectives are combined. Here we present a unified framework that encompasses previously proposed efficient coding models and extends to unique regimes. We show that optimizing neural responses to encode predictive information can lead them to either correlate or decorrelate their inputs, depending on the stimulus statistics; in contrast, at low noise, efficiently encoding the past always predicts decorrelation. Later, we investigate coding of naturalistic movies and show that qualitatively different types of visual motion tuning and levels of response sparsity are predicted, depending on whether the objective is to recover the past or predict the future. Our approach promises a way to explain the observed diversity of sensory neural responses, as due to multiple functional goals and constraints fulfilled by different cell types and/or circuits.

Neural Encoding and Decoding with Deep Learning for Dynamic Natural Vision.

Convolutional neural network (CNN) driven by image recognition has been shown to be able to explain cortical responses to static pictures at ventral-stream areas. Here, we further showed that such CNN could reliably predict and decode functional magnetic resonance imaging data from humans watching natural movies, despite its lack of any mechanism to account for temporal dynamics or feedback processing. Using separate data, encoding and decoding models were developed and evaluated for describing the bi-directional relationships between the CNN and the brain. Through the encoding models, the CNN-predicted areas covered not only the ventral stream, but also the dorsal stream, albeit to a lesser degree; single-voxel response was visualized as the specific pixel pattern that drove the response, revealing the distinct representation of individual cortical location; cortical activation was synthesized from natural images with high-throughput to map category representation, contrast, and selectivity. Through the decoding models, fMRI signals were directly decoded to estimate the feature representations in both visual and semantic spaces, for direct visual reconstruction and semantic categorization, respectively. These results corroborate, generalize, and extend previous findings, and highlight the value of using deep learning, as an all-in-one model of the visual cortex, to understand and decode natural vision.

Enhancing early consolidation of human episodic memory by theta EEG neurofeedback

Consolidation of newly formed memories is readily disrupted, but can it be enhanced? Given the prominent role of hippocampal theta oscillations in memory formation and retrieval, we hypothesized that upregulating theta power during early stages of consolidation might benefit memory stability and persistence. We used EEG neurofeedback to enable participants to selectively increase theta power in their EEG spectra following episodic memory encoding, while other participants engaged in low beta-focused neurofeedback or passively viewed a neutral nature movie. Free recall assessments immediately following the interventions, 24h later and 7d later all indicated benefit to memory of theta neurofeedback, relative to low beta neurofeedback or passive movie-viewing control conditions. The degree of benefit to memory was correlated with the extent of theta power modulation, but not with other spectral changes. Theta enhancement may provide optimal conditions for stabilization of new hippocampus-dependent memories.

Perceptual inference of dynamic emotion in natural movies

Perceptual inference of dynamic emotion in natural movies

Effects of spatial fMRI resolution on the classification of naturalistic movies

Studies involving multivariate pattern analysis (MVPA) of BOLD fMRI data generally attribute the success of the information-theoretic approach to BOLD signal contrast on the fine spatial scale of millimeters facilitating the classification or decoding of perceptual stimuli. However, to date MVPA studies that have actually explored fMRI resolutions at less than 2 mm voxel size are rare and limited to small sets of unnatural stimuli (like visual gratings) as well as specific sub-regions of the brain, notably the primary somatosensory cortices. To investigate what spatial scale best supports high information extraction under more general conditions this study combined naturalistic movie stimuli with high-resolution fMRI at 7 T and linear discriminant analysis (LDA) of global and local BOLD signal patterns.Contrary to predictions, LDA and similar classifiers reached a maximum in classification accuracy (CA) at a smoothed resolution close to 3 mm, well above the 1.2 mm voxel size of the fMRI acquisition. Maximal CAs around 90% were contingent upon global fMRI signal patterns comprising 4 k–16 k of the most reactive voxels distributed sparsely throughout the occipital and ventro-temporal cortices. A Searchlight analysis of local fMRI patterns largely confirmed the global results, but also revealed a small subset of brain regions in early visual cortex showing limited increases in CA with higher resolution. Principal component analysis of the global and local fMRI signal patterns suggested that reproducible neuronal contributions were spatially auto-correlated and smooth, while other components of higher spatial frequency were likely related to physiological noise and responsible for the reduced CA at higher resolution. Systematic differences between experiments and subjects suggested that higher CA was significantly correlated with more consistent behavior revealed by eye tracking. Thus, the optimal resolution of fMRI data for MVPA was mainly limited by physiological noise of high spatial frequency as well as behavioral (in-)consistency.

Novel method of extracting motion from natural movies

BackgroundThe visual system in primates can be segregated into motion and shape pathways. Interaction occurs at multiple stages along these pathways. Processing of shape-from-motion and biological motion is considered to be a higher-order integration process involving motion and shape information. However, relatively limited types of stimuli have been used in previous studies on these integration processes. New methodWe propose a new algorithm to extract object motion information from natural movies and to move random dots in accordance with the information. The object motion information is extracted by estimating the dynamics of local normal vectors of the image intensity projected onto the x-y plane of the movie. ResultsAn electrophysiological experiment on two adult common marmoset monkeys (Callithrix jacchus) showed that the natural and random dot movies generated with this new algorithm yielded comparable neural responses in the middle temporal visual area. Comparison with existing methodsIn principle, this algorithm provided random dot motion stimuli containing shape information for arbitrary natural movies. This new method is expected to expand the neurophysiological and psychophysical experimental protocols to elucidate the integration processing of motion and shape information in biological systems. ConclusionsThe novel algorithm proposed here was effective in extracting object motion information from natural movies and provided new motion stimuli to investigate higher-order motion information processing.

Decoding naturalistic experiences from human brain activity via distributed representations of words

Natural visual scenes induce rich perceptual experiences that are highly diverse from scene to scene and from person to person. Here, we propose a new framework for decoding such experiences using a distributed representation of words. We used functional magnetic resonance imaging (fMRI) to measure brain activity evoked by natural movie scenes. Then, we constructed a high-dimensional feature space of perceptual experiences using skip-gram, a state-of-the-art distributed word embedding model. We built a decoder that associates brain activity with perceptual experiences via the distributed word representation. The decoder successfully estimated perceptual contents consistent with the scene descriptions by multiple annotators. Our results illustrate three advantages of our decoding framework: (1) three types of perceptual contents could be decoded in the form of nouns (objects), verbs (actions), and adjectives (impressions) contained in 10,000 vocabulary words; (2) despite using such a large vocabulary, we could decode novel words that were absent in the datasets to train the decoder; and (3) the inter-individual variability of the decoded contents co-varied with that of the contents of scene descriptions. These findings suggest that our decoding framework can recover diverse aspects of perceptual experiences in naturalistic situations and could be useful in various scientific and practical applications.

Relative Time Compression for Slow-Motion Stimuli through Rapid Recalibration.

A number of psychophysical studies have shown that moving stimuli appear to last longer than static stimuli. Here, we report that the perceived duration for slow moving stimuli can be shorter than for static stimuli under specific circumstances. Observers were tested using natural movies presented at various speeds (0.0× = static, 0.25× = slow, or 1.9× = fast, relative to original speed) and indicated whether test duration was perceived as longer or shorter than comparison movies presented at their original speed. While fast movies were perceived as longer than slow and static movies (in accordance with previous studies), we found that slow movies were perceived as shorter (i.e., time compressed) compared to static images. Similar results were obtained for artificial stimuli consisting of drifting gratings. However, time compression for slow stimuli disappeared if comparison stimuli were replaced by a white static disk that removed repetitive exposures to moving stimuli. Results suggest that duration estimation is modulated by contextual effects induced by the specific diet – or distribution – of prior visual stimuli to which observers are exposed. A simple model, which includes a rapid recalibration of human time estimation via adaptation to preceding stimuli, succeeds in reproducing our experimental data.

Basal forebrain activation enhances between-trial reliability of low-frequency local field potentials (LFP) and spiking activity in tree shrew primary visual cortex (V1).

Brain state has profound effects on neural processing and stimulus encoding in sensory cortices. While the synchronized state is dominated by low-frequency local field potential (LFP) activity, low-frequency LFP power is suppressed in the desynchronized state, where a concurrent enhancement in gamma power is observed. Recently, it has been shown that cortical desynchronization co-occurs with enhanced between-trial reliability of spiking activity in sensory neurons, but it is currently unclear whether this effect is also evident in LFP signals. Here, we address this question by recording both spike trains and LFP in primary visual cortex during natural movie stimulation, and using isoflurane anesthesia and basal forebrain (BF) electrical activation as proxies for synchronized and desynchronized brain states. We show that indeed, low-frequency LFP modulations ("LFP events") also occur more reliably following BF activation. Interestingly, while being more reliable, these LFP events are smaller in amplitude compared to those generated in the synchronized brain state. We further demonstrate that differences in reliability of spiking activity between cortical states can be linked to amplitude and probability of LFP events. The correlated temporal dynamics between low-frequency LFP and spiking response reliability in visual cortex suggests that these effects may both be the result of the same neural circuit activation triggered by BF stimulation, which facilitates switching between processing of incoming sensory information in the desynchronized and reverberation of internal signals in the synchronized state.

PCA meets RG.

A system with many degrees of freedom can be characterized by a covariance matrix; principal components analysis (PCA) focuses on the eigenvalues of this matrix, hoping to find a lower dimensional description. But when the spectrum is nearly continuous, any distinction between components that we keep and those that we ignore becomes arbitrary; it then is natural to ask what happens as we vary this arbitrary cutoff. We argue that this problem is analogous to the momentum shell renormalization group (RG). Following this analogy, we can define relevant and irrelevant operators, where the role of dimensionality is played by properties of the eigenvalue density. These results also suggest an approach to the analysis of real data. As an example, we study neural activity in the vertebrate retina as it responds to naturalistic movies, and find evidence of behavior controlled by a nontrivial fixed point. Applied to financial data, our analysis separates modes dominated by sampling noise from a smaller but still macroscopic number of modes described by a non-Gaussian distribution.

Restoration of vision with ectopic expression of human rod opsin

BackgroundInherited retinal degenerations that lead to irreversible blindness due to progressive loss of rods and cones in the outer retina affect 1 in 2500 people worldwide. However, despite the substantial photoreceptor loss, the inner retinal neurons can survive, making them potentially amenable to emerging optogenetic therapies. The aim of this study was to determine whether it is possible to recreate vision in blind mice using ectopic expression of human rod opsin. MethodsThe rod opsin expressing adeno-associated virus serotype 2 vector, driven by either a non-selective or ON-bipolar cell-specific promoter, was injected intravitreally into adult rd1 mice, a mouse model of advanced retinal degeneration. Retinal function was assessed in vitro with multielectrode array recordings, and in vivo with electrophysiological recordings from the visual thalamus. Behavioural studies were developed to test the visual responses in treated mice under a range of light conditions. FindingsEctopically expressed human rod opsin, driven by either a non-selective or ON-bipolar cell-specific promoter, functioned outside native photoreceptors and restored visual function in rd1 mice. Electrophysiological recordings from retinal explants and the visual thalamus revealed changes in firing induced by simple light pulses, luminance increases, and naturalistic movies in treated mice. These responses could be elicited at light intensities within the physiological range and substantially below those required by other optogenetic strategies. Mice with rod opsin expression driven by the ON-bipolar specific promoter displayed behavioural responses to increases in luminance, flicker, coarse spatial patterns, and elements of a natural movie at natural levels of contrast and illuminance. InterpretationThese data reveal that virally mediated ectopic expression of human rod opsin can restore vision under natural viewing conditions and at moderate light intensities. Given the inherent advantages of using a human protein, the simplicity of this intervention, and the quality of vision restored, we suggest that rod opsin merits consideration as an optogenetic actuator for treating patients with advanced retinal degeneration. FundingMedical Research Council (clinical research training fellowship to JCK), Medical Research Council Confidence in Concept Award, European Research Council.

Eye movement-invariant representations in the human visual system.

During natural vision, humans make frequent eye movements but perceive a stable visual world. It is therefore likely that the human visual system contains representations of the visual world that are invariant to eye movements. Here we present an experiment designed to identify visual areas that might contain eye-movement-invariant representations. We used functional MRI to record brain activity from four human subjects who watched natural movies. In one condition subjects were required to fixate steadily, and in the other they were allowed to freely make voluntary eye movements. The movies used in each condition were identical. We reasoned that the brain activity recorded in a visual area that is invariant to eye movement should be similar under fixation and free viewing conditions. In contrast, activity in a visual area that is sensitive to eye movement should differ between fixation and free viewing. We therefore measured the similarity of brain activity across repeated presentations of the same movie within the fixation condition, and separately between the fixation and free viewing conditions. The ratio of these measures was used to determine which brain areas are most likely to contain eye movement-invariant representations. We found that voxels located in early visual areas are strongly affected by eye movements, while voxels in ventral temporal areas are only weakly affected by eye movements. These results suggest that the ventral temporal visual areas contain a stable representation of the visual world that is invariant to eye movements made during natural vision.

Pyrcca: Regularized Kernel Canonical Correlation Analysis in Python and Its Applications to Neuroimaging.

In this article we introduce Pyrcca, an open-source Python package for performing canonical correlation analysis (CCA). CCA is a multivariate analysis method for identifying relationships between sets of variables. Pyrcca supports CCA with or without regularization, and with or without linear, polynomial, or Gaussian kernelization. We first use an abstract example to describe Pyrcca functionality. We then demonstrate how Pyrcca can be used to analyze neuroimaging data. Specifically, we use Pyrcca to implement cross-subject comparison in a natural movie functional magnetic resonance imaging (fMRI) experiment by finding a data-driven set of functional response patterns that are similar across individuals. We validate this cross-subject comparison method in Pyrcca by predicting responses to novel natural movies across subjects. Finally, we show how Pyrcca can reveal retinotopic organization in brain responses to natural movies without the need for an explicit model.

Decoding the Semantic Content of Natural Movies from Human Brain Activity.

One crucial test for any quantitative model of the brain is to show that the model can be used to accurately decode information from evoked brain activity. Several recent neuroimaging studies have decoded the structure or semantic content of static visual images from human brain activity. Here we present a decoding algorithm that makes it possible to decode detailed information about the object and action categories present in natural movies from human brain activity signals measured by functional MRI. Decoding is accomplished using a hierarchical logistic regression (HLR) model that is based on labels that were manually assigned from the WordNet semantic taxonomy. This model makes it possible to simultaneously decode information about both specific and general categories, while respecting the relationships between them. Our results show that we can decode the presence of many object and action categories from averaged blood-oxygen level-dependent (BOLD) responses with a high degree of accuracy (area under the ROC curve > 0.9). Furthermore, we used this framework to test whether semantic relationships defined in the WordNet taxonomy are represented the same way in the human brain. This analysis showed that hierarchical relationships between general categories and atypical examples, such as organism and plant, did not seem to be reflected in representations measured by BOLD fMRI.

Functional Subdomains within Scene-Selective Cortex: Parahippocampal Place Area, Retrosplenial Complex, and Occipital Place Area.

Visual scene perception is a critical ability to survive in the real world. It is therefore reasonable to assume that the human brain contains neural circuitry selective for visual scenes. Here we show that responses in three scene-selective areas-identified in previous studies-carry information about many object and action categories encountered in daily life. We identify two subregions in each area: one that is selective for categories of man-made objects, and another that is selective for vehicles and locomotion-related action categories that appear in dynamic scenes. This consistent functional subdivision may reflect an anatomical organization into two processing streams, one biased toward static stimuli and one biased toward dynamic stimuli.

Specificity and timescales of cortical adaptation as inferences about natural movie statistics.

Adaptation is a phenomenological umbrella term under which a variety of temporal contextual effects are grouped. Previous models have shown that some aspects of visual adaptation reflect optimal processing of dynamic visual inputs, suggesting that adaptation should be tuned to the properties of natural visual inputs. However, the link between natural dynamic inputs and adaptation is poorly understood. Here, we extend a previously developed Bayesian modeling framework for spatial contextual effects to the temporal domain. The model learns temporal statistical regularities of natural movies and links these statistics to adaptation in primary visual cortex via divisive normalization, a ubiquitous neural computation. In particular, the model divisively normalizes the present visual input by the past visual inputs only to the degree that these are inferred to be statistically dependent. We show that this flexible form of normalization reproduces classical findings on how brief adaptation affects neuronal selectivity. Furthermore, prior knowledge acquired by the Bayesian model from natural movies can be modified by prolonged exposure to novel visual stimuli. We show that this updating can explain classical results on contrast adaptation. We also simulate the recent finding that adaptation maintains population homeostasis, namely, a balanced level of activity across a population of neurons with different orientation preferences. Consistent with previous disparate observations, our work further clarifies the influence of stimulus-specific and neuronal-specific normalization signals in adaptation.

Internal vs. external determinants of human speed discrimination with natural image movies

Internal vs. external determinants of human speed discrimination with natural image movies