Visual Scene Information Research Articles

Factorized visual representations in the primate visual system and deep neural networks.

Object classification has been proposed as a principal objective of the primate ventral visual stream and has been used as an optimization target for deep neural network models (DNNs) of the visual system. However, visual brain areas represent many different types of information, and optimizing for classification of object identity alone does not constrain how other information may be encoded in visual representations. Information about different scene parameters may be discarded altogether ('invariance'), represented in non-interfering subspaces of population activity ('factorization') or encoded in an entangled fashion. In this work, we provide evidence that factorization is a normative principle of biological visual representations. In the monkey ventral visual hierarchy, we found that factorization of object pose and background information from object identity increased in higher-level regions and strongly contributed to improving object identity decoding performance. We then conducted a large-scale analysis of factorization of individual scene parameters - lighting, background, camera viewpoint, and object pose - in a diverse library of DNN models of the visual system. Models which best matched neural, fMRI, and behavioral data from both monkeys and humans across 12 datasets tended to be those which factorized scene parameters most strongly. Notably, invariance to these parameters was not as consistently associated with matches to neural and behavioral data, suggesting that maintaining non-class information in factorized activity subspaces is often preferred to dropping it altogether. Thus, we propose that factorization of visual scene information is a widely used strategy in brains and DNN models thereof.

Testing a Vision-Based Autonomous Drone Navigation Model in a Forest Environment

Drones play a pivotal role in various industries of Industry 4.0. For achieving the application of drones in a dynamic environment, finding a clear path for their autonomous flight requires more research. This paper addresses the problem of finding a navigation path for an autonomous drone based on visual scene information. A deep learning-based object detection approach can localize obstacles detected in a scene. Considering this approach, we propose a solution framework that includes masking with a color-based segmentation method to identify an empty area where the drone can fly. The scene is described using segmented regions and localization points. The proposed approach can be used to remotely guide drones in dynamic environments that have poor coverage from global positioning systems. The simulation results show that the proposed framework with object detection and the proposed masking technique support drone navigation in a dynamic environment based only on the visual input from the front field of view.

3SHNet: Boosting image–sentence retrieval via visual semantic–spatial self-highlighting

In this paper, we propose a novel visual Semantic-Spatial Self-Highlighting Network (termed 3SHNet) for high-precision, high-efficiency and high-generalization image–sentence retrieval. 3SHNet highlights the salient identification of prominent objects and their spatial locations within the visual modality, thus allowing the integration of visual semantics–spatial interactions and maintaining independence between two modalities. This integration effectively combines object regions with the corresponding semantic and position layouts derived from segmentation to enhance the visual representation. And the modality-independence guarantees efficiency and generalization. Additionally, 3SHNet utilizes the structured contextual visual scene information from segmentation to conduct the local (region-based) or global (grid-based) guidance and achieve accurate hybrid-level retrieval. Extensive experiments conducted on MS-COCO and Flickr30K benchmarks substantiate the superior performances, inference efficiency and generalization of the proposed 3SHNet when juxtaposed with contemporary state-of-the-art methodologies. Specifically, on the larger MS-COCO 5K test set, we achieve 16.3%, 24.8%, and 18.3% improvements in terms of rSum score, respectively, compared with the state-of-the-art methods using different image representations, while maintaining optimal retrieval efficiency. Moreover, our performance on cross-dataset generalization improves by 18.6%.

A Mobile Image Aesthetics Processing System with Intelligent Scene Perception

Image aesthetics processing (IAP) is used primarily to enhance the aesthetic quality of images. However, IAP faces several issues, including its failure to analyze the influence of visual scene information and the difficulty of deploying IAP capabilities to mobile devices. This study proposes an automatic IAP system (IAPS) for mobile devices that integrates machine learning and traditional image-processing methods. First, we employ an extremely computation-efficient deep learning model, ShuffleNet, designed for mobile devices as our scene recognition model. Then, to enable computational inferencing on resource-constrained edge devices, we use a modern mobile machine-learning library, TensorFlow Lite, to convert the model type to TFLite format. Subsequently, we adjust the image contrast and color saturation using group filtering, respectively. These methods enable us to achieve maximal aesthetic enhancement of images with minimal parameter adjustments. Finally, we use the InceptionResNet-v2 aesthetic evaluation model to rate the images. Even when employing the benchmark model with an accuracy of 70%, the score of the IAPS processing image is verified to be higher and more effective compared with a state-of-the-art smartphone’s beautification function. Additionally, an anonymous questionnaire survey with 100 participants is conducted, and the result shows that IAPS enhances the aesthetic appeal of images based on the public’s preferences.

The Development of Human Cortical Scene Processing

Decades of research have uncovered the neural basis of place (or “scene”) processing in adulthood, revealing a set of three regions that respond selectively to visual scene information, each hypothesized to support distinct functions within scene processing (e.g., recognizing a particular kind of place versus navigating through it). Despite this considerable progress, surprisingly little is known about how these cortical regions develop. Here we review the limited evidence to date, highlighting the first few studies exploring the origins of cortical scene processing in infancy and the several studies addressing when the scene regions reach full maturity, unfortunately with inconsistent findings. This inconsistency likely stems from common pitfalls in pediatric functional magnetic resonance imaging, and accordingly, we discuss how these pitfalls may be avoided. Furthermore, we point out that almost all studies to date have focused only on general scene selectivity and argue that greater insight could be gleaned by instead exploring the more distinct functions of each region as well as their connectivity. Finally, with this last point in mind, we offer a novel hypothesis that scene regions supporting navigation (including the occipital place area and retrosplenial complex) mature later than those supporting scene categorization (including the parahippocampal place area).

No-Reference Video Quality Assessment Using Local Structural and Quality-Aware Deep Features

Due to the growing demand for high-quality video services in 4G and 5G applications, measuring the quantitative quality of video services is expected to become a major vital task. The no-reference video quality assessment (NR-VQA) work published so far regresses computationally complex statistical transforms or convolutional neural network (CNN) features to predict a quality score. In this paper, we propose a novel NR-VQA scheme using systematic sampling of spatiotemporal planes (XY, XT, and YT) based on the high standard deviation (σ) of their high-frequency bands to represent distortion. Human visual system (HVS) is highly sensitive to structural information in visual scenes, and distortions disrupt the structural properties. The proposed scheme encodes two-level, three-dimensional structural video information using novel Local Spatiotemporal Tetra Patterns (LSTP) on the sampled highest σ planes from each block of planes. Besides, we extract quality-aware deep features from the second highest σ sampled video frames (XY-spatial) from each block using a fine-tuned CNN model. The extracted LSTP and deep quality-aware features of the two highest σ frames are average pooled and concatenated with the top hundred σ values of other frames to form video-level final features. Finally, the concatenated features are fed to a support vector regressor (SVR) to predict the perceptual quality scores of test videos. The proposed method is evaluated on ten publicly available standard exhaustive VQA databases containing synthetic, authentic, and mixed distortions. Comprehensive, robust, and extensive experiments indicate that the proposed model outperforms all the state-of-the-art VQA models and is consistent with human subjective assessment.

Fast, Multi-Bit, and Vis-Infrared Broadband Nonvolatile Optoelectronic Memory with MoS2 /2D-Perovskite Van der Waals Heterojunction.

Nonvolatile optoelectronic memory (NVOM) integrating the functions of optical sensing and long-term memory can efficiently process and store a large amount of visual scene information, which has become the core requirement of multiple intelligence scenarios. However, realizing NVOM with vis-infrared broadband response is still challenging. Herein, the room temperature vis-infrared broadband NVOM based on few-layer MoS2 /2D Ruddlesden-Popper perovskite (2D-RPP) van der Waals heterojunction is realized. It is found that the 2D-RPP converts the initial n-type MoS2 into p-type and facilitates hole transfer between them. Furthermore, the 2D-RPP rich in interband states serves as an effective electron trapping layer as well as broadband photoresponsive layer. As a result, the dielectric-free MoS2 /2D-RPP heterojunction enables the charge to transfer quickly under external field, which enables a large memory window (104V), fast write speed of 20 µs, and optical programmable characteristics from visible light (405nm) to telecommunication wavelengths (i.e., 1550nm) at room temperature. Trapezoidal optical programming can produce up to 100 recognizable states (>6 bits), with operating energy as low as 5.1 pJ per optical program. These results provide a route to realize fast, low power, multi-bit optoelectronic memory from visible to the infrared wavelength.

Full-field fMRI: a novel approach to study immersive vision

In everyday vision, we experience an >180 degree view of the world in front of us. However, traditional functional magnetic resonance imaging (fMRI) setups are limited to presenting scenes like postcards in the central 10-15 degrees of the visual field. Here, we develop a method for ultra-wide angle visual presentation in the scanner, and explore how the brain processes visual scene information when presented with immersive first-person views. To accomplish wide-angle projection, we bounced the image off two angled mirrors directly into the scanner bore onto a custom-built curved screen, creating an unobstructed visual presentation of over 175 degrees. Additionally, we presented images that depicted a compatible wide field-of-view, rendered from 3D scenes built in Unity software; using standard scene images led to distorted perceptions of the environment. With this setup, we measured brain responses to a range of stimuli, including scene images presented in the full-field and at a typical smaller visual size. We found that all classic scene areas (parahippocampal place area, retrosplenial cortex, and occipital place area) were activated significantly more to the full-field scenes compared to the postcard scenes, indicating their preference for the far-periphery. Crucially, we found that a large swath of cortex connecting these areas was also strongly activated by the full-field more than the postcard scenes, forming a ring-shape around the parieto-occipital sulcus. Theoretically, these findings raise an intriguing possibility that there are representational principles unifying what are currently considered separate scene-selective regions within a common large-scale organization. Methodologically, our approach provides a novel avenue to test hypotheses relating to the foveal-peripheral organization of higher-level visual areas, and measure scene processing mechanisms with an immersive experience of scale.

Visual object detection biases escape trajectories following acoustic startle in larval zebrafish

In this study, we investigated whether the larval zebrafish is sensitive to the presence of obstacles in its environment. Zebrafish execute fast escape swims when in danger of predation. We posited that collisions with solid objects during escape would be maladaptive to the fish, and therefore, the direction of escape swims should be informed by the locations of barriers. To test this idea, we developed a closed-loop imaging rig outfitted with barriers of various qualities. We show that when larval zebrafish escape in response to a non-directional vibrational stimulus, they use visual scene information to avoid collisions with obstacles. Our study demonstrates that barrier avoidance rate corresponds to the absolute distance of obstacles, as distant barriers outside of collision range elicit less bias than nearby collidable barriers that occupy the same amount of visual field. The computation of barrier avoidance is covert: the fact that fish will avoid barriers during escape cannot be predicted by its routine swimming behavior in the barrier arena. Finally, two-photon laser ablation experiments suggest that excitatory bias is provided to the Mauthner cell ipsilateral to approached barriers, either via direct excitation or a multi-step modulation process. We ultimately propose that zebrafish detect collidable objects via an integrative visual computation that is more complex than retinal occupancy alone, laying a groundwork for understanding how cognitive physical models observed in humans are implemented in an archetypal vertebrate brain. VIDEO ABSTRACT.

Submillimeter fMRI reveals an extensive, fine-grained and functionally-relevant scene-processing network in monkeys.

Primates are endowed with a dedicated cortical network for processing visual scene information, which is critical for navigation and object retrieval. Previous studies showed that this scene network encompasses three to maximally five cortical regions in humans and monkeys. Using submillimeter resolution fMRI (0.22 mm3 voxels), and two entirely different but carefully controlled stimulus sets, we demonstrate a robust, fine-grained, yet three-fold more extensive scene-processing network in macaques compared to previous studies. The core network, selective for both familiar and unfamiliar scenes, encompasses eleven patches distributed over all cerebral lobes and is surprisingly elaborated in frontal cortex. Five additional non-core scene-selective patches show scene selectivity, but only for places familiar to the monkeys. Notably, resting-state fMRI revealed that the frontal and temporo-parietal scene-selective patches form an intrinsically-connected network, largely segregated from other category-selective networks. Moreover, the strength of the functional connectivity across nodes of the network is a predictor of functional scene responses of nodes belonging to this network. Hence, this scene processing network is functionally-relevant. In summary, the scene-processing system is considerably more complex than previously documented, consisting of functionally interconnected patches throughout all cortical lobes.

Scene image and human skeleton-based dual-stream human action recognition

The dual stream-based human action recognition model offers the advantage of high recognition accuracy, but the algorithm is less robust in case of lighting changes. The human skeleton has a strong ability to express human behavior and actions; however, the scene information is ignored. Drawing on the idea of the dual-stream model, this paper proposes a human skeleton and scene image-based dual-stream model for human action recognition. The motion features are extracted through the spatio-temporal graph convolution of the human skeleton, and a scene recognition model is proposed based on the sparse frame sampling of video and video-level consensus strategy to process the scene video and gather the visual scene information. The proposed model exploits the advantages of skeleton information in motion expression and the superiority of the image in scene presentation. The scene information and spatio-temporal graph convolution-based human skeleton limbs are fused complementarily to achieve human action recognition. Compared to the conventional optical flow-based dual-stream action recognition method, this model is verified by experimenting under unstable light conditions, and the performance of human action recognition is robust and promising.

Open Domain Dialogue Generation with Latent Images

We consider grounding open domain dialogues with images. Existing work assumes that both an image and a textual context are available, but image-grounded dialogues by nature are more difficult to obtain than textual dialogues. Thus, we propose learning a response generation model with both image-grounded dialogues and textual dialogues by assuming that the visual scene information at the time of a conversation can be represented by an image, and trying to recover the latent images of the textual dialogues through text-to-image generation techniques. The likelihood of the two types of dialogues is then formulated by a response generator and an image reconstructor that are learned within a conditional variational auto-encoding framework. Empirical studies are conducted in both image-grounded conversation and text-based conversation. In the first scenario, image-grounded dialogues, especially under a low-resource setting, can be effectively augmented by textual dialogues with latent images; while in the second scenario, latent images can enrich the content of responses and at the same time keep them relevant to contexts.

Ganglion Cells with Sustained Activity in the Fish Retina and Their Possible Function in Evaluation of Visual Scenes

Background extracellular spike activity of single ganglion cells was recorded from axon terminals in the optic tectum of living immobilized fish. The sizes of the receptive fields of ON and OFF units with sustained responses (USR) amounted to 4–5° and were comparable with those of feature detectors. Generation of spike discharges by USR required contrast between the center and periphery of the receptive field. When there was no contrast, no spike activity appeared. The magnitude of the reaction was monotonically dependent on the level of this contrast. USR of both the ON and OFF types were connected with three types of cone (L, M, S). Both the center and periphery of the receptive field displayed color opponency, the center and periphery of the receptive field being opponent in terms of this characteristic. In other words, USR were double opponent and may thus take part in color discrimination. The simultaneous operation of feature detectors and ganglion cells with baseline activity separated into ON and OFF channels is represented retinotopically and may provide tectum opticum neurons with the visual scene information required for their function of controlling external attention.

M37. Automatic Visual Processing of Social Information in Schizophrenia

Background: It is well documented that people with Schizophrenia perform poorly on explicit tests of social cognition. Less is known of implicit social cognition in this disorder. Our group has previously found that people with Schizophrenia demonstrate reduced automatic attentional capture by visual social information (away from fixation). We aimed to advance this line of research using visual images that were balanced with regard to their social and nonsocial content. Methods: Twenty people with Schizophrenia (15 male, 5 female) and 20 healthy control participants (age and gender matched) passively viewed natural scenes that contained both people and nonsocial objects equidistant from an initial fixation point. Participants’ eye movements were tracked using an Eyelink 1000 gaze-monitoring system as they viewed photographs. Results: Eye-tracking data revealed no group differences in time taken to make the first eye-movement away from fixation. Overall, participants with Schizophrenia spent significantly less time looking at the social information in the photograph when compared to controls. They also showed a trend towards initially fixating on the nonsocial information more than controls. Conclusion: Results suggest that social information in visual scenes is not as automatically salient for people with Schizophrenia compared to controls.

Scene Construction, Visual Foraging, and Active Inference.

This paper describes an active inference scheme for visual searches and the perceptual synthesis entailed by scene construction. Active inference assumes that perception and action minimize variational free energy, where actions are selected to minimize the free energy expected in the future. This assumption generalizes risk-sensitive control and expected utility theory to include epistemic value; namely, the value (or salience) of information inherent in resolving uncertainty about the causes of ambiguous cues or outcomes. Here, we apply active inference to saccadic searches of a visual scene. We consider the (difficult) problem of categorizing a scene, based on the spatial relationship among visual objects where, crucially, visual cues are sampled myopically through a sequence of saccadic eye movements. This means that evidence for competing hypotheses about the scene has to be accumulated sequentially, calling upon both prediction (planning) and postdiction (memory). Our aim is to highlight some simple but fundamental aspects of the requisite functional anatomy; namely, the link between approximate Bayesian inference under mean field assumptions and functional segregation in the visual cortex. This link rests upon the (neurobiologically plausible) process theory that accompanies the normative formulation of active inference for Markov decision processes. In future work, we hope to use this scheme to model empirical saccadic searches and identify the prior beliefs that underwrite intersubject variability in the way people forage for information in visual scenes (e.g., in schizophrenia).

Category-specific learned attentional bias to object parts

Humans can selectively attend to information in visual scenes. Learning from previous experiences plays a role in how visual attention is subsequently deployed. For example, visual search times are faster in areas that are statistically more likely to contain a target (Jiang and Swallow in Cognition, 126(3), 378-390, 2013). Here, we examined whether similar attentional biases can be created for different locations on complex objects as a function of their category, based on a history of these locations containing a target. Subjects performed a visual search task in the context of novel objects called Greebles. The target appeared in one half (e.g., top) of the Greebles 89 % of the time and in the other half (e.g., bottom) 11 % of the time. We found a reaction time advantage when the target was located in a "target-rich" region, even after target location probabilities were equated. This indicates that attentional biases can be associated not only with regions of space but also with specific object features, or at least with locations in an object-based frame of reference.

Egocentric Direction and Position Perceptions are Dissociable Based on Only Static Lane Edge Information.

When observers perceive several objects in a space, at the same time, they should effectively perceive their own position as a viewpoint. However, little is known about observers’ percepts of their own spatial location based on the visual scene information viewed from them. Previous studies indicate that two distinct visual spatial processes exist in the locomotion situation: the egocentric position perception and egocentric direction perception. Those studies examined such perceptions in information rich visual environments where much dynamic and static visual information was available. This study examined these two perceptions in information of impoverished environments, including only static lane edge information (i.e., limited information). We investigated the visual factors associated with static lane edge information that may affect these perceptions. Especially, we examined the effects of the two factors on egocentric direction and position perceptions. One is the “uprightness factor” that “far” visual information is seen at upper location than “near” visual information. The other is the “central vision factor” that observers usually look at “far” visual information using central vision (i.e., foveal vision) whereas ‘near’ visual information using peripheral vision. Experiment 1 examined the effect of the “uprightness factor” using normal and inverted road images. Experiment 2 examined the effect of the “central vision factor” using normal and transposed road images where the upper half of the normal image was presented under the lower half. Experiment 3 aimed to replicate the results of Experiments 1 and 2. Results showed that egocentric direction perception is interfered with image inversion or image transposition, whereas egocentric position perception is robust against these image transformations. That is, both “uprightness” and “central vision” factors are important for egocentric direction perception, but not for egocentric position perception. Therefore, the two visual spatial perceptions about observers’ own viewpoints are fundamentally dissociable.

Contextual Feedback to Superficial Layers of V1

SummaryNeuronal cortical circuitry comprises feedforward, lateral, and feedback projections, each of which terminates in distinct cortical layers [1–3]. In sensory systems, feedforward processing transmits signals from the external world into the cortex, whereas feedback pathways signal the brain’s inference of the world [4–11]. However, the integration of feedforward, lateral, and feedback inputs within each cortical area impedes the investigation of feedback, and to date, no technique has isolated the feedback of visual scene information in distinct layers of healthy human cortex. We masked feedforward input to a region of V1 cortex and studied the remaining internal processing. Using high-resolution functional brain imaging (0.8 mm3) and multivoxel pattern information techniques, we demonstrate that during normal visual stimulation scene information peaks in mid-layers. Conversely, we found that contextual feedback information peaks in outer, superficial layers. Further, we found that shifting the position of the visual scene surrounding the mask parametrically modulates feedback in superficial layers of V1. Our results reveal the layered cortical organization of external versus internal visual processing streams during perception in healthy human subjects. We provide empirical support for theoretical feedback models such as predictive coding [10, 12] and coherent infomax [13] and reveal the potential of high-resolution fMRI to access internal processing in sub-millimeter human cortex.

Intrinsic and extrinsic contributions to heavy tails in visual foraging

Eyes move over visual scenes to gather visual information. Studies have found heavy-tailed distributions in measures of eye movements during visual search, which raises questions about whether these distributions are pervasive to eye movements, and whether they arise from intrinsic or extrinsic factors. Three different measures of eye movement trajectories were examined during visual foraging of complex images, and all three were found to exhibit heavy tails: Spatial clustering of eye movements followed a power law distribution, saccade length distributions were lognormally distributed, and the speeds of slow, small amplitude movements occurring during fixations followed a 1/f spectral power law relation. Images were varied to test whether the spatial clustering of visual scene information is responsible for heavy tails in eye movements. Spatial clustering of eye movements and saccade length distributions were found to vary with image type and task demands, but no such effects were found for eye movement speeds during fixations. Results showed that heavy-tailed distributions are general and intrinsic to visual foraging, but some of them become aligned with visual stimuli when required by task demands. The potentially adaptive value of heavy-tailed distributions in visual foraging is discussed.

Bottom-up attention orienting in young children with autism.

We examined the impact of simultaneous bottom-up visual influences and meaningful social stimuli on attention orienting in young children with autism spectrum disorders (ASDs). Relative to typically-developing age and sex matched participants, children with ASDs were more influenced by bottom-up visual scene information regardless of whether social stimuli and bottom-up scene properties were congruent or competing. This initial reliance on bottom-up strategies correlated with severity of social impairment as well as receptive language impairments. These data provide support for the idea that there is enhanced reliance on bottom-up attention strategies in ASDs, and that this may have a negative impact on social and language development.