Raw Visual Input Research Articles

Invited Session III: Neural network models of the visual system: Reverse-engineering neural code in the language of objects and generative models.

When we open our eyes, we do not see a jumble of light or colorful patterns. There lies a great distance from the raw inputs sensed at our retinas to what we experience as the contents of our perception. How in the brain are incoming sense inputs transformed into rich, discrete structures that we can think about and plan with? These "world models" include representations of objects with kinematic and dynamical properties, scenes with navigational affordances, and events with temporally demarcated dynamics. Real world scenes are complex, but given a momentary task, only a fraction of this complexity is relevant to the observer. Attention allows us to selectively form these world models, driving flexible action as task-driven, simulatable state-spaces. How in the mind and brain do we build and use such internal models of the world from raw visual inputs? In this talk, I will begin to address this question by presenting two new computational modeling frameworks. First, in high-level vision, I will show how we can reverse-engineer population-level neural activity in the macaque visual cortex in the language of three-dimensional objects and computer graphics, by combining generative models with deep neural networks. Second, I will present a novel account of attention based on adaptive computation that situates vision in the broader context of an agent with goals, and show how it explains internal representations and implicit goals underlying the selectivity of scene perception.

A Time-Delay Feedback Neural Network for Discriminating Small, Fast-Moving Targets in Complex Dynamic Environments.

Discriminating small moving objects within complex visual environments is a significant challenge for autonomous micro-robots that are generally limited in computational power. By exploiting their highly evolved visual systems, flying insects can effectively detect mates and track prey during rapid pursuits, even though the small targets equate to only a few pixels in their visual field. The high degree of sensitivity to small target movement is supported by a class of specialized neurons called small target motion detectors (STMDs). Existing STMD-based computational models normally comprise four sequentially arranged neural layers interconnected via feedforward loops to extract information on small target motion from raw visual inputs. However, feedback, another important regulatory circuit for motion perception, has not been investigated in the STMD pathway and its functional roles for small target motion detection are not clear. In this article, we propose an STMD-based neural network with feedback connection (feedback STMD), where the network output is temporally delayed, then fed back to the lower layers to mediate neural responses. We compare the properties of the model with and without the time-delay feedback loop and find that it shows a preference for high-velocity objects. Extensive experiments suggest that the feedback STMD achieves superior detection performance for fast-moving small targets, while significantly suppressing background false positive movements which display lower velocities. The proposed feedback model provides an effective solution in robotic visual systems for detecting fast-moving small targets that are always salient and potentially threatening.

A target-driven visual navigation method based on intrinsic motivation exploration and space topological cognition

Target-driven visual navigation is essential for many applications in robotics, and it has gained increasing interest in recent years. In this work, inspired by animal cognitive mechanisms, we propose a novel navigation architecture that simultaneously learns exploration policy and encodes environmental structure. First, to learn exploration policy directly from raw visual input, we use deep reinforcement learning as the basic framework and allow agents to create rewards for themselves as learning signals. In our approach, the reward for the current observation is driven by curiosity and calculated by a count-based approach and temporal distance. While agents learn exploration policy, we use temporal distance to find waypoints in observation sequences and incrementally describe the structure of the environment in a way that integrates episodic memory. Finally, space topological cognition is integrated into the model as a path planning module and combined with a locomotion network to obtain a more generalized approach to navigation. We test our approach in the DMlab, a visually rich 3D environment, and validate its exploration efficiency and navigation performance through extensive experiments. The experimental results show that our approach can explore and encode the environment more efficiently and has better capability in dealing with stochastic objects. In navigation tasks, agents can use space topological cognition to effectively reach the target and guide detour behaviour when a path is unavailable, exhibiting good environmental adaptability.

Deep Predictive Learning in Neocortex and Pulvinar.

How do humans learn from raw sensory experience? Throughout life, but most obviously in infancy, we learn without explicit instruction. We propose a detailed biological mechanism for the widely embraced idea that learning is driven by the differences between predictions and actual outcomes (i.e., predictive error-driven learning). Specifically, numerous weak projections into the pulvinar nucleus of the thalamus generate top-down predictions, and sparse driver inputs from lower areas supply the actual outcome, originating in Layer 5 intrinsic bursting neurons. Thus, the outcome representation is only briefly activated, roughly every 100 msec (i.e., 10 Hz, alpha), resulting in a temporal difference error signal, which drives local synaptic changes throughout the neocortex. This results in a biologically plausible form of error backpropagation learning. We implemented these mechanisms in a large-scale model of the visual system and found that the simulated inferotemporal pathway learns to systematically categorize 3-D objects according to invariant shape properties, based solely on predictive learning from raw visual inputs. These categories match human judgments on the same stimuli and are consistent with neural representations in inferotemporal cortex in primates.

Context-dependent extinction learning emerging from raw sensory inputs: a reinforcement learning approach

The context-dependence of extinction learning has been well studied and requires the hippocampus. However, the underlying neural mechanisms are still poorly understood. Using memory-driven reinforcement learning and deep neural networks, we developed a model that learns to navigate autonomously in biologically realistic virtual reality environments based on raw camera inputs alone. Neither is context represented explicitly in our model, nor is context change signaled. We find that memory-intact agents learn distinct context representations, and develop ABA renewal, whereas memory-impaired agents do not. These findings reproduce the behavior of control and hippocampal animals, respectively. We therefore propose that the role of the hippocampus in the context-dependence of extinction learning might stem from its function in episodic-like memory and not in context-representation per se. We conclude that context-dependence can emerge from raw visual inputs.

Efficient policy detecting and reusing for non-stationarity in Markov games

One challenging problem in multiagent systems is to cooperate or compete with non-stationary agents that change behavior from time to time. An agent in such a non-stationary environment is usually supposed to be able to quickly detect the other agents’ policy during online interaction, and then adapt its own policy accordingly. This article studies efficient policy detecting and reusing techniques when playing against non-stationary agents in cooperative or competitive Markov games. We propose a new deep Bayesian policy reuse algorithm, a.k.a. DPN-BPR+, by extending the recent BPR+ algorithm with a neural network as the value-function approximator. To detect policy accurately, we propose the rectified belief model taking advantage of the opponent model to infer the other agents’ policy from reward signals and its behavior. Instead of directly storing individual policies as BPR+, we introduce distilled policy network that serves as the policy library, and policy distillation to achieve efficient online policy learning and reuse. DPN-BPR+ inherits all the advantages of BPR+. In experiments, we evaluate DPN-BPR+ in terms of detection accuracy, cumulative reward and speed of convergence in four complex Markov games with raw visual inputs, including two cooperative games and two competitive games. Empirical results show that our proposed DPN-BPR+ approach has better performance than existing algorithms in all these Markov games.

Learning Semantic Keypoint Representations for Door Opening Manipulation

We consider a scenario where a robot is capable of autonomously opening previously unseen doors. Prior works either use model-based methods that rely strongly on accurate kinematic models, or learn a policy from scratch through trial-and-error, which cannot generalize to large variations in shape, and location of doors. In this letter, we propose a novel method for opening unseen doors with no prior knowledge of door model, which leverages semantic 3D keypoints as door handle representations to generate the end-effector trajectory from a motion planner. The keypoint representations are predicted from raw visual input by a deep neural network, which can provide a concise, and semantic description of the handle to determine the grasp pose, and subsequent motion planning. In contrast to existing works that require known object models or significant manual effort on data collection, we present a data augmentation technique to automatically generate large amounts of realistic-looking synthetic data with almost no human labeling effort. An augmented dataset, consisting of large amounts of synthetic data, and small amounts of real data, is used to train the network. Qualitative results show that our proposed method outperforms the state-of-the-art pose-based methods on real test sets in terms of perception metrics. Hardware experiments demonstrate that our proposed method can achieve 94.2% success rate on opening 6 previously unseen doors with significant shape variations under different environments, and conditions.

Efficient Multiagent Policy Optimization Based on Weighted Estimators in Stochastic Cooperative Environments

Multiagent deep reinforcement learning (MA-DRL) has received increasingly wide attention. Most of the existing MA-DRL algorithms, however, are still inefficient when faced with the non-stationarity due to agents changing behavior consistently in stochastic environments. This paper extends the weighted double estimator to multiagent domains and proposes an MA-DRL framework, named Weighted Double Deep Q-Network (WDDQN). By leveraging the weighted double estimator and the deep neural network, WDDQN can not only reduce the bias effectively but also handle scenarios with raw visual inputs. To achieve efficient cooperation in multiagent domains, we introduce a lenient reward network and scheduled replay strategy. Empirical results show that WDDQN outperforms an existing DRL algorithm (double DQN) and an MA-DRL algorithm (lenient Q-learning) regarding the averaged reward and the convergence speed and is more likely to converge to the Pareto-optimal Nash equilibrium in stochastic cooperative environments.

Deep intrinsically motivated continuous actor-critic for efficient robotic visuomotor skill learning

Abstract In this paper, we present a new intrinsically motivated actor-critic algorithm for learning continuous motor skills directly from raw visual input. Our neural architecture is composed of a critic and an actor network. Both networks receive the hidden representation of a deep convolutional autoencoder which is trained to reconstruct the visual input, while the centre-most hidden representation is also optimized to estimate the state value. Separately, an ensemble of predictive world models generates, based on its learning progress, an intrinsic reward signal which is combined with the extrinsic reward to guide the exploration of the actor-critic learner. Our approach is more data-efficient and inherently more stable than the existing actor-critic methods for continuous control from pixel data. We evaluate our algorithm for the task of learning robotic reaching and grasping skills on a realistic physics simulator and on a humanoid robot. The results show that the control policies learned with our approach can achieve better performance than the compared state-of-the-art and baseline algorithms in both dense-reward and challenging sparse-reward settings.

Deep imitation learning for 3D navigation tasks

Deep learning techniques have shown success in learning from raw high-dimensional data in various applications. While deep reinforcement learning is recently gaining popularity as a method to train intelligent agents, utilizing deep learning in imitation learning has been scarcely explored. Imitation learning can be an efficient method to teach intelligent agents by providing a set of demonstrations to learn from. However, generalizing to situations that are not represented in the demonstrations can be challenging, especially in 3D environments. In this paper, we propose a deep imitation learning method to learn navigation tasks from demonstrations in a 3D environment. The supervised policy is refined using active learning in order to generalize to unseen situations. This approach is compared to two popular deep reinforcement learning techniques: deep-Q-networks and Asynchronous actor-critic (A3C). The proposed method as well as the reinforcement learning methods employ deep convolutional neural networks and learn directly from raw visual input. Methods for combining learning from demonstrations and experience are also investigated. This combination aims to join the generalization ability of learning by experience with the efficiency of learning by imitation. The proposed methods are evaluated on 4 navigation tasks in a 3D simulated environment. Navigation tasks are a typical problem that is relevant to many real applications. They pose the challenge of requiring demonstrations of long trajectories to reach the target and only providing delayed rewards (usually terminal) to the agent. The experiments show that the proposed method can successfully learn navigation tasks from raw visual input while learning from experience methods fail to learn an effective policy. Moreover, it is shown that active learning can significantly improve the performance of the initially learned policy using a small number of active samples.

Multiagent cooperation and competition with deep reinforcement learning

Evolution of cooperation and competition can appear when multiple adaptive agents share a biological, social, or technological niche. In the present work we study how cooperation and competition emerge between autonomous agents that learn by reinforcement while using only their raw visual input as the state representation. In particular, we extend the Deep Q-Learning framework to multiagent environments to investigate the interaction between two learning agents in the well-known video game Pong. By manipulating the classical rewarding scheme of Pong we show how competitive and collaborative behaviors emerge. We also describe the progression from competitive to collaborative behavior when the incentive to cooperate is increased. Finally we show how learning by playing against another adaptive agent, instead of against a hard-wired algorithm, results in more robust strategies. The present work shows that Deep Q-Networks can become a useful tool for studying decentralized learning of multiagent systems coping with high-dimensional environments.

Image Correspondence Based on Interest Point Correlation in Difference Streams: Method and Applications to Mobile Robot Localization

[abstFig src='/00280002/14.jpg' width=""280"" text='Similarity map between image sequences' ]Visual recognition of previously visited places is a basic cognitive skill for a wide variety of living beings, including humans. This requires a method to extract relevant cues from visual input and successfully match them to memories of known locations, disregarding environmental variations such as lighting changes, viewer pose differences, moving objects and scene occlusion.Interest point correlationis a visual place recognition method inspired by results from neuroscience and psychology; specifically, it addresses those challenges by converting raw visual inputs to a low-variance representation, selecting regions-of-interest for representation matching, and identifying consistent matching trends. Real-world experiments employing a mobile robot demonstrate that interest point correlation is robust to visual changes, suggesting its founding principles are sound.

Modeling place field activity with hierarchical slow feature analysis.

What are the computational laws of hippocampal activity? In this paper we argue for the slowness principle as a fundamental processing paradigm behind hippocampal place cell firing. We present six different studies from the experimental literature, performed with real-life rats, that we replicated in computer simulations. Each of the chosen studies allows rodents to develop stable place fields and then examines a distinct property of the established spatial encoding: adaptation to cue relocation and removal; directional dependent firing in the linear track and open field; and morphing and scaling the environment itself. Simulations are based on a hierarchical Slow Feature Analysis (SFA) network topped by a principal component analysis (ICA) output layer. The slowness principle is shown to account for the main findings of the presented experimental studies. The SFA network generates its responses using raw visual input only, which adds to its biological plausibility but requires experiments performed in light conditions. Future iterations of the model will thus have to incorporate additional information, such as path integration and grid cell activity, in order to be able to also replicate studies that take place during darkness.

A method for analyzing the dimensions of preattentive visual sensitivity

A fundamental question in vision science is: Which physical differences in the visual input are spontaneously visible and which are not? At present this question has only been partially answered. We propose that spontaneously visible variations are coded in “field-capture channels” that compute statistics on the raw visual input and pass them on to higher level processes. We describe a psychophysical method for exhaustively deriving the sensitivities of perceptually-available field-capture channels and thereby determining the dimensionality of early visual processes. The description of the field-capture channels resident in human vision will take the form of a compendium of dimensions of preattentive visual sensitivity. Here we demonstrate a method for deriving this compendium. In particular, we apply the method in a domain of physical variation (textures defined by randomly scrambled mixtures of different gray levels) for which the experimental data are available. A simulation shows that the method can (1) determine the number of field-capture channels that are differentially sensitive to variations in the domain and (2) derive a set of basis functions of the space of physical variations to which those channels are sensitive.

Neurovisual Control in the Quake II Environment

A wide variety of tasks may be performed by humans using only visual data as input. Creating artificial intelligence that adequately uses visual data allows controllers to use single cameras for input and to interact with computer games by merely reading the screen render. In this research, we use the Quake II game environment to compare various techniques that train neural network (NN) controllers to perform a variety of behaviors using only raw visual input. First, it is found that a humanlike retina, which has greater acuity in the center and less in the periphery, is more useful than a uniform acuity retina, both having the same number of inputs and interfaced to the same NN structure, when learning to attack a moving opponent in a visually simple room. Next, we use the same humanlike retina and NN in a more visually complex room, but, finding it is unable to learn successfully, we use a Lamarckian learning algorithm with a nonvisual hand-coded controller as a supervisor to help train the visual controller via backpropagation. Last, we replace the hand-coded supervising nonvisual controller with an evolved nonvisual NN controller, eliminating the human aspect from the supervision, and it solves a problem for which a solution was not previously known.

Retinal vision applied to facial features detection and face authentication

Retinotopic sampling and the Gabor decomposition have a well-established role in computer vision in general as well as in face authentication. The concept of Retinal Vision we introduce aims at complementing these biologically inspired tools with models of higher-order visual process, specifically the Human Saccadic System. We discuss the Saccadic Search strategy, a general purpose attentional mechanism that identifies semantically meaningful structures in images by performing “jumps” ( saccades) between relevant locations. Saccade planning relies on a priori knowledge encoded by SVM classifiers. The raw visual input is analysed by means of a log-polar retinotopic sensor, whose receptive fields consist in a vector of modified Gabor filters designed in the log-polar frequency plane. Applicability to complex cognitive tasks is demonstrated by facial landmark detection and authentication experiments over the M2VTS and Extended M2VTS (XM2VTS) databases.