Object Affordances Research Articles

Spontaneous object recognition: a promising approach to the comparative study of memory.

Spontaneous recognition of a novel object is a popular measure of exploratory behavior, perception and recognition memory in rodent models. Because of its relative simplicity and speed of testing, the variety of stimuli that can be used, and its ecological validity across species, it is also an attractive task for comparative research. To date, variants of this test have been used with vertebrate and invertebrate species, but the methods have seldom been sufficiently standardized to allow cross-species comparison. Here, we review the methods necessary for the study of novel object recognition in mammalian and non-mammalian models, as well as the results of these experiments. Critical to the use of this test is an understanding of the organism’s initial response to a novel object, the modulation of exploration by context, and species differences in object perception and exploratory behaviors. We argue that with appropriate consideration of species differences in perception, object affordances, and natural exploratory behaviors, the spontaneous object recognition test can be a valid and versatile tool for translational research with non-mammalian models.

Object affordance modulates visual responses in the macaque medial posterior parietal cortex.

Area V6A is a visuomotor area of the dorsomedial visual stream that contains cells modulated by object observation and by grip formation. As different objects have different shapes but also evoke different grips, the response selectivity during object presentation could reflect either the coding of object geometry or object affordances. To clarify this point, we here investigate neural responses of V6A cells when monkeys observed two objects with similar visual features but different contextual information, such as the evoked grip type. We demonstrate that many V6A cells respond to the visual presentation of objects and about 30% of them by the object affordance. Given that area V6A is an early stage in the visuomotor processes underlying grasping, these data suggest that V6A may participate in the computation of object affordances. These results add some elements in the recent literature about the role of the dorsal visual stream areas in object representation and contribute in elucidating the neural correlates of the extraction of action-relevant information from general object properties, in agreement with recent neuroimaging studies on humans showing that vision of graspable objects activates action coding in the dorsomedial visual steam.

Human Activity-Understanding: A Multilayer Approach Combining Body Movements and Contextual Descriptors Analysis

A deep understanding of human activity is key to successful human-robot interaction (HRI). The translation of sensed human behavioural signals/cues and context descriptors into an encoded human activity remains a challenge because of the complex nature of human actions. In this paper, we propose a multilayer framework for the understanding of human activity to be implemented in a mobile robot. It consists of a perception layer which exploits a D-RGB-based skeleton tracking output used to simulate a physical model of virtual human dynamics in order to compensate for the inaccuracy and inconsistency of the raw data. A multi-support vector machine (MSVM) model trained with features describing the human motor coordination through temporal segments in combination with environment descriptors (object affordance) is used to recognize each sub-activity (classification layer). The interpretation of sequences of classified elementary actions is based on discrete hidden Markov models (DHMMs) (interpretation layer). The framework assessment was performed on the Cornell Activity Dataset (CAD-120) [ 1 ]. The performances of our method are comparable with those presented in [ 2 ] and clearly show the relevance of this model-based approach.

Shape representations in the primate dorsal visual stream

The primate visual system extracts object shape information for object recognition in the ventral visual stream. Recent research has demonstrated that object shape is also processed in the dorsal visual stream, which is specialized for spatial vision and the planning of actions. A number of studies have investigated the coding of 2D shape in the anterior intraparietal area (AIP), one of the end-stage areas of the dorsal stream which has been implicated in the extraction of affordances for the purpose of grasping. These findings challenge the current understanding of area AIP as a critical stage in the dorsal stream for the extraction of object affordances. The representation of three-dimensional (3D) shape has been studied in two interconnected areas known to be critical for object grasping: area AIP and area F5a in the ventral premotor cortex (PMv), to which AIP projects. In both areas neurons respond selectively to 3D shape defined by binocular disparity, but the latency of the neural selectivity is approximately 10 ms longer in F5a compared to AIP, consistent with its higher position in the hierarchy of cortical areas. Furthermore, F5a neurons were more sensitive to small amplitudes of 3D curvature and could detect subtle differences in 3D structure more reliably than AIP neurons. In both areas, 3D-shape selective neurons were co-localized with neurons showing motor-related activity during object grasping in the dark, indicating a close convergence of visual and motor information on the same clusters of neurons.

The role of object affordances and center of gravity in eye movements toward isolated daily-life objects.

The purpose of the current study was to investigate to what extent low-level versus high-level effects determine where the eyes land on isolated daily-life objects. We operationalized low-level effects as eye movements toward an object's center of gravity (CoG) or the absolute object center (OC) and high-level effects as visuomotor priming by object affordances. In two experiments, we asked participants to make saccades toward peripherally presented photographs of graspable objects (e.g., a hammer) and to either categorize them (Experiment 1) or to discriminate them from visually matched nonobjects (Experiment 2). Objects were rotated such that their graspable part (e.g., the hammer's handle) pointed toward either the left or the right whereas their action-performing part (e.g., the hammer's head) pointed toward the other side. We found that early-triggered saccades were neither biased toward the object's graspable part nor toward its action-performing part. Instead, participants' eyes landed near the CoG/OC. Only longer-latency initial saccades and refixations were subject to high-level influences, being significantly biased toward the object's action-performing part. Our comparison with eye movements toward visually matched nonobjects revealed that the latter was not merely the consequence of a low-level effect of shape, texture, asymmetry, or saliency. Instead, we interpret it as a higher-level, object-based affordance effect that requires time, and to some extent also foveation, in order to build up and to overcome default saccadic-programming mechanisms.

Interaction between affordance and handedness recognition: a chronometric study.

The visualization of tools and manipulable objects activates motor-related areas in the cortex, facilitating possible actions toward them. This pattern of activity may underlie the phenomenon of object affordance. Some cortical motor neurons are also covertly activated during the recognition of body parts such as hands. One hypothesis is that different subpopulations of motor neurons in the frontal cortex are activated in each motor program; for example, canonical neurons in the premotor cortex are responsible for the affordance of visual objects, while mirror neurons support motor imagery triggered during handedness recognition. However, the question remains whether these subpopulations work independently. This hypothesis can be tested with a manual reaction time (MRT) task with a priming paradigm to evaluate whether the view of a manipulable object interferes with the motor imagery of the subject's hand. The MRT provides a measure of the course of information processing in the brain and allows indirect evaluation of cognitive processes. Our results suggest that canonical and mirror neurons work together to create a motor plan involving hand movements to facilitate successful object manipulation.

The Role of Functional Affordances in Socializing Robots

Just as humans behave according to the social norms of their groups, autonomous systems that become part of these groups also need to behave in socially-expected and accepted ways. For humans these social norms are learned through interaction with members of the group. In this work, we propose that the functional affordances of objects, what objects are meant to be used for, provide us with a starting point for the socialization of such agents. We model these functional affordances in description logics and show how this enables the socially-expected human behavior of substituting objects as needed to achieve a goal. In addition, we propose to combine these affordances with conceptual similarity and proximity in order to make more complex substitutions, which are socially acceptable in their given context. Finally, we describe how their use would allow the agent to take advantage of opportunities and how they are modified and extended through interaction with humans.

On working relations with the citizen

Why do people resist acting upon climate change? The article approaches this much-debated question through a study of the failed attempt to implement congestion charging in Tromsø, Norway. Taking this event as its starting-point, the article sets out to examine how climate change is rendered governable and how this in turn affords the citizen rather limited political agency. Arguing that this in itself created resistance towards the congestion charging scheme, the article moves to look at how a specific object sought governed through – the car – can inform ideas of the citizen and its agentic capacities. It is here argued that whilst the concept of relational agency has sensitized political analysis towards the doings of non-human entities, the orientation towards symmetrical accounts of agency has also left more or less unexplored what it is that constitutes the subject. There is, I thus suggest, a need to distinguish between symmetry and sameness and, further, to find ways of teasing out difference that does not simultaneously re/produce asymmetries. Approaching this by way of object multiplicity and affordances, the article concludes with a discussion of how subjectivities can be understood as a ‘gathering’ of relations and encounters that constantly re-enact the subject. Arguing that this represents a form of ‘compression’ through which potentials of ‘the subject multiple’ are made more proximate and thus also concentrated, the article suggests that rather than to work through perceived mechanisms of the subject, authorities should instead work with it.

Affordance Learning Based on Subtask's Optimal Strategy

Affordances define the relationships between the robot and environment, in terms of actions that the robot is able to perform. Prior work is mainly about predicting the possibility of a reactive action, and the object's affordance is invariable. However, in the domain of dynamic programming, a robot's task could often be decomposed into several subtasks, and each subtask could limit the search space. As a result, the robot only needs to replan its sub-strategy when an unexpected situation happens, and an object's affordance might change over time depending on the robot's state and current subtask. In this paper, we propose a novel affordance model linking the subtask, object, robot state and optimal action. An affordance represents the first action of the optimal strategy under the current subtask when detecting an object, and its influence is promoted from a primitive action to the subtask strategy. Furthermore, hierarchical reinforcement learning and state abstraction mechanism are introduced to learn the task graph and reduce state space. In the navigation experiment, the robot equipped with a camera could learn the objects' crucial characteristics, and gain their affordances in different subtasks.

Language Shapes Thought: Rethinking on Linguistic Relativity

We reviewed the researches on linguistic relativity in color, space and time domains, and rethink the relationship between language and perception to support the idea that language interact with other cognitive processes and shape the perception and thought of human. Language, perception and action are not separated systems, but are closely interconnected and highly interacted to some extent. And language plays a constructive role in object affordances, which do not only require low-level processes of action and motor control, but depend on the language knowledge as well.

Action and Language—Action-Based Neurolinguistics Approach

There are a lot of increasing evidences to support the strong links between action and language. Action based approach proposed the hierarchical structure of language based on action control and concerned with action goal, context, intention, mirror neurons and object affordances. Action semantics proposed that action control and use of object need not only low-level action control, but also semantic knowledge. Action Semantics are hierarchically organized and selectively activated depending on the action intention of the actor and the context. This framework integrated declarative and procedural action knowledge, thus challenging the traditional divide between semantics and pragmatics and they might draw on common representational resources.

Self-Supervised Online Learning of Basic Object Push Affordances

Continuous learning of object affordances in a cognitive robot is a challenging problem, the solution to which arguably requires a developmental approach. In this paper, we describe scenarios where robotic systems interact with household objects by pushing them using robot arms while observing the scene with cameras, and which must incrementally learn, without external supervision, both the effect classes that emerge from these interactions as well as a discriminative model for predicting them from object properties. We formalize the scenario as a multi-view learning problem where data co-occur over two separate data views over time, and we present an online learning framework that uses a self-supervised form of learning vector quantization to build the discriminative model. In various experiments, we demonstrate the effectiveness of this approach in comparison with related supervised methods using data from experiments performed using two different robotic platforms.

No need to match: a comment on Bach, Nicholson and Hudson's "Affordance-Matching Hypothesis".

No need to match: a comment on Bach, Nicholson and Hudson's "Affordance-Matching Hypothesis".

Perceiving, learning, and exploiting object affordances for autonomous pile manipulation

Autonomous manipulation in unstructured environments will enable a large variety of exciting and important applications. Despite its promise, autonomous manipulation remains largely unsolved. Even the most rudimentary manipulation task--such as removing objects from a pile--remains challenging for robots. We identify three major challenges that must be addressed to enable autonomous manipulation: object segmentation, action selection, and motion generation. These challenges become more pronounced when unknown man-made or natural objects are cluttered together in a pile. We present a system capable of manipulating unknown objects in such an environment. Our robot is tasked with clearing a table by removing objects from a pile and placing them into a bin. To that end, we address the three aforementioned challenges. Our robot perceives the environment with an RGB-D sensor, segmenting the pile into object hypotheses using non-parametric surface models. Our system then computes the affordances of each object, and selects the best affordance and its associated action to execute. Finally, our robot instantiates the proper compliant motion primitive to safely execute the desired action. For efficient and reliable action selection, we developed a framework for supervised learning of manipulation expertise. To verify the performance of our system, we conducted dozens of trials and report on several hours of experiments involving more than 1,500 interactions. The results show that our learning-based approach for pile manipulation outperforms a common sense heuristic as well as a random strategy, and is on par with human action selection.

Where culture takes hold: "overimitation" and its flexible deployment in Western, Aboriginal, and Bushmen children.

Children often "overimitate," comprehensively copying others' actions despite manifest perceptual cues to their causal ineffectuality. The inflexibility of this behavior renders its adaptive significance difficult to apprehend. This study explored the boundaries of overimitation in 3- to 6-year-old children of three distinct cultures: Westernized, urban Australians (N=64 in Experiment 1; N=19 in Experiment 2) and remote communities of South African Bushmen (N=64) and Australian Aborigines (N=19). Children overimitated at high frequency in all communities and generalized what they had learned about techniques and object affordances from one object to another. Overimitation thus provides a powerful means of acquiring and flexibly deploying cultural knowledge. The potency of such social learning was also documented compared to opportunities for exploration and practice.

Cortical processing of object affordances for self and others' action.

The perception of objects does not rely only on visual brain areas, but also involves cortical motor regions. In particular, different parietal and premotor areas host neurons discharging during both object observation and grasping. Most of these cells often show similar visual and motor selectivity for a specific object (or set of objects), suggesting that they might play a crucial role in representing the “potential motor act” afforded by the object. The existence of such a mechanism for the visuomotor transformation of object physical properties in the most appropriate motor plan for interacting with them has been convincingly demonstrated in humans as well. Interestingly, human studies have shown that visually presented objects can automatically trigger the representation of an action provided that they are located within the observer's reaching space (peripersonal space). The “affordance effect” also occurs when the presented object is outside the observer's peripersonal space, but inside the peripersonal space of an observed agent. These findings recently received direct support by single neuron studies in monkey, indicating that space-constrained processing of objects in the ventral premotor cortex might be relevant to represent objects as potential targets for one's own or others' action.

Multisensory integration in action control.

The integration of multisensory information is an essential mechanism in perception and action control. Research in multisensory integration is concerned with how the information from the different sensory modalities, such as the senses of vision, hearing, smell, taste, touch, and proprioception, are integrated to a coherent representation of objects (for an overview, see e.g., Calvert et al., 2004). The combination of information from the different senses is central for action control. For instance, when you grasp for a rubber duck, you can see its size, feel its compliance and hear the sound it produces. Moreover, identical physical properties of an object can be provided by different senses. You can both see and feel the size of the rubber duck. Even when you grasp for the rubber duck with a tool (e.g., with tongs), the information from the proximal hand, from the effective part of the distal tool and from the eyes are integrated in a manner to act successfully (for limitations of this integration see Sutter et al., 2013). Over the recent decade a surge of interest in multisensory integration and action control has been witnessed, especially in connection with the idea of a statistically optimized integration of multiple sensory sources. The human information processing system is assumed to adjust moment-by-moment the relative contribution of each sense's estimate to a multisensory task. The sense's contribution depends on its variance, so that the total variance of the multisensory estimate is lower than that for each sense alone. Accordingly, the validity of a statistically optimized multisensory integration has been demonstrated by extensive empirical research (e.g., Ernst and Banks, 2002; Alais and Burr, 2004; Reuschel et al., 2010), also in applied setting such as tool-use (e.g., Takahashi et al., 2009; in the present research topic: Takahashi and Watt, 2014). For this perspective to mature it will be helpful to delve deeper into the multisensory information processing mechanisms and their neural correlates, asking about the range and constraints of these mechanisms, about its localization and involved networks. The contributions to the present research topic range from how information from different senses and action control are linked and modulated by object affordances (Garrido-Vasquez and Schubo, 2014), by task-irrelevant information (Juravle et al., 2013; Wendker et al., 2014; for a review see Wesslein et al., 2014), by temporal and spatial coupling within and between senses (Cameron et al., 2014; Mueller and Fiehler, 2014; Rieger et al., 2014; Sugano et al., 2014) to childhood development of multisensory mechanisms (Jovanovic and Drewing, 2014). Correspondences between the information from different senses play an important role for multisensory integration. Integration does, for instance, not take place when vision and touch are spatially separated (e.g., Gepshtein et al., 2005). However, cognitive approaches on action effect control assume that information from different senses is still coded and represented within the same cognitive domain, when the information concerns the same action (e.g., Musseler, 1999; Hommel et al., 2001). The present research topic also addresses the corresponding issue of modality-specific action control (Boutin et al., 2013; Grunwald et al., 2014). Overall, the present research topic broadens our view on how multisensory mechanisms add to action control. We thank all authors and all reviewers for their valuable contributions.

Observed manipulation enhances left fronto-parietal activations in the processing of unfamiliar tools.

Tools represent a special class of objects, as functional details of tools can afford certain actions. In addition, information gained via prior experience with tools can be accessed on a semantic level, providing a basis for meaningful object interactions. Conceptual representations of tools also encompass knowledge about tool manipulation which can be acquired via direct (active manipulation) or indirect (observation of others manipulating objects) motor experience. The present study aimed to explore the impact of observation of manipulation on the neural processing of previously unfamiliar, manipulable objects. Brain activity was assessed by means of functional magnetic resonance imaging while participants accomplished a visual matching task involving pictures of the novel objects before and after they received object-related training. Three training session in which subjects observed an experimenter manipulating one set of objects and visually explored another set of objects were used to make subjects familiar with the tools and to allow the formation of new tool representations. A control object set was not part of the training. Training-related brain activation increases were found for observed manipulation objects compared to not trained objects in a left-hemispheric network consisting of inferior frontal gyrus (iFG) pars opercularis and triangularis and supramarginal/angular gyrus. This illustrates that direct manipulation experience is not required to elicit tool-associated activation changes in the action system. While the iFG activation might indicate a close relationship between the areas involved in tool representation and those involved in observational knowledge acquisition, the parietal activation is discussed in terms of non-semantic effects of object affordances and hand-tool spatial relationships.

Not just playing around: Infants’ behaviors with objects reflect ability, constraints, and object properties

This study describes infants’ behaviors with objects in relation to age, body position, and object properties. Object behaviors were assessed longitudinally in 22 healthy infants supine, prone, and sitting from birth through 2 years. Results reveal: (1) infants learn to become intense and sophisticated explorers within the first 6 months of life; (2) young infants dynamically and rapidly shift among a variety of behavioral combinations to gather information; (3) behaviors on objects develop along different trajectories so that behavioral profiles vary across time; (4) object behaviors are generally similar in supine and sitting but diminished in prone; and (5) infants begin matching certain behaviors to object properties as newborns. These data demonstrate how infants learn to match their emerging behaviors with changing positional constraints and object affordances.

The scope and limits of action semantics: Reply to comments on ‘Action semantics: A unifying conceptual framework for the selective use of multimodal and modality-specific object knowledge’

In our target paper [1], we have provided an integrative review of the current state of affairs regarding research on the use of conceptual knowledge for action. In short, we argue that humans have developed declarative and procedural knowledge about objects, i.e. action semantics that enable them to use objects in a purposeful and effective manner. We were impressed by both the quantity and the quality of the commentaries on our position paper. We would like to thank the commentators for their valuable contributions, pointing out supportive evidence for our framework, indicating additional and complementary strands of research in this field and raising important conceptual and theoretical challenges for our proposed model. Here we take the opportunity to briefly indicate how additional lines of research could be integrated in our framework and how potentially conflicting findings could be reconciled. Joachim Hermsdorfer suggests that the overlap between the kinematics of pantomimed and object-directed actions points towards the existence of a common motor program, while at the same time task context also modulates the low-level features of action execution [2]. The suggestion that the context may exert both an indirect effect (i.e. through a top-down selection of motor programs) and a direct effect on movement execution (e.g. by providing participants with stronger task constraints and sensory feedback through a direct interaction with an object) provides an excellent illustration of the role of context, as proposed in our model of action semantics. A similar role of context on the activation of motor programs is proposed by Anna Borghi [3], who showed that the context (i.e. provided by the task, the inclusion of other objects, semantic information or the presence of other persons) can have a strong influence on the activation of object affordances.