Sensorimotor Contingencies Research Articles

Active head movements contribute to spatial updating across gaze shifts.

Keeping visual space constant across movements of the eye and head is a not yet fully understood feature of perception. To understand the mechanisms that update the internal coordinates of space, research has mostly focused on eye movements. However, in natural vision, head movements are an integral part of gaze shifts that enlarge the field of vision. Here, we directly compared spatial updating for eye and head movements. In a virtual reality environment, participants had to localize the position of a stimulus across the execution of a gaze shift. We found that performing head movements increased the accuracy of spatial localization. By manipulating the speed of the visual scene displacement that a head movement produced, we found that spatial updating takes into account the sensorimotor contingencies of vision. When we presented gaze-contingent visual motion, subjects overestimated the position of stimuli presented across gaze shifts. The overestimation decreased if subjects were allowed to perform eye movements during the head movement. We conclude that head movements contribute to stabilizing visual space across gaze shifts and that contingencies of head movements, rather than being cancelled, facilitate the updating.

Decision-related activity and movement selection in primate visual cortex.

Fluctuations in the activity of sensory neurons often predict perceptual decisions. This connection can be quantified with a metric called choice probability (CP), and there is a longstanding debate about whether CP reflects a causal influence on decisions or an echo of decision-making activity elsewhere in the brain. Here, we show that CP can reflect a third variable, namely, the movement used to indicate the decision. In a standard visual motion discrimination task, neurons in the middle temporal (MT) area of primate cortex responded more strongly during trials that involved a saccade toward their receptive fields. This variability accounted for much of the CP observed across the neuronal population, and it arose through training. Moreover, pharmacological inactivation of MT biased behavioral responses away from the corresponding visual field locations. These results demonstrate that training on a task with fixed sensorimotor contingencies introduces movement-related activity in sensory brain regions and that this plasticity can shape the neural circuitry of perceptual decision-making.

Differential representation of sensory information and behavioral choice across layers of the mouse auditory cortex

The activity of neurons in sensory areas sometimes covaries with upcoming choices in decision-making tasks. However, the prevalence, causal origin, and functional role of choice-related activity remain controversial. Understanding the circuit-logic of decision signals in sensory areas will require understanding their laminar specificity, but simultaneous recordings of neural activity across the cortical layers in forced-choice discrimination tasks have not yet been performed. Here, we describe neural activity from such recordings in the auditory cortex of mice during a frequency discrimination task with delayed report, which, as we show, requires the auditory cortex. Stimulus-related information was widely distributed across layers but disappeared very quickly after stimulus offset. Choice selectivity emerged toward the end of the delay period-suggesting a top-down origin-but only in the deep layers. Early stimulus-selective and late choice-selective deep neural ensembles were correlated, suggesting that the choice-selective signal fed back to the auditory cortex is not just action specific but develops as a consequence of the sensory-motor contingency imposed by the task.

Mesoscale simulations predict the role of synergistic cerebellar plasticity during classical eyeblink conditioning.

According to the motor learning theory by Albus and Ito, synaptic depression at the parallel fibre to Purkinje cells synapse (pf-PC) is the main substrate responsible for learning sensorimotor contingencies under climbing fibre control. However, recent experimental evidence challenges this relatively monopolistic view of cerebellar learning. Bidirectional plasticity appears crucial for learning, in which different microzones can undergo opposite changes of synaptic strength (e.g. downbound microzones-more likely depression, upbound microzones-more likely potentiation), and multiple forms of plasticity have been identified, distributed over different cerebellar circuit synapses. Here, we have simulated classical eyeblink conditioning (CEBC) using an advanced spiking cerebellar model embedding downbound and upbound modules that are subject to multiple plasticity rules. Simulations indicate that synaptic plasticity regulates the cascade of precise spiking patterns spreading throughout the cerebellar cortex and cerebellar nuclei. CEBC was supported by plasticity at the pf-PC synapses as well as at the synapses of the molecular layer interneurons (MLIs), but only the combined switch-off of both sites of plasticity compromised learning significantly. By differentially engaging climbing fibre information and related forms of synaptic plasticity, both microzones contributed to generate a well-timed conditioned response, but it was the downbound module that played the major role in this process. The outcomes of our simulations closely align with the behavioural and electrophysiological phenotypes of mutant mice suffering from cell-specific mutations that affect processing of their PC and/or MLI synapses. Our data highlight that a synergy of bidirectional plasticity rules distributed across the cerebellum can facilitate finetuning of adaptive associative behaviours at a high spatiotemporal resolution.

Exploring Saliency for Learning Sensory-Motor Contingencies in Loco-Manipulation Tasks

The objective of this paper is to propose a framework for a robot to learn multiple Sensory-Motor Contingencies from human demonstrations and reproduce them. Sensory-Motor Contingencies are a concept that describes intelligent behavior of animals and humans in relation to their environment. They have been used to design control and planning algorithms for robots capable of interacting and adapting autonomously. However, enabling a robot to autonomously develop Sensory-Motor Contingencies is challenging due to the complexity of action and perception signals. This framework leverages tools from Learning from Demonstrations to have the robot memorize various sensory phases and corresponding motor actions through an attention mechanism. This generates a metric in the perception space, used by the robot to determine which sensory-motor memory is contingent to the current context. The robot generalizes the memorized actions to adapt them to the present perception. This process creates a discrete lattice of continuous Sensory-Motor Contingencies that can control a robot in loco-manipulation tasks. Experiments on a 7-dof collaborative robotic arm with a gripper, and on a mobile manipulator demonstrate the functionality and versatility of the framework.

Hearing elliptic movements reveals the imprint of action on prototypical geometries

Within certain categories of geometric shapes, prototypical exemplars that best characterize the category have been evidenced. These geometric prototypes are classically identified through the visual and haptic perception or motor production and are usually characterized by their spatial dimension. However, whether prototypes can be recalled through the auditory channel has not been formally investigated. Here we address this question by using auditory cues issued from timbre-modulated friction sounds evoking human drawing elliptic movements. Since non-spatial auditory cues were previously found useful for discriminating distinct geometric shapes such as circles or ellipses, it is hypothesized that sound dynamics alone can evoke shapes such as an exemplary ellipse. Four experiments were conducted and altogether revealed that a common elliptic prototype emerges from auditory, visual, and motor modalities. This finding supports the hypothesis of a common coding of geometric shapes according to biological rules with a prominent role of sensory-motor contingencies in the emergence of such prototypical geometry.

Brain Patterns Shaping Embodied Activities of Their Bodily Limbs in Perception and Cognition

This essay aims to expose the metaphysical underpinnings of enactivism. While enactivism relies heavily on rejecting the traditional mind-body problem by excluding the familiar thought experiments that favor phenomenal dualism, the crucial point that is overlooked is instead the brain-body problem, specifically the crucial interaction between the brain and the bodily limbs in their embodied activities of perception and cognition. If enactivism is correct, differences in sensory experience necessarily entail differences in embodied activity—this is the metaphysical core of enactivism, which we think is entirely wrong. We will argue that a physical or biological body ("Körper") is configured or shaped into a perceptually living body ("Leib") by brain patterns and not by embodied activity or by the so-called sensorimotor contingencies. Undoubtedly, embodied actions influence sensory perception. However, since variations in embodied actions do not necessarily lead to variations in sensory experience, they are not metaphysically constitutive of these experiences (supervenience claim).

Effects of the self-perceived sensorimotor demand and immersion during video gaming on visual-attention skills.

Playing specific genres of video games (e.g., action video games) has been linked to improvements in cognitive skills mostly related to attentional phenomena. Nonetheless, do video games have features or dimensions in common that impact cognitive improvements beyond the game genre? Here, we argue that the sensorimotor demand-the amount of demand for precise coordination between movement and perception-is a key element in the improvements associated with playing video games. We conducted a two-part study to test this hypothesis: a self-report online gaming instrument development and validation and an in-lab behavioural and electrophysiological study. In the first study, data from 209 participants were used to devise the sensorimotor demand instrument (SMDI). The SMDI was split into three dimensions of video game playing: sensorimotor contingency, immersion and unfocused gaming. Criterion validity related to video gamers' characteristics supported that the SMDI is sensitive to the input device (e.g., keyboard or touchscreens), and the most recent experience gained during gaming sessions while not being sensitive to the game genre. In the second study, data from 20 participants who performed four visual-attentional tasks previously reported in the literature showed that the SMDI's dimensions were associated with behavioural performance measures and the latency and amplitude of event-related potentials (N1, P2 and P3). Despite the challenge of studying the video gamer population, our study remarks on the relevance of sensorimotor demands in the performance of attentional tasks and its potential use as a dimension to characterize the experience of playing video games beyond the game genre.

"Brainets" Shaping The Embodied Activities

This paper explores the crucial connection between the brain and other organs of the body and the concept of embodied sensorimotor contingencies. Our contention is this: enactivism in all its forms reverses the conventional order of things. It is not the embodied sensorimotor actions that shape and configure brain patterns, but rather the "brainets" (synchronizations of neural activities between different brains) that configure or shape a physical and biological body ("Körper") as a living body ("Leib").

Serial dependencies between locomotion and visual space

How do we know the spatial distance of objects around us? Only by physical interaction within an environment can we measure true physical distances. Here, we investigated the possibility that travel distances, measured during walking, could be used to calibrate visual spatial perception. The sensorimotor contingencies that arise during walking were carefully altered using virtual reality and motion tracking. Participants were asked to walk to a briefly highlighted location. During walking, we systematically changed the optic flow, i.e., the ratio between the visual and physical motion speed. Although participants remained unaware of this manipulation, they walked a shorter or longer distance as a function of the optic flow speed. Following walking, participants were required to estimate the perceived distance of visual objects. We found that visual estimates were serially dependent on the experience of the manipulated flow in the previous trial. Additional experiments confirmed that to affect visual perception, both visual and physical motion are required. We conclude that the brain constantly uses movements to measure space for both, actions, and perception.

Using Extended Reality to Study the Experience of Presence.

Extended reality (XR), encompassing various forms of virtual reality (VR) and augmented reality (AR), has become a powerful experimental tool in consciousness research due to its capability to create holistic and immersive experiences of oneself and surrounding environments through simulation. One hallmark of a successful XR experience is when it elicits a strong sense of presence, which can be thought of as a subjective sense of reality of the self and the world. Although XR research has shed light on many factors that may influence presence (or its absence) in XR environments, there remains much to be discovered about the detailed and diverse phenomenology of presence, and the neurocognitive mechanisms that underlie it. In this chapter, we analyse the concept of presence and relate it to the way in which humans may generate and maintain a stable sense of reality during both natural perception and virtual experiences. We start by reviewing the concept of presence as developed in XR research, covering both factors that may influence presence and potential ways of measuring presence. We then discuss the phenomenological characteristics of presence in human consciousness, drawing on clinical examples where presence is disturbed. Next, we describe two experiments using XR that investigated the effects of sensorimotor contingency and affordances on a specific form of presence related to the sense of objects as really existing in the world, referred to as 'objecthood'. We then go beyond perceptual presence to discuss the concept of 'conviction about reality', which corresponds to people's beliefs about the reality status of their perceptual experiences. We finish by exploring how the novel XR method of 'Substitutional Reality' can allow experimental investigation of these topics, opening new experimental directions for studying presence beyond the 'as-if' experience of fully simulated environments.

Using enactive robotics to think outside of the problem-solving box: How sensorimotor contingencies constrain the forms of emergent autononomous habits.

We suggest that the influence of biology in 'biologically inspired robotics' can be embraced at a deeper level than is typical, if we adopt an enactive approach that moves the focus of interest from how problems are solved to how problems emerge in the first place. In addition to being inspired by mechanisms found in natural systems or by evolutionary design principles directed at solving problems posited by the environment, we can take inspiration from the precarious, self-maintaining organization of living systems to investigate forms of cognition that are also precarious and self-maintaining and that thus also, like life, have their own problems that must be be addressed if they are to persist. In this vein, we use a simulation to explore precarious, self-reinforcing sensorimotor habits as a building block for a robot's behavior. Our simulations of simple robots controlled by an Iterative Deformable Sensorimotor Medium demonstrate the spontaneous emergence of different habits, their re-enactment and the organization of an ecology of habits within each agent. The form of the emergent habits is constrained by the sensory modality of the robot such that habits formed under one modality (vision) are more similar to each other than they are to habits formed under another (audition). We discuss these results in the wider context of: (a) enactive approaches to life and mind, (b) sensorimotor contingency theory, (c) adaptationist vs. structuralist explanations in biology, and (d) the limits of functionalist problem-solving approaches to (artificial) intelligence.

Affordances of musical instruments: Conceptual consideration.

While the concept of affordances has been applied in music research, it has not been satisfyingly developed regarding musical instruments. The resulting vagueness restricts the potential of the concept to guide exploration, discussion, and development of new approaches towards musical learning. Also, the concept of affordances comes with strong ontological claims and thus prompts the researcher to be careful when merging it with other theoretical domains or applying it in empirical studies. Consequently, the present article aims at contributing to a conceptualization of affordances of musical instruments by highlighting and discussing components that are necessary to consider in such a project. The first part consists of an overview of key elements of ecological psychology and more recent theoretical contributions, which are of relevance to the aim of the article: Material Engagement Theory, Skilled Intentionality Framework, and Sensorimotor Contingency Theory. A brief review of examples on how the concept of affordances has been applied in music research is presented. The main section of the article discusses four components, vital to further theoretical developments on affordances of musical instruments: the musical niche, spatial networks, sensorimotor relationship, and the amodal nature of affordances. Central to the argument is an understanding of affordances as relational, limited in scope and bound up with the physical interaction between musician and instrument. Accordingly, it is proposed that analytical focus in studies of musical instruments should be the sensorimotor relationship, spatiotemporally unfolding through a musical event. The article is concluded with comments upon educational implications of the presented perspective and suggestions on further research on the topic.

Denken uit gewoonten: een belichaamd perspectief

Thinking out of habit: an embodied perspective In the second half of the twentieth century, habit had received little attention in the cognitive sciences and philosophy of cognition. This despite the extensive theoretical attention habit received in phenomenology and pragmatism. This is because due to influence of behaviorism and the cognitivist revolution, habit was reduced to mechanical stimulus-response reaction that is learned through drill and repetition, and therefore habit cannot be considered intelligent. In this article I argue that a richer and more accurate notion of habit is possible through an embodied vision of cognition, namely enactivism and ecological psychology. This means that we can consider habit as intelligent without equating habit with reflective thought. Such a notion is possible because both enactivism and ecological psychology have their conceptual roots in pragmatism and phenomenology. Secondly, an embodied vison of cognition can describe how our habits are formed from sensorimotor contingencies and are self-organizing patterns of behavior in interaction with an environment. This can be described through the metaphor of ‘laying down a path’, whereby taking a shortcut across a field of grass a path is formed, which enables walking across the field of grass. Laying down a path is more than a metaphor, but also an example of the sociomaterial dimension of habit. Habits are always formed in an interaction between an organism and an environment that is, in our case both material and social.

Learning limb-specific contingencies in early infancy.

Most research with the mobile paradigm has the underlying assumption that young infants can selectively move the limb causing the contingent feedback from the mobile while avoiding irrelevant motor responses. Contrary to this long-held belief, others have argued that such differentiation ability is not fully developed early in life. In the current study, we revisited the traditional mobile paradigm with a contemporary research approach (using high-precision motion capture techniques, a yoked-control design, and a large sample size) to investigate whether response differentiation ability emerges before 5months of age. The data collected from 76 infants (aged between 115 and 159days) revealed that infants can learn sensorimotor contingencies by increasing the movement of the connected leg relative to their baseline level. However, they did not differentially increase the movement of the leg causing an effect in the environment compared with that of other limbs. Our results illustrate that response differentiation ability emerges later than previously suggested.

How to teach a blind person to hear colours? Multi-method training for a colour-to-sound sensory substitution device – design and evaluation

Sensory substitution is thought to be a promising non-invasive assistive technology for people with complete loss of sight because it provides inaccessible visual information via a preserved modality. However, Sensory Substitution Devices (SSDs) are still rarely used by visually impaired persons, possibly due to a lack of structured and supervised training that could be offered alongside these devices. Here, we developed and evaluated a training program that supports the usage of a recently developed colour-to-sound SSD – the Colorophone. Following our recently proposed theoretical model of SSD development, we propose that this training should help people with complete loss of sight to learn how to efficiently use the device by developing relationships between the components of the user-environment-technology system. We applied systematic case studies combined with a mixed-method approach to evaluate the efficacy of this SSD training program. Five blind users underwent ca. 22 h of training, divided into four main parts: identification of the users’ individual characteristics and adaptations; sensorimotor training with the device; semi-structured explorations with the device; and evaluation of the training. We demonstrated that this training allows users to successfully acquire a set of skills (i.e., master the sensorimotor contingencies required by the device, develop visual-like perceptual skills, as well as learn about colours) and progress along developmental trajectories (e.g., switch from serial to parallel information processing, recognize more complex colours, increase environment and task complexity). Importantly, we identified individual differences in learning strategies (i.e., sensorimotor vs. metacognitive strategy) that had an impact on the users’ training progress and required the training assistants (TAs) to apply different assistive strategies. Additionally, we described the crucial role of a (non-professional) training assistant in the training progress: this person facilitates the development of relationships between elements of the user-environment-technology system by supporting a metacognitive learning strategy, thereby reducing the risk of abandonment of the SSD. Our study shows the importance for SSD development of well-designed, tailored training, and it provides new insights into the process of SSD-related perceptual learning.

Learning torsional eye movements through active efficient coding

The human eye has three rotational degrees of freedom: azimuthal, elevational, and torsional. Although torsional eye movements have the most limited excursion, Hering and Helmholtz have argued that they play an important role in optimizing visual information processing. In humans, the relationship between gaze direction and torsional eye angle is described by Listing’s law. However, it is still not clear how this behavior initially develops and remains calibrated during growth. Here we present the first computational model that enables an autonomous agent to learn and maintain binocular torsional eye movement control. In our model, two neural networks connected in series: one for sensory encoding followed by one for torsion control, are learned simultaneously as the agent behaves in the environment. Learning is based on the active efficient coding (AEC) framework, a generalization of Barlow’s efficient coding hypothesis to include action. Both networks adapt by minimizing the prediction error of the sensory representation, subject to a sparsity constraint on neural activity. The policies that emerge follow the predictions of Listing’s law. Because learning is driven by the sensorimotor contingencies experienced by the agent as it interacts with the environment, our system can adapt to the physical configuration of the agent as it changes. We propose that AEC provides the most parsimonious expression to date of Hering’s and Helmholtz’s hypotheses. We also demonstrate that it has practical implications in autonomous artificial vision systems, by providing an automatic and adaptive mechanism to correct orientation misalignments between cameras in a robotic active binocular vision head. Our system’s use of fairly low resolution (100 × 100 pixel) image windows and perceptual representations amenable to event-based input paves a pathway towards the implementation of adaptive self-calibrating robot control on neuromorphic hardware.

Time to imagine moving: Simulated motor activity affects time perception

Sensing the passage of time is important for countless daily tasks, yet time perception is easily influenced by perception, cognition, and emotion. Mechanistic accounts of time perception have traditionally regarded time perception as part of central cognition. Since proprioception, action execution, and sensorimotor contingencies also affect time perception, perception-action integration theories suggest motor processes are central to the experience of the passage of time. We investigated whether sensory information and motor activity may interactively affect the perception of the passage of time. Two prospective timing tasks involved timing a visual stimulus display conveying optical flow at increasing or decreasing velocity. While doing the timing tasks, participants were instructed to imagine themselves moving at increasing or decreasing speed, independently of the optical flow. In the direct-estimation task, the duration of the visual display was explicitly judged in seconds while in the motor-timing task, participants were asked to keep a constant pace of tapping. The direct-estimation task showed imagining accelerating movement resulted in relative overestimation of time, or time dilation, while decelerating movement elicited relative underestimation, or time compression. In the motor-timing task, imagined accelerating movement also accelerated tapping speed, replicating the time-dilation effect. The experiments show imagined movement affects time perception, suggesting a causal role of simulated motor activity. We argue that imagined movements and optical flow are integrated by temporal unfolding of sensorimotor contingencies. Consequently, as physical time is relative to spatial motion, so too is perception of time relative to imaginary motion.

From STEM to STEAM: An Enactive and Ecological Continuum

STEM and STEAM education promotes the integration between science, technology, engineering, mathematics, and the arts. The latter aims at favoring deep and collaborative learning on students, through curricular integration in K-12 science education. The enactive and ecological psychology approach to education puts attention on the role of the teacher, learning context and socio-cultural environment in shaping lived learning experiences. The approach describes education as a process of embodied cognitive assemblage of guided perception and action. The latter process depends on the interaction of learners with digital and/or analogue learning affordances existing within the socio-technological environment. This article proposes that the scope of an enactive-ecological approach can be extended to the domain of learning science, technology, engineering, arts, and mathematics (STEAM), especially when it comes to understanding deep roots of the learning process. We first present an exhaustive literature review regarding the foundations of both the enactive and the ecological learning theories, along with their differences and key similarities. We then describe the fundamentals and latest research advances of an integrated STEAM pedagogy, followed by the notion of mixed reality (XR) as an emerging educational technology approach, offering an understanding of its current foundations and general disposition on how to understand digital immersion from ecological psychology. Next, we propose a systems theoretical approach to integrate the enactive-ecological approach in STEAM pedagogy, framed in the Santiago school of cognition attending to the interactive dynamics occurring between learners and their interaction with learning affordances existing within their educational medium, establishing that sensorimotor contingencies and attentional anchors are important to restrict sensory variety and stabilize learning concepts. Finally, we consider two empirical studies, one from Chile and the other from New Zealand, in which we demonstrate how the enactive-ecological approach built upon a systems theory perspective can contribute to understanding the roots of STEAM learning and inform its learning design.

Socializing Sensorimotor Contingencies.

The aim of this review is to highlight the idea of grounding social cognition in sensorimotor interactions shared across agents. We discuss an action-oriented account that emerges from a broader interpretation of the concept of sensorimotor contingencies. We suggest that dynamic informational and sensorimotor coupling across agents can mediate the deployment of action-effect contingencies in social contexts. We propose this concept of socializing sensorimotor contingencies (socSMCs) as a shared framework of analysis for processes within and across brains and bodies, and their physical and social environments. In doing so, we integrate insights from different fields, including neuroscience, psychology, and research on human–robot interaction. We review studies on dynamic embodied interaction and highlight empirical findings that suggest an important role of sensorimotor and informational entrainment in social contexts. Furthermore, we discuss links to closely related concepts, such as enactivism, models of coordination dynamics and others, and clarify differences to approaches that focus on mentalizing and high-level cognitive representations. Moreover, we consider conceptual implications of rethinking cognition as social sensorimotor coupling. The insight that social cognitive phenomena like joint attention, mutual trust or empathy rely heavily on the informational and sensorimotor coupling between agents may provide novel remedies for people with disturbed social cognition and for situations of disturbed social interaction. Furthermore, our proposal has potential applications in the field of human–robot interaction where socSMCs principles might lead to more natural and intuitive interfaces for human users.