Task-relevant Actions Research Articles

Task parse tree: Learning task policy from videos with task-irrelevant components

A good task policy should explicitly interpret the preconditions of actions and the composition structure of task. We aim to automatically learn such a task policy from videos which remains challenging at present. This issue can be further aggravated when task-irrelevant components are involved in videos, such as unoperated objects and small actions. Task-irrelevant objects may introduce disruptive visual relations, and task-irrelevant actions would lead to misleading and even failed task planning. Solving both issues simultaneously is beyond the scope of existing methods. To this end, we propose Task Parse Tree (TPT) as a novel task policy representation, distinguishing task-relevant actions with definite preconditions and clear execution order. The automatic generation of TPT relies on two core designs, where spatio-temporal graph (STG) seizes the vital changes in visual relations of objects and their attributes both spatially and temporally, and conjugate action graph (CAG) models the execution logic of actions in a graph. We collect a dataset of a real-world task, Make Tea, and experiment results on the dataset show that TPT realizes both accurate and interpretable task planning in two different scenarios.

Task-Dependent Warping of Semantic Representations during Search for Visual Action Categories.

Object and action perception in cluttered dynamic natural scenes relies on efficient allocation of limited brain resources to prioritize the attended targets over distractors. It has been suggested that during visual search for objects, distributed semantic representation of hundreds of object categories is warped to expand the representation of targets. Yet, little is known about whether and where in the brain visual search for action categories modulates semantic representations. To address this fundamental question, we studied brain activity recorded from five subjects (one female) via functional magnetic resonance imaging while they viewed natural movies and searched for either communication or locomotion actions. We find that attention directed to action categories elicits tuning shifts that warp semantic representations broadly across neocortex and that these shifts interact with intrinsic selectivity of cortical voxels for target actions. These results suggest that attention serves to facilitate task performance during social interactions by dynamically shifting semantic selectivity toward target actions and that tuning shifts are a general feature of conceptual representations in the brain.SIGNIFICANCE STATEMENT The ability to swiftly perceive the actions and intentions of others is a crucial skill for humans that relies on efficient allocation of limited brain resources to prioritize the attended targets over distractors. However, little is known about the nature of high-level semantic representations during natural visual search for action categories. Here, we provide the first evidence showing that attention significantly warps semantic representations by inducing tuning shifts in single cortical voxels, broadly spread across occipitotemporal, parietal, prefrontal, and cingulate cortices. This dynamic attentional mechanism can facilitate action perception by efficiently allocating neural resources to accentuate the representation of task-relevant action categories.

Gated feedforward inhibition in the frontal cortex releases goal-directed action.

Cortical circuits process both sensory and motor information in animals performing perceptual tasks. However, it is still unclear how sensory inputs are transformed into motor signals in the cortex to initiate goal-directed actions. In this study, we found that a visual-to-motor inhibitory circuit in the anterior cingulate cortex (ACC) triggers precise action in mice performing visual Go/No-go tasks. Three distinct features of ACC neurons-visual amplitudes of sensory neurons, suppression times of motor neurons and network activity from other neurons-predicted response times of the mice. Moreover, optogenetic activation of visual inputs in the ACC, which drives fast-spiking sensory neurons, prompted task-relevant actions in mice by suppressing ACC motor neurons and disinhibiting downstream striatal neurons. Notably, when mice terminated actions in response to stop signals, both motor neuron and network activity increased. Collectively, our data demonstrate that visual inputs to the frontal cortex trigger gated feedforward inhibition to initiate goal-directed actions.

Grasping the world from a cockpit: perspectives on embodied neural mechanisms underlying human performance and ergonomics in aviation context

ABSTRACTA great challenge for cognitive neuroscience is studying human behavior in its complexity as it manifests in the real world. The field of aviation provides a unique opportunity to investigate how perception, action and cognition interact in complex yet controlled ecologically valid environments. We suggest a novel cross-domain approach that combines insights from ecological psychology and embodied cognition with a neurophysiological framework to explain patterns of human performance across a variety of aviation contexts. Specifically, we argue that studying the interaction between an agent and the environment, as manifest in the Mirror Neuron system as a neural correlate, is key to understanding complex behavior. We can describe the experience and skills involved with task-relevant actions – like flying an airplane – using brain mechanisms of motor simulation of the observed action. With this direct coupling between perception and action, the automatic implicit nature of the Mirror Neuron system can be harnessed to improve human factor and ergonomics. This analysis offers three areas for future study and application: (1) enhancing flight training by isolating specific agent-environment relations; (2) tracking training progression based on brain signatures of flight expertise; and (3) neuroscientific-inspired ecological design of next-generation human–machine interfaces in flight decks.

Caudate Microstimulation Increases Value of Specific Choices

Value-based decision-making involves an assessment of the value of items available and the actions required to obtain them. The basal ganglia are highly implicated in action selection and goal-directed behavior [1-4], and the striatum in particular plays a critical role in arbitrating between competing choices [5-9]. Previous work has demonstrated that neural activity in the caudate nucleus is modulated by task-relevant action values [6, 8]. Nonetheless, how value is represented and maintained in the striatum remains unclear since decision-making in these tasks relied on spatially lateralized responses, confounding the ability to generalize to a more abstract choice task [6, 8, 9]. Here, we investigate striatal value representations by applying caudate electrical stimulation in macaque monkeys (n= 3) to bias decision-making in a task that divorces the value of a stimulus from motor action. Electrical microstimulation is known to induce neural plasticity [10,11], and caudate microstimulation in primates has been shown to accelerate associative learning [12, 13]. Our results indicate that stimulation paired with a particular stimulus increases selection of that stimulus, and this effect was stimulus dependent and action independent. The modulation of choice behavior using microstimulation was best modeled as resulting from changes in stimulus value. Caudate neural recordings (n= 1) show that changes in value-coding neuron activity arestimulus value dependent. We argue that caudate microstimulation can differentially increase stimulus values independent of action, and unilateral manipulations of value are sufficient to mediate choice behavior. These results support potential future applications of microstimulation to correct maladaptive plasticity underlying dysfunctional decision-making related to neuropsychiatric conditions.

Learning, remembering, and predicting how to use tools: Distributed neurocognitive mechanisms: Comment on Osiurak and Badets (2016).

The reasoning-based approach championed by Francois Osiurak and Arnaud Badets (Osiurak & Badets, 2016) denies the existence of sensory-motor memories of tool use except in limited circumstances, and suggests instead that most tool use is subserved solely by online technical reasoning about tool properties. In this commentary, I highlight the strengths and limitations of the reasoning-based approach and review a number of lines of evidence that manipulation knowledge is in fact used in tool action tasks. In addition, I present a "two route" neurocognitive model of tool use called the "Two Action Systems Plus (2AS+)" framework that posits a complementary role for online and stored information and specifies the neurocognitive substrates of task-relevant action selection. This framework, unlike the reasoning based approach, has the potential to integrate the existing psychological and functional neuroanatomic data in the tool use domain. (PsycINFO Database Record

A Quantitative Analysis of Context-Dependent Remapping of Medial Frontal Cortex Neurons and Ensembles.

The frontal cortex has been implicated in a number of cognitive and motivational processes, but understanding how individual neurons contribute to these processes is particularly challenging as they respond to a broad array of events (multiplexing) in a manner that can be dynamically modulated by the task context, i.e., adaptive coding (Duncan, 2001). Fundamental questions remain, such as how the flexibility gained through these mechanisms is balanced by the need for consistency and how the ensembles of neurons are coherently shaped by task demands. In the present study, ensembles of medial frontal cortex neurons were recorded from rats trained to perform three different operant actions either in two different sequences or two different physical environments. Single neurons exhibited diverse mixtures of responsivity to each of the three actions and these mixtures were abruptly altered by context/sequence switches. Remarkably, the overall responsivity of the population remained highly consistent both within and between context/sequences because the gains versus losses were tightly balanced across neurons and across the three actions. These data are consistent with a reallocation mixture model in which individual neurons express unique mixtures of selectivity for different actions that become reallocated as task conditions change. However, because the allocations and reallocations are so well balanced across neurons, the population maintains a low but highly consistent response to all actions. The frontal cortex may therefore balance consistency with flexibility by having ensembles respond in a fixed way to task-relevant actions while abruptly reconfiguring single neurons to encode "actions in context." Flexible modes of behavior involve performance of similar actions in contextually relevant ways. The present study quantified the changes in how rat medial frontal cortex neurons respond to the same actions when performed in different task contexts (sequences or environments). Most neurons altered the mixture of actions they were responsive to in different contexts or sequences. Nevertheless, the responsivity profile of the ensemble remained fixed as did the ability of the ensemble to differentiate between the three actions. These mechanisms may help to contextualize the manner in which common events are represented across different situations.

The Decoupled Mind: Mind-wandering Disrupts Cortical Phase-locking to Perceptual Events

The mind flows in a "stream of consciousness," which often neglects immediate sensory input in favor of focusing on intrinsic, self-generated thoughts or images. Although considerable research has documented the disruptive influences of task-unrelated thought for perceptual processing and task performance, the brain dynamics associated with these phenomena are not well understood. Here we investigate the possibility, suggested by several convergent lines of research, that task-unrelated thought is associated with a reduction in the trial-to-trial phase consistency of the oscillatory neural signal in response to perceptual input. Using an experience sampling paradigm coupled with continuous high-density electroencephalography, we observed that task-unrelated thought was associated with a reduction of the P1 ERP, replicating prior observations that mind-wandering is accompanied by a reduction of the brain-evoked response to sensory input. Time-frequency analysis of the oscillatory neural response revealed a decrease in theta-band cortical phase-locking, which peaked over parietal scalp regions. Furthermore, we observed that task-unrelated thought impacted the oscillatory mode of the brain during the initiation of a task-relevant action, such that more cortical processing was required to meet task demands. Together, these findings document that the attenuation of perceptual processing that occurs during task-unrelated thought is associated with a reduction in the temporal fidelity with which the brain responds to a stimulus and suggest that increased neural processing may be required to recouple attention to a task. More generally, these data provide novel confirmatory evidence for the mechanisms through which attentional states facilitate the neural processing of sensory input.

Actions speak louder with words: The roles of action and pedagogical language for grounding mathematical proof

Theories of grounded and embodied cognition posit that situated actions are central constituents in cognitive processes. We investigate whether grounding actions influence reasoning, and how pedagogical language influences the action–cognition relationship. Undergraduate students (N = 120) generated proofs for two mathematical tasks after performing either grounding or non-grounding actions. Grounding actions facilitated key mathematical insights for both tasks, but did not lead to superior proofs. Pedagogical language in the form of prompts (prospective statements) and hints (retrospective statements) accompanying grounding actions enhanced proof performance on one task but not the other. Results from transfer tasks suggested that participants learned to apply their mathematical insights to new contexts. The findings suggest that relations between action and cognition are reciprocal: actions facilitate insight, while pedagogical language strengthens the influence of task-relevant actions for proof production. Pedagogically supported grounding actions offer alternative ways of fostering mathematical reasoning.

Interactive Multiple Object Tracking (iMOT)

We introduce a new task for exploring the relationship between action and attention. In this interactive multiple object tracking (iMOT) task, implemented as an iPad app, participants were presented with a display of multiple, visually identical disks which moved independently. The task was to prevent any collisions during a fixed duration. Participants could perturb object trajectories via the touchscreen. In Experiment 1, we used a staircase procedure to measure the ability to control moving objects. Object speed was set to 1°/s. On average participants could control 8.4 items without collision. Individual control strategies were quite variable, but did not predict overall performance. In Experiment 2, we compared iMOT with standard MOT performance using identical displays. Object speed was set to 2°/s. Participants could reliably control more objects (M = 6.6) than they could track (M = 4.0), but performance in the two tasks was positively correlated. In Experiment 3, we used a dual-task design. Compared to single-task baseline, iMOT performance decreased and MOT performance increased when the two tasks had to be completed together. Overall, these findings suggest: 1) There is a clear limit to the number of items that can be simultaneously controlled, for a given speed and display density; 2) participants can control more items than they can track; 3) task-relevant action appears not to disrupt MOT performance in the current experimental context.

Multi-speaker utterances and co-ordination in task-oriented dialogue

We investigate task-oriented dialogue where utterances are produced by several agents. Starting out from an example where a directive is initiated by one agent and completed by the addressee, we explain under which conditions co-ordinations of this type can be successful. The explanatory devices developed are ‘action schema’ and ‘sufficiently informative proposition’. We argue that propositions embedded in directives must be sufficiently informative for an agent to execute a task-relevant action. We further suggest that agents’ co-ordination on sufficiently informative propositions may be taken as a measure of co-ordination in dialogue. Using this idea, we show how co-ordination in a full task-oriented dialogue develops. Finally, we demonstrate that agents’ “pointwise” co-ordination can be explained using Asher–Morreau defeasible inference, a version of the Lascarides–Asher Practical Syllogism (PS), as well as additional principles such as a Co-operativity Principle and a Principle of Transparency, saying ‘Make assumed conformity of interests and intentions publicly known’. Our proposals would entail that any theory of dialogue would have to be based on co-ordination; in addition, the proposals would have repercussions on the notions of proposition, mutuality, and common ground.

Action-based mechanisms of attention.

Actions, which have effects in the external world, must be spatiotopically represented in the brain. The brain is capable of representing space in many different forms (e.g. retinotopic-, environment-, head- or shoulder-centred), but we maintain that actions are represented in action-centred space, meaning that, at the cellular level, the direction of movement is defined by the activity of cells. In reaching, for example, object location is defined as the direction and distance between the origin of the hand and the target. Most importantly, we argue that more than one task-relevant action can be evoked at any moment in time. Therefore, highly efficient selection processes that accurately link vision and action have had to evolve. Research is reviewed which supports the notion of action-based inhibitory mechanisms that select the target from competing distractors.