Task-relevant Representations Research Articles

Learning task-relevant representations via rewards and real actions for reinforcement learning

The input of visual reinforcement learning often contains redundant information, which will reduce the decision efficiency and decrease the performance of the agent. To address this issue, task-relevant representations of the input are usually learned, where only task-related information is preserved for the decision making. While in the literature, auxiliary tasks that are constructed by using reward signals, optimal policy or by extracting controllable elements in the input are commonly adopted to learn the task-relevant representations, the methods based on reward signals do not work well in sparse reward environments, the effectiveness of methods using optimal policy relies heavily on the optimality degree of the given policy, and the methods by extracting the controllable elements will ignore the uncontrollable task-relevant information in the input. To alleviate these problems and to learn better task-relevant representations, in this paper, we first encourage the encoder to encode controllable parts in the input by maximizing the conditional mutual information between the representations and agent’s real actions. And then as reward signals are directly related to the underlying tasks , they are used to make more task-related information encoded irrespective of whether such information is controllable or not. Finally, a temporal coherence constraint is incorporated into the whole framework to reduce the task-irrelevant information in the representations. Experiments on Distracting DeepMind Control Suite and autonomous driving simulator CARLA show that our proposed approach can achieve better performances than some state-of-the-art (SOTA) baselines, which demonstrates the method’s effectiveness in enhancing the agent’s decision efficiency and overall performances. Code is available at https://github.com/DMU-XMU/Learning-Task-relevant-Representations-via-Rewards-and-Real-Actions-for-Reinforcement-Learning.git.

A Model-Based Reinforcement Learning Method with Conditional Variational Auto-Encoder

Model-based reinforcement learning can effectively improve the sample efficiency of reinforcement learning, but the environment model in this method has errors. The model errors can mislead the policy optimization, leading to suboptimal policy. To improve the generalization ability of the environment model, existing methods often use ensemble models or Bayesian models to build the environment model. However, these methods are computationally intensive and complex to update. Since the generated model can describe the stochastic nature of the environment, this paper proposes a model-based reinforcement learning method based on a conditional variational auto-encoder. In this paper, we use a conditional variational auto-encoder to learn task-related representations and apply the generative model to predict environmental changes. Considering the problem of multi-step error accumulation, model adaptation is utilized to minimize the difference between simulated and real data distributions. Furthermore, the experiments verified that the proposed method can learn task-relevant representations and accelerate policy learning.

Be prepared for interruptions: EEG correlates of anticipation when dealing with task interruptions and the role of aging

Dealing with task interruptions requires the flexible use of working memory and attentional control mechanisms, which are prone to age-related changes. We investigated effects of age on dealing with task interruptions and potential advantages of anticipating an interruption using EEG and a retrospective cueing (retro-cue) paradigm. Thirty-two young (18–30 years) and 28 older (55–70 years) participants performed a visual working memory task, where they had to report the orientation of a target following a retro-cue. Within blocks of 10 trials, they were always, never, or randomly interrupted with an arithmetic task before the onset of the retro-cue. The interruption-induced decline in primary task performance was more pronounced in older participants, while only these benefited from anticipation. The EEG analysis revealed reduced theta and alpha/beta response to the retro-cue following interruptions, especially for the older participants. In both groups, anticipated interruptions were associated with increased theta and alpha/beta power prior and during the interruption, and stronger beta suppression to the retro-cue. The results indicate that interruptions impede the refocusing of attention on the task-relevant representation of the primary task, especially in older people, while anticipation facilitates preparation for the interruption task and resumption of the primary task.

The emergence of task-relevant representations in a nonlinear decision-making task

This paper describes the relationship between performance in a decision-making task and the emergence of task-relevant representations. Participants learnt two tasks in which the appropriate response depended on multiple relevant stimuli and the underlying stimulus-outcome associations were governed by a latent feature that participants could discover. We divided participants into good and bad performers based on their overall classification rate and computed behavioural accuracy for each feature value. We found that participants with better performance had a better representation of the latent feature space. We then used representation similarity analysis on Electroencephalographic (EEG) data to identify when these representations emerge. We were able to decode task-relevant representations in a time window emerging 700 ms after stimulus presentation, but only for participants with good task performance. Our findings suggest that, in order to make good decisions, it is necessary to create and extract a low-dimensional representation of the task at hand.

Minority-Oriented Vicinity Expansion with Attentive Aggregation for Video Long-Tailed Recognition

A dramatic increase in real-world video volume with extremely diverse and emerging topics naturally forms a long-tailed video distribution in terms of their categories, and it spotlights the need for Video Long-Tailed Recognition (VLTR). In this work, we summarize the challenges in VLTR and explore how to overcome them. The challenges are: (1) it is impractical to re-train the whole model for high-quality features, (2) acquiring frame-wise labels requires extensive cost, and (3) long-tailed data triggers biased training. Yet, most existing works for VLTR unavoidably utilize image-level features extracted from pretrained models which are task-irrelevant, and learn by video-level labels. Therefore, to deal with such (1) task-irrelevant features and (2) video-level labels, we introduce two complementary learnable feature aggregators. Learnable layers in each aggregator are to produce task-relevant representations, and each aggregator is to assemble the snippet-wise knowledge into a video representative. Then, we propose Minority-Oriented Vicinity Expansion (MOVE) that explicitly leverages the class frequency into approximating the vicinity distributions to alleviate (3) biased training. By combining these solutions, our approach achieves state-of-the-art results on large-scale VideoLT and synthetically induced Imbalanced-MiniKinetics200. With VideoLT features from ResNet-50, it attains 18% and 58% relative improvements on head and tail classes over the previous state-of-the-art method, respectively. Code and dataset are available at https://github.com/wjun0830/MOVE.

From Biological Synapses to “Intelligent” Robots

This selective review explores biologically inspired learning as a model for intelligent robot control and sensing technology on the basis of specific examples. Hebbian synaptic learning is discussed as a functionally relevant model for machine learning and intelligence, as explained on the basis of examples from the highly plastic biological neural networks of invertebrates and vertebrates. Its potential for adaptive learning and control without supervision, the generation of functional complexity, and control architectures based on self-organization is brought forward. Learning without prior knowledge based on excitatory and inhibitory neural mechanisms accounts for the process through which survival-relevant or task-relevant representations are either reinforced or suppressed. The basic mechanisms of unsupervised biological learning drive synaptic plasticity and adaptation for behavioral success in living brains with different levels of complexity. The insights collected here point toward the Hebbian model as a choice solution for “intelligent” robotics and sensor systems.

Task-Relevant Representations and Cognitive Control Demands Modulate Functional Connectivity from Ventral Occipito-Temporal Cortex During Object Recognition Tasks.

The left ventral occipito-temporal cortex (vOTC) supports extraction and processing of visual features. However, it has remained unclear whether left vOTC-based functional connectivity (FC) differs according to task-relevant representations (e.g., lexical, visual) and control demands imposed by the task, even when similar visual-semantic processing is required for object identification. Here, neural responses to the same set of pictures of meaningful objects were measured, while the type of task that participants had to perform (picture naming versus size-judgment task), and the level of cognitive control required by the picture naming task (high versus low interference contexts) were manipulated. Explicit retrieval of lexical representations in the picture naming task facilitated activation of lexical/phonological representations, modulating FC between left vOTC and dorsal anterior-cingulate-cortex/pre-supplementary-motor-area. This effect was not observed in the size-judgment task, which did not require explicit word-retrieval of object names. Furthermore, retrieving the very same lexical/phonological representation in the high versus low interference contexts during picture naming increased FC between left vOTC and left caudate. These findings support the proposal that vOTC functional specialization emerges from interactions with task-relevant brain regions.

Alpha suppression indexes a spotlight of visual-spatial attention that can shine on both perceptual and memory representations.

Although researchers have been recording the human electroencephalogram (EEG) for almost a century, we still do not completely understand what cognitive processes are measured by the activity of different frequency bands. The 8- to 12-Hz activity in the alpha band has long been a focus of this research, but our understanding of its links to cognitive mechanisms has been rapidly evolving recently. Here, we review and discuss the existing evidence for two competing perspectives about alpha activity. One view proposes that the suppression of alpha-band power following the onset of a stimulus array measures attentional selection. The competing view is that this same activity measures the buffering of the task-relevant representations in working memory. We conclude that alpha-band activity following the presentation of stimuli appears to be due to the operation of an attentional selection mechanism, with characteristics that mirror the classic views of attention as selecting both perceptual inputs and representations already stored in memory.

Simulating and Predicting Dynamical Systems With Spatial Semantic Pointers

While neural networks are highly effective at learning task-relevant representations from data, they typically do not learn representations with the kind of symbolic structure that is hypothesized to support high-level cognitive processes, nor do they naturally model such structures within problem domains that are continuous in space and time. To fill these gaps, this work exploits a method for defining vector representations that bind discrete (symbol-like) entities to points in continuous topological spaces in order to simulate and predict the behavior of a range of dynamical systems. These vector representations are spatial semantic pointers (SSPs), and we demonstrate that they can (1) be used to model dynamical systems involving multiple objects represented in a symbol-like manner and (2) be integrated with deep neural networks to predict the future of physical trajectories. These results help unify what have traditionally appeared to be disparate approaches in machine learning.

Concurrent neuroimaging and neurostimulation reveals a causal role for dlPFC in coding of task-relevant information

Dorsolateral prefrontal cortex (dlPFC) is proposed to drive brain-wide focus by biasing processing in favour of task-relevant information. A longstanding debate concerns whether this is achieved through enhancing processing of relevant information and/or by inhibiting irrelevant information. To address this, we applied transcranial magnetic stimulation (TMS) during fMRI, and tested for causal changes in information coding. Participants attended to one feature, whilst ignoring another feature, of a visual object. If dlPFC is necessary for facilitation, disruptive TMS should decrease coding of attended features. Conversely, if dlPFC is crucial for inhibition, TMS should increase coding of ignored features. Here, we show that TMS decreases coding of relevant information across frontoparietal cortex, and the impact is significantly stronger than any effect on irrelevant information, which is not statistically detectable. This provides causal evidence for a specific role of dlPFC in enhancing task-relevant representations and demonstrates the cognitive-neural insights possible with concurrent TMS-fMRI-MVPA.

The ASM Journals Committee Values the Contributions of Black Microbiologists.

Abstract Oscillations are ubiquitous features of brain dynamics that undergo task-related changes in synchrony, power, and frequency. The impact of those changes on target networks is poorly understood. In this work, we used a biophysically detailed model of prefrontal cortex (PFC) to explore the effects of varying the spike rate, synchrony, and waveform of strong oscillatory inputs on the behavior of cortical networks driven by them. Interacting populations of excitatory and inhibitory neurons with strong feedback inhibition are inhibition-based network oscillators that exhibit resonance (i.e., larger responses to preferred input frequencies). We quantified network responses in terms of mean firing rates and the population frequency of network oscillation; and characterized their behavior in terms of the natural response to asynchronous input and the resonant response to oscillatory inputs. We show that strong feedback inhibition causes the PFC to generate internal (natural) oscillations in the beta/gamma frequency range (>15 Hz) and to maximize principal cell spiking in response to external oscillations at slightly higher frequencies. Importantly, we found that the fastest oscillation frequency that can be relayed by the network maximizes local inhibition and is equal to a frequency even higher than that which maximizes the firing rate of excitatory cells; we call this phenomenon population frequency resonance. This form of resonance is shown to determine the optimal driving frequency for suppressing responses to asynchronous activity. Lastly, we demonstrate that the natural and resonant frequencies can be tuned by changes in neuronal excitability, the duration of feedback inhibition, and dynamic properties of the input. Our results predict that PFC networks are tuned for generating and selectively responding to beta- and gamma-rhythmic signals due to the natural and resonant properties of inhibition-based oscillators. They also suggest strategies for optimizing transcranial stimulation and using oscillatory networks in neuromorphic engineering. Author Summary The prefrontal cortex (PFC) flexibly encodes task-relevant representations and outputs biases to mediate higher cognitive functions. The relevant neural ensembles undergo task-related changes in oscillatory dynamics at beta- and gamma frequencies. Using a computational model of the PFC network, we show that strong feedback inhibition causes the PFC to generate internal oscillations and to prefer external oscillations at similar frequencies. The precise frequencies that are generated and preferred can be flexibly tuned by varying the synchrony and strength of input network activity, the level of background excitation, and neuromodulation of intrinsic ion currents. We also show that the peak output frequency in response to external oscillations, which depends on the synchrony and strength of the input as well as the strong inhibitory feedback, is faster than the internally generated frequency, and that this difference enables exclusive response to oscillatory inputs. These properties enable changes in oscillatory dynamics to govern the selective processing and gating of task-relevant signals in service of cognitive control.

Concurrent guidance of attention by multiple working memory items: Behavioral and computational evidence

During visual search, task-relevant representations in visual working memory (VWM), known as attentional templates, are assumed to guide attention. A current debate concerns whether only one (Single-Item-Template hypothesis; SIT) or multiple (Multiple-Item-Template hypothesis; MIT) items can serve as attentional templates simultaneously. The current study was designed to test these two hypotheses. Participants memorized two colors, prior to a visual-search task in which the target and the distractor could match or not match the colors held in VWM. Robust attentional guidance was observed when one of the memory colors was presented as the target (reduced response times (RTs) on target-match trials) or the distractor (increased RTs on distractor-match trials). We constructed two drift-diffusion models that implemented the MIT and SIT hypotheses, which are similar in their predictions about overall RTs, but differ in their predictions about RTs on individual trials. Critically, simulated RT distributions and error rates revealed a better match of the MIT hypothesis to the observed data than the SIT hypothesis. Taken together, our findings provide behavioral and computational evidence for the concurrent guidance of attention by multiple items in VWM.

The Role of the Temporoparietal Junction in Self-Other Distinction.

Being able to discriminate between what originates from ourselves and what originates from others is critical for efficient interactions with our social environment. However, it remains an open question whether self-other distinction is a domain-general mechanism that is involved in various social-cognitive functions or whether specific 'self-other distinction mechanisms' exist for each of these functions. On the neural level, there is evidence that self-other distinction is related to a specific brain region at the border of the superior temporal and inferior parietal cortex, the temporoparietal junction (TPJ). Demonstrating that the TPJ plays a role in social processes that require self-other distinction would support the idea of a domain-general mechanism of self-other distinction. In the present paper, we review evidence coming from clinical observations, neuroimaging experiments and a meta-analysis indicating the involvement of the TPJ in various cognitive operations requiring self-other distinction. At the perceptual level, we discuss the human ability to identify one's own body and to distinguish it from others. At the action level, we review research on the human ability to experience agency and the control of imitative response tendencies. Finally, at the mental-state level, we discuss the ability to attribute mental states to others. Based on this integrative review, we suggest that the TPJ, and in particular its dorsal part, supports a domain general ability to enhance task-relevant representations when self-related and other-related representations are in conflict. Finally, this conception allows us to propose a unifying architecture for the emergence of numerous socio-cognitive abilities.

A large-scale analysis of task switching practice effects across the lifespan

An important feature of human cognition is the ability to flexibly and efficiently adapt behavior in response to continuously changing contextual demands. We leverage a large-scale dataset from Lumosity, an online cognitive-training platform, to investigate how cognitive processes involved in cued switching between tasks are affected by level of task practice across the adult lifespan. We develop a computational account of task switching that specifies the temporal dynamics of activating task-relevant representations and inhibiting task-irrelevant representations and how they vary with extended task practice across a number of age groups. Practice modulates the level of activation of the task-relevant representation and improves the rate at which this information becomes available, but has little effect on the task-irrelevant representation. While long-term practice improves performance across all age groups, it has a greater effect on older adults. Indeed, extensive task practice can make older individuals functionally similar to less-practiced younger individuals, especially for cognitive measures that focus on the rate at which task-relevant information becomes available.

Oscillatory correlates of prioritization of emotional stimuli in WM: The interaction between bottom-up and top-down processes

Working memory can be enhanced by directing attention to task-relevant representations. Alpha oscillations are a neural correlate of spatial attention either to the perceptual or the mnemonic domain. Specifically, an enhancement of alpha power is observed in the ipsilateral posterior cortex to the locus of attention, along with a suppression in the contralateral hemisphere. In this study we aim to unravel the contribution of bottom-up processes in the top-down guidance of attention inside WM. In this way, emotionality of the memoranda was manipulated in a retro-cue task. Behaviorally, we found a recognition advantage for emotional cued items, and significantly, that the emotionality of the non-cued items did not influence accuracy. We found that bilateral alpha power was greater for the emotional irrelevant condition, which could be reflecting the greater demands to suppress an emotional item from the focus of attention. Critically, we found that alpha power lateralization was not modulated by neither the emotionality of the cued or the non-cued items. However, we found that the latency at which alpha lateralization emerged was modulated by the emotionality of the memory representations. In conclusion, we propose that while alpha power lateralization might be reflecting a general spatial orienting mechanism, in our experiment it is influenced by the selection of relevant information within WM.

Abstract WP91: Chemogenetic Activation of Cholinergic Neurons in the Mouse Nucleus Basalis During Stroke Recovery

The cholinergic projection neurons of the nucleus basalis (NB) mediate learning and cortical plasticity. Artificial activation of NB during tone presentation has been shown to increase the representation of paired frequencies in the auditory cortex, and when NB is stimulated during motor behavior it increases task-relevant representations of the cortical motor map. NB activation during successful movements also increases performance after training. We tested the hypothesis that increased activity of NB neurons during rehabilitative motions can improve the extent of stroke recovery. We injected a cre-dependent Gq DREADD (Designer Receptors Activated by Designer Drugs) or control GFP virus bilaterally into the nucleus basalis of transgenic C57/BL6 mice expressing cre in cholinergic neurons. Photothrombotic stroke was then targeted to sensorimotor representations of the forelimb, and performance in the automated reach task used to assess functional deficits and recovery. From weeks 2-6 after stroke, mice received and injection of the DREADD-activating drug clozapine-N-oxide before rehabilitative sessions, and final drug-free performance was assessed at week 7. We see a biologically significant trend toward greater improvement at week 7 after stroke in the average performance of DREADD-injected mice compared to the control group. Methods for improved replication and increasing effect size will be discussed.

Top-down modulation of alpha power and pattern similarity for threatening representations in visual short-term memory

Recent studies have shown that top-down attention biases task-relevant representations in visual short-term memory (VSTM). Accumulating evidence has also revealed the modulatory effects of emotional arousal on attentional processing. However, it remains unclear how top-down attention interacts with emotional memoranda in VSTM. In this study, we investigated the mechanisms of alpha oscillations and their spatiotemporal characteristics that underlie top-down attention to threatening representations during VSTM maintenance with electroencephalography. Participants were instructed to remember a threatening object and a neutral object in a cued variant delayed response task. Retrospective cues (retro-cues) were presented to direct attention to the hemifield of a threatening object (i.e., cue-to-threat trials) or a neutral object (i.e., cue-to-neutral trials) during a retention interval prior to the probe test. We found a significant retro-cue-related alpha lateralisation over posterior regions during VSTM maintenance. The novel finding was that the magnitude of alpha lateralisation was greater for cue-to-threat objects compared to cue-to-neutral ones. These results indicated that directing attention towards threatening representations compared to neutral representations could result in greater regulation of alpha activity contralateral to the cued hemifield. Importantly, we estimated the spatiotemporal pattern similarity in alpha activity and found significantly higher similarity indexes for the posterior regions relative to the anterior regions and for the cue-to-threat objects relative to cue-to-neutral objects over the posterior regions. Together, our findings provided the oscillatory evidence of greater top-down modulations of alpha lateralisation and spatiotemporal pattern similarity for attending to threatening representations in VSTM.

Subtle eye movement metrics reveal task-relevant representations prior to visual search.

Visual search is thought to be guided by an active visual working memory (VWM) representation of the task-relevant features, referred to as the search template. In three experiments using a probe technique, we investigated which eye movement metrics reveal which search template is activated prior to the search, and distinguish it from future relevant or no longer relevant VWM content. Participants memorized a target color for a subsequent search task, while being instructed to keep central fixation. Before the search display appeared, we briefly presented two task-irrelevant colored probe stimuli to the left and right from fixation, one of which could match the current target template. In all three experiments, participants made both more and larger eye movements towards the probe matching the target color. The bias was predominantly expressed in microsaccades, 100-250 ms after probe onset. Experiment 2 used a retro-cue technique to show that these metrics distinguish between relevant and dropped representations. Finally, Experiment 3 used a sequential task paradigm, and showed that the same metrics also distinguish between current and prospective search templates. Taken together, we show how subtle eye movements track task-relevant representations for selective attention prior to visual search.

Enhancing links between visual short term memory, visual attention and cognitive control processes through practice: An electrophysiological insight

The operation of attention on visible objects involves a sequence of cognitive processes. The current study firstly aimed to elucidate the effects of practice on neural mechanisms underlying attentional processes as measured with both behavioural and electrophysiological measures. Secondly, it aimed to identify any pattern in the relationship between Event-Related Potential (ERP) components which play a role in the operation of attention in vision. Twenty-seven participants took part in two recording sessions one week apart, performing an experimental paradigm which combined a match-to-sample task with a memory-guided efficient visual-search task within one trial sequence. Overall, practice decreased behavioural response times, increased accuracy, and modulated several ERP components that represent cognitive and neural processing stages. This neuromodulation through practice was also associated with an enhanced link between behavioural measures and ERP components and with an enhanced cortico-cortical interaction of functionally interconnected ERP components. Principal component analysis (PCA) of the ERP amplitude data revealed three components, having different rostro-caudal topographic representations. The first component included both the centro-parietal and parieto-occipital mismatch triggered negativity – involved in integration of visual representations of the target with current task-relevant representations stored in visual working memory – loaded with second negative posterior-bilateral (N2pb) component, involved in categorising specific pop-out target features. The second component comprised the amplitude of bilateral anterior P2 – related to detection of a specific pop-out feature – loaded with bilateral anterior N2, related to detection of conflicting features, and fronto-central mismatch triggered negativity. The third component included the parieto-occipital N1 – related to early neural responses to the stimulus array – which loaded with the second negative posterior-contralateral (N2pc) component, mediating the process of orienting and focusing covert attention on peripheral target features. We discussed these three components as representing different neurocognitive systems modulated with practice within which the input selection process operates.

Increased experience amplifies the activation of task-irrelevant category representations.

Prior research has demonstrated the benefits (i.e., task-relevant attentional selection) and costs (i.e., task-irrelevant attentional capture) of prior knowledge on search for an individual target or multiple targets from a category. This study investigated whether the level of experience with particular categories predicts the degree of task-relevant and task-irrelevant activation of item and category representations. Adults with varying levels of dieting experience (measured via 3 subscales of Disinhibition, Restraint, Hunger; Stunkard & Messick, Journal of Psychosomatic Research, 29(1), 71-83, 1985) searched for targets defined as either a specific food item (e.g., carrots), or a category (i.e., any healthy or unhealthy food item). Apart from the target-present trials, in the target-absent "foil" trials, when searching for a specific item (e.g., carrots), irrelevant items from the target's category (e.g., squash) were presented. The ERP (N2pc) results revealed that the activation of task-relevant representations (measured via Exemplar and Category N2pc amplitudes) did not differ based on the degree of experience. Critically, however, increased dieting experience, as revealed by lower Disinhibition scores, predicted activation of task-irrelevant representations (i.e., attentional capture of foils from the target item category). Our results suggest that increased experience with particular categories encourages the rapid activation of category representations even when category information is task irrelevant, and that the N2pc in foil trials could potentially serve as an indication of experience level in future studies on categorization.