Visual Working Memory Representations Research Articles

Action Planning Renders Objects in Working Memory More Attentionally Salient.

A rapidly growing body of work suggests that visual working memory (VWM) is fundamentally action oriented. Consistent with this, we recently showed that attention is more strongly biased by VWM representations of objects when we plan to act on those objects in the future. Using EEG and eye tracking, here, we investigated neurophysiological correlates of the interactions between VWM and action. Participants (n = 36) memorized a shape for a subsequent VWM test. At test, a probe was presented along with a secondary object. In the action condition, participants gripped the actual probe if it matched the memorized shape, whereas in the control condition, they gripped the secondary object. Crucially, during the VWM delay, participants engaged in a visual selection task, in which they located a target as fast as possible. The memorized shape could either encircle the target (congruent trials) or a distractor (incongruent trials). Replicating previous findings, we found that eye gaze was biased toward the VWM-matching shape and, importantly, more so when the shape was directly associated with an action plan. Moreover, the ERP results revealed that during the selection task, future action-relevant VWM-matching shapes elicited (1) a stronger Ppc (posterior positivity contralateral), signaling greater attentional saliency; (2) an earlier PD (distractor positivity) component, suggesting faster suppression; (3) a larger inverse (i.e., positive) sustained posterior contralateral negativity in incongruent trials, consistent with stronger suppression of action-associated distractors; and (4) an enhanced response-locked positivity over left motor regions, possibly indicating enhanced inhibition of the response associated with the memorized item during the interim task. Overall, these results suggest that action planning renders objects in VWM more attentionally salient, supporting the notion of selection-for-action in working memory.

A certain future strengthens the past: Knowing ahead how to act on an object prioritizes its visual working memory representation.

Findings from recent studies indicate that planning an action toward an object strengthens its visual working memory (VWM) representation, emphasizing the importance of sensorimotor links in VWM. In the present study, we investigated to what extent such sensorimotor links are modulated by how well-defined an action plan is. In three eye-tracking experiments, we asked participants to memorize a visual stimulus for a subsequent memory test, whereby they performed a specific hand movement toward memory-matching probes. We manipulated action uncertainty so that in the defined action condition, participants knew before the memory delay what specific action they would have to perform at the memory test, while in the undefined action condition, they were informed about the specific action on the object in VWM only after the delay. Importantly, during the delay, participants were presented with a visual detection task, designed to measure any attentional biases toward the memorized object. Across the three experiments, we found moderate evidence that knowing in advance how to act on an object prioritized its mnemonic representation, as expressed in an increased attentional bias toward it. Our results support the idea that knowing what action to perform on an object strengthens its representation in VWM, and further highlight the importance of considering action in the study of VWM. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Supplemental Material for A Certain Future Strengthens the Past: Knowing Ahead How to Act on an Object Prioritizes Its Visual Working Memory Representation

Supplemental Material for A Certain Future Strengthens the Past: Knowing Ahead How to Act on an Object Prioritizes Its Visual Working Memory Representation

Similar Quality of Visual Working Memory Representations between Negative and Positive Attentional Templates.

Visual working memory (VWM) plays an important role during visual search, with some theories suggesting an equivalence between working memory representations and guidance from attentional templates. However, recent work has shown that participants can also use 'negative templates', the foreknowledge of distractor-features stored in VWM, to guide attention away from distractors during visual search. These negative templates must also be represented in working memory, but the question remains whether the quality of the working memory representations underlying negative and positive templates are similar, in spite of their opposite impacts on attention. In this study, participants (N = 33) engaged in a visual search task for a shape-defined target after receiving a positive cue (target color), negative cue (distractor color) or neutral cue (non-informative). In 20% of the trials, a color-wheel probe was presented instead of a search array to measure the quality of the cue representation stored in VWM. Our results revealed that participants were more likely to guess in response to neutral cues than negative cues. Yet, the comparison between positive and negative cues showed no significant differences. However, we found no difference in memory precision for the three cue types. More interestingly, the more the VWM quality is boosted by the negative cue, the greater the ability to guide attention away from distractors. Such a pattern of results might map to recent evidence of between-individuals differences in utilization of negative cues. These findings highlight the distinction between attentional templates and simple maintenance in working memory.

Decomposing the multiple encoding benefit in visual long-term memory: Primary contributions by the number of encoding opportunities.

Although access to the seemingly infinite capacity of our visual long-term memory (VLTM) can be restricted by visual working memory (VWM) capacity at encoding and retrieval, access can be improved with repeated encoding. This leads to the multiple encoding benefit (MEB), the finding that VLTM performance improves as the number of opportunities to encode the same information increases over time. However, as the number of encoding opportunities increases, so do other factors such as the number of identical encoded VWM representations and chances to engage in successful retrieval during each opportunity. Thus, across two experiments, we disentangled the contributions of each of these factors to the MEB by having participants encode a varying number of identical objects across multiple encoding opportunities. Along with behavioural data, we also examined two established EEG correlates that track the number of maintained VWM representations, namely the posterior alpha suppression and the negative slow wave. Here, we identified that the primary mechanism behind the MEB was the number of encoding opportunities. That is, recognition memory performance was higher following an increase in the number of encoding opportunities, and this could not be attributed solely to an increase in the number of encoded VWM representations or successful retrieval. Our results thus contribute to the understanding of the fundamental mechanisms behind the influence of VWM on VLTM encoding.

The impact of sleep quality on visual working memory varied with the duration of maintenance.

Sleep quality can affect the performance of visual working memory. However, the effect of sleep quality on the maintenance stage, which is the key to maintain the quality and efficiency of visual working memory representation, remains unclear. This study is the first to explore the effect of sleep quality on the maintenance of visual working memory information. 60 healthy college students completed the Pittsburgh Sleep Quality Index (PSQI) and color recall task of visual working memory. A mixed experimental design of sleep quality (high or low) and delay duration (1, 4, or 6 s) was used to assess the effect of sleep quality on the maintenance phase of visual working memory. The main effects of sleep quality were significant on visual working memory quantity, precision and offset indexes. Among the quantity index, the interaction between sleep quality and delay duration was also significant. This suggests that prolonging the delay time in the maintenance phase leads to difficulty in maintaining attention to the task for those with lower sleep quality, which results in poorer working memory quantitative representations. Increases in the delay duration of the maintenance phase in visual working memory intensify the impact of sleep quality on task performance. Our study provides evidence to reveal the relationship between sleep quality and visual working memory and offers recommendations for improving sleep quality and cognitive functioning in individuals.

Representation and computation in visual working memory.

The ability to sustain internal representations of the sensory environment beyond immediate perception is a fundamental requirement of cognitive processing. In recent years, debates regarding the capacity and fidelity of the working memory (WM) system have advanced our understanding of the nature of these representations. In particular, there is growing recognition that WM representations are not merely imperfect copies of a perceived object or event. New experimental tools have revealed that observers possess richer information about the uncertainty in their memories and take advantage of environmental regularities to use limited memory resources optimally. Meanwhile, computational models of visuospatial WM formulated at different levels of implementation have converged on common principles relating capacity to variability and uncertainty. Here we review recent research on human WM from a computational perspective, including the neural mechanisms that support it.

Visual working memories are abstractions of percepts.

During perception, decoding the orientation of gratings depends on complex interactions between the orientation of the grating, aperture edges, and topographic structure of the visual map. Here, we aimed to test how aperture biases described during perception affect working memory (WM) decoding. For memoranda, we used gratings multiplied by radial and angular modulators to generate orthogonal aperture biases for identical orientations. Therefore, if WM representations are simply maintained sensory representations, they would have similar aperture biases. If they are abstractions of sensory features, they would be unbiased and the modulator would have no effect on orientation decoding. Neural patterns of delay period activity while maintaining the orientation of gratings with one modulator (e.g. radial) were interchangeable with patterns while maintaining gratings with the other modulator (e.g. angular) in visual and parietal cortex, suggesting that WM representations are insensitive to aperture biases during perception. Then, we visualized memory abstractions of stimuli using models of visual field map properties. Regardless of aperture biases, WM representations of both modulated gratings were recoded into a single oriented line. These results provide strong evidence that visual WM representations are abstractions of percepts, immune to perceptual aperture biases, and compel revisions of WM theory.

Visual working memories are abstractions of percepts

During perception, decoding the orientation of gratings depends on complex interactions between the orientation of the grating, aperture edges, and topographic structure of the visual map. Here, we aimed to test how aperture biases described during perception affect working memory (WM) decoding. For memoranda, we used gratings multiplied by radial and angular modulators to generate orthogonal aperture biases for identical orientations. Therefore, if WM representations are simply maintained sensory representations, they would have similar aperture biases. If they are abstractions of sensory features, they would be unbiased and the modulator would have no effect on orientation decoding. Neural patterns of delay period activity while maintaining the orientation of gratings with one modulator (e.g. radial) were interchangeable with patterns while maintaining gratings with the other modulator (e.g. angular) in visual and parietal cortex, suggesting that WM representations are insensitive to aperture biases during perception. Then, we visualized memory abstractions of stimuli using models of visual field map properties. Regardless of aperture biases, WM representations of both modulated gratings were recoded into a single oriented line. These results provide strong evidence that visual WM representations are abstractions of percepts, immune to perceptual aperture biases, and compel revisions of WM theory.

Proactive interference of visual working memory chunks implicates long-term memory.

Visual working memory (VWM) is a limited cognitive resource that can be functionally expanded through chunking (Miller, 1956). For example, participants can hold an increasing number of colours in mind as they learn to chunk reliably paired combinations (Bradyet al.,2009). We investigated whether this benefit is mediated through the in situ compression of VWM representations (Brady et al.,2009) or the offloading of chunks to long-term memory (LTM; Huang & Awh, 2018; Ngiam et al., 2019) by asking if a vulnerability of LTM - proactive interference - influences VWM performance. We adapted previous designs using deterministic (Experiment 1, N = 60) and probabilistic pairings (Experiments 2 and 3, N = 64 and 80, respectively), to include colour pairings that swapped in sequence along with pairings that were consistent in sequence. Generally, participants reported colours from consistent pairs more accurately than from swapping pairs, which we designed to drive interference in LTM (Experiments 1 and 2). The error profiles also pointed to proactive interference between swapping pairs in all three experiments. Moreover, participants who had explicit awareness of frequent colour pairings had higher VWM accuracy, and their errors reflected more proactive interference than their unaware counterparts (Experiment 3). This pattern of long-term proactive interference in a VWM task lends support for accounts of VWM chunking that propose LTM offloading.

Time course of attentional guidance by visual and verbal working memory representations

ABSTRACT Visual and verbal representations in working memory guide visual attention to a memory-matching item. It is well documented that visual cues have an advantage in target detection over verbal cues; however, little research has explored the differences in visual search processes and between attentional guidance by visual and verbal working memory. To address this issue, here we investigated the time course of attentional guidance through visual and verbal representations by comparing the inhibition of return effects. Previous research shows that both types of working memory items produce early facilitation effects and later inhibition effects, but there are lesser facilitation effects for verbal than for visual working memory guidance of attention. Results showed that in all conditions, attention was guided by memory representations, but no inhibition of return was observed. Thus, the robustness of previous findings on the time course of attentional guidance by working memory is low.

ERPs and alpha oscillations track the encoding and maintenance of object-based representations in visual working memory.

When memorizing an integrated object such as a Kanizsa figure, the completion of parts into a coherent whole is attained by grouping processes which render a whole-object representation in visual working memory (VWM). The present study measured event-related potentials (ERPs) and oscillatory amplitudes to track these processes of encoding and representing multiple features of an object in VWM. To this end, a change detection task was performed, which required observers to memorize both the orientations and colors of six "pacman" items while inducing configurations of the pacmen that systematically varied in terms of their grouping strength. The results revealed an effect of object configuration in VWM despite physically constant visual input: change detection for both orientation and color features was more accurate with increased grouping strength. At the electrophysiological level, the lateralized ERPs and alpha activity mirrored this behavioral pattern. Perception of the orientation features gave rise to the encoding of a grouped object as reflected by the amplitudes of the Ppc. The grouped object structure, in turn, modulated attention to both orientation and color features as indicated by the enhanced N1pc and N2pc. Finally, during item retention, the representation of individual objects and the concurrent allocation of attention to these memorized objects were modulated by grouping, as reflected by variations in the CDA amplitude and a concurrent lateralized alpha suppression, respectively. These results indicate that memorizing multiple features of grouped, to-be-integrated objects involves multiple, sequential stages of processing, providing support for a hierarchical model of object representations in VWM.

Scaling models of visual working memory to natural images

Over the last few decades, psychologists have developed precise quantitative models of human recall performance in visual working memory (VWM) tasks. However, these models are tailored to a particular class of artificial stimulus displays and simple feature reports from participants (e.g., the color or orientation of a simple object). Our work has two aims. The first is to build models that explain people’s memory errors in continuous report tasks with natural images. Here, we use image generation algorithms to generate continuously varying response alternatives that differ from the stimulus image in natural and complex ways, in order to capture the richness of people’s stored representations. The second aim is to determine whether models that do a good job of explaining memory errors with natural images also explain errors in the more heavily studied domain of artificial displays with simple items. We find that: (i) features taken from state-of-the-art deep encoders predict trial-level difficulty in natural images better than several reasonable baselines; and (ii) the same visual encoders can reproduce set-size effects and response bias curves in the artificial stimulus domains of orientation and color. Moving forward, our approach offers a scalable way to build a more generalized understanding of VWM representations by combining recent advances in both AI and cognitive modeling.

Cognitive mechanisms underlying the formation of offline representations in visual working memory

Cognitive mechanisms underlying the formation of offline representations in visual working memory

When "looking at nothing" imparts something: Retrospective gaze cues flexibly direct prioritization in visual working memory.

Previous research has shown that retrospective gaze cues direct attention to internally maintained representations in visual working memory (vWM). Here, we aimed to differentiate the dual nature of gaze and accordingly proposed two hypotheses regarding the gaze-induced prioritization in vWM. The directional cueing hypothesis claims a constant attentional shifting to the gazed-at direction. By contrast, the referential cueing hypothesis proposes that gaze cues selectively orient attention toward their referents. To test these hypotheses, we employed an adapted change-detection task wherein gaze cues were presented during the retention interval. Critically, the cue character was positioned between two barriers, which could be either opaque (the blocked condition) or transparent (the unblocked condition). Polygons previously presented at the gazed-at (vs. gazed-away) location were better memorized, but not when the visual perspective of the character was obstructed (i.e., the blocked condition, Experiment 1). Subsequent experiments demonstrated that physical motion cues (Experiment 2) and inverted face cues (Experiment 3), which disrupted the extraction of referential signals, were immune to barrier settings. In Experiment 4, we generalize this selective cueing effect to faces with fearful expressions. These consistent findings support the referential cueing hypothesis and emphasize the distinctiveness of social attention. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Dissociable online integration processes in visual working memory.

Visual working memory has severe capacity limits, creating a bottleneck for active processing. A key way of mitigating this limitation is by chunking, i.e. compressing several pieces of information into one visual working memory representation. However, despite decades of research, chunking efficiency remains debated because of mixed evidence. We propose that there are actually 2 integration mechanisms: Grouping combines several objects to one representation, and object-unification merges the parts of a single object. Critically, we argue that the fundamental distinction between the 2 processes is their differential use of the pointer system, the indexing process connecting visual working memory representations with perception. In grouping, the objects that are represented together still maintain independent pointers, making integration costly but highly flexible. Conversely, object-unification fuses the pointers as well as the representations, with the single pointer producing highly efficient integration but blocking direct access to individual parts. We manipulated integration cues via task-irrelevant movement, and monitored visual working memory's online electrophysiological marker. Uniquely colored objects were flexibly grouped and ungrouped via independent pointers (experiment 1). If objects turned uniformly black, object-integration could not be undone (experiment 2), requiring visual working memory to reset before re-individuation. This demonstrates 2 integration levels (representational-merging versus pointer-compression) and establishes the dissociation between visual working memory representations and their underlying pointers.

Assessing the interaction between working memory and perception through time.

Content maintained in visual working memory changes concurrent visual processing, suggesting that visual working memory may recruit an overlapping neural representation with visual perception. However, it remains unclear whether visual working memory representations persist as a sensory code through time, or are recoded later into an abstract code. Here, we directly contrasted a temporal decay + visual code account and a temporal decay + abstract code account within the temporal dynamics of the interaction between working memory and perception. By manipulating the ISI (inter-stimulus interval) between working memory encoding and a perceptual discrimination task, we found that task-relevant and therefore actively maintained perceptual information parametrically altered participants' ability to discriminate perceptual stimuli even 4 s after encoding, whereas task-irrelevant information caused only an acutely transient effect. While continuously present, the size of this shift in discrimination thresholds gradually decreased over time. Concomitantly, the size of the bias in working memory reports increased over time. The opposing directions of threshold and bias effects are consistent with the local maintenance of information in perceptual areas, explained by a temporal decay + visual code account. As the maintained representation decays over time, its ability to alter incoming perceptual signals decreases (reduced threshold effects) while its likelihood of being impacted by those same signals increases (increased bias effects).Altogether, these results suggest that the readout of working memory relies on a sensory representation at a cost of increased interference by ongoing perception.

Alternating capture of attention by multiple visual working memory representations

Can we look for multiple objects simultaneously? Previous studies have proposed that the representation of an item in visual working memory (VWM) can bias the deployment of attention to memory-matching items in visual search. However, it is still controversial whether multiple VWM item representations are able to capture attention. In the present study, we adopted an eye-tracking technique to reveal this issue. In Experiment 1, we replicated Chen and Du’s Experiment 2 and adopted an eye-tracking technique to determine whether multiple VWM item representations are able to bias attention. In Chen and Du’s paradigm, the memory test was always followed by the search test, and the participants might intentionally prepare for the search task, which can affect the results of the research. Thus, in Experiment 2, we prevented participants from predicting the temporal occurrence of the visual search task by randomly conducting a search test or a memory test after memoranda. The findings of the present study suggested that only one working memory item at a time influences attention and multiple working memory items may affect attention through alternation.Protocol registrationThe stage 1 protocol for this Registered Report was accepted in principle on September 5, 2022. The protocol, as accepted by the journal, can be found at: https://doi.org/10.17605/OSF.IO/BE529.

Bend but don't break: Prioritization protects working memory from displacement but leaves it vulnerable to distortion from distraction

Perceptual distraction distorts visual working memory representations. Previous research has shown that memory responses are systematically biased towards passively viewed visual distractors that are similar to the memoranda. However, it remains unclear whether the prioritization of one working memory representation over another reduces the impact of perceptual distractors. We designed a study with five different types of visual distraction that varied in engagement and found evidence for both subtle distortions and catastrophic failures of memory. Importantly, prioritization protected working memories from catastrophic loss (fewer “swap errors”) but rendered them more vulnerable to distortion (greater attractive “biases” towards the distractor). Our findings demonstrate that prioritization does not simply protect working memory from any and all interference, but rather it reduces the likelihood of catastrophic disruption from perceptual distraction at the cost of an increased likelihood of distortion.

The effects of saccades on visual working memory representations

The effects of saccades on visual working memory representations