Contents Of Visual Working Memory Research Articles

Delineating resetting and updating in visual working memory based on the object-to-representation correspondence

When an object we represent in visual working memory (VWM) changes, its representation is modified accordingly. VWM can either access and change the existing representation by an updating process, or it can reset, by encoding the object in its novel status as a new representation. Our goal was to show that the determining factor of updating versus resetting is the availability of a stable correspondence between the object and its VWM representation. Here, we demonstrate that updating relies on the object-to-representation mapping to access and modify the appropriate representation, while losing this mapping triggers a resetting process. We compared very similar situations of object separation that either allowed the mapping to hold, or caused it to be lost. When an object that was mapped to one representation separated, VWM reset, manifested by a sharp drop in the contralateral delay activity (CDA) amplitude (an electrophysiological marker of VWM contents; Experiment 1), and a behavioral cost to detect salient changes that co-occurred with the resetting-triggering event (Experiment 2). When each part was mapped to a different representation, the separation resulted in updating, with a gradual rise in CDA amplitude (Experiment 1), and a reduced behavioral cost (Experiment 2). Thus, while updating and resetting resulted in similar final representations (corresponding to the post-change objects), their dynamics were different, depending on the availability of the mapping. Our results reveal the triggering conditions of resetting and updating, establish methods to study these online processes, and highlight the importance of the object-to-representation correspondence in VWM.

Strategic inhibition of distractors with visual working memory contents after involuntary attention capture

Previous research has suggested that visual working memory (VWM) contents had a guiding effect on selective attention, and once participants realized that the distractors shared the same information with VWM contents in the search task, they would strategically inhibit the potential distractors with VWM contents. However, previous behavioral studies could not reveal the way how distractors with VWM contents are inhibited strategically. By employing the eye-tracking technique and a dual-task paradigm, we manipulated the probability of memory items occurring as distractors to explore this issue. Consistent with previous behavioral studies, the results showed that the inhibitory effect occurred only in the high-probability condition, while the guiding effect emerged in the low-probability condition. More importantly, the eye-movement results indicated that in the high-probability condition, once few (even one) distractors with VWM contents were captured at first, all the remaining distractors with VWM contents would be rejected as a whole. However, in the low-probability condition, attention could be captured by the majority of distractors with VWM contents. These results suggested that the guiding effect of VWM contents on attention is involuntary in the early stage of visual search. After the completion of this involuntary stage, the guiding effect of task-irrelevant VWM contents on attention could be strategically controlled.

Visual working memory can selectively reset a subset of its representations.

The visual working memory (VWM) resetting process is triggered when the mapping between an object in the environment and its corresponding VWM representation becomes irrelevant. Resetting involves discarding the no longer relevant representations, and encoding novel representations and mappings. We examined how resetting operates on VWM's contents. Specifically, we tested whether losing only part of the encoded mappings led to resetting all of the VWM representations. Subjects monitored moving polygons for an abrupt shape-change. Occasionally, a polygon separated into two halves that continued to move independently, making the original single mapping irrelevant. This loss of mapping triggered a resetting process, producing a performance cost: subjects missed shape-changes when they occurred during resetting, but not when the changes occurred before or after resetting. Critically, the cost was (1) specific to the separated item, (2) larger when more mappings were lost, and (3) unaffected by the set-size. This suggests that resetting is a "local" process: VWM removes only the representations whose mappings are lost.

Executive and perceptual distraction in visual working memory.

The contents of visual working memory are likely to reflect the influence of both executive control resources and information present in the environment. We investigated whether executive attention is critical in the ability to exclude unwanted stimuli by introducing concurrent potentially distracting irrelevant items to a visual working memory paradigm, and manipulating executive load using simple or more demanding secondary verbal tasks. Across 7 experiments varying in presentation format, timing, stimulus set, and distractor number, we observed clear disruptive effects of executive load and visual distraction, but relatively minimal evidence supporting an interactive relationship between these factors. These findings are in line with recent evidence using delay-based interference, and suggest that different forms of attentional selection operate relatively independently in visual working memory.

The content of visual working memory alters processing of visual input prior to conscious access: evidence from pupillometry

Visual working memory (VWM) allows for keeping relevant visual information available after termination of its sensory input. Storing information in VWM, however, affects concurrent conscious perception of visual input: initially suppressed visual input gains prioritized access to consciousness when it matches the content of VWM (Gayet et al., 2013). Recently, there has been a debate whether such modulations of conscious access operate prior to conscious perception or, rather, during a transition period from non-conscious to conscious perception. Here, we used pupil size measurements to track the influence of VWM on visual input continuously, and dissociate between these possibilities. Participants were sequentially presented with two shapes drawn from different shape categories (ellipses, rectangles, or triangles) and a retro-cue, indicating which of the two shapes should be remembered for subsequent recall. During the retention interval, participants were instructed to report whether a target shape, which either matched or mismatched the concurrently memorized item, was presented left or right of fixation. Critically, the target shape was initially suppressed from consciousness by continuous flash suppression, and could therefore only be responded to once it was consciously accessible. Analyses of response times revealed that targets were released from suppression faster when they matched compared to when they mismatched the memorized shape. This behavioral effect was paralleled by a differential pupillary response such, that pupil constriction was more pronounced when visual input matched compared to when it mismatched the content of VWM. Importantly, this difference in pupil size emerged already 500ms after target onset, and almost two seconds before participants could report the location of the target shape. We conclude that the content of VWM affects processing of visual input when it is not yet consciously accessible, thereby allowing it to reach prioritized conscious access. Meeting abstract presented at VSS 2017.

Does an unexpected task reset the contents of visual working memory?

Does an unexpected task reset the contents of visual working memory?

The cost of making an eye movement: A direct link between visual working memory and saccade execution.

To facilitate visual continuity across eye movements, the visual system must presaccadically acquire information about the future foveal image. Previous studies have indicated that visual working memory (VWM) affects saccade execution. However, the reverse relation, the effect of saccade execution on VWM load is less clear. To investigate the causal link between saccade execution and VWM, we combined a VWM task and a saccade task. Participants were instructed to remember one, two, or three shapes and performed either a No Saccade-, a Single Saccade- or a Dual (corrective) Saccade-task. The results indicate that items stored in VWM are reported less accurately if a single saccade-or a dual saccade-task is performed next to retaining items in VWM. Importantly, the loss of response accuracy for items retained in VWM by performing a saccade was similar to committing an extra item to VWM. In a second experiment, we observed no cost of executing a saccade for auditory working memory performance, indicating that executing a saccade exclusively taxes the VWM system. Our results suggest that the visual system presaccadically stores the upcoming retinal image, which has a similar VWM load as committing one extra item to memory and interferes with stored VWM content. After the saccade, the visual system can retrieve this item from VWM to evaluate saccade accuracy. Our results support the idea that VWM is a system which is directly linked to saccade execution and promotes visual continuity across saccades.

Subtle eye movement metrics reveal task-relevant representations prior to visual search.

Visual search is thought to be guided by an active visual working memory (VWM) representation of the task-relevant features, referred to as the search template. In three experiments using a probe technique, we investigated which eye movement metrics reveal which search template is activated prior to the search, and distinguish it from future relevant or no longer relevant VWM content. Participants memorized a target color for a subsequent search task, while being instructed to keep central fixation. Before the search display appeared, we briefly presented two task-irrelevant colored probe stimuli to the left and right from fixation, one of which could match the current target template. In all three experiments, participants made both more and larger eye movements towards the probe matching the target color. The bias was predominantly expressed in microsaccades, 100-250 ms after probe onset. Experiment 2 used a retro-cue technique to show that these metrics distinguish between relevant and dropped representations. Finally, Experiment 3 used a sequential task paradigm, and showed that the same metrics also distinguish between current and prospective search templates. Taken together, we show how subtle eye movements track task-relevant representations for selective attention prior to visual search.

Visual Working Memory Enhances the Neural Response to Matching Visual Input.

Visual working memory (VWM) is used to maintain visual information available for subsequent goal-directed behavior. The content of VWM has been shown to affect the behavioral response to concurrent visual input, suggesting that visual representations originating from VWM and from sensory input draw upon a shared neural substrate (i.e., a sensory recruitment stance on VWM storage). Here, we hypothesized that visual information maintained in VWM would enhance the neural response to concurrent visual input that matches the content of VWM. To test this hypothesis, we measured fMRI BOLD responses to task-irrelevant stimuli acquired from 15 human participants (three males) performing a concurrent delayed match-to-sample task. In this task, observers were sequentially presented with two shape stimuli and a retro-cue indicating which of the two shapes should be memorized for subsequent recognition. During the retention interval, a task-irrelevant shape (the probe) was briefly presented in the peripheral visual field, which could either match or mismatch the shape category of the memorized stimulus. We show that this probe stimulus elicited a stronger BOLD response, and allowed for increased shape-classification performance, when it matched rather than mismatched the concurrently memorized content, despite identical visual stimulation. Our results demonstrate that VWM enhances the neural response to concurrent visual input in a content-specific way. This finding is consistent with the view that neural populations involved in sensory processing are recruited for VWM storage, and it provides a common explanation for a plethora of behavioral studies in which VWM-matching visual input elicits a stronger behavioral and perceptual response.SIGNIFICANCE STATEMENT Humans heavily rely on visual information to interact with their environment and frequently must memorize such information for later use. Visual working memory allows for maintaining such visual information in the mind's eye after termination of its retinal input. It is hypothesized that information maintained in visual working memory relies on the same neural populations that process visual input. Accordingly, the content of visual working memory is known to affect our conscious perception of concurrent visual input. Here, we demonstrate for the first time that visual input elicits an enhanced neural response when it matches the content of visual working memory, both in terms of signal strength and information content.

View-Independent Working Memory Representations of Artificial Shapes in Prefrontal and Posterior Regions of the Human Brain.

Traditional views of visual working memory postulate that memorized contents are stored in dorsolateral prefrontal cortex using an adaptive and flexible code. In contrast, recent studies proposed that contents are maintained by posterior brain areas using codes akin to perceptual representations. An important question is whether this reflects a difference in the level of abstraction between posterior and prefrontal representations. Here, we investigated whether neural representations of visual working memory contents are view-independent, as indicated by rotation-invariance. Using functional magnetic resonance imaging and multivariate pattern analyses, we show that when subjects memorize complex shapes, both posterior and frontal brain regions maintain the memorized contents using a rotation-invariant code. Importantly, we found the representations in frontal cortex to be localized to the frontal eye fields rather than dorsolateral prefrontal cortices. Thus, our results give evidence for the view-independent storage of complex shapes in distributed representations across posterior and frontal brain regions.

How visual working memory contents influence priming of visual attention.

Recent evidence shows that when the contents of visual working memory overlap with targets and distractors in a pop-out search task, intertrial priming is inhibited (Kristjánsson, Sævarsson & Driver, Psychon Bull Rev 20(3):514-521, 2013, Experiment 2, Psychonomic Bulletin and Review). This may reflect an interesting interaction between implicit short-term memory-thought to underlie intertrial priming-and explicit visual working memory. Evidence from a non-pop-out search task suggests that it may specifically be holding distractors in visual working memory that disrupts intertrial priming (Cunningham & Egeth, Psychol Sci 27(4):476-485, 2016, Experiment 2, Psychological Science). We examined whether the inhibition of priming depends on whether feature values in visual working memory overlap with targets or distractors in the pop-out search, and we found that the inhibition of priming resulted from holding distractors in visual working memory. These results are consistent with separate mechanisms of target and distractor effects in intertrial priming, and support the notion that the impact of implicit short-term memory and explicit visual working memory can interact when each provides conflicting attentional signals.

Competitive interactions in visual working memory drive access to awareness

Models of biased competition assume that pre-activating a visual representation in visual working memory (VWM) biases perception towards memory-matching objects. Consistent with this, it has been shown that targets suppressed by interocular competition gain prioritized access to awareness when they match VWM content. Thus far, these VWM biases during interocular suppression have been investigated with minimal levels of competition, as there was always only one target stimulus and observers only held a single item in VWM. In the current study we investigated how VWM-based modulation of access to awareness is influenced by a) multiple-item competition within the stimulus display and b) multiple-item competition within VWM. Using the method of breaking continuous flash suppression (b-CFS), we replicated the finding that information matching the content of VWM is released from interocular suppression faster than non-matching information. This VWM-based facilitation was significantly reduced, though still present, when VWM load increased from one to two items, demonstrating a clear competitive constraint on the top-down modulation by VWM. Furthermore, we manipulated inter-stimulus competition by varying the presence of distractors. When distractors were present, VWM-based facilitation was no longer specific to interocular suppression, but also occurred for monocular displays. The results demonstrate that VWM-based visual biases occur in response to competition, whether between or within the eyes, and reconcile findings from different paradigms.

Between-Subject Variability in the Breaking Continuous Flash Suppression Paradigm: Potential Causes, Consequences, and Solutions.

A recent focus in the field of consciousness research involves investigating the propensity of initially non-conscious visual information to gain access to consciousness. A critical tool for measuring conscious access is the so-called breaking continuous flash suppression paradigm (b-CFS). In this paradigm, a high contrast dynamic pattern is presented to one eye, thereby temporarily suppressing a target stimulus that is presented to the other eye. The time it takes for observers to report (e.g., the location of) the initially suppressed stimulus provides a measure of conscious access. Typical observations in b-CFS studies include the finding that upright faces are released from suppression faster than inverted faces, and the finding that stimuli that match the current content of visual working memory are released from suppression faster than mismatching stimuli. Interestingly, the extent to which observers exhibit these effects varies extensively (in the range of hundreds of milliseconds). By re-analyzing existing datasets and a new dataset we establish that the difference in RTs between conditions in b-CFS tasks (i.e., the effect of interest) is highly correlated with participants' overall suppression durations, and with their trial-to-trial variability in RTs. We advocate the usage of a simple latency- normalization method, which (1) removes the between-subject variability in suppression duration from the effect of interest, while (2) providing distributions of RT differences that are better suited for parametric testing. We next compare this latency-normalization method to two other transformations that are widely applied on within-subject RT data (z-transformations and log-transformations). Finally, we tentatively discuss how trial-to-trial variability and overall suppression duration might relate to prolonged phases of shallow suppression that are more prone to modulations of conscious access.

Fechner's law in metacognition: A quantitative model of visual working memory confidence.

Although visual working memory (VWM) has been studied extensively, it is unknown how people form confidence judgments about their memories. Peirce (1878) speculated that Fechner's law-which states that sensation is proportional to the logarithm of stimulus intensity-might apply to confidence reports. Based on this idea, we hypothesize that humans map the precision of their VWM contents to a confidence rating through Fechner's law. We incorporate this hypothesis into the best available model of VWM encoding and fit it to data from a delayed-estimation experiment. The model provides an excellent account of human confidence rating distributions as well as the relation between performance and confidence. Moreover, the best-fitting mapping in a model with a highly flexible mapping closely resembles the logarithmic mapping, suggesting that no alternative mapping exists that accounts better for the data than Fechner's law. We propose a neural implementation of the model and find that this model also fits the behavioral data well. Furthermore, we find that jointly fitting memory errors and confidence ratings boosts the power to distinguish previously proposed VWM encoding models by a factor of 5.99 compared to fitting only memory errors. Finally, we show that Fechner's law also accounts for metacognitive judgments in a word recognition memory task, which is a first indication that it may be a general law in metacognition. Our work presents the first model to jointly account for errors and confidence ratings in VWM and could lay the groundwork for understanding the computational mechanisms of metacognition. (PsycINFO Database Record

Competition in saccade target selection reveals attentional guidance by simultaneously active working memory representations.

The content of visual working memory (VWM) guides attention, but whether this interaction is limited to a single VWM representation or functional for multiple VWM representations is under debate. To test this issue, we developed a gaze-contingent search paradigm to directly manipulate selection history and examine the competition between multiple cue-matching saccade target objects. Participants first saw a dual-color cue followed by two pairs of colored objects presented sequentially. For each pair, participants selectively fixated an object that matched one of the cued colors. Critically, for the second pair, the cued color from the first pair was presented either with a new distractor color or with the second cued color. In the latter case, if two cued colors in VWM interact with selection simultaneously, we expected the second cued color object to generate substantial competition for selection, even though the first cued color was used to guide attention in the immediately previous pair. Indeed, in the second pair, selection probability of the first cued color was substantially reduced in the presence of the second cued color. This competition between cue-matching objects provides strong evidence that both VWM representations interacted simultaneously with selection. (PsycINFO Database Record

Neural and Behavioral Evidence for an Online Resetting Process in Visual Working Memory.

Visual working memory (VWM) maintains task-relevant information in an online state. Previous studies showed that VWM representations are accessed and modified after changes in the environment. Here, we show that this updating process critically depends on an ongoing mapping between the representations and the objects in the environment. When this mapping breaks, VWM cannot access the old representations and instead resets. The novel resetting process that we introduce removes the existing representations instead of modifying them and this process is accompanied by a unique neural marker. During the resetting process, VWM was blind to salient changes in the object's shape. The resetting process highlights the flexibility of our cognitive system in handling the dynamic environment by abruptly abandoning irrelevant schemas.

Recognition Decisions From Visual Working Memory Are Mediated by Continuous Latent Strengths.

Making recognition decisions often requires us to reference the contents of working memory, the information available for ongoing cognitive processing. As such, understanding how recognition decisions are made when based on the contents of working memory is of critical importance. In this work we examine whether recognition decisions based on the contents of visual working memory follow a continuous decision process of graded information about the correct choice or a discrete decision process reflecting only knowing and guessing. We find a clear pattern in favor of a continuous latent strength model of visual working memory-based decision making, supporting the notion that visual recognition decision processes are impacted by the degree of matching between the contents of working memory and the choices given. Relation to relevant findings and the implications for human information processing more generally are discussed.

Visual input that matches the content of visual working memory requires less (not faster) evidence sampling to reach conscious access

The content of visual working memory (VWM) affects the processing of concurrent visual input. Recently, it has been demonstrated that stimuli are released from interocular suppression faster when they match rather than mismatch a color that is memorized for subsequent recall. In order to investigate the nature of the interaction between visual representations elicited by VWM and visual representations elicited by retinal input, we modeled the perceptual processes leading up to this difference in suppression durations. We replicated the VWM modulation of suppression durations, and fitted sequential sampling models (linear ballistic accumulators) to the response time data. Model comparisons revealed that the data was best explained by a decrease in threshold for visual input that matches the content of VWM. Converging evidence was obtained by fitting similar sequential sampling models (shifted Wald model) to published datasets. Finally, to confirm that the previously observed threshold difference reflected processes occurring before rather than after the stimuli were released from suppression, we applied the same procedure to the data of an experiment in which stimuli were not interocularly suppressed. Here, we found no decrease in threshold for stimuli that match the content of VWM. We discuss our findings in light of a preactivation hypothesis, proposing that matching visual input taps into the same neural substrate that is already activated by a representation concurrently maintained in VWM, thereby reducing its threshold for reaching visual awareness.

Attending and Inhibiting Stimuli That Match the Contents of Visual Working Memory: Evidence from Eye Movements and Pupillometry

When you keep a red apple in working memory, your attention is usually—but not always—attracted by other red objects. The conditions under which the contents of visual working memory guide visual attention are still unclear. We conducted two experiments to test whether attention is indeed biased toward memory-match stimuli, and, if so, whether this bias is transient or sustained. We used a new pupillometric technique, which exploits that the pupil is larger when you covertly attend to dark, compared to bright, stimuli. This allowed us to test whether and when attention was biased toward memory-match stimuli, by placing them either on a dark or a bright background, while measuring pupil size over time. In addition, we looked at gaze errors, that is, whether and when participants looked at memory-match stimuli (despite instructions to maintain central fixation). We found that the eyes were captured by memory-match stimuli early on in the trial. However, the pupillary data suggested that there was no sustained attention bias toward memory-match stimuli later in time; rather, attention appeared to be biased away from them. Together, our results suggest that: (1) memory-match stimuli automatically capture attention; but (2) they do so only briefly; and (3) they may even become inhibited later in time. Meeting abstract presented at VSS 2016

Tracking the dynamics of working memory representations through the eyes

The search template is a specific type of visual working memory (VWM) representation that facilitates search by guiding attention toward matching features when they appear in view. In a series of experiments we tested whether the pupil response and microsaccades can be used to track attentional guidance by the search template and differentiate this process from recognition of other VWM contents. Participants memorized either one or two colored circles to be searched for later on. When two circles were presented, a retro-cue indicated which one of the two was relevant for the upcoming search task (the template). Crucially, before the search display appeared, we briefly presented one or two colored probe stimuli. These probes could match the color of either the relevant-template color, the irrelevant color, or it could be a new color unrelated to the items in memory. We measured the pupil response to the probe display and the amount of microsaccades to the template matching probe as a signature of attentional guidance. Overall, the pupil showed significantly more constriction in response to memory matching probes than for unrelated colors. However, this effect seems to be driven by the recognition of the memory items in the probe display rather than by the relevance of each item. In contrast, microsaccades were consistently directed towards probes matching the relevant color. Taken together, both the pupil and microsaccades were sensitive to when information was previously presented but only microsaccades could distinguish the search template from other VWM representations. Meeting abstract presented at VSS 2016