Rapid Serial Visual Presentation Sequence Research Articles

Semantic image sorting method for RSVP presentation

Objective. The rapid serial visual presentation (RSVP) paradigm, which is based on the electroencephalogram (EEG) technology, is an effective approach for object detection. It aims to detect the event-related potentials (ERP) components evoked by target images for rapid identification. However, the object detection performance within this paradigm is affected by the visual disparity between adjacent images in a sequence. Currently, there is no objective metric to quantify this visual difference. Consequently, a reliable image sorting method is required to ensure the generation of a smooth sequence for effective presentation. Approach. In this paper, we propose a novel semantic image sorting method for sorting RSVP sequences, which aims at generating sequences that are perceptually smoother in terms of the human visual experience. Main results. We conducted a comparative analysis between our method and two existing methods for generating RSVP sequences using both qualitative and quantitative assessments. A qualitative evaluation revealed that the sequences generated by our method were smoother in subjective vision and were more effective in evoking stronger ERP components than those generated by the other two methods. Quantitatively, our method generated semantically smoother sequences than the other two methods. Furthermore, we employed four advanced approaches to classify single-trial EEG signals evoked by each of the three methods. The classification results of the EEG signals evoked by our method were superior to those of the other two methods. Significance. In summary, the results indicate that the proposed method can significantly enhance the object detection performance in RSVP-based sequences.

Slippage of the attentional beam when searching in space and in time

“Slippage” of attention in time and space has been studied separately, using visual search (e.g., Snyder, 1972) and rapid serial visual presentation (RSVP) (e.g., McLean, Broadbent, & Broadbent, 1982). The primary purpose of the current study was to see if we could replicate these findings of slippage and if we did, to use individual differences to explore relationships between slippage in the temporal and spatial domains. The participants identified and localized targets in visual search and in RSVP sequences. In Experiment 1, we used visual search and RSVP tasks closely replicating the methods of Snyder and McLean et al. In Experiment 2, we closely equated the two tasks as far as possible while maintaining the crucial space/time difference. Consistent with the previous studies, and reflecting binding errors (or slippage) in both space and time, erroneously reported identities were predominantly from items adjacent to the targets. Correlations between measures of the slippage in space (visual search) and time (RSVP) were near zero, suggesting that different attentional ‘beams’ bind features in space and time, a possibility that is consistent with other behavioural as well as neuropsychological evidence.

Perceived Duration Depends Upon Target Detection in Rapid Serial Visual Presentation Sequence.

It is well known that the perceived duration for a given time period decreases with the reduction of the number of perceived events. We examined whether target detection failures in viewing Rapid Serial Visual Presentation (RSVP) sequence, caused by attentional blink, affect this reduction of perceived duration. In two experiments, trials consisted of displays of two series of RSVP sequences; in the first sequence (the comparison), two, one, or no numerals were presented as targets embedded within the string of letters, while in the second sequence (the standard), only alphabetic letters were presented. In each trial, participants judged whether the duration of the comparison is perceived as longer than that of the standard (Experiment 1), or whether the number of frames in the comparison is perceived as more than that in the standard (Experiment 2). Results showed that perceived duration was inflated with target detection, but not with the increment of presented frames although number of perceived frames was inflated with both target detection and increment of presented frames. These results suggest that perceived duration in viewing RSVP sequences is determined by the cognitive load necessary to accomplish target detection rather than by the number of perceived frames.

All-or-none face categorization in the human brain

Visual categorization is integral for our interaction with the natural environment. In this process, similar selective responses are produced to a class of variable visual inputs. Whether categorization is supported by partial (graded) or absolute (all-or-none) neural responses in high-level human brain regions is largely unknown. We address this issue with a novel frequency-sweep paradigm probing the evolution of face categorization responses between the minimal and optimal stimulus presentation times. In a first experiment, natural images of variable non-face objects were progressively swept from 120 to 3 ​Hz (8.33–333 ​ms duration) in rapid serial visual presentation sequences. Widely variable face exemplars appeared every 1 ​s, enabling an implicit frequency-tagged face-categorization electroencephalographic (EEG) response at 1 ​Hz. Face-categorization activity emerged with stimulus durations as brief as 17 ​ms (17–83 ​ms across individual participants) but was significant with 33 ​ms durations at the group level. The face categorization response amplitude increased until 83 ​ms stimulus duration (12 ​Hz), implying graded categorization responses. In a second EEG experiment, faces appeared non-periodically throughout such sequences at fixed presentation rates, while participants explicitly categorized faces. A strong correlation between response amplitude and behavioral accuracy across frequency rates suggested that dilution from missed categorizations, rather than a decreased response to each face stimulus, accounted for the graded categorization responses as found in Experiment 1. This was supported by (1) the absence of neural responses to faces that participants failed to categorize explicitly in Experiment 2 and (2) equivalent amplitudes and spatio-temporal signatures of neural responses to behaviorally categorized faces across presentation rates. Overall, these observations provide original evidence that high-level visual categorization of faces, starting at about 100 ​ms following stimulus onset in the human brain, is variable across observers tested under tight temporal constraints, but occurs in an all-or-none fashion.

Relating categorization to set summary statistics perception

Two cognitive processes have been explored that compensate for the limited information that can be perceived and remembered at any given moment. The first parsimonious cognitive process is object categorization. We naturally relate objects to their category, assume they share relevant category properties, often disregarding irrelevant characteristics. Another scene organizing mechanism is representing aspects of the visual world in terms of summary statistics. Spreading attention over a group of objects with some similarity, one perceives an ensemble representation of the group. Without encoding detailed information of individuals, observers process summary data concerning the group, including set mean for various features (from circle size to face expression). Just as categorization may include/depend on prototype and intercategory boundaries, so set perception includes property mean and range. We now explore common features of these processes. We previously investigated summary perception of low-level features with a rapid serial visual presentation (RSVP) paradigm and found that participants perceive both the mean and range extremes of stimulus sets, automatically, implicitly, and on-the-fly, for each RSVP sequence, independently. We now use the same experimental paradigm to test category representation of high-level objects. We find participants perceive categorical characteristics better than they code individual elements. We relate category prototype to set mean and same/different category to in/out-of-range elements, defining a direct parallel between low-level set perception and high-level categorization. The implicit effects of mean or prototype and set or category boundaries are very similar. We suggest that object categorization may share perceptual-computational mechanisms with set summary statistics perception.

Reward and emotion influence attentional bias in rapid serial visual presentation.

Facial emotion constitutes an important source of information, and rapid processing of this information may bring adaptive advantages. Previous evidence suggests that emotional faces are sometimes prioritised for cognitive processing. Three experiments used an emotion-induced blindness task to examine whether this prioritisation occurs in a purely stimulus-driven fashion or whether it emerges only when the faces are task-relevant. Angry or neutral faces appeared as distractors in a rapid serial visual presentation sequence, shortly before a target that participants were required to identify. Either the emotion (Experiment 1) or gender (Experiments 2 and 3) of the distractor face indicated whether a correct/incorrect response to the target would produce reward/punishment, or not. The three experiments found that reward-related faces impaired subsequent target identification, replicating previous results. Target identification accuracy was also impaired following angry faces, compared with neutral faces, demonstrating an emotion-induced attentional bias. Importantly, this impairment was observed even when face emotion was entirely irrelevant to the participants' ongoing task (in Experiments 2 and 3), suggesting that rapid processing of the facial emotion might arise (at least in part) from the operation of relatively automatic cognitive-perceptual processes.

Perceiving set mean and range: Automaticity and precision

To compensate for the limited visual information that can be perceived and remembered at any given moment, many aspects of the visual world are represented as summary statistics. We acquire ensemble representations of element groups as a whole, spreading attention over objects, for which we encode no detailed information. Previous studies found that different features of items (from size/orientation to facial expression/biological motion) are summarized to their mean, over space or time. Summarizing is economical, saving time and energy when the environment is too rich and complex to encode each stimulus separately. We investigated set perception using rapid serial visual presentation sequences. Following each sequence, participants viewed two stimuli, member and nonmember, indicating the member. Sometimes, unbeknownst to participants, one stimulus was the set mean, and or the nonmember was outside the set range. Participants preferentially chose stimuli at/near the mean, a "mean effect," and more easily rejected out-of-range stimuli, a "range effect." Performance improved with member proximity to the mean and nonmember distance from set mean and edge, though they were instructed only to remember presented stimuli. We conclude that participants automatically encode both mean and range boundaries of stimulus sets, avoiding capacity limits and speeding perceptual decisions.

Rapid Serial Visual Presentation (RSVP) of Emotional Faces Generates Substantial Emotion Aftereffect as the Average Face of the RSVP Sequence

Rapid Serial Visual Presentation (RSVP) of Emotional Faces Generates Substantial Emotion Aftereffect as the Average Face of the RSVP Sequence

Character Decomposition and Transposition Processes in Chinese Compound Words Modulates Attentional Blink

The attentional blink (AB) is the phenomenon in which the identification of the second of two targets (T2) is attenuated if it is presented less than 500 ms after the first target (T1). Although the AB is eliminated in canonical word conditions, it remains unclear whether the character order in compound words affects the magnitude of the AB. Morpheme decomposition and transposition of Chinese two-character compound words can provide an effective means to examine AB priming and to assess combinations of the component representations inherent to visual word identification. In the present study, we examined the processing of consecutive targets in a rapid serial visual presentation (RSVP) paradigm using Chinese two-character compound words in which the two characters were transposed to form meaningful words or meaningless combinations (reversible, transposed, or canonical words). We found that when two Chinese characters that form a compound word, regardless of their order, are presented in an RSVP sequence, the likelihood of an AB for the second character is greatly reduced or eliminated compared to when the two characters constitute separate words rather than a compound word. Moreover, the order of the report for the two characters is more likely to be reversed when the normal order of the two characters in a compound word is reversed, especially when the interval between the presentation of the two characters is extremely short. These findings are more consistent with the cognitive strategy hypothesis than the resource-limited hypothesis during character decomposition and transposition of Chinese two-character compound words. These results suggest that compound characters are perceived as a unit, rather than two separate words. The data further suggest that readers could easily understand the text with character transpositions in compound words during Chinese reading.

Failure to detect meaning in RSVP at 27ms per picture.

The human visual system has the remarkable ability to rapidly detect meaning from visual stimuli. Potter, Wyble, Hagmann, and McCourt (Attention, Perception, & Psychophysics, 76, 270-279, 2014) tested the minimum viewing time required to obtain meaning from a stream of pictures shown in a rapid serial visual presentation (RSVP) sequence containing either six or 12 pictures. They reported that observers could detect the presence of a target picture specified by name (e.g., smiling couple) even when the pictures in the sequence were presented for just 13ms each. Potter et al. claimed that this was insufficient time for feedback processing to occur, so feedforward processing alone must be able to generate conscious awareness of the target pictures. A potential confound in their study is that the pictures in the RSVP sequence sometime contained areas with no high-contrast edges, and so may not have adequately masked each other. Consequently, iconic memories of portions of the target pictures may have persisted in the visual system, thereby increasing the effective presentation time. Our study addressed this issue by redoing the Potter et al. study, but using four different types of masks. We found that when adequate masking was used, no evidence emerged that observers could detect the presence of a specific target picture, even when each picture in the RSVP sequence was presented for 27ms. On the basis of these findings, we cannot rule out the possibility that feedback processing is necessary for individual pictures to be recognized.

The attentional blink impairs detection and delays encoding of visual information: evidence from human electrophysiology.

This article explores the time course of the functional interplay between detection and encoding stages of information processing in the brain and the role they play in conscious visual perception. We employed a multitarget rapid serial visual presentation (RSVP) approach and examined the electrophysiological P3 component elicited by a target terminating an RSVP sequence. Target-locked P3 activity was detected both at frontal and parietal recording sites and an independent component analysis confirmed the presence of two distinct P3 components. The posterior P3b varied with intertarget lag, with diminished amplitude and postponed latency at short relative to long lags-an electroencephalographic signature of the attentional blink (AB). Under analogous conditions, the anterior P3a was also reduced in amplitude but did not vary in latency. Collectively, the results provide an electrophysiological record of the interaction between frontal and posterior components linked to detection (P3a) and encoding (P3b) of visual information. Our findings suggest that, although the AB delays target encoding into working memory, it does not slow down detection of a target but instead reduces the efficacy of this process. A functional characterization of P3a in attentive tasks is discussed with reference to current models of the AB phenomenon.

Rapid serial processing of natural scenes: Color modulates detection but neither recognition nor the attentional blink

The exact function of color vision for natural-scene perception has remained puzzling. In rapid serial visual presentation (RSVP) tasks, categorically defined targets (e.g., animals) are detected typically slightly better for color than for grayscale stimuli. Here we test the effect of color on animal detection, recognition, and the attentional blink. We present color and grayscale RSVP sequences with up to two target images (animals) embedded. In some conditions, we modify either the hue or the intensity of each pixel. We confirm a benefit of color over grayscale images for animal detection over a range of stimulus onset asynchronies (SOAs), with improved hit rates from 50 to 120 ms and overall improved performance from 90 to 120 ms. For stimuli in which the hue is inverted, performance is similar to grayscale for small SOAs and indistinguishable from original color only for large SOAs. For subordinate category discrimination, color provides no additional benefit. Color and grayscale sequences show an attentional blink, but differences between color and grayscale are fully explained by single-target differences, ruling out the possibility that the color benefit is purely attentional.

Single-trial classification of neural responses evoked in rapid serial visual presentation: effects of stimulus onset asynchrony and stimulus repetition.

Rapid serial visual presentation (RSVP) tasks, in which participants are presented with a continuous sequence of images in one location, have been used in combination with electroencephalography (EEG) in a variety of Brain-Machine Interface (BMI) applications. The RSVP task is advantageous because it can be performed at a high temporal rate. The rate of the RSVP sequence is controlled by the stimulus onset asynchrony (SOA) between subsequent stimuli. When used within the context of a BMI, an RSVP task with short SOA could increase the information throughput of the system while also allowing for stimulus repetitions. However, reducing the SOA also increases the perceptual degradation caused by presenting two stimuli in close succession, and it decreases the target-to-target interval (TTI), which can increase the cognitive demands of the task. These negative consequences of decreasing the SOA could affect on the EEG signal measured in the task and degrade the performance of the BMI. Here we systematically investigate the effects of SOA and stimulus repetition (r) on single-trial target detection in an RSVP task. Ten healthy volunteers participated in an RSVP task in four conditions that varied in SOA and repetitions (SOA=500 ms, r=1; SOA=250 ms, r=2; SOA=166 ms, r=3; and SOA=100 ms, r=5) while processing time across conditions was controlled. There were two key results: First, when controlling for the number of repetitions, single-trial performance increases when the SOA decreases. Second, when the repetitions were combined, the best performance (AUC=0.967) was obtained with the shortest SOA (100 ms). These results suggest that shortening the SOA in an RSVP task has the benefit of increasing the performance relative to longer SOAs, and it also allows a higher number of repetitions of the stimuli in a limited amount of time.

Twelve-picture RSVP sequences support feed-forward models of detection at 75 Hz

Potter, Wyble, and McCourt previously reported that viewers show above-chance detection of a picture in a rapid serial visual presentation (RSVP) when they are given a verbal title (e.g., cut-up fruit) just before or immediately after the sequence, even at a duration as short as 13 ms/picture (75 Hz). We took these results as support for a feed-forward model of detection. However, given that there were only six pictures in each sequence, was detection only possible because of the minimal input load or incomplete masking? Here we report the results for twelve-picture sequences. We presented color photographs of a wide variety of scenes at rates of 80, 53, 27, and 13 ms/picture, with no ISI. The target was never the first or the last picture. In one group the target name was presented before the sequence and in the other group, immediately after the sequence. All pictures were new to the participants and none were repeated in the experiment. Comparing hit and false alarm rates, performance in both groups was significantly above chance at all durations. Whether or not the participant reported detection of the target, each target trial was followed by a forced choice test between two pictures, both of which matched the target title but only one of which had been in the sequence. Given correct detection, forced choice was above chance. The finding that both pre- and post-detection are robust with 12-picture sequences gives further support for feed-forward models of high-level perception. Meeting abstract presented at VSS 2013

Remapping time across space

Multiple lines of evidence indicate that visual attention's temporal properties differ between the left and right visual fields (LVF and RVF). Notably, recent electroencephalograph recordings indicate that event-related potentials peak earlier for LVF than for RVF targets on bilateral-stream rapid serial visual presentation (RSVP) identification tasks. Might this hastened neural response render LVF targets perceptually available sooner than RVF targets? If so, how might the visual system reconcile these timing differences to estimate simultaneity across the LVF and RVF? We approached these questions by presenting bilateral-stream RSVP displays that contained opposite-hemifield targets and requiring participants to judge both the targets' temporal order and simultaneity. The temporal order judgments (TOJs) revealed that participants perceived LVF targets ∼134 ms sooner than RVF targets. This LVF hastening approximates a full cycle of visual attention's canonical ∼10 Hz (∼100 ms) temporal resolution. In contrast, performance on the simultaneity task did not exhibit the LVF hastening observed on the TOJ task, despite identical retinal stimulation across the two tasks. This finding rules out a stimulus-driven "bottom-up" explanation for the task-specific behavior. Moreover, error patterns across the two tasks revealed that, within the decision stage of simultaneity judgments, participants remapped LVF targets, but not RVF targets, to a later time in the RSVP sequence. Such hemifield-specific remapping would effectively compensate for the cross-hemifield asymmetries in neural response latencies that could otherwise impair simultaneity estimates.

The processing of emotional stimuli during periods of limited attentional resources in schizophrenia.

When participants are asked to attend to two target stimuli in a rapid serial visual presentation sequence, the successful identification of the 1st target (T1) leads to transient impairment in reporting the 2nd target (T2)--this effect is known as the attentional blink (AB). In healthy individuals, this AB effect is either diminished or accentuated when emotional stimuli are presented in the T2 or T1 positions, respectively, suggesting that affective content influences bottom-up (i.e., exogenous) attention. In the current study, we conducted two separate experiments using the Emotional Attentional Blink paradigm where emotional words were presented in the T2 or T1 position to determine whether schizophrenia patients with high and low negative symptoms differ from controls in the extent to which emotional stimuli influence bottom-up attentional processes. Participants included 33 schizophrenia patients and 28 controls in Experiment 1 (T2 Task), and 30 schizophrenia patients and 24 controls in Experiment 2 (T1 Task). In both experiments, patients were divided into high (HI-NEG) and low (LOW-NEG) negative symptom subgroups using the Scale for the Assessment of Negative Symptoms. Results of Experiment 1 indicated that controls and LOW-NEG patients displayed the typical pattern of AB sparing at early lags for emotional relative to neutral words; however, HI-NEG patients showed no difference in T2 accuracy between emotional and neutral stimuli. Results of Experiment 2 indicated that controls and LOW-NEG patients displayed reduced T2 accuracy following unpleasant T1 stimuli, while HI-NEG patients showed no decrement in T2 accuracy after emotional T1s. Across both experiments, findings suggest that emotional stimuli have a bottom-up competitive advantage in LOW-NEG patients and controls; however, this bottom-up advantage is absent in HI-NEG patients.

Attentional Selection Dilates Perceived Duration

How do observers judge the passage of time at a short time-scale? Humans are not equipped with a dedicated sensory system for perceiving durations in the same way as they are equipped with systems for perceiving light and sound. Thus, subjective duration depends on the sensory and cognitive processes triggered by sensory input, eg visual or auditory stimuli. Previous studies have demonstrated that the dynamics of this sensory input (eg the rate of stimulus presentation) affect duration judgments. However, it is yet unclear whether automatic or attentive processing of such dynamics accounts for their effect on subjective duration. Automatic and attentive stimulus processing can be distinguished when stimuli are presented in a rapid serial visual presentation (RSVP) paradigm. The second of two targets embedded in an RSVP stream often fails to attract participants' attention and escapes conscious detection, in spite of being automatically processed at a perceptual level. In the present study, we presented RSVP streams and combined a target detection task with a prospective duration judgment task. We demonstrate in three experiments that the number of subjectively perceived target stimuli (and not the number of objectively presented targets) determines subjective duration of the entire RSVP sequence. Target stimuli which escape attentional selection did not affect perceived duration. This finding indicates that attentive rather than automatic processing of stimulus dynamics leads to the subjective time dilation of dynamic stimuli.

Working Memory Encoding Delays Top–Down Attention to Visual Cortex

The encoding of information from one event into working memory can delay high-level, central decision-making processes for subsequent events [e.g., Jolicoeur, P., & Dell'Acqua, R. The demonstration of short-term consolidation. Cognitive Psychology, 36, 138-202, 1998, doi:10.1006/cogp.1998.0684]. Working memory, however, is also believed to interfere with the deployment of top-down attention [de Fockert, J. W., Rees, G., Frith, C. D., & Lavie, N. The role of working memory in visual selective attention. Science, 291, 1803-1806, 2001, doi:10.1126/science.1056496]. It is, therefore, possible that, in addition to delaying central processes, the engagement of working memory encoding (WME) also postpones perceptual processing as well. Here, we tested this hypothesis with time-resolved fMRI by assessing whether WME serially postpones the action of top-down attention on low-level sensory signals. In three experiments, participants viewed a skeletal rapid serial visual presentation sequence that contained two target items (T1 and T2) separated by either a short (550 msec) or long (1450 msec) SOA. During single-target runs, participants attended and responded only to T1, whereas in dual-target runs, participants attended and responded to both targets. To determine whether T1 processing delayed top-down attentional enhancement of T2, we examined T2 BOLD response in visual cortex by subtracting the single-task waveforms from the dual-task waveforms for each SOA. When the WME demands of T1 were high (Experiments 1 and 3), T2 BOLD response was delayed at the short SOA relative to the long SOA. This was not the case when T1 encoding demands were low (Experiment 2). We conclude that encoding of a stimulus into working memory delays the deployment of attention to subsequent target representations in visual cortex.

Closing the loop in cortically-coupled computer vision: a brain–computer interface for searching image databases

We describe a closed-loop brain–computer interface that re-ranks an image database by iterating between user generated ‘interest’ scores and computer vision generated visual similarity measures. The interest scores are based on decoding the electroencephalographic (EEG) correlates of target detection, attentional shifts and self-monitoring processes, which result from the user paying attention to target images interspersed in rapid serial visual presentation (RSVP) sequences. The highest scored images are passed to a semi-supervised computer vision system that reorganizes the image database accordingly, using a graph-based representation that captures visual similarity between images. The system can either query the user for more information, by adaptively resampling the database to create additional RSVP sequences, or it can converge to a ‘done’ state. The done state includes a final ranking of the image database and also a ‘guess’ of the user's chosen category of interest. We find that the closed-loop system's re-rankings can substantially expedite database searches for target image categories chosen by the subjects. Furthermore, better reorganizations are achieved than by relying on EEG interest rankings alone, or if the system were simply run in an open loop format without adaptive resampling.

Can race enhance perceptual awareness? Evidence from the attentional blink paradigm

When two visual targets are presented in a rapid serial visual presentation sequence, people typically demonstrate a brief impairment in subjective awareness for the second visual target, a phenomenon known as the ‘attentional blink. Here we investigated the influence of race as a salient visual cue to modulate this temporary perceptual impairment for faces when shown in a rapid visual presentation sequence. In this experiment, Black and White participants viewed a rapid serial visual stream of distractor images and intact Black and White target faces, which were presented sequentially at varying degrees of temporal lag. When the second visual target was a face of a different race, Black and White participants demonstrated an attenuation of the attentional blink. This finding demonstrates that race can enhance subjective awareness and highlights the importance of race in shaping the initial perceptual experience of faces.