Rapid Serial Visual Presentation Research Articles

Semantic image sorting method for RSVP presentation

Objective. The rapid serial visual presentation (RSVP) paradigm, which is based on the electroencephalogram (EEG) technology, is an effective approach for object detection. It aims to detect the event-related potentials (ERP) components evoked by target images for rapid identification. However, the object detection performance within this paradigm is affected by the visual disparity between adjacent images in a sequence. Currently, there is no objective metric to quantify this visual difference. Consequently, a reliable image sorting method is required to ensure the generation of a smooth sequence for effective presentation. Approach. In this paper, we propose a novel semantic image sorting method for sorting RSVP sequences, which aims at generating sequences that are perceptually smoother in terms of the human visual experience. Main results. We conducted a comparative analysis between our method and two existing methods for generating RSVP sequences using both qualitative and quantitative assessments. A qualitative evaluation revealed that the sequences generated by our method were smoother in subjective vision and were more effective in evoking stronger ERP components than those generated by the other two methods. Quantitatively, our method generated semantically smoother sequences than the other two methods. Furthermore, we employed four advanced approaches to classify single-trial EEG signals evoked by each of the three methods. The classification results of the EEG signals evoked by our method were superior to those of the other two methods. Significance. In summary, the results indicate that the proposed method can significantly enhance the object detection performance in RSVP-based sequences.

Nasal inhalation does not improve memory of visual repetitions.

Several studies suggest that breathing entrains neural oscillations and thereby improves visual detection and memory performance during nasal inhalation. However, the evidence for this association is mixed, with some studies finding no, minor, or opposite effects. Here, we tested whether nasal breathing phase influences memory of repeated images presented in a rapid serial visual presentation (RSVP) task. The RSVP task is ideal for studying the effects of respiratory-entrained oscillations on visual memory because it engages critical aspects of sensory encoding that depend on oscillatory activity, such as fast processing of natural images, repetition detection, memory encoding, and retrieval. It also enables the presentation of a large number of stimuli during each phase of the breathing cycle. In two separate experiments (n = 72 and n = 142, respectively) where participants were explicitly asked to breathe through their nose, we found that nasal breathing phase at target presentation did not significantly affect memory performance. An exploratory analysis in the first experiment suggested a potential benefit for targets appearing approximately 1 s after inhalation. However, this finding was not replicated in the pre-registered second experiment with a larger sample. Thus, in two large sample experiments, we found no measurable impact of breathing phase on memory performance in the RSVP task. These results suggest that the natural breathing cycle does not have a significant impact on memory for repeated images and raise doubts about the idea that visual memory is broadly affected by the breathing phase.

PSAEEGNet: pyramid squeeze attention mechanism-based CNN for single-trial EEG classification in RSVP task.

Accurate classification of single-trial electroencephalogram (EEG) is crucial for EEG-based target image recognition in rapid serial visual presentation (RSVP) tasks. P300 is an important component of a single-trial EEG for RSVP tasks. However, single-trial EEG are usually characterized by low signal-to-noise ratio and limited sample sizes. Given these challenges, it is necessary to optimize existing convolutional neural networks (CNNs) to improve the performance of P300 classification. The proposed CNN model called PSAEEGNet, integrates standard convolutional layers, pyramid squeeze attention (PSA) modules, and deep convolutional layers. This approach arises the extraction of temporal and spatial features of the P300 to a finer granularity level. Compared with several existing single-trial EEG classification methods for RSVP tasks, the proposed model shows significantly improved performance. The mean true positive rate for PSAEEGNet is 0.7949, and the mean area under the receiver operating characteristic curve (AUC) is 0.9341 (p < 0.05). These results suggest that the proposed model effectively extracts features from both temporal and spatial dimensions of P300, leading to a more accurate classification of single-trial EEG during RSVP tasks. Therefore, this model has the potential to significantly enhance the performance of target recognition systems based on EEG, contributing to the advancement and practical implementation of target recognition in this field.

Event-related Potentials Indicate Target Processing in the Absence of Distractor Suppression during Rapid Serial Visual Presentation

Event-related Potentials Indicate Target Processing in the Absence of Distractor Suppression during Rapid Serial Visual Presentation

Contribution of low-level image statistics to EEG decoding of semantic content in multivariate and univariate models with feature optimization

Spatio-temporal patterns of evoked brain activity contain information that can be used to decode and categorize the semantic content of visual stimuli. However, this procedure can be biased by low-level image features independently of the semantic content present in the stimuli, prompting the need to understand the robustness of different models regarding these confounding factors. In this study, we trained machine learning models to distinguish between concepts included in the publicly available THINGS-EEG dataset using electroencephalography (EEG) data acquired during a rapid serial visual presentation paradigm. We investigated the contribution of low-level image features to decoding accuracy in a multivariate model, utilizing broadband data from all EEG channels. Additionally, we explored a univariate model obtained through data-driven feature selection applied to the spatial and frequency domains. While the univariate models exhibited better decoding accuracy, their predictions were less robust to the confounding effect of low-level image statistics. Notably, some of the models maintained their accuracy even after random replacement of the training dataset with semantically unrelated samples that presented similar low-level content. In conclusion, our findings suggest that model optimization impacts sensitivity to confounding factors, regardless of the resulting classification performance. Therefore, the choice of EEG features for semantic decoding should ideally be informed by criteria beyond classifier performance, such as the neurobiological mechanisms under study.

A novel deep transfer learning framework integrating general and domain-specific features for EEG-based brain–computer interface

The statistical variability of electroencephalographic (EEG) signals across individuals poses a common challenge for brain–computer interfaces (BCI). Specifically, the reuse of pre-recorded data from previous subjects presents a significant obstacle in decoding new subjects. To address these challenges, this paper introduces EEGTransferNet, an end-to-end modular transfer learning framework that enhances the similarity of statistical distributions across subjects. It utilizes convolutional neural networks as backbones for feature extraction and incorporates statistical distribution alignment and domain adaptation in different network layers to learn general features and domain-specific features respectively. The robust generalization design of EEGTransferNet enables it to handle multiple BCI paradigms using a single network making it efficient and effective. Its modular structure facilitates customization of backbone and transfer modules for diverse tasks. The optimal structure of EEGTransferNet was determined through analysis and extensive experimentation. We validated it on motor imagery (MI), error-related negativity (ERN), and rapid serial visual presentation (RSVP) datasets, which exhibits competitive performance compared to state-of-the-art methods. The experimental results confirm the effectiveness and superiority of EEGTransferNet.

Multiband task related components enhance rapid cognition decoding for both small and similar objects

The cortically-coupled target recognition system based on rapid serial visual presentation (RSVP) has a wide range of applications in brain computer interface (BCI) fields such as medical and military. However, in the complex natural environment backgrounds, the identification of event-related potentials (ERP) of both small and similar objects that are quickly presented is a research challenge. Therefore, we designed corresponding experimental paradigms and proposed a multi-band task related components matching (MTRCM) method to improve the rapid cognitive decoding of both small and similar objects. We compared the areas under the receiver operating characteristic curve (AUC) between MTRCM and other 9 methods under different numbers of training sample using RSVP-ERP data from 50 subjects. The results showed that MTRCM maintained an overall superiority and achieved the highest average AUC (0.6562 ± 0.0091). We also optimized the frequency band and the time parameters of the method. The verification on public data sets further showed the necessity of designing MTRCM method. The MTRCM method provides a new approach for neural decoding of both small and similar RSVP objects, which is conducive to promote the further development of RSVP-BCI.

Attention and feature binding in the temporal domain.

Previous studies have shown that illusory conjunction can emerge for both spatially and temporally proximal objects. However, the mechanisms involved in binding in the temporal domain are not yet fully understood. In the current study, we investigated the role of attentional processes in correct and incorrect temporal binding, and specifically how feature binding is affected by the speed of attentional engagement. In two experiments, participants searched for a target in a rapid serial visual presentation stream and reported its colour and alphanumeric identity. Temporal binding errors were frequent. Critically, when participants reported the identity of a distractor instead of a target, they were also more likely to report the colour of this distractor. This association was observed both within and between individuals. These findings suggest that attentional engagement facilitates the binding of temporally co-occurring features. We discuss these results within a 'diachronic' framework of selective attention, and also consider other factors that contribute to temporal binding errors.

Total sleep deprivation effects on the attentional blink.

The Attentional Blink (AB) is a phenomenon that reflects difficulty in detecting or identifying the second of two successive targets (T1 and T2) that are presented in rapid succession, between 200-500ms apart. The AB involves indicators of attentional and temporal integration mechanisms related to the early stages of visual processing. The aim of this study was to identify the effects of 24-h of sleep deprivation (total sleep deprivation, TSD) on the attentional and temporal integration mechanisms of the AB. Twenty-two undergraduate students were recorded during five successive days, in these three conditions: baseline (two days), TSD (one day), and recovery (two days). Each day, at around 12:00h, participants responded to a Rapid Serial Visual Presentation task (RSVP) that presented two targets separated by random intervals from 100 to 1000ms. The attentional mechanisms were assessed by the AB presence, the AB magnitude, and the AB interval, while the temporal integration mechanisms were evaluated by lag-1 sparing and order reversal responses. TSD negatively affected the attentional mechanisms, which is expressed by an overall reduction in performance, an extended AB interval, and a reduced AB magnitude. TSD also negatively affected the temporal integration mechanisms, manifested by an absence of lag-1 sparing and an increase in order reversals. These results suggest that people are still able to respond to two successive stimuli after 24h without sleep. However, it becomes more difficult to respond to both stimuli because the attentional and temporal integration mechanisms of the AB are impaired.

Modulation of attentional blink with sound in children with attention deficit hyperactivity disorder (ADHD)

Although it has been found that children with attention deficit hyperactivity disorder (ADHD) with impaired ability of inhibition control often showed greater attentional blink (AB), there is minimal information in regards to attention deficit in the interaction of AB enhancement with sound induction. A rapid serial visual presentation (RSVP) task was used by presenting an audiology pure tone evaluation simultaneously with T2 to investigate how auditory-driven visual perception enhances AB in children with ADHD. The results showed that children with ADHD had greater AB, and they performed worse than typically developing children (TD) in AB amplitude and recovery time. In addtion, the sound enhanced AB in TD children but interfered in children with ADHD. These results indicate that children with ADHD with faulty attentional allocation have fewer resources available for targets in the cross-modality AB, thus leading to a severe interference effect and a larger AB. It demonstrates that sound can only enhance AB when attentional resources are previously invested and the ability to suppress interference are still sufficient after dividing attention in cross-modality.

Pupillometry Analysis of Rapid Serial Visual Presentation at Five Presentation Rates

Pupillometry Analysis of Rapid Serial Visual Presentation at Five Presentation Rates

Effect of Cognitive Processing Style on Attentional Blink during Analogue Trauma

Background: This study attempted to examine the impact of each cognitive processing style (bottom-up, top-down) on attention patterns following a traumatic experience by measuring attentional blink (AB).Methods: Participants were 37 university students with no direct experience of traffic accidents. They were randomly assigned to either a single-task (bottom-up processing) group or a dual-task (top-down processing) group, who performed the Rapid Serial Visual Presentation (RSVP) after watching an analogue trauma video. The correct response rate to target stimuli was then compared between groups based on stimulus type and stimulus presentation interval.Results: In the dual-task group, no significant differences were found. However, in the single-task group, a trend towards a decrease in the correct response rate to the target stimulus was observed 200 ms after the presentation of the traumatic stimulus (i.e., AB).Conclusions: This tendency can be explained by the phenomenon of automatic attentional capture by traumatic stimuli, suggesting a relationship between bottom-up processing and attentional bias.

The Effects of a Novel Treatment for Hemianopic Dyslexia on Reading, Symptom Load, and Return to Work.

Reading disorders are frequent in homonymous hemianopia and are termed hemianopic dyslexia (HD). The existing treatment methods have shown improvements in reading speed, accuracy, and eye movements during reading. Yet, little is known about the transfer effects of such treatments on functional, reading-related tasks of daily life, e.g., reading phone numbers, finding typing errors or text memory. In addition, little is known about the effects on symptom load and return to work. Here, we examined a new reading therapy entailing three different methods-floating text, rapid serial visual presentation (RSVP) of single words, and the moving window technique-and evaluated their efficacy. Twenty-seven chronic HD patients were treated in a baseline design with treatment-free intervals before and after a treatment period of several months. HD was assessed with a battery of reading tests and a questionnaire about subjective symptom load at four time-points. Patients received all three reading therapies over several weeks. The results show significant and stable improvements during treatment within all measures. Approximately 63% of treated patients returned to work after the therapy. We concluded that our novel HD treatment led to widespread and lasting improvements in reading performance, generalized to functional reading tasks and reduced symptom load, and the majority of patients were able to return to work.

A toolbox for decoding BCI commands based on event-related potentials.

Commands in brain-computer interface (BCI) applications often rely on the decoding of event-related potentials (ERP). For instance, the P300 potential is frequently used as a marker of attention to an oddball event. Error-related potentials and the N2pc signal are further examples of ERPs used for BCI control. One challenge in decoding brain activity from the electroencephalogram (EEG) is the selection of the most suitable channels and appropriate features for a particular classification approach. Here we introduce a toolbox that enables ERP-based decoding using the full set of channels, while automatically extracting informative components from relevant channels. The strength of our approach is that it handles sequences of stimuli that encode multiple items using binary classification, such as target vs. nontarget events typically used in ERP-based spellers. We demonstrate examples of application scenarios and evaluate the performance of four openly available datasets: a P300-based matrix speller, a P300-based rapid serial visual presentation (RSVP) speller, a binary BCI based on the N2pc, and a dataset capturing error potentials. We show that our approach achieves performances comparable to those in the original papers, with the advantage that only conventional preprocessing is required by the user, while channel weighting and decoding algorithms are internally performed. Thus, we provide a tool to reliably decode ERPs for BCI use with minimal programming requirements.

Expectation Modifies the Representational Fidelity of Complex Visual Objects

Abstract Prediction has been shown to play a fundamental role in facilitating efficient perception of simple visual features such as orientation and motion, but it remains unclear whether expectations modulate neural representations of more complex stimuli. Here, we addressed this issue by characterising patterns of brain activity evoked by two-dimensional images of familiar, real-world objects which were either expected or unexpected based on a preceding cue. Participants (n = 30) viewed stimuli in rapid serial visual presentation (RSVP) streams which contained both high-fidelity and degraded (diffeomorphically warped) object images. Multivariate pattern analyses of electroencephalography (EEG) data were used to quantify and compare the degree of information represented in neural activity when stimuli were random (unpredictable), expected, or unexpected. Degraded images elicited reduced representational fidelity relative to high-fidelity images. However, degraded images were represented with improved fidelity when they were presented in expected relative to random sequence positions; and stimuli in unexpected sequence positions yielded reduced representational fidelity relative to random presentations. Most notably, neural responses to unexpected stimuli contained information pertaining to the expected (but not presented) stimulus. Debriefing at the conclusion of the experiment revealed that participants were not aware of the relationship between cue and target stimuli within the RSVP streams, suggesting that the differences in stimulus decoding between conditions arose in the absence of explicit predictive knowledge. Our findings extend fundamental understanding of how the brain detects and employs predictive relationships to modulate high-level visual perception.

Multi-source domain adaptation based tempo-spatial convolution network for cross-subject EEG classification in RSVP task

Objective. Many subject-dependent methods were proposed for electroencephalogram (EEG) classification in rapid serial visual presentation (RSVP) task, which required a large amount of data from new subject and were time-consuming to calibrate system. Cross-subject classification can realize calibration reduction or zero calibration. However, cross-subject classification in RSVP task is still a challenge. Approach. This study proposed a multi-source domain adaptation based tempo-spatial convolution (MDA-TSC) network for cross-subject RSVP classification. The proposed network consisted of three modules. First, the common feature extraction with multi-scale tempo-spatial convolution was constructed to extract domain-invariant features across all subjects, which could improve generalization of the network. Second, the multi-branch domain-specific feature extraction and alignment was conducted to extract and align domain-specific feature distributions of source and target domains in pairs, which could consider feature distribution differences among source domains. Third, the domain-specific classifier was exploited to optimize the network through loss functions and obtain prediction for the target domain. Main results. The proposed network was evaluated on the benchmark RSVP dataset, and the cross-subject classification results showed that the proposed MDA-TSC network outperformed the reference methods. Moreover, the effectiveness of the MDA-TSC network was verified through both ablation studies and visualization. Significance. The proposed network could effectively improve cross-subject classification performance in RSVP task, and was helpful to reduce system calibration time.

Single-Trial EEG Classification Using Spatio-Temporal Weighting and Correlation Analysis for RSVP-Based Collaborative Brain Computer Interface.

Since single brain computer interface (BCI) is limited in performance, it is necessary to develop collaborative BCI (cBCI) systems which integrate multi-user electroencephalogram (EEG) information to improve system performance. However, there are still some challenges in cBCI systems, including effective discriminant feature extraction of multi-user EEG data, fusion algorithms, time reduction of system calibration, etc. Methods: This study proposed an event-related potential (ERP) feature extraction and classification algorithm of spatio-temporal weighting and correlation analysis (STC) to improve the performance of cBCI systems. The proposed STC algorithm consisted of three modules. First, source extraction and interval modeling were used to overcome the problem of inter-trial variability. Second, spatio-temporal weighting and temporal projection were utilized to extract effective discriminant features for multi-user information fusion and cross-session transfer. Third, correlation analysis was conducted to match target/non-target templates for classification of multi-user and cross-session datasets. The collaborative cross-session datasets of rapid serial visual presentation (RSVP) from 14 subjects were used to evaluate the performance of the EEG classification algorithm. For single-user/collaborative EEG classification of within-session and cross-session datasets, STC had significantly higher performance than the existing state-of-the-art machine learning algorithms. It was demonstrated that STC was effective to improve the classification performance of multi-user collaboration and cross-session transfer for RSVP-based BCI systems, and was helpful to reduce the system calibration time.

Development of a novel rodent rapid serial visual presentation task reveals dissociable effects of stimulant versus nonstimulant treatments on attentional processes

The rapid serial visual presentation (RSVP) task and continuous performance tasks (CPT) are used to assess attentional impairments in patients with psychiatric and neurological conditions. This study developed a novel touchscreen task for rats based on the structure of a human RSVP task and used pharmacological manipulations to investigate their effects on different performance measures. Normal animals were trained to respond to a target image and withhold responding to distractor images presented within a continuous sequence. In a second version of the task, a false-alarm image was included, so performance could be assessed relative to two types of nontarget distractors. The effects of acute administration of stimulant and nonstimulant treatments for ADHD (amphetamine and atomoxetine) were tested in both tasks. Methylphenidate, ketamine, and nicotine were tested in the first task only. Amphetamine made animals more impulsive and decreased overall accuracy but increased accuracy when the target was presented early in the image sequence. Atomoxetine improved accuracy overall with a specific reduction in false-alarm responses and a shift in the attentional curve reflecting improved accuracy for targets later in the image sequence. However, atomoxetine also slowed responding and increased omissions. Ketamine, nicotine, and methylphenidate had no specific effects at the doses tested. These results suggest that stimulant versus nonstimulant treatments have different effects on attention and impulsive behaviour in this rat version of an RSVP task. These results also suggest that RSVP-like tasks have the potential to be used to study attention in rodents.

The Effect of Jittered Stimulus Onset Interval on Electrophysiological Markers of Attention in a Brain–Computer Interface Rapid Serial Visual Presentation Paradigm

Brain responses to discrete stimuli are modulated when multiple stimuli are presented in sequence. These alterations are especially pronounced when the time course of an evoked response overlaps with responses to subsequent stimuli, such as in a rapid serial visual presentation (RSVP) paradigm used to control a brain–computer interface (BCI). The present study explored whether the measurement or classification of select brain responses during RSVP would improve through application of an established technique for dealing with overlapping stimulus presentations, known as irregular or “jittered” stimulus onset interval (SOI). EEG data were collected from 24 healthy adult participants across multiple rounds of RSVP calibration and copy phrase tasks with varying degrees of SOI jitter. Analyses measured three separate brain signals sensitive to attention: N200, P300, and occipitoparietal alpha attenuation. Presentation jitter visibly reduced intrusion of the SSVEP, but in general, it did not positively or negatively affect attention effects, classification, or system performance. Though it remains unclear whether stimulus overlap is detrimental to BCI performance overall, the present study demonstrates that single-trial classification approaches may be resilient to rhythmic intrusions like SSVEP that appear in the averaged EEG.

Performance-based attentional control, but not self-reported attentional control, predicts changes in depressive symptoms in short-term psychotherapy

ObjectiveAlthough impairments in attentional control are pervasive across psychopathology, there is substantial individual differences. In the current study, we examined whether individual differences in self-reported and performance-based measures of attentional control predict changes in depressive symptoms and well-being in a diagnostically diverse sample of patients attending a CBT-based partial hospital program. MethodParticipants were 89 patients (56.2% men, 75.3% non-Hispanic White). At baseline, patients completed the self-reported Attentional Control Scale and the Rapid Serial Visual Presentation task (RSVP), a behavioral measure of attentional control. Depressive symptoms were assessed daily using the Patient Health Questionnaire and well-being was assessed using the Mental Health Continuum Short Form. ResultsOn average, greater self-reported attentional control was significantly associated with lower depressive symptoms, β = −0.49, t(52) = 4.84, p < .001, 95% CIs [−0.69, −0.29], and greater well-being, β = 0.45, t(53) = 3.90, p < .001, 95% CIs [0.22, 0.67]. More accurate task-based performance was associated with a decline in depressive symptoms over time, β = −0.02, t(32) = 2.50, p = .02, 95% CIs [−0.04, −0.01]. Neither self-reported nor performance-based measures of attentional control predicted changes in well-being. Finally, exploratory analyses suggest that depressive symptoms also improved over time for individuals who underestimated self-reported attentional control abilities relative to task-based performance, β = −0.19, t(32) = 2.23, p = .03, 95% CIs [−0.36, −0.02]. ConclusionsResults suggest that performance-based attentional control may be an important target for assessment and intervention, as well as a potential mechanism underlying risk and recovery.