Event Boundaries Research Articles

The relationship between event boundary strength and pattern shifts across the cortical hierarchy during naturalistic movie-viewing.

Our continuous experience is spontaneously segmented by the brain into discrete events. However, the beginning of a new event (an event boundary) is not always sharply identifiable: phenomenologically, event boundaries vary in salience. How are the response profiles of cortical areas at event boundaries modulated by boundary strength during complex, naturalistic movie-viewing? Do cortical responses scale in a graded manner with boundary strength, or do they merely detect boundaries in a binary fashion? We measured "cortical boundary shifts" as transient changes in multi-voxel patterns at event boundaries with different strengths (weak, moderate, and strong), determined by across-subject agreement. Cortical regions with different processing timescales were examined. In auditory areas, which have short timescales, cortical boundary shifts exhibited a clearly graded profile both in group-level and individual-level analyses. In cortical areas with long timescales, including the default mode network, boundary strength modulated pattern shift magnitude at the individual subject level. We also observed a positive relationship between boundary strength and the extent of temporal alignment of boundary shifts across different levels of the cortical hierarchy. Additionally, hippocampal activity was highest at event boundaries for which cortical boundary shifts were most aligned across hierarchical levels. Overall, we found that event boundary strength modulated cortical pattern shifts strongly in sensory areas and more weakly in higher-level areas, and that stronger boundaries were associated with greater alignment of these shifts across the cortical hierarchy.

Investigating the Congestion Levels on a Mesoscopic Scale During Outdoor Events

In event management, preventing excessive overcrowding is not only essential for providing comfort but also crucial for ensuring safety. However, understanding the crowd dynamics of participants in outdoor events can be challenging. One of the primary reasons is the limited availability of sensing systems suitable for outdoor use. Challenges include the need for power outlets and adapting to dynamic environmental conditions and unclear event boundaries. Consequently, there is still uncertainty about which measurements can be conducted to scientifically manage crowding based on sound principles. Therefore, there is a need for systems that are capable of discerning spatial and temporal heterogeneity in density and accurately estimating the number of people in regions of interest in both sparse and congested areas. In this study, we propose a novel approach for measuring and understanding crowd states at outdoor events. We designed a highly portable measurement system utilizing Bluetooth technology to monitor crowd density in real time, ensuring uninterrupted data collection even in remote event locations. This system stands out for its ability to operate effectively under diverse weather and lighting conditions without power outlets, making it highly adaptable to various outdoor settings. In our experiments, conducted at four distinct outdoor event locations, we used a 360° camera and LiDARs to validate the system. For instance, we deployed the system at 40-m intervals in a shopping district during a high-density parade. This deployment enabled us to capture the movement of the crowd and estimate the total number of people within the district. A key finding was the system’s capability to detect temporal and spatial congestion in both sparse and crowded areas. The system’s potential to estimate crowd sizes and manage diverse outdoor events marks an advancement over traditional methods like cameras and LiDARs.

Event Boundaries Stretched and Compressed by Aspect: Temporal Segmentation in a First and a Second Language

AbstractEvent segmentation tests have shown substantial overlaps in how adults recognize starts and endpoints as events unfold. However, far less is known about what role different language systems play in the process. Variations in grammatical aspect have been shown to influence event processing. We tested how closely first language (L1) speakers of Mandarin and English versus Mandarin learners of English as a second language (L2) align event boundaries with event‐internal changes. We used two event boundary marking tasks (online/offline) and a sorting task. Participants saw 60 animations; their task was to indicate starts and endpoints. For punctual events (e.g., breaking a wall), Mandarin L1 speakers and Mandarin learners of English L2 were significantly further from event transitions than English L1 speakers. This pattern was replicated in the untimed experiment but not in sorting transitions, jointly suggesting that Mandarin L1 and Mandarin learners of English L2 may be less attentive to segmentation of phases surrounding transitions than English L1 speakers. We argue that this variation reflects L1‐specific encoding of ongoingness.

DiffSED: Sound Event Detection with Denoising Diffusion

Sound Event Detection (SED) aims to predict the temporal boundaries of all the events of interest and their class labels, given an unconstrained audio sample. Taking either the split-and-classify (i.e., frame-level) strategy or the more principled event-level modeling approach, all existing methods consider the SED problem from the discriminative learning perspective. In this work, we reformulate the SED problem by taking a generative learning perspective. Specifically, we aim to generate sound temporal boundaries from noisy proposals in a denoising diffusion process, conditioned on a target audio sample. During training, our model learns to reverse the noising process by converting noisy latent queries to the ground-truth versions in the elegant Transformer decoder framework. Doing so enables the model generate accurate event boundaries from even noisy queries during inference. Extensive experiments on the Urban-SED and EPIC-Sounds datasets demonstrate that our model significantly outperforms existing alternatives, with 40+% faster convergence in training. Code: https://github.com/Surrey-UPLab/DiffSED

Multimodal single-neuron, intracranial EEG, and fMRI brain responses during movie watching in human patients

We present a multimodal dataset of intracranial recordings, fMRI, and eye tracking in 20 participants during movie watching. Recordings consist of single neurons, local field potential, and intracranial EEG activity acquired from depth electrodes targeting the amygdala, hippocampus, and medial frontal cortex implanted for monitoring of epileptic seizures. Participants watched an 8-min long excerpt from the video “Bang! You’re Dead” and performed a recognition memory test for movie content. 3 T fMRI activity was recorded prior to surgery in 11 of these participants while performing the same task. This NWB- and BIDS-formatted dataset includes spike times, field potential activity, behavior, eye tracking, electrode locations, demographics, and functional and structural MRI scans. For technical validation, we provide signal quality metrics, assess eye tracking quality, behavior, the tuning of cells and high-frequency broadband power field potentials to familiarity and event boundaries, and show brain-wide inter-subject correlations for fMRI. This dataset will facilitate the investigation of brain activity during movie watching, recognition memory, and the neural basis of the fMRI-BOLD signal.

People can reliably detect action changes and goal changes during naturalistic perception.

As a part of ongoing perception, the human cognitive system segments others' activities into discrete episodes (event segmentation). Although prior research has shown that this process is likely related to changes in an actor's actions and goals, it has not yet been determined whether untrained observers can reliably identify action and goal changes as naturalistic activities unfold, or whether the changes they identify are tied to visual features of the activity (e.g., the beginnings and ends of object interactions). This study addressed these questions by examining untrained participants' identification of action changes, goal changes, and event boundaries while watching videos of everyday activities that were presented in both first-person and third-person perspectives. We found that untrained observers can identify goal changes and action changes consistently, and these changes are not explained by visual change and the onsets or offsets of contact with objects. Moreover, the action and goal changes identified by untrained observers were associated with event boundaries, even after accounting for objective visual features of the videos. These findings suggest that people can identify action and goal changes consistently and with high agreement, that they do so by using sensory information flexibly, and that the action and goal changes they identify may contribute to event segmentation.

Emotional arousal lingers in time to bind discrete episodes in memory

ABSTRACT Temporal stability and change in neutral contexts can transform continuous experiences into distinct and memorable events. However, less is known about how shifting emotional states influence these memory processes, despite ample evidence that emotion impacts non-temporal aspects of memory. Here, we examined if emotional stimuli influence temporal memory for recent event sequences. Participants encoded lists of neutral images while listening to auditory tones. At regular intervals within each list, participants heard emotional positive, negative, or neutral sounds, which served as “emotional event boundaries” that divided each sequence into discrete events. Temporal order memory was tested for neutral item pairs that either spanned an emotional sound or were encountered within the same auditory event. Encountering a highly arousing event boundary led to faster response times for items encoded within the next event. Critically, we found that highly arousing sounds had different effects on binding ongoing versus ensuing sequential representations in memory. Specifically, highly arousing sounds were significantly more likely to enhance temporal order memory for ensuing information compared to information that spanned those boundaries, especially for boundaries with negative valence. These findings suggest that within aversive emotional contexts, fluctuations in arousal help shape the temporal organisation of events in memory.

Associations between mind wandering, viewer interactions, and the meaningful structure of educational videos

Over the last years, and especially in the context of the COVID-19 pandemic, online videos have become an integral part of education. Therefore, it is essential to understand how to design such videos to provide an efficient and engaging learning experience for the viewers. Regarding learning success, the occurrence of mind wandering is of particular relevance. Thus, this preregistered research investigates the perception of online videos by analyzing the relationship between viewer behavior (such as pause button presses), mind wandering, and event segmentation by employing logfiles of a large German online video platform for educational videos combined with data collected in two online studies. Contrary to our expectations, we found a significant positive correlation between pausing, seek and resume viewer behavior, and mind wandering. Furthermore, replicating previous research, we identified a significant positive correlation between pausing behavior and meaningful event boundaries in the video. Additionally, a significant positive correlation between perceived event boundaries and mind wandering was shown. The triangulation of three data sources suggests that mind-wandering co-occurs with perceived event boundaries and observable, logged behavior (e.g., pressing the pause button). Given that the overall mind-wandering occurrence was also positively related to self-reported learning, a possible interpretation of the data is that event boundaries may trigger task-related mind-wandering that might even be conducive to learning.

Context-Aware Proposal-Boundary Network With Structural Consistency for Audiovisual Event Localization.

Audiovisual event localization aims to localize the event that is both visible and audible in a video. Previous works focus on segment-level audio and visual feature sequence encoding and neglect the event proposals and boundaries, which are crucial for this task. The event proposal features provide event internal consistency between several consecutive segments constructing one proposal, while the event boundary features offer event boundary consistency to make segments located at boundaries be aware of the event occurrence. In this article, we explore the proposal-level feature encoding and propose a novel context-aware proposal-boundary (CAPB) network to address audiovisual event localization. In particular, we design a local-global context encoder (LGCE) to aggregate local-global temporal context information for visual sequence, audio sequence, event proposals, and event boundaries, respectively. The local context from temporally adjacent segments or proposals contributes to event discrimination, while the global context from the entire video provides semantic guidance of temporal relationship. Furthermore, we enhance the structural consistency between segments by exploiting the above-encoded proposal and boundary representations. CAPB leverages the context information and structural consistency to obtain context-aware event-consistent cross-modal representation for accurate event localization. Extensive experiments conducted on the audiovisual event (AVE) dataset show that our approach outperforms the state-of-the-art methods by clear margins in both supervised event localization and cross-modality localization.

Event Integration and Temporal Differentiation: How Hierarchical Knowledge Emerges in Hippocampal Subfields through Learning.

Everyday life is composed of events organized by changes in contexts, with each event containing an unfolding sequence of occurrences. A major challenge facing our memory systems is how to integrate sequential occurrences within events while also maintaining their details and avoiding over-integration across different contexts. We asked if and how distinct hippocampal subfields come to hierarchically and, in parallel, represent both event context and subevent occurrences with learning. Female and male human participants viewed sequential events defined as sequences of objects superimposed on shared color frames while undergoing high-resolution fMRI. Importantly, these events were repeated to induce learning. Event segmentation, as indexed by increased reaction times at event boundaries, was observed in all repetitions. Temporal memory decisions were quicker for items from the same event compared to across different events, indicating that events shaped memory. With learning, hippocampal CA3 multivoxel activation patterns clustered to reflect the event context, with more clustering correlated with behavioral facilitation during event transitions. In contrast, in the dentate gyrus (DG), temporally proximal items that belonged to the same event became associated with more differentiated neural patterns. A computational model explained these results by dynamic inhibition in the DG. Additional similarity measures support the notion that CA3 clustered representations reflect shared voxel populations, while DG's distinct item representations reflect different voxel populations. These findings suggest an interplay between temporal differentiation in the DG and attractor dynamics in CA3. They advance our understanding of how knowledge is structured through integration and separation across time and context.

Event Segmentation in the Audio Description of Films

To make the content of films available to a visually impaired audience, a sighted translator can provide audio description (AD), a verbal description of visual events. To achieve this goal, the audio describer needs to select what to describe, when to describe it, and how to describe it, as well as to express the information aurally. The efficacy of this communication is critically dependent upon basic cognitive processes of how the sighted audio describer perceives and segments the film’s unfolding chain of events and in what way the visually impaired end users conceive the structure, content, and segmentation of such events in relation to the produced AD. There is, however, virtually no research on this interplay in relation to AD. In this study, we scrutinize live AD of a film from two trained audio describers and examine how events are structured, segmented and construed in their AD. Results demonstrate that the event segmentation experienced from the film is indeed a fundamental part of how AD is structured and construed. It was found that AD at event boundaries was highly sensitive to different spatiotemporal circumstances and this relationship depends on semantic resources for expressing AD. Lay summary In our everyday lives, we naturally organize and remember our experiences as meaningful sequences of events. This happens the same way when we watch movies – we see them as a series of events happening in different places, times, and with changing character dynamics. Now, think about making this cinematic experience accessible to people with visual impairments. This is where a sighted translator, called an audio describer, becomes really important. To make movies understandable for a visually impaired audience, a sighted audio describer describes crucial visual events in the film. Their goal is to create a "narrative equivalence" between the original film and the audio-described version. Thus, the main challenges for audio describers are deciding what information to describe, when to do it, and how to articulate it effectively. In the current case study, we looked at two audio descriptions for the Swedish film "Skumtimmen" (English title: "Echoes from the Dead"). These descriptions were created by trained audio describers. We wanted to see how these narrators organized, broke down, and verbalized the course of events in their audio descriptions. To achieve this, we focused on two important parts of storytelling – changes in time and location – and paid special attention to how often and in what way audio describers verbalized these event boundaries. What we found is that both describers indeed verbalized most event boundaries. This shows that these boundaries are a big part of how audio descriptions are put together. We also learned that the descriptions at these event boundaries were very sensitive to when and where key events unfolded in the story. The way these boundaries were described depended on fundamental cognitive and linguistic resources to talk about space and time. Our findings help us understand how breaking down events in a film is a crucial part of shaping audio descriptions. Plus, our research highlights how important it is to study how events are broken down and verbalized, giving useful insights to improve audio descriptions and train future audio describers. The ultimate goal is to make the experience better for visually impaired viewers, contributing to a more inclusive and enjoyable audiovisual world.

False memories for ending of events.

Memories are not perfect recordings of the past and can be subject to systematic biases. Memory distortions are often caused by our experience of what typically happens in a given situation. However, it is unclear whether memory for events is biased by the knowledge that events usually have a predictable structure (a beginning, middle, and an end). Using video clips of everyday situations, we tested how interrupting events at unexpected time points affects memory of how those events ended. In four free recall experiments (1, 2, 4, and 5), we found that interrupting clips just before a salient piece of action was completed, resulted in the false recall of details about how the clip might have ended. We refer to this as "event extension." On the other hand, interrupting clips just after one scene had ended and a new scene started, resulted in omissions of details about the true ending of the clip (Experiments 4 and 5). We found that these effects were present, albeit attenuated, when testing memory shortly after watching the video clips compared to a week later (Experiments 5a and 5b). The event extension effect was not present when memory was tested with a recognition paradigm (Experiment 3). Overall, we conclude that when people watch videos that violate their expectations of typical event structure, they show a bias to later recall the videos as if they had ended at a predictable event boundary, exhibiting event extension or the omission of details depending on where the original video was interrupted. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

The psychophysiology of Mastermind: Characterizing response times and blinking in a high-stakes television game show.

Television game shows have proven to be a valuable resource for studying human behavior under conditions of high stress and high stakes. However, previous work has focused mostly on choices-ignoring much of the rich visual information that is available on screen. Here, we take a first step to extracting more of this information by investigating the response times and blinking of contestants in the BBC show Mastermind. In Mastermind, contestants answer rapid-fire quiz questions while a camera slowly zooms in on their faces. By labeling contestants' behavior and blinks from 25 episodes, we asked how accuracy, response times, and blinking varied over the course of the game. For accuracy and response times, we tested whether contestants responded more accurately and more slowly after an error-exhibiting the "post-error increase in accuracy" and "post-error slowing" which has been repeatedly observed in the lab. For blinking, we tested whether blink rates varied according to the cognitive demands of the game-decreasing during periods of cognitive load, such as when pondering a response, and increasing at event boundaries in the task, such as the start of a question. In contrast to the lab, evidence for post-error changes in accuracy and response time was weak, with only marginal effects observed. In line with the lab, blinking varied over the course of the game much as we predicted. Overall, our findings demonstrate the potential of extracting dynamic signals from game shows to study the psychophysiology of behavior in the real world.

Hippocampal-cortical interactions during event boundaries support retention of complex narrative events

According to most memory theories, encoding involves continuous communication between the hippocampus and neocortex, but recent work has shown that key moments at the end of an event, called event boundaries, may be especially critical for memory formation. We sought to determine how communication between the hippocampus and neocortical regions during the encoding of naturalistic events related to subsequent retrieval of those events and whether this was particularly important at event boundaries. Participants encoded and recalled two cartoon movies during fMRI scanning. Higher functional connectivity between the hippocampus and the posterior medial network (PMN) at an event's offset is related to the subsequent successful recall of that event. Furthermore, hippocampal-PMN offset connectivity also predicted the amount of detail retrieved after a 2-day delay. These data demonstrate that the relationship between memory encoding and hippocampal-neocortical interaction is dynamic and biased toward boundaries.

Event segmentation structures temporal experience: Simultaneous dilation and contraction in rhythmic reproductions.

We experience the world in terms of both (continuous) time and (discrete) events, but time seems especially primitive-since we cannot perceive events without an underlying temporal medium. It is all the more intriguing, then, to discover that event segmentation can itself influence how we perceive the passage of time. We demonstrated this using a novel "rhythmic reproduction" task, in which people listened to irregular sequences of musical tones, and then immediately reproduced those rhythmic patterns from memory. Each sequence contained a single salient (and entirely task-irrelevant) perceptual event boundary, but the temporal placement of that boundary varied across multiple trials in which people reproduced the same underlying rhythmic pattern. Reproductions were systematically influenced by event boundaries in two complementary ways: tones immediately following event boundaries were delayed (being effectively played "too late" in the reproductions), while tones immediately preceding event boundaries were sped up (being effectively played "too early"). This demonstrates how event segmentation influences time perception in subtle and nonuniform ways that go beyond global temporal distortions-with dilation across events, but contraction within events. Events structure temporal experience, facilitating a give-and-take between the subjective expansion and contraction of time. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Local Compressed Video Stream Learning for Generic Event Boundary Detection

Local Compressed Video Stream Learning for Generic Event Boundary Detection

Opposite effects of emotion and event segmentation on temporal order memory and object-context binding

ABSTRACT Our daily lives unfold continuously, yet our memories are organised into distinct events, situated in a specific context of space and time, and chunked when this context changes (at event boundaries). Previous research showed that this process, termed event segmentation, enhances object-context binding but impairs temporal order memory. Physiologically, peaks in pupil dilation index event segmentation, similar to emotion-induced bursts of autonomic arousal. Emotional arousal also modulates object-context binding and temporal order memory. Yet, these two critical factors have not been systematically studied together. To address this gap, we ran a behavioural experiment using a paradigm validated to study event segmentation and extended it with emotion manipulation. During encoding, we sequentially presented greyscale objects embedded in coloured frames (colour changes defining events), with a neutral or aversive sound. During retrieval, we tested participants’ memory of temporal order memory and object-colour binding. We found opposite effects of emotion and event segmentation on episodic memory. While event segmentation enhanced object-context binding, emotion impaired it. On the contrary, event segmentation impaired temporal order memory, but emotion enhanced it. These findings increase our understanding of episodic memory organisation in laboratory settings, and potentially in real life with perceptual changes and emotion fluctuations constantly interacting.

Visual event boundaries restrict anchoring effects in decision-making.

Research on higher-level thought has revealed many principles of reasoning and decision-making but has rarely made contact with how we perceive the world in the first place. Here we show how a lower-level property of perception-the spontaneous and task-irrelevant segmentation of continuous visual stimulation into discrete events-can restrict one of the most notorious biases in decision-making: numerical anchoring. Subjects walked down a long room in an immersive three dimensional (3D) animation and then made a numerical judgment (e.g., of how much a suitcase is worth, or of how many hours of community service a minor crime deserved). Critically, some subjects passed through a doorway (a visual event boundary) during their virtual walk, while others did not-equating time, distance traveled, and visual complexity. The anchoring manipulation was especially innocuous, not appearing to be part of the experiment at all. Before the online trial began, subjects reported the two-digit numerical value from a visually distorted "CAPTCHA" ("to verify that you are human")-where this task-irrelevant anchor was either low (e.g., 29) or high (e.g., 92). With no doorway, we observed reliable anchoring effects: Higher CAPTCHA values produced higher estimates. With the doorway, however, such effects were attenuated or even eliminated. This generalized across tasks involving item valuations, factual questions, and legal judgments and in tests of both incidental and explicit anchoring. This demonstrates how spontaneous visual event segmentation can have profound consequences for higher-level thought.

Img2fmri: a python package for predicting group-level fMRI responses to visual stimuli using deep neural networks.

Here we introduce a new python package, img2fmri, to predict group-level fMRI responses to individual images. This prediction model uses an artificial deep neural network (DNN), as DNNs have been successful at predicting cortical responses in the human visual cortex when trained on real world visual categorization tasks. To validate our model, we predict fMRI responses to images our model has not previously seen from a new dataset. We then show how our frame-by-frame prediction model can be extended to a continuous visual stimulus by predicting an fMRI response to Pixar Animation Studio's short film Partly Cloudy. In analyzing the timepoint-timepoint similarity of our predicted fMRI response around human-annotated event boundaries in the movie, we find that our model outperforms the baseline model in describing the dynamics of the real fMRI response around these event boundaries, particularly in the timepoints just before and at an event. These analyses suggest that in visual areas of the brain, at least some of the temporal dynamics we see in the brain's processing of continuous, naturalistic stimuli can be explained by dynamics in the stimulus itself, since they can be predicted from our frame-by-frame model. All code, analyses, tutorials, and installation instructions can be found at https://github.com/dpmlab/img2fmri.

Bayesian Surprise Predicts Human Event Segmentation in Story Listening.

Event segmentation theory posits that people segment continuous experience into discrete events and that event boundaries occur when there are large transient increases in prediction error. Here, we set out to test this theory in the context of story listening, by using a deep learning language model (GPT-2) to compute the predicted probability distribution of the next word, at each point in the story. For three stories, we used the probability distributions generated by GPT-2 to compute the time series of prediction error. We also asked participants to listen to these stories while marking event boundaries. We used regression models to relate the GPT-2 measures to the human segmentation data. We found that event boundaries are associated with transient increases in Bayesian surprise but not with a simpler measure of prediction error (surprisal) that tracks, for each word in the story, how strongly that word was predicted at the previous time point. These results support the hypothesis that prediction error serves as a control mechanism governing event segmentation and point to important differences between operational definitions of predictionerror.