Event-based Video Research Articles

Learning Adaptive Parameter Representation for Event-Based Video Reconstruction

Learning Adaptive Parameter Representation for Event-Based Video Reconstruction

HyperE2VID: Improving Event-Based Video Reconstruction via Hypernetworks.

Event-based cameras are becoming increasingly popular for their ability to capture high-speed motion with low latency and high dynamic range. However, generating videos from events remains challenging due to the highly sparse and varying nature of event data. To address this, in this study, we propose HyperE2VID, a dynamic neural network architecture for event-based video reconstruction. Our approach uses hypernetworks to generate per-pixel adaptive filters guided by a context fusion module that combines information from event voxel grids and previously reconstructed intensity images. We also employ a curriculum learning strategy to train the network more robustly. Our comprehensive experimental evaluations across various benchmark datasets reveal that HyperE2VID not only surpasses current state-of-the-art methods in terms of reconstruction quality but also achieves this with fewer parameters, reduced computational requirements, and accelerated inference times.

Event-based video deblurring based on image and event feature fusion

Event-based video deblurring is a method that performs deblurring by taking the event sequence data obtained from an event camera, which is composed of bio-inspired sensors, along with blurry frames as input. Event-based video deblurring has gained attention as a method that can overcome the limitations of conventional frame-based video deblurring. In this study, we propose a novel event-based video deblurring network based on convolution neural networks (CNNs). Unlike the existing event-based deblurring methods that only use event data, the proposed method fuses all the available information from current blurry frames, previously recovered sharp frames, and event data to deblur a video. Specifically, we propose an image and event feature fusion (IEFF) module to fuse event data with current intensity frame information. Additionally, we propose a current-frame reconstruction from previous-frame (CRP) module for acquiring a pseudo sharp frame from a previously recovered sharp frame and a fusion-based residual estimation (FRE) module, which fuses the event features with the image features of the previous sharp frame extracted from the CRP module. We demonstrate through a verification experiment using synthetic and real datasets that the proposed method has superior quantitative and qualitative results compared to state-of-the-art methods.

Multi-Stage Network for Event-Based Video Deblurring with Residual Hint Attention

Video deblurring aims at removing the motion blur caused by the movement of objects or camera shake. Traditional video deblurring methods have mainly focused on frame-based deblurring, which takes only blurry frames as the input to produce sharp frames. However, frame-based deblurring has shown poor picture quality in challenging cases of video restoration where severely blurred frames are provided as the input. To overcome this issue, recent studies have begun to explore the event-based approach, which uses the event sequence captured by an event camera for motion deblurring. Event cameras have several advantages compared to conventional frame cameras. Among these advantages, event cameras have a low latency in imaging data acquisition (0.001 ms for event cameras vs. 10 ms for frame cameras). Hence, event data can be acquired at a high acquisition rate (up to one microsecond). This means that the event sequence contains more accurate motion information than video frames. Additionally, event data can be acquired with less motion blur. Due to these advantages, the use of event data is highly beneficial for achieving improvements in the quality of deblurred frames. Accordingly, the results of event-based video deblurring are superior to those of frame-based deblurring methods, even for severely blurred video frames. However, the direct use of event data can often generate visual artifacts in the final output frame (e.g., image noise and incorrect textures), because event data intrinsically contain insufficient textures and event noise. To tackle this issue in event-based deblurring, we propose a two-stage coarse-refinement network by adding a frame-based refinement stage that utilizes all the available frames with more abundant textures to further improve the picture quality of the first-stage coarse output. Specifically, a coarse intermediate frame is estimated by performing event-based video deblurring in the first-stage network. A residual hint attention (RHA) module is also proposed to extract useful attention information from the coarse output and all the available frames. This module connects the first and second stages and effectively guides the frame-based refinement of the coarse output. The final deblurred frame is then obtained by refining the coarse output using the residual hint attention and all the available frame information in the second-stage network. We validated the deblurring performance of the proposed network on the GoPro synthetic dataset (33 videos and 4702 frames) and the HQF real dataset (11 videos and 2212 frames). Compared to the state-of-the-art method (D2Net), we achieved a performance improvement of 1 dB in PSNR and 0.05 in SSIM on the GoPro dataset, and an improvement of 1.7 dB in PSNR and 0.03 in SSIM on the HQF dataset.

Eventmd: High-Speed Moving Object Detection Based on Event-Based Video Frames

Eventmd: High-Speed Moving Object Detection Based on Event-Based Video Frames

Profound Learning Approach for Shot Boundary Location

Distinguishing shot boundaries and Gradual shot change happen to be critical research areas in the field of video retrieval, summarization and segmentation. Identification of Video shot boundaries is generally a significant and first step for ordering, retrieval, video segmentation and event-based video analysis and numerous other such aspects. There has been extraordinary research to improve the accuracy of SBD calculations. Advance research on this work is reported towards interpretable highlights of edges. In this paper, we projected an identification of video shot structure dependent on Convolutional Neural Systems (CNNs). The method proposed in this paper uses RAI Dataset with the CNN network with improved results than traditional practices. We have tested the proposed system on the TRECVID IACC.3 dataset and made use of Keras and TensorFlow algorithms. Segmentation of videos in to fragment which extract video shots and separate the video shot boundary. This sort of preprocessing strategy helps in improving both the speed and precision of the SBD calculation. In the next part, shots are extracted through semantic mark which is produced during shot detection. The projected algorithm performs well when compared with the Adaptive algorithm in terms of recall, Precision, and F1 measure. The data set consists of standard videos (Movie Gods of Egypt, Tennis Sports, Bulletin news, Traffic Video playback, Krishna’s Carton).

Wood export varies among decadal, annual, seasonal, and daily scale hydrologic regimes in a large, Mediterranean climate, mountain river watershed

The dynamics that move wood through and out of watersheds are complex and not yet fully understood. In this study, climatic conditions, hydrologic responses, and watershed processes were explored to better understand variations in wood export using aerial imagery, event-based video monitoring, and field measurements from the 1097km2 mountainous Mediterranean climate North Yuba River, California, watershed and its reservoir near the downstream outlet. Over a 30-year study period, 198\\5–2014, volumetric estimates of annual wood export into the reservoir, available for a subset of years, were used to investigate watershed-scale wood export dynamics. Variations in annual peak discharge explained 79% of the variance in interannual wood export, with 84% of total observed wood export (ca. >10,000m3 of wood per event) delivered by three discharge events of 19-year, 21.5-year, and 60-year flood recurrence intervals. Continuous video monitoring conducted during snowmelt season periods in 2010 and 2011 yielded wood discharge observations at minima 15% of statistical bankfull flow, while maximum daily discharge explained 55% of observed daily wood piece variation. No statistically significant wood discharge differences were found in snowmelt season observations, likely because of domination of the hydrograph by diurnal pulses within the seasonal cycle. A conceptual model and functional framework are introduced in support of a watershed-scale explanation of wood export, transport, and storage processes applicable to large, Mediterranean-climate, mountain watershed settings.

Guest editorial: Event-based video analysis/retrieval

Identification of events from visual cues is in general an arduous task because of complexmotion, cluttered backgrounds, occlusions, and geometric and photometric variations of the physical objects. This is even more challenging in case of detection of a logical chain of events, i.e., of a sequence of events called a workflow, and in case of the presence of multiple workflows of events in the environment, able to interact one with the other and affect one the outcome of the other. The recent research advances in computer vision and pattern recognition society have stimulated the development of a series of innovative algorithms, tools and methods for salient object detection and tracking in still images/video streams. These techniques are framed with appropriate descriptors (usually with invariance properties) such as the Scale-Invariant Feature Transform (SIFT) or the Speeded Up Robust Features (SURF), or the MPEG-7 visual descriptors. All these research methods can be considered as initial steps towards the ultimate goal for behavior/event understanding. However, automatic comprehension of someone’s behavior within a scene or even automatic supervision of workflows (e.g., industrial processes) is a complex research field of great attention but with limited results so far. Most of the current approaches presented involve machine learning theories, such as supervised or semi-supervised methods, object tracking algorithms, adaptation mechanisms to handle complex, dynamic and abrupt visual conditions and application-specific analysis topics. On the other hand, during the past few years, more and more people have been coping with the so called “Information Overload” phenomenon. On the basis of the i) diversity and plenitude of media information currently available on the web and ii) the gradual but quick role shift of the users from being solely content consumers to acting both as content consumers Multimed Tools Appl (2014) 69:247–251 DOI 10.1007/s11042-013-1726-z

Temporal-Based Video Event Detection and Retrieval

The fast proliferation of video data archives has increased the need for automatic video content analysis and semantic video retrieval. Since temporal information is critical in conveying video content, in this chapter, an effective temporal-based event detection framework is proposed to support high-level video indexing and retrieval. The core is a temporal association mining process that systematically captures characteristic temporal patterns to help identify and define interesting events. This framework effectively tackles the challenges caused by loose video structure and class imbalance issues. One of the unique characteristics of this framework is that it offers strong generality and extensibility with the capability of exploring representative event patterns with little human interference. The temporal information and event detection results can then be input into our proposed distributed video retrieval system to support the high-level semantic querying, selective video browsing and event-based video retrieval.

The dynamics of ensemble: the case for flamenco

We provide behavioural and quantitative analyses of coordination and synchronization in ensemble performance, using flamenco as a case study. The concept of ensemble is introduced and developed with the aid of connected network models, then applied to the flamenco ensemble. Flamenco performance is described in terms of ensemble interaction, including both the types of rhythmical accompaniment used by the individual classes of performer participating in the ensemble and the main sociocultural factors governing control of rhythm and cuing changes. These factors involve hierarchies relating to the degree to which each performer commands audience attention (focus) and the status of ensemble members within the company. We describe an observational approach to ensemble coordination with an event-based video analysis of a four-member ensemble performing a flamenco piece and a correlation-based motion analysis of two-person performance of simple rhythm.

Event based indexing of broadcasted sports video by intermodal collaboration

In this paper, we propose event-based video indexing, which is a kind of indexing by its semantical contents. Because video data is composed of multimodal information streams such as visual, auditory, and textual [closed caption (CC)] streams, we introduce a strategy of intermodal collaboration, i.e., collaborative processing taking account of the semantical dependency between these streams. Its aim is to improve the reliability and efficiency in contents analysis of video. Focusing here on temporal correspondence between visual and CC streams, the proposed method attempts to seek for time spans in which events are likely to take place through extraction of keywords from the CC stream and then to index shots in the visual stream. The experimental results for broadcasted sports video of American football games indicate that intermodal collaboration is effective for video indexing by the events such as touchdown (TD) and field goal (FG).

A semantic event-detection approach and its application to detecting hunts in wildlife video

We propose a three-level video-event detection methodology and apply it to animal-hunt detection in wildlife documentaries. The first level extracts color, texture, and motion features, and detects shot boundaries and moving object blobs. The mid-level employs a neural network to determine the object class of the moving object blobs. This level also generates shot descriptors that combine features from the first level and inferences from the mid-level. The shot descriptors are then used by the domain-specific inference process at the third level to detect video segments that match the user defined event model. The proposed approach has been applied to the detection of hunts in wildlife documentaries. Our method can be applied to different events by adapting the classifier at the intermediate level and by specifying a new event model at the highest level. Event-based video indexing, summarization, and browsing are among the applications of the proposed approach.