Event Streams Research Articles

Seasonal meteorological forcing controls runoff generation at multiple scales in a Mediterranean forested mountain catchment

Understanding hydrological processes during dry periods in Mediterranean mountain catchments is critical due to the increasing frequency of drought episodes. In this work, we aimed at characterizing the effect of the seasonal variability of meteorological forcing on the hydrological response of a small mountain forested catchment in the Mediterranean region. We analyzed the hydrological response and its timing based on hydrometric and electrical conductivity (EC) data for a year in the nested (0.31–2 km2) Re della Pietra catchment, in Central Italy. We used a soil moisture-based metric to distinguish between wet and dry periods and performed EC-based hydrograph separations during these two periods. The results revealed the important role of seasonality as a meteorological forcing affecting soil moisture, groundwater, streamflow response, and stream event water fractions. Wet and dry periods were distinctly characterized by different streamflow, soil moisture, and groundwater responses. Event water fractions in streamflow also highlight the relevance of the seasonality in the meteorological forcing on runoff generation. Particularly, at the rainfall-runoff event scale, the combination of antecedent soil moisture and precipitation depth controlled the non-linear response of streamflow, groundwater, and different event water fractions in the wet and dry periods.Stream stages and event water fractions also varied across nested spatial scales. Antecedent moisture conditions triggered a faster streamflow response due to higher connectivity along the hillslope in the wet period, with higher event water fractions in the upper sub-catchments (25 %) compared to the lower sub-catchments (15 %). Conversely, in the dry period, higher event water fractions were registered at the outlet (11 %) than at the headwaters (7 %). Time lags between peak flows observed across the nested catchment showed a complex pattern, suggesting the interaction of multiple factors controlling the timing of streamflow peaks in the study area. These findings contributed to improve our mechanistic insights into the elusive seasonal hydrological patterns observed in Mediterranean mountain forested catchments.

ANOMALY DETECTION USING MODIFIED LINEARITY BASED GREY SWARM OPTIMIZATION ALGORITHM AND EFFICIENT HYPER PARAMETER OPTIMIZATION TECHNIQUES

Attacks and anomalies are quiet common in today’s technological world. However it is important to identify these anomalies as soon as possible since these anomalies can cause various problem. Therefore anomaly detection plays a vital role in detecting these outliers. Anomaly detection detects suspicious activity which fundamentally falls outside of the respected pattern of behaviour. Anomaly detection aims at identifying the unpredicted and unexpected events in data streams which are commonly referred to as anomalous events. Therefore proposed modified linearity based grey swarm algorithm and hyperparameter optimization techniques for improving the performance of the proposed model. Hyperparameter optimization techniques implemented in the proposed model are genetic algorithm (GA), Grid Search and Random Search. Feature selection is implemented in the proposed model is modified linearity based grey swarm algorithm since it has the ability to reduce the convergence. In addition to modified linearity based grey swarm algorithm, modified objective function is employed in the proposed model as it has the capability to find solutions to problems which are difficult to solve. Finally the performance of the proposed model is evaluated and assessed using various performance metrics such as accuracy, recall, precision and F1 score. At last, proposed model is compared with the existing models and found that proposed model outperformed other existing models.

Bayesian neuromorphic imaging for single-photon LiDAR.

This paper proposes a Bayesian approach to enable single photon avalanche diode (SPAD) arrays to be used as pseudo event cameras that report changes in the scene. Motivated by the working principle of event cameras, which produce sparse events associated with light flux changes, we adopt a changepoint detection strategy to generate intensity and depth change event streams from direct time-of-flight (dToF) sequences measured by SPAD arrays. Although not our main goal, the algorithm also produces as a by-product, intensity and depth estimates. Unlike the output of passive event cameras that only correspond to light flux changes, the change events detected from the sequential dToFs can relate to changes in light flux and/or depth. The integration of the proposed Bayesian approach with single-photon LiDAR (SPL) systems provides a novel solution to achieve active neuromorphic 3D imaging that offers the advantages of significantly reduced output redundancy and in particular the capacity to report scene depth changes. For each pixel of the SPAD array, asynchronous events are generated by performing online Bayesian inference to detect changepoints and estimate the model parameters simultaneously from individual single-photon measurements. Experiments are conducted on synthetic data and real dToF measurements acquired by a 172×126 pixel SPAD camera to demonstrate the feasibility and efficiency of the proposed Bayesian approach.

The influence of recall direction on judgments of subjective temporal distance from the beginning of the COVID-19 pandemic lockdowns

IntroductionIn a series of 5 studies, Lam and Buehler found that first-year university students felt closer to a target event (the day they learned that they were accepted into university) when they recalled a stream of related events in a backward direction (a reverse-chronological order ending with the target event) than when they recalled those events in a forward direction (a forward-chronological order beginning with the target event).MethodsIn a conceptual replication of their Study 2, we asked participants how close they felt to the first day that lockdowns were imposed in response to the Covid-19 pandemic in the U.S. (federally mandated on March 13, 2020) following either backward or forward recall of a stream of related events.ResultsThe results of the present study ran directly counter to those of Lam and Buehler: participants rated the first day of lockdowns as feeling closer following forward recall than following backward recall.DiscussionPotential explanations for this reversal of Lam and Buehler's effect are discussed that focus on the temporal distortions that people have been found to experience when they think about autobiographical events that occurred at the beginning of the pandemic.

Event-based depth estimation with dense occlusion.

Occlusions pose a significant challenge to depth estimation in various fields, including automatic driving, remote sensing observation, and video surveillance. In this Letter, we propose a novel, to the best of our knowledge, depth estimation method for dense occlusion to estimate the depth behind occlusions. We design a comprehensive procedure using an event camera that consists of two steps: rough estimation and precise estimation. In the rough estimation, we reconstruct two segments of the event stream to remove occlusions and subsequently employ a binocular intersection measurement to estimate the rough depth. In the precise estimation, we propose a criterion that the maximum total length of edges of reconstructed images corresponds to the actual depth and search for the precise depth around the rough depth. The experimental results demonstrate that our method is implemented with relative errors of depth estimation below 1.05%.

Multimodal Shot Prediction Based on Spatial-Temporal Interaction between Players in Soccer Videos

Sports data analysis has significantly advanced and become an indispensable technology for planning strategy and enhancing competitiveness. In soccer, shot prediction has been realized on the basis of historical match situations, and its results contribute to the evaluation of plays and team tactics. However, traditional event prediction methods required tracking data acquired with expensive instrumentation and event stream data annotated by experts, and the benefits were limited to only some professional athletes. To tackle this problem, we propose a novel shot prediction method using soccer videos. Our method constructs a graph considering player relationships with audio and visual features as graph nodes. Specifically, by introducing players’ importance into the graph edge based on their field positions and team information, our method enables the utilization of knowledge that reflects the detailed match situation. Next, we extract latent features considering spatial–temporal interactions from the graph and predict event occurrences with uncertainty based on the probabilistic deep learning method. In comparison with several baseline methods and ablation studies using professional soccer match data, our method was confirmed to be effective as it demonstrated the highest average precision of 0.948, surpassing other methods.

DecoPa: Query Decomposition for Parallel Complex Event Processing

Systems for Complex Event Processing (CEP) enable the detection of predefined patterns in event streams. While the evaluation of CEP queries is computationally hard, scalability may be achieved by parallelization. Yet, existing approaches for parallel CEP are driven by static query properties, such as partitioning keys and states of the evaluation model. They largely neglect the rates with which processing units may ingest and compare events for query evaluation. In this paper, we present an approach for parallel CEP that is based on a flexible decomposition of CEP queries. Our idea is to guide the decomposition by the sustainable throughput of each processing unit, in order to maximize the overall performance. To this end, we introduce DecoPa plans for parallel CEP, provide a cost model for them, elaborate on their correctness and optimality, and present an algorithm for their construction. Experiments using a DecoPa implementation in Flink illustrate throughput gains of up to 12 orders of magnitude compared to state-of-the-art approaches.

LECalib: Line-based event camera calibration

Camera calibration is an essential prerequisite for event-based vision applications. Current event camera calibration methods typically involve using flashing patterns, reconstructing intensity images, and utilizing the features extracted from events. Existing methods are generally time-consuming and require manually placed calibration objects, which cannot meet the needs of rapidly changing scenarios. In this paper, we propose a line-based event camera calibration framework exploiting the geometric lines of commonly-encountered objects in man-made environments, e.g., doors, windows, boxes, etc. Different from previous methods, our method detects lines directly from event streams and leverages an event-line calibration model to generate the initial guess of camera parameters, which is suitable for both planar and non-planar lines. Then, a non-linear optimization is adopted to refine camera parameters. Both simulation and real-world experiments have demonstrated the feasibility and accuracy of our method, with validation performed on monocular and stereo event cameras. The source code is released at https://github.com/Zibin6/line_based_event_camera_calib.

Structural vibration frequency monitoring based on event camera

Compared with traditional cameras, event cameras (ECs) have the significant advantages of high temporal resolution, low data redundancy, and microsecond delay, which are beneficial in structural monitoring to extract the dense response of structures in both spatial and temporal dimensions. In this paper, the vibration frequency detection method based on ECs is studied. This study investigates vibration frequency detection methods based on ECs, and proposes two algorithms for vibration frequency detection based on event streams: marker tracking and event count. Experimental verification is conducted through forced vibration experiments. The results indicate that the event count method achieves high-precision measurement of vibration frequencies in the range of 10–190 Hz for different vibration scales, with a maximum relative error of 1% and an average relative error of 0.673%. The marker tracking method demonstrates a maximum relative error of 1.43% and an average relative error of 0.575% in frequency measurement for large-amplitude vibrations. However, as the amplitude decreases, the frequency measurement error increases. When the amplitude is less than 3 pixels, the frequency measurement error exceeds 30%, rendering the measurement results unreliable. This research provides technical support for high-precision structural vibration frequency monitoring and further expands the application of ECs in structural monitoring.

Event-based asynchronous HDR imaging by temporal incident light modulation

Dynamic range (DR) is a pivotal characteristic of imaging systems. Current frame-based cameras struggle to achieve high dynamic range imaging due to the conflict between globally uniform exposure and spatially variant scene illumination. In this paper, we propose AsynHDR, a pixel-asynchronous HDR imaging system, based on key insights into the challenges in HDR imaging and the unique event-generating mechanism of dynamic vision sensors (DVS). Our proposed AsynHDR system integrates the DVS with a set of LCD panels. The LCD panels modulate the irradiance incident upon the DVS by altering their transparency, thereby triggering the pixel-independent event streams. The HDR image is subsequently decoded from the event streams through our temporal-weighted algorithm. Experiments under the standard test platform and several challenging scenes have verified the feasibility of the system in HDR imaging tasks.

Multi-Connectivity for Multicast Video Streaming in Cellular Networks

The escalating demand for high-quality video streaming poses a major challenge for communication networks today. Catering to these bandwidth-hungry video streaming services places a huge burden on the limited spectral resources of communication networks, limiting the resources available for other services as well. Large volumes of video traffic can lead to severe network congestion, particularly during live streaming events, which require sending the same content to a large number of users simultaneously. For such applications, multicast transmission can effectively combat network congestion while meeting the demands of all the users by serving groups of users requesting the same content over shared spectral resources. Streaming services can further benefit from multi-connectivity, which allows users to receive content from multiple base stations simultaneously. Integrating multi-connectivity within multicast streaming can improve the system resource utilization while also providing seamless connectivity to multicast users. Toward this end, this work studied the impact of using multi-connectivity (MC) alongside wireless multicast for meeting the resource requirements of video streaming. Our findings show that MC substantially enhances the performance of multicast streaming, particularly benefiting cell-edge users who often experience poor channel conditions. We especially considered the number of users that can be simultaneously served by multi-connected multicast systems. It was observed that about 60% of the users that are left unserved under single-connectivity multicast are successfully served using the same resources by employing multi-connectivity in multicast transmissions. We prove that the optimal resource allocation problem for MC multicast is NP-hard. As a solution, we present a greedy approximation algorithm with an approximation factor of (1−1/e). Furthermore, we establish that no other polynomial-time algorithm can offer a superior approximation. To generate realistic video traffic patterns in our simulations, we made use of traces from actual videos. Our results clearly demonstrate that multi-connectivity leads to significant enhancements in the performance of multicast streaming.

E-DQN-Based Path Planning Method for Drones in Airsim Simulator under Unknown Environment.

To improve the rapidity of path planning for drones in unknown environments, a new bio-inspired path planning method using E-DQN (event-based deep Q-network), referring to introducing event stream to reinforcement learning network, is proposed. Firstly, event data are collected through an airsim simulator for environmental perception, and an auto-encoder is presented to extract data features and generate event weights. Then, event weights are input into DQN (deep Q-network) to choose the action of the next step. Finally, simulation and verification experiments are conducted in a virtual obstacle environment built with an unreal engine and airsim. The experiment results show that the proposed algorithm is adaptable for drones to find the goal in unknown environments and can improve the rapidity of path planning compared with that of commonly used methods.

Voxel-Based Multi-Scale Transformer Network for Event Stream Processing

Voxel-Based Multi-Scale Transformer Network for Event Stream Processing

Enhancing Robotic Perception through Synchronized Simulation and Physical Common-Sense Reasoning.

We introduce both conceptual and empirical findings arising from the amalgamation of a robotics cognitive architecture with an embedded physics simulator, aligning with the principles outlined in the intuitive physics literature. The employed robotic cognitive architecture, named CORTEX, leverages a highly efficient distributed working memory known as deep state representation. This working memory inherently encompasses a fundamental ontology, state persistency, geometric and logical relationships among elements, and tools for reading, updating, and reasoning about its contents. Our primary objective is to investigate the hypothesis that the integration of a physics simulator into the architecture streamlines the implementation of various functionalities that would otherwise necessitate extensive coding and debugging efforts. Furthermore, we categorize these enhanced functionalities into broad types based on the nature of the problems they address. These include addressing challenges related to occlusion, model-based perception, self-calibration, scene structural stability, and human activity interpretation. To demonstrate the outcomes of our experiments, we employ CoppeliaSim as the embedded simulator and both a Kinova Gen3 robotic arm and the Open-Manipulator-P as the real-world scenarios. Synchronization is maintained between the simulator and the stream of real events. Depending on the ongoing task, numerous queries are computed, and the results are projected into the working memory. Participating agents can then leverage this information to enhance overall performance.

Closing the loop: High-speed robotics with accelerated neuromorphic hardware.

The BrainScaleS-2 system is an established analog neuromorphic platform with versatile applications in the diverse fields of computational neuroscience and spike-based machine learning. In this work, we extend the system with a configurable realtime event interface that enables a tight coupling of its distinct analog network core to external sensors and actuators. The 1,000-fold acceleration of the emulated nerve cells allows us to target high-speed robotic applications that require precise timing on a microsecond scale. As a showcase, we present a closed-loop setup for commuting brushless DC motors: we utilize PyTorch to train a spiking neural network emulated on the analog substrate to control an electric motor from a sensory event stream. The presented system enables research in the area of event-driven controllers for high-speed robotics, including self-supervised and biologically inspired online learning for such applications.

Flow-Event Autoencoder: Event Stream Object Recognition Dataset Generation with Arbitrary High Temporal Resolution

Event camera has unique advantages in high temporal resolution and dynamic range and has shown potentials in several computer vision tasks. However, due to the novelty of this hardware, there’s a lack of large benchmark DVS event-stream datasets, including datasets for object recognition. In this work, we proposed an encoder-decoder method to augment event stream dataset from image and optical flow with arbitrary temporal resolution for object recognition task. We believe this proposed method can be generalized well in augmenting event stream vision data for object recognition and will help advance the development of event vision paradigm.

HARDVS: Revisiting Human Activity Recognition with Dynamic Vision Sensors

The main streams of human activity recognition (HAR) algorithms are developed based on RGB cameras which usually suffer from illumination, fast motion, privacy preservation, and large energy consumption. Meanwhile, the biologically inspired event cameras attracted great interest due to their unique features, such as high dynamic range, dense temporal but sparse spatial resolution, low latency, low power, etc. As it is a newly arising sensor, even there is no realistic large-scale dataset for HAR. Considering its great practical value, in this paper, we propose a large-scale benchmark dataset to bridge this gap, termed HARDVS, which contains 300 categories and more than 100K event sequences. We evaluate and report the performance of multiple popular HAR algorithms, which provide extensive baselines for future works to compare. More importantly, we propose a novel spatial-temporal feature learning and fusion framework, termed ESTF, for event stream based human activity recognition. It first projects the event streams into spatial and temporal embeddings using StemNet, then, encodes and fuses the dual-view representations using Transformer networks. Finally, the dual features are concatenated and fed into a classification head for activity prediction. Extensive experiments on multiple datasets fully validated the effectiveness of our model. Both the dataset and source code will be released at https://github.com/Event-AHU/HARDVS.

Color Event Enhanced Single-Exposure HDR Imaging

Single-exposure high dynamic range (HDR) imaging aims to reconstruct the wide-range intensities of a scene by using its single low dynamic range (LDR) image, thus providing significant efficiency. Existing methods pay high attention to restoring the luminance by inversing the tone-mapping process, while the color in the over-/under-exposed area cannot be well restored due to the information loss of the single LDR image. To address this issue, we introduce color events into the imaging pipeline, which record asynchronous pixel-wise color changes in a high dynamic range, enabling edge-like scene perception under challenging lighting conditions. Specifically, we propose a joint framework that incorporates color events and a single LDR image to restore both content and color of an HDR image, where an exposureaware transformer (EaT) module is designed to propagate the informative hints, provided by the normal-exposed LDR regions and the event streams, to the missing areas. In this module, an exposure-aware mask is estimated to suppress distractive information and strengthen the restoration of the over-/under-exposed regions. To our knowledge, we are the first to use color events to enhance single-exposure HDR imaging. We also contribute corresponding datasets, consisting of synthesized datasets and a real-world dataset collected by a DAVIS346-color camera. The datasets can be found at https://www.kaggle.com/datasets/mengyaocui/ce-hdr. Extensive experiments demonstrate the effectiveness of the proposed method.

Memristor-CMOS Hybrid Circuits Implementing Event-Driven Neural Networks for Dynamic Vision Sensor Camera.

For processing streaming events from a Dynamic Vision Sensor camera, two types of neural networks can be considered. One are spiking neural networks, where simple spike-based computation is suitable for low-power consumption, but the discontinuity in spikes can make the training complicated in terms of hardware. The other one are digital Complementary Metal Oxide Semiconductor (CMOS)-based neural networks that can be trained directly using the normal backpropagation algorithm. However, the hardware and energy overhead can be significantly large, because all streaming events must be accumulated and converted into histogram data, which requires a large amount of memory such as SRAM. In this paper, to combine the spike-based operation with the normal backpropagation algorithm, memristor-CMOS hybrid circuits are proposed for implementing event-driven neural networks in hardware. The proposed hybrid circuits are composed of input neurons, synaptic crossbars, hidden/output neurons, and a neural network's controller. Firstly, the input neurons perform preprocessing for the DVS camera's events. The events are converted to histogram data using very simple memristor-based latches in the input neurons. After preprocessing the events, the converted histogram data are delivered to an ANN implemented using synaptic memristor crossbars. The memristor crossbars can perform low-power Multiply-Accumulate (MAC) calculations according to the memristor's current-voltage relationship. The hidden and output neurons can convert the crossbar's column currents to the output voltages according to the Rectified Linear Unit (ReLU) activation function. The neural network's controller adjusts the MAC calculation frequency according to the workload of the event computation. Moreover, the controller can disable the MAC calculation clock automatically to minimize unnecessary power consumption. The proposed hybrid circuits have been verified by circuit simulation for several event-based datasets such as POKER-DVS and MNIST-DVS. The circuit simulation results indicate that the neural network's performance proposed in this paper is degraded by as low as 0.5% while saving as much as 79% in power consumption for POKER-DVS. The recognition rate of the proposed scheme is lower by 0.75% compared to the conventional one, for the MNIST-DVS dataset. In spite of this little loss, the power consumption can be reduced by as much as 75% for the proposed scheme.

An event-oriented diffusion-refinement method for sparse events completion

Event cameras or dynamic vision sensors (DVS) record asynchronous response to brightness changes instead of conventional intensity frames, and feature ultra-high sensitivity at low bandwidth. The new mechanism demonstrates great advantages in challenging scenarios with fast motion and large dynamic range. However, the recorded events might be highly sparse due to either limited hardware bandwidth or extreme photon starvation in harsh environments. To unlock the full potential of event cameras, we propose an inventive event sequence completion approach conforming to the unique characteristics of event data in both the processing stage and the output form. Specifically, we treat event streams as 3D event clouds in the spatiotemporal domain, develop a diffusion-based generative model to generate dense clouds in a coarse-to-fine manner, and recover exact timestamps to maintain the temporal resolution of raw data successfully. To validate the effectiveness of our method comprehensively, we perform extensive experiments on three widely used public datasets with different spatial resolutions, and additionally collect a novel event dataset covering diverse scenarios with highly dynamic motions and under harsh illumination. Besides generating high-quality dense events, our method can benefit downstream applications such as object classification and intensity frame reconstruction.