Event Cameras Research Articles

Fourier‐Based Action Recognition for Wildlife Behavior Quantification with Event Cameras

Event cameras are novel bioinspired vision sensors that measure pixel‐wise brightness changes asynchronously instead of images at a given frame rate. They offer promising advantages, namely, a high dynamic range, low latency, and minimal motion blur. Modern computer vision algorithms often rely on artificial neural network approaches, which require image‐like representations of the data and cannot fully exploit the characteristics of event data. Herein, approaches to action recognition based on the Fourier transform are proposed. The approaches are intended to recognize oscillating motion patterns commonly present in nature. In particular, the approaches are applied to a recent dataset of breeding penguins annotated for “ecstatic display,” a behavior where the observed penguins flap their wings at a certain frequency. It is found that the approaches are both simple and effective, producing slightly lower results than a deep neural network (DNN) while relying just on a tiny fraction of the parameters compared to the DNN (five orders of magnitude fewer parameters). They work well despite the uncontrolled, diverse data present in the dataset. It is hoped that this work opens a new perspective on event‐based processing and action recognition.

High-frequency Structure Deformation Measurement Based on an Event Camera

High-frequency Structure Deformation Measurement Based on an Event Camera

Hyper real-time flame detection: Dynamic insights from event cameras and FlaDE dataset

Hyper real-time flame detection: Dynamic insights from event cameras and FlaDE dataset

Spatiotemporal Focus and Lightweight Memory Network for Continuous Object Detection With Event Camera

Spatiotemporal Focus and Lightweight Memory Network for Continuous Object Detection With Event Camera

SGE: Structured Light System Based on Gray Code with an Event Camera

SGE: Structured Light System Based on Gray Code with an Event Camera

High-performance surface defect detection of aluminum substrate based on event camera

Abstract Traditional industrial surface defect detection often employs CCD/CMOS cameras, but they are difficult to detect the minute defects on aluminum substrates in highly dynamic industrial scenes due to their nature. Event camera is a novel high-resolution vision sensor which measures per-pixel brightness changes in an asynchronous manner and outputs as event information flow (EIF). Small and weak defects on aluminum substrate can be captured by event camera effectively, but the EIF contain a large amount of noise, making it difficult to perform accurate and high-precision defect detection. To address this problem, we propose a frame aggregation method to realize good event information flow processing (EIFP), and then use an improved circle detection method to locate the aluminum substrate in each frame, removing abundant event information outside the aluminum substrate. Subsequently, we enhance the event signals under different frames based on optical flow tracking using multiple features, and construct a semi-supervised detector based on pseudo-labels to achieve high-precision defect localization. Finally, considering the small inter-class differences in defects on the surface of aluminum substrates, we construct a defect class corrector based on ensemble learning to enhance the ability to determine defect classes, achieving high-precision automatic quality inspection of aluminum substrate surfaces. The performance of our method is compared with other advanced methods based on event camera data of aluminum substrates in real industrial scenarios. The experimental results show that our method has improved the detection accuracy by about 10% and the classification accuracy by about 25% compared to the original state-of-the-art methods.

How Many Events are Needed for One Reconstructed Image Using an Event Camera?

Abstract. Event cameras offer significant advantages over traditional cameras, including high temporal resolution, high dynamic range, and low power consumption. However, reconstructing images from the asynchronous events generated by these cameras presents unique challenges. This paper investigates the optimal number of events needed for high-quality image reconstruction using event cameras. We evaluate two primary reconstruction strategies—fixed time window and fixed number of events—across various dynamic and static scenes. Our study includes scenarios with different lighting conditions and camera movements. Using the state-of-the-art E2VID algorithm, we perform both qualitative and quantitative analyses of the reconstructed images, comparing them with reference frames from a traditional RGB camera. Our results demonstrate the trade-offs between temporal resolution and image quality for each reconstruction strategy, providing insights into the optimal settings for different applications. This research offers practical guidelines for selecting appropriate reconstruction parameters to achieve the better image quality from event cameras.

Asynchronous Visual-Inertial Odometry for Event Cameras

Abstract. Event cameras are bio-inspired visual sensors that output changes in pixel-level brightness asynchronously instead of standard intensity frames at a fixed rate. These cameras offer reliable visual information in high-speed motion and high dynamic range (HDR) scenes, addressing the limitations of traditional cameras in such scenarios. Therefore, research of integrating event cameras into established visual algorithms holds significant value. In this study, based on traditional Visual-Inertial Odometry (VIO) frameworks, we proposed an innovative asynchronous monocular event-based inertial odometry method to fully exploit the benefits of event cameras. First, the corner features are extracted separately from the raw event stream and the time surface map, followed by uniform feature selection to accurately describe three-dimensional spatial geometry. Then, feature tracking is achieved by integrating these two event representation methods. In addition, our method obtains stable and high frequency state estimation by fusing event and IMU measurements through graph optimization. We validate the effectiveness of our proposed approach, comparing with several state-of-the-art EVIO systems and VIO systems.

Event Camera Denoising Using Asynchronous Spatio-Temporal Event Denoising Neural Network

Abstract. Compared to conventional cameras, event cameras represent a noteworthy advancement in neuromorphic imaging technology, garnering considerable attention from researchers due to their distinct advantages. However, event cameras are susceptible to significant levels of measurement noise, which can detrimentally affect the performance of algorithms reliant on event stream for tasks such as perception and navigation. In this study, we introduce a novel method for denoising event stream, aiming to filter out events that do not accurately reflect genuine logarithmic intensity changes within the observed scene. Our approach focuses on the asynchronous nature and spatiotemporal properties of events, culminating in the development of a novel Asynchronous Spatio- Temporal Event Denoising neural Network(ASTEDNet). This network operates directly on event streams, circumventing the need to convert event stream into denser formats like image frames, thereby preserving their inherent asynchronous nature. Drawing upon principles from graph encoding and temporal convolutional networks, we incorporate spatiotemporal feature attention mechanisms to capture the temporal and spatial correlations between events. This enables the classification of each active event pixel in the original stream as either representing a genuine intensity change or noise. Comparative evaluations conducted on multiple datasets against state-of-the-art methods demonstrate the remarkable efficacy and robustness of our proposed algorithm in noise removal while retaining meaningful event information within the scene.

Strong Robust LIO System Based on Event Camera Assistance

Abstract. At present, the LiDAR system can achieve high precision and is extensively used for indoor and outdoor mobile positioning and mapping. However, LiDAR systems still face issues in cluttered environments where strong features are absent, leading to a degradation of the LiDAR-based solution. When the carrier movement involves high-speed or prolonged exposure to the mirror wall, it can cause severe degradation issues or even positioning failures in the laser slam system. Event cameras are vision sensors inspired by biology that exhibit strong robustness in high dynamic and low texture environments, potentially leading to better performance in such environments. However, some issues with event cameras remain unresolved. In this paper, a multi-source fusion method based on EVIO, LIO and IMU trajectory layer post-processing method is proposed, which will fully consider the robustness of event camera in high dynamic environment and the high precision advantage of Lidar in conventional environment, and use an algorithm based on normalized uncertainty. The elastic multi-source fusion of event camera and LiDAR is realized and tested in realde environment. Experimental results show that the proposed algorithm can effectively improve the accuracy of event camera and LiDAR. Compared to the current more advanced algorithms, the proposed algorithm effectively addresses the problem of LIO in degraded environments. Additionally, it mitigates the scale inaccuracy and divergence of the EVIO trajectory to some extent. Compared to LIO, the algorithm can reduce the maximum position error by approximately 30% and increase the overall position accuracy by 32%. Additionally, it can significantly constrain the divergence of errors in the Y direction, improving its accuracy by about 75% and 65% compared to the LIO and EVIO algorithms, respectively.

Event Stream Denoising Method Based on Spatio-Temporal Density and Time Sequence Analysis.

An event camera is a neuromimetic sensor inspired by the human retinal imaging principle, which has the advantages of high dynamic range, high temporal resolution, and low power consumption. Due to the interference of hardware and software and other factors, the event stream output from the event camera usually contains a large amount of noise, and traditional denoising algorithms cannot be applied to the event stream. To better deal with different kinds of noise and enhance the robustness of the denoising algorithm, based on the spatio-temporal distribution characteristics of effective events and noise, an event stream noise reduction and visualization algorithm is proposed. The event stream enters fine filtering after filtering the BA noise based on spatio-temporal density. The fine filtering performs time sequence analysis on the event pixels and the neighboring pixels to filter out hot noise. The proposed visualization algorithm adaptively overlaps the events of the previous frame according to the event density difference to obtain clear and coherent event frames. We conducted denoising and visualization experiments on real scenes and public datasets, respectively, and the experiments show that our algorithm is effective in filtering noise and obtaining clear and coherent event frames under different event stream densities and noise backgrounds.

Hierarchical Event-RGB Interaction Network for single-eye expression recognition

The Single-eye Expression Recognition task stands as a crucial vision task, aimed at decoding human emotional states through careful examination of the eye region. Nevertheless, traditional cameras face challenges in detecting and capturing relevant biological information, especially under demanding lighting conditions such as dim environments, high exposure scenarios, or when other radiation sources are present. In this regard, we use a new type of sensor data that can resist extreme lighting conditions, namely event camera data, to improve the performance of single-eye expression recognition. To this end, we propose a novel Hierarchical Event-RGB Interaction Network (HI-Net), to fully integrate RGB and event data to overcome the extreme lighting challenges faced by the single-eye expression recognition task. The HI-Net contains two novel designs: Event-RGB Semantic Interaction Mechanism (ER-SIM) and Hierarchical Semantics Modeling (HSM) Scheme. The former aims to achieve interaction between Event and RGB modality semantics, while the latter aims to obtain high-quality modality semantic representations. In the ER-SIM, we employ an effective cross-attention mechanism to facilitate information fusion, to adaptively integrate and complement multi-scale Event and RGB semantics to cope with extreme lighting conditions. In HSM Scheme, we first explore multi-scale contextual semantics for the event modality and the RGB modality respectively. Then, we perform a semantics interaction strategy for these multi-scale contextual semantics, to enhance each modality's semantic representation. Extensive experiments demonstrate that our HI-Net significantly outperforms many state-of-the-art methods on the single-eye expression recognition task, especially under degraded lighting conditions.

Exploring event-based human pose estimation with 3D event representations

Human pose estimation is a fundamental and appealing task in computer vision. Although traditional cameras are commonly applied, their reliability decreases in scenarios under high dynamic range or heavy motion blur, where event cameras offer a robust solution. Predominant event-based methods accumulate events into frames, ignoring the asynchronous and high temporal resolution that is crucial for distinguishing distinct actions. To address this issue and to unlock the 3D potential of event information, we introduce two 3D event representations: the Rasterized Event Point Cloud (RasEPC) and the Decoupled Event Voxel (DEV). The RasEPC aggregates events within concise temporal slices at identical positions, preserving their 3D attributes along with statistical information, thereby significantly reducing memory and computational demands. Meanwhile, the DEV representation discretizes events into voxels and projects them across three orthogonal planes, utilizing decoupled event attention to retrieve 3D cues from the 2D planes. Furthermore, we develop and release EV-3DPW, a synthetic event-based dataset crafted to facilitate training and quantitative analysis in outdoor scenes. Our methods are tested on the DHP19 public dataset, MMHPSD dataset, and our EV-3DPW dataset, with further qualitative validation via a derived driving scene dataset EV-JAAD and an outdoor collection vehicle. Our code and dataset have been made publicly available at https://github.com/MasterHow/EventPointPose.

Hypergraph-Based Multi-View Action Recognition Using Event Cameras.

Action recognition from video data forms a cornerstone with wide-ranging applications. Single-view action recognition faces limitations due to its reliance on a single viewpoint. In contrast, multi-view approaches capture complementary information from various viewpoints for improved accuracy. Recently, event cameras have emerged as innovative bio-inspired sensors, leading to advancements in event-based action recognition. However, existing works predominantly focus on single-view scenarios, leaving a gap in multi-view event data exploitation, particularly in challenges like information deficit and semantic misalignment. To bridge this gap, we introduce HyperMV, a multi-view event-based action recognition framework. HyperMV converts discrete event data into frame-like representations and extracts view-related features using a shared convolutional network. By treating segments as vertices and constructing hyperedges using rule-based and KNN-based strategies, a multi-view hypergraph neural network that captures relationships across viewpoint and temporal features is established. The vertex attention hypergraph propagation is also introduced for enhanced feature fusion. To prompt research in this area, we present the largest multi-view event-based action dataset THUMV-EACT-50, comprising 50 actions from 6 viewpoints, which surpasses existing datasets by over tenfold. Experimental results show that HyperMV significantly outperforms baselines in both cross-subject and cross-view scenarios, and also exceeds the state-of-the-arts in frame-based multi-view action recognition.

"Seeing" ENF From Neuromorphic Events: Modeling and Robust Estimation.

Most artificial lights exhibit subtle fluctuations in intensity and frequency in response to the influence of the grid's alternating current, providing the potential to estimate the Electric Network Frequency (ENF) from conventional frame-based videos. Nevertheless, the performance of Video-based ENF (V-ENF) estimation largely relies on the imaging quality and thus may suffer from significant interference caused by non-ideal sampling, scene diversity, motion interference, and extreme lighting conditions. In this paper, we show that the ENF can be extracted without the above limitations from a new modality provided by the so-called event camera, a neuromorphic sensor that encodes the light intensity variations and asynchronously emits events with extremely high temporal resolution and high dynamic range. Specifically, we formulate and validate the physical mechanism for the ENF captured in events and then propose a simple yet robust Event-based ENF (E-ENF) estimation method through mode filtering and harmonic enhancement. To validate the effectiveness, we build the first Event-Video ENF Dataset (EV-ENFD) and its extension EV-ENFD+ with diverse scenarios, including static, dynamic, and extreme lighting scenes. Comprehensive experiments have been conducted on our proposed datasets, showcasing that our proposed E-ENF significantly outperforms the V-ENF in extracting accurate ENF traces, especially in challenging environments.

EvHandPose: Event-Based 3D Hand Pose Estimation With Sparse Supervision.

Event camera shows great potential in 3D hand pose estimation, especially addressing the challenges of fast motion and high dynamic range in a low-power way. However, due to the asynchronous differential imaging mechanism, it is challenging to design event representation to encode hand motion information especially when the hands are not moving (causing motion ambiguity), and it is infeasible to fully annotate the temporally dense event stream. In this paper, we propose EvHandPose with novel hand flow representations in Event-to-Pose module for accurate hand pose estimation and alleviating the motion ambiguity issue. To solve the problem under sparse annotation, we design contrast maximization and hand-edge constraints in Pose-to-IWE (Image with Warped Events) module and formulate EvHandPose in a weakly-supervision framework. We further build EvRealHands, the first large-scale real-world event-based hand pose dataset on several challenging scenes to bridge the real-synthetic domain gap. Experiments on EvRealHands demonstrate that EvHandPose outperforms previous event-based methods under all evaluation scenes, achieves accurate and stable hand pose estimation with high temporal resolution in fast motion and strong light scenes compared with RGB-based methods, generalizes well to outdoor scenes and another type of event camera, and shows the potential for the hand gesture recognition task.

EV-TIFNet: lightweight binocular fusion network assisted by event camera time information for 3D human pose estimation

EV-TIFNet: lightweight binocular fusion network assisted by event camera time information for 3D human pose estimation

Neural image re-exposure

Images and videos often suffer from issues such as motion blur, video discontinuity, or rolling shutter artifacts. Prior studies typically focus on designing specific algorithms to address individual issues. In this paper, we highlight that these issues, albeit differently manifested, fundamentally stem from sub-optimal exposure processes. With this insight, we propose a paradigm termed re-exposure, which resolves the aforementioned issues by performing exposure simulation. Following this paradigm, we design a new architecture, which constructs visual content representation from images and event camera data, and performs exposure simulation in a controllable manner. Experiments demonstrate that, using only a single model, the proposed architecture can effectively address multiple visual issues, including motion blur, video discontinuity, and rolling shutter artifacts, even when these issues co-occur.

Eye Tracking Based on Event Camera and Spiking Neural Network

An event camera generates an event stream based on changes in brightness, retaining only the characteristics of moving objects, and addresses the high power consumption associated with using high-frame-rate cameras for high-speed eye-tracking tasks. However, the asynchronous incremental nature of event camera output has not been fully utilized, and there are also issues related to missing event datasets. Combining the temporal information encoding and state-preserving properties of a spiking neural network (SNN) with an event camera, a near-range eye-tracking algorithm is proposed as well as a novel event-based dataset for validation and evaluation. According to experimental results, the proposed solution outperforms artificial neural network (ANN) algorithms, while computational time remains only 12.5% of that of traditional SNN algorithms. Furthermore, the proposed algorithm allows for self-adjustment of time resolution, with a maximum achievable resolution of 0.081 ms, enhancing tracking stability while maintaining accuracy.

Application of Event Cameras and Neuromorphic Computing to VSLAM: A Survey.

Simultaneous Localization and Mapping (SLAM) is a crucial function for most autonomous systems, allowing them to both navigate through and create maps of unfamiliar surroundings. Traditional Visual SLAM, also commonly known as VSLAM, relies on frame-based cameras and structured processing pipelines, which face challenges in dynamic or low-light environments. However, recent advancements in event camera technology and neuromorphic processing offer promising opportunities to overcome these limitations. Event cameras inspired by biological vision systems capture the scenes asynchronously, consuming minimal power but with higher temporal resolution. Neuromorphic processors, which are designed to mimic the parallel processing capabilities of the human brain, offer efficient computation for real-time data processing of event-based data streams. This paper provides a comprehensive overview of recent research efforts in integrating event cameras and neuromorphic processors into VSLAM systems. It discusses the principles behind event cameras and neuromorphic processors, highlighting their advantages over traditional sensing and processing methods. Furthermore, an in-depth survey was conducted on state-of-the-art approaches in event-based SLAM, including feature extraction, motion estimation, and map reconstruction techniques. Additionally, the integration of event cameras with neuromorphic processors, focusing on their synergistic benefits in terms of energy efficiency, robustness, and real-time performance, was explored. The paper also discusses the challenges and open research questions in this emerging field, such as sensor calibration, data fusion, and algorithmic development. Finally, the potential applications and future directions for event-based SLAM systems are outlined, ranging from robotics and autonomous vehicles to augmented reality.