Event Streams Research Articles

Neuromorphic Vision-Based Fall Localization in Event Streams With Temporal-Spatial Attention Weighted Network.

Falling down is a serious problem for health and has become one of the major etiologies of accidental death for the elderly living alone. In recent years, many efforts have been paid to fall recognition based on wearable sensors or standard vision sensors. However, the prior methods have the risk of privacy leaks, and almost all these methods are based on video clips, which cannot localize where the falls occurred in long videos. For these reasons, in this article, the bioinspired vision sensor-based falls temporal localization framework is proposed. The bioinspired vision sensors, such as dynamic and active-pixel vision sensor (DAVIS) camera applied in this work responds to pixels' brightness change, and each pixel works independently and asynchronously compared to the standard vision sensors. This property makes it have a very high dynamic range and privacy preserving. First, to better represent event data, compared with the typical constant temporal window mechanism, an adaptive temporal window conversion mechanism is developed. The temporal localization framework follows a proven proposal and classification paradigm. Second, for the high-efficient and recall proposal generation, different from the traditional sliding window scheme, the event temporal density as the actionness score is set and the 1D-watershed algorithm to generate proposals is applied. In addition, we combine the temporal and spatial attention mechanism with our feature extraction network to temporally model the falls. Finally, to evaluate the performance of our framework, 30 volunteers are recruited to join the simulated fall experiments. According to the results of experiments, our framework can realize precise falls temporal localization and achieve the state-of-the-art performance.

Examining the Role of Dignity in the Experience of Meaningfulness: a Process-Relational View on Meaningful Work

The objective of the present study is to examine the ethical grounding and process-relational nature of meaningful work through the relationship of dignity and meaningfulness. Adopting a practice lens, we show how a shift from methodological individualism to a process-relational worldview allows meaningful work to be understood through organizational activities rather than individual characteristics. Building on practice-based theorization, we present a process-relational model of meaningful work that 1) examines meaningfulness as a flow of experience in the stream of work activity events; 2) highlights how experiencing meaningfulness is embedded in social practices, distinguishing it as a social phenomenon that is defined by this embeddedness; 3) delineates situationality, historicity, and contextuality of meaningfulness; and 4) shows how meaningful work is grounded in the prioritization of dignity in the logic of practice. Accordingly, our model enables a more holistic understanding of how dignity functions as the ethical basis for the experience of meaningfulness in the context of work and organization.

A Bio-Inspired Hierarchical Spiking Neural Network With Reward-Modulated STDP Learning Rule for AER Object Recognition

The inherent spike-based and event-driven computation makes spiking neural networks (SNNs) naturally suitable to provide efficient and low-latency solution in neuromorphic vision processing. In this work we propose an address-event representation (AER) object recognition system which consists of a noise filter, an event stream segmentation module and a hierarchical convolutional SNN. In order to improve the feature learning capacity of bio-inspired shallow hierarchical SNNs for AER object recognition, we apply a convolutional SNN with R-STDP learning rule and first-spike coding to further extract features after primary feature learning using Gabor filter-based event-driven convolution. The R-STDP learning rule enables the network to adjust weights by combining local STDP learning rule with a global reward signal and to make classification without using an external classifier. Experimental results show that our method significantly outperforms the existing hierarchical SNN-based recognition methods on four popular AER datasets. Moreover, our method has much better recognition ability when using very short input event stream, and can effectively learn from small size training sets. In addition, the robustness of the network to input noise is improved. Our method is very beneficial for developing neuromorphic-vision object recognition algorithm in resource-constrained applications when event streams are short and training samples are limited.

Runtime verification of real-time event streams using the tool HStriver

Runtime verification of real-time event streams using the tool HStriver

Multimodal Event Processing: A Neural-Symbolic Paradigm for the Internet of Multimedia Things

With the Internet of Multimedia Things (IoMT) becoming a reality, new approaches are needed to process real-time multimodal event streams. Existing approaches to event processing have limited consideration for the challenges of multimodal events, including the need for complex content extraction, and increased computational and memory costs. This article explores event processing as a basis for processing real-time IoMT data. This article introduces the multimodal event processing (MEP) paradigm, which provides a formal basis for native approaches to neural multimodal content analysis (i.e., computer vision, linguistics, and audio) with symbolic event processing rules to support real-time queries over multimodal data streams using the multimodal event processing language to express single, primitive multimodal, and complex multimodal event patterns. The content of multimodal streams is represented using multimodal event knowledge graphs to capture the semantic, spatial, and temporal content of the multimodal streams. The approach is implemented and evaluated within a MEP engine using single and multimodal queries achieving near real-time performance with a throughput of ~30 frames processed per second (fps) and subsecond latency of 0.075–0.30 s for video streams of 30 fps input rate. Support for high input stream rates (45 fps) is achieved through content-aware load-shedding techniques with a ~127X latency improvement resulting in only a minor decrease in accuracy.

Rainer Waser – A Pioneer of Fundamentals of Resistive Switching Memories

Rainer Waser – A Pioneer of Fundamentals of Resistive Switching Memories

Event stream learning using spatio-temporal event surface

Event cameras sense changes in light intensity and record them as an asynchronous event stream. Efficiently encoding and learning spatiotemporal information of the event streams remain challenging. In this paper, we propose a novel event descriptor to encode the spatio-temporal features for event streams and a local-search based multi-spike learning algorithm for spiking neural networks to classify encoded features. The spatio-temporal event surface (STES) descriptor explicitly captures both spatial and temporal correlations among events, and thus can characterize spatiotemporal features more accurately than existing feature descriptors that focus only on temporal or spatial information. In classification with multi-spike learning, we introduce a local search and gradient clipping mechanism to ensure the efficiency and stability of learning, which avoids other multi-spike learning rules’ time-consuming global search and the gradient explosion problem. Experimental results demonstrate the superior classification performance of our proposed model, especially for event streams with rich spatiotemporal dynamics.

Image De-occlusion via Event-enhanced Multi-modal Fusion Hybrid Network

Seeing through dense occlusions and reconstructing scene images is an important but challenging task. Traditional frame-based image de-occlusion methods may lead to fatal errors when facing extremely dense occlusions due to the lack of valid information available from the limited input occluded frames. Event cameras are bio-inspired vision sensors that record the brightness changes at each pixel asynchronously with high temporal resolution. However, synthesizing images solely from event streams is ill-posed since only the brightness changes are recorded in the event stream, and the initial brightness is unknown. In this paper, we propose an event-enhanced multi-modal fusion hybrid network for image de-occlusion, which uses event streams to provide complete scene information and frames to provide color and texture information. An event stream encoder based on the spiking neural network (SNN) is proposed to encode and denoise the event stream efficiently. A comparison loss is proposed to generate clearer results. Experimental results on a large-scale event-based and frame-based image de-occlusion dataset demonstrate that our proposed method achieves state-of-the-art performance.

MDOE: A Spatiotemporal Event Representation Considering the Magnitude and Density of Events

Event-based sensors (e.g., DVS cameras) are capable of higher dynamic range, higher temporal resolution, lower time latency, and better power efficiency compared to conventional devices (e.g., RGB cameras). However, learning from these sensors remains challenging; event-based sensors output a stream of asynchronous events, which cannot be directly used by state-of-the-art convolutional neural networks (CNNs). In this paper, we present a novel event-based representation called MDOE that considers both the magnitude and density of events. Compared to existing representations, which discard one or more types of information about event polarity, temporal information, and/or density, MDOE contains richer information about events. It has two benefits: (i) it is a conceptually-simple generic representation that is task-independent; (ii) it achieves superior performance relative to existing representations on a variety of event-based datasets.

Did Covid‐19 transform the European Union into a ‘community of destiny’?

Did Covid‐19 transform the European Union into a ‘community of destiny’?

Event-Based Line SLAM in Real-Time

Event-based cameras generate asynchronous streams of events, triggered proportionally to the logarithmic change of brightness in the scene. These cameras have very low latency and high dynamic range suitable to address challenging motion scenarios in robotics. In this work, we explore a new event-based line-SLAM approach following a parallel tracking and mapping philosophy. Our fast tracking algorithm, produces accurate camera pose estimates at a high rate by minimizing the event-line reprojection error with an error-state Kalman filter formulated entirely with Lie theory. The mapping thread leverages the natural edge highlighting strength of events to recover and optimize straight lines in human-made scenarios. The proper manipulation of matrix sparsity as well as the information sharing between tracking and mapping nodes allow us to achieve real-time performance on a standard multi-core CPU. This system was tested on several scenarios rich in straight edge objects, and compared against, ground truth and frame and event based state-of-the-art approaches.

Fractional Model of the Deformation Process

The article considers the fractional Poisson process as a mathematical model of deformation activity in a seismically active region. The dislocation approach is used to describe five modes of the deformation process. The change in modes is determined by the change in the intensity of the event stream, the regrouping of dislocations, and the change in and the appearance of stable connections between dislocations. Modeling of the change of deformation modes is carried out by changing three parameters of the proposed model. The background mode with independent events is described by a standard Poisson process. To describe variations from the background mode of seismic activity, when connections are formed between dislocations, the fractional Poisson process and the Mittag–Leffler function characterizing it are used. An approximation of the empirical cumulative distribution function of waiting time of the foreshocks obtained as a result of processing the seismic catalog data was carried out on the basis of the proposed model. It is shown that the model curves, with an appropriate choice of the Mittag–Leffler function’s parameters, gives results close to the experimental ones and can be allowed to characterize the deformation process in the seismically active region under consideration.

Evaluation Goals for Online Process Mining: A Concept Drift Perspective

Online process mining refers to a class of techniques for analyzing in real-time event streams generated by the execution of business processes. These techniques are crucial in the reactive monitoring of business processes, timely resource allocation and detection/prevention of dysfunctional behavior. Many interesting advances have been made by the research community in recent years, but there is no consensus on the exact set of properties these techniques have to achieve. This article fills the gap by identifying a set of evaluation goals for online process mining and examining their fulfillment in the state of the art. We discuss parameters and techniques regulating the balance between conflicting goals and outline research needed for their improvement. Concept drift detection is crucial in this sense but, as demonstrated by our experiments, it is only partially supported by current solutions.

ProgressiveMotionSeg: Mutually Reinforced Framework for Event-Based Motion Segmentation

Dynamic Vision Sensor (DVS) can asynchronously output the events reflecting apparent motion of objects with microsecond resolution, and shows great application potential in monitoring and other fields. However, the output event stream of existing DVS inevitably contains background activity noise (BA noise) due to dark current and junction leakage current, which will affect the temporal correlation of objects, resulting in deteriorated motion estimation performance. Particularly, the existing filter-based denoising methods cannot be directly applied to suppress the noise in event stream, since there is no spatial correlation. To address this issue, this paper presents a novel progressive framework, in which a Motion Estimation (ME) module and an Event Denoising (ED) module are jointly optimized in a mutually reinforced manner. Specifically, based on the maximum sharpness criterion, ME module divides the input event into several segments by adaptive clustering in a motion compensating warp field, and captures the temporal correlation of event stream according to the clustered motion parameters. Taking temporal correlation as guidance, ED module calculates the confidence that each event belongs to real activity events, and transmits it to ME module to update energy function of motion segmentation for noise suppression. The two steps are iteratively updated until stable motion segmentation results are obtained. Extensive experimental results on both synthetic and real datasets demonstrate the superiority of our proposed approaches against the State-Of-The-Art (SOTA) methods.

Linking Transformer to Hawkes Process for Information Cascade Prediction (Student Abstract)

Information cascade is typically formalized as a process of (simplified) discrete sequence of events, and recent approaches have tackled its prediction via variants of recurrent neural networks. However, the information diffusion process is essentially an evolving directed acyclic graph (DAG) in the continuous-time domain. In this paper, we propose a transformer enhanced Hawkes process (Hawkesformer), which links the hierarchical attention mechanism with Hawkes process to model the arrival stream of discrete events continuously. A two-level attention architecture is used to parameterize the intensity function of Hawkesformer, which captures the long-term dependencies between nodes in graph and better embeds the cascade evolution rate for modeling short-term outbreaks. Experimental results demonstrate the significant improvements of Hawkesformer over the state-of-the-art.

Event-Image Fusion Stereo Using Cross-Modality Feature Propagation

Event cameras asynchronously output the polarity values of pixel-level log intensity alterations. They are robust against motion blur and can be adopted in challenging light conditions. Owing to these advantages, event cameras have been employed in various vision tasks such as depth estimation, visual odometry, and object detection. In particular, event cameras are effective in stereo depth estimation to find correspondence points between two cameras under challenging illumination conditions and/or fast motion. However, because event cameras provide spatially sparse event stream data, it is difficult to obtain a dense disparity map. Although it is possible to estimate disparity from event data at the edge of a structure where intensity changes are likely to occur, estimating the disparity in a region where event occurs rarely is challenging. In this study, we propose a deep network that combines the features of an image with the features of an event to generate a dense disparity map. The proposed network uses images to obtain spatially dense features that are lacking in events. In addition, we propose a spatial multi-scale correlation between two fused feature maps for an accurate disparity map. To validate our method, we conducted experiments using synthetic and real-world datasets.

Online incremental updating for model enhancement based on multi-perspective trusted intervals

Model incremental updating enhances the initial model by analysing discrepancy parts of the system to improve the model's adaptability for new scenarios. These discrepancy components originate from the deviation between the increasing business process operational state and the outdated original planning model. Considerable domain-specific knowledge is required to determine the threshold points for selecting appropriate activities, but process analysts rarely know each scenario's domain knowledge. Moreover, most analytical processes only focussed on the control flow level. Aiming at these issues, this paper proposes a Hybrid Behavioural and Resource Trusted intervals Updating algorithm (BI&RI Updating) for model enhancement based on control level and resource level of online event streams. First, analyse the reference model to construct multi-perspective trusted interval constraints in the offline stage. From a control-flow perspective, the behavioural relationships between activities are researched using a deep clustering approach. Moreover, from a data-flow standpoint, resource co-occurrence relationships are analysed based on association rules. Next, the incremental update algorithm in online scenarios is proposed to update the model by filtering the event streams and iteratively optimising the trusted intervals. Finally, the proposed algorithm is implemented based on the PM4PY framework and evaluated using real logs with both outcome model quality and execution efficiency. The results show that the algorithm can execute quickly and improve the quality of the model even in a non-ideal state where the logs contain noise.

ALVS: Adaptive Live Video Streaming using deep reinforcement learning

Achieving a high Quality of Experience (QoE) in live event streaming is a challenging problem given a low-latency requirement and time-varying network conditions. Adaptive video bitrate and adaptive playback speed techniques are two separate control knobs to address this challenge. In this paper, we consider these two control parameters in a joint optimization problem and present a deep reinforcement learning (DRL) framework to maximize QoE for live streaming without any assumption about the environment or fixed rule-based heuristics. With the proposed DRL framework, our approach (ALVS) constructs the inference model to make a joint decision of adaptive playback speed and video quality level for the next video segment. Simulation results through real network traces show that ALVS outperforms both state-of-the-art DRL-based and rule-based algorithms in terms of QoE without sacrificing live latency and skipping any content.

Flow variation at multiple scales filters fish life histories and constrains community diversity in desert streams

AbstractEnvironmental regimes, which encompass decadal‐scale or longer variation in climate and disturbance, shape communities by selecting for adaptive life histories, behaviors, and morphologies. In turn, at ecological timescales, extreme events may cause short‐term changes in composition and structure via mortality and recolonization of the species pool. Here, we illustrate how short‐term variation in desert stream fish communities following floods and droughts depends on the context of the long‐term flow regime through ecological filtering of life history strategies. Using quarterly measures of fish populations in streams spanning a 10‐fold gradient in flow variation in Arizona, USA, we quantified temporal change in community composition and life history strategies. In streams with highly variable flow regimes, fish communities were less diverse, fluctuation in species richness was the principle mechanism of temporal change in diversity, and communities were dominated by opportunistic life history strategies. Conversely, relatively stable flow regimes resulted in more diverse communities with greater species replacement and dominance of periodic and equilibrium strategies. Importantly, the effects of anomalous high‐ and low‐flow events depended on flow regime. Under more stable flow regimes, fish diversity was lower following large floods than after seasons without floods, whereas diversity was independent of high‐flow events in streams with flashier flow regimes. Likewise, community life history composition was more dependent on antecedent anomalous events in stable compared to more temporally variable regimes. These findings indicate that extreme events may be a second‐level filter on community composition, with effects contingent on the long‐term properties of the disturbance regime (e.g., overall degree of variation) in which extremes take place. Ongoing changes to global environmental regimes will likely drive new patterns of community response to extreme events.

Business Process Outcome Prediction Based on Deep Latent Factor Model

Business process outcome prediction plays an essential role in business process monitoring. It continuously analyzes completed process events to predict the executing cases’ outcome. Most of the current outcome prediction focuses only on the activity information in historical logs and less on the embedded and implicit knowledge that has not been explicitly represented. To address these issues, this paper proposes a Deep Latent Factor Model Predictor (DLFM Predictor) for uncovering the implicit factors affecting system operation and predicting the final results of continuous operation cases based on log behavior characteristics and resource information. First, the event logs are analyzed from the control flow and resource perspectives to construct composite data. Then, the stack autoencoder model is trained to extract the data’s main feature components for improving the training data’s reliability. Next, we capture the implicit factors at the control and data flow levels among events and construct a deep implicit factor model to optimize the parameter settings. After that, an expansive prefix sequence construction method is proposed to realize the outcome prediction of online event streams. Finally, the proposed algorithm is implemented based on the mainstream framework of neural networks and evaluated by real logs. The results show that the algorithm performs well under several evaluation metrics.