Streaming Manner Research Articles

KF-PEV: a causal Kalman filter-based particle event velocimetry

Event-based pixel sensors asynchronously report changes in log-intensity in microsecond-order resolution. Its exceptional speed, cost effectiveness, and sparse event stream make it an attractive imaging modality for particle tracking velocimetry. In this work, we propose a causal Kalman filter-based particle event velocimetry (KF-PEV). Using the Kalman filter model to track the events generated by the particles seeded in the flow medium, KF-PEV yields the linear least squares estimate of the particle track velocities corresponding to the flow vector field. KF-PEV processes events in a computationally efficient and streaming manner (i.e., causal and iteratively updating). Our simulation-based benchmarking study with synthetic particle event data confirms that the proposed KF-PEV outperforms the conventional frame-based particle image/tracking velocimetry as well as the state-of-the-art event-based particle velocimetry methods. In a real-world water tunnel event-based sensor data experiment performed on what we believe to be the widest field view ever reported, KF-PEV accurately predicted the expected flow field of the SD7003 wing, including details such as the lower velocity in the wake and the flow separation around the underside of an angled wing.

Streaming prediction-error filters

Prediction-error filters (PEFs) are essential for seismic deconvolution and other geophysical estimation problems. We find that nonstationary multidimensional PEFs can be computed in a “streaming” manner, wherein the filter is updated incrementally by accepting one new data point at a time. The computational cost of estimating a streaming PEF is reduced to the cost of a single convolution. In other words, the cost of the PEF design while filtering is equivalent to the cost of applying the filter. Moreover, the nonlinear operation of finding and applying a streaming PEF is invertible at a similar cost, which enables a fast approach for missing data interpolation.

Continual Vision-Language Retrieval via Dynamic Knowledge Rectification

The recent large-scale pre-trained models like CLIP have aroused great concern in vision-language tasks. However, when required to match image-text data collected in a streaming manner, namely Continual Vision-Language Retrieval (CVRL), their performances are still limited due to the catastrophic forgetting of the learned old knowledge. To handle this issue, advanced methods are proposed to distill the affinity knowledge between images and texts from the old model to the new one for anti-forgetting. Unfortunately, existing approaches neglect the impact of incorrect affinity, which prevents the balance between the anti-forgetting of old knowledge and the acquisition of new knowledge. Therefore, we propose a novel framework called Dynamic Knowledge Rectification (DKR) that simultaneously achieves incorrect knowledge filtering and rectification. Specifically, we first filter the incorrect affinity knowledge calculated by the old model on the new data. Then, a knowledge rectification method is designed to rectify the incorrect affinities while preserving the correct ones. In particular, for the new data that can only be correctly retrieved by the new model, we rectify them with the corresponding new affinity to protect them from negative transfer. Additionally, for those that can not be retrieved by either the old or the new model, we introduce paired ground-truth labels to promote the acquisition of both old and new knowledge. Extensive experiments on several benchmark datasets demonstrate the effectiveness of our DKR and its superiority against state-of-the-art methods.

AdaStreamLite

Streaming speech recognition aims to transcribe speech to text in a streaming manner, providing real-time speech interaction for smartphone users. However, it is not trivial to develop a high-performance streaming speech recognition system purely running on mobile platforms, due to the complex real-world acoustic environments and the limited computational resources of smartphones. Most existing solutions lack the generalization to unseen environments and have difficulty to work with streaming speech. In this paper, we design AdaStreamLite, an environment-adaptive streaming speech recognition tool for smartphones. AdaStreamLite interacts with its surroundings to capture the characteristics of the current acoustic environment to improve the robustness against ambient noise in a lightweight manner. We design an environment representation extractor to model acoustic environments with compact feature vectors, and construct a representation lookup table to improve the generalization of AdaStreamLite to unseen environments. We train our system using large speech datasets publicly available covering different languages. We conduct experiments in a large range of real acoustic environments with different smartphones. The results show that AdaStreamLite outperforms the state-of-the-art methods in terms of recognition accuracy, computational resource consumption and robustness against unseen environments.

Multiview Deep Online Clustering: An Application to Online Research Topic Modeling and Recommendations

In today&#x2019;s scenario, a large number of scientific articles on various domains are being published everyday, resulting in a rapid change in the trends of research topics. Retrieving the trending topics, evaluating the trends and extracting the scope of topics could be beneficial to young researchers, which can be recommended for future scope. Publication of articles is a continuous process, and so is the evolution of topics as well as the scope. The dynamic behavior of topics can be handled by continuously updating the partitioning of incoming articles arriving in a streaming manner. This article proposes an online clustering approach utilizing the autoencoder, which is capable of handling the online stream of articles. The training is performed by considering the multiple views of articles and posing the topic modeling (TM) problem as a multiview (MV) clustering problem. In addition, we employ an evolutionary-based approach to the latent representation of data to automatically determine the number of clusters. To learn the nonlinear mapping appropriately and to generate clusters effectively, the model simultaneously optimizes the reconstruction loss and clustering loss in an MV framework. The developed method has experimented on <i>ArXiv</i> dataset to determine the trending topics, scope, and recommendation of topics. The superiority of the proposed method in generating clusters as well as in determining scope is shown by comparing the results with many existing and baseline methods.

Online continual learning for human activity recognition

Sensor-based human activity recognition (HAR), with the ability to recognise human activities from wearable or embedded sensors, has been playing an important role in many applications including personal health monitoring, smart home, and manufacturing. The real-world, long-term deployment of these HAR systems drives a critical research question: how to evolve the HAR model automatically over time to accommodate changes in an environment or activity patterns. This paper presents an online continual learning (OCL) scenario for HAR, where sensor data arrives in a streaming manner which contains unlabelled samples from already learnt activities or new activities. We propose a technique, OCL-HAR, making a real-time prediction on the streaming sensor data while at the same time discovering and learning new activities. We have empirically evaluated OCL-HAR on four third-party, publicly available HAR datasets. Our results have shown that this OCL scenario is challenging to state-of-the-art continual learning techniques that have significantly underperformed. Our technique OCL-HAR has consistently outperformed them in all experiment setups, leading up to 0.17 and 0.23 improvements in micro and macro F1 scores.

ML-KnockoffGAN: Deep online feature selection for multi-label learning

Many online platforms now generate data in a streaming manner, resulting in the continuous production of new features. Multi-label data generation has also surged in recent years, making feature selection for online multi-label data essential. However, existing feature selection methods are mainly based on single-label data or offline selection approaches. Only a few methods exist for multi-label data in an online framework, and most of these methods use classical or evolutionary-based techniques, paying little attention to deep learning. In this study, we propose a novel deep-learning feature selection technique that utilizes generative adversarial nets (GANs). We develop a framework, called ML-KnockoffGAN, which generates knockoff features in a multi-label setting, and then features are selected by considering both the generated knockoff features and real features together. As the features arrive online in a continuous fashion, our proposed method incorporates online features and selects them in a group-wise manner. We tested our method on various multi-label data sets from different domains, including text, biology, and audio, and our results show that our approach outperforms existing methods, with an average improvement of 7.1−16.3% for all evaluation metrics. Our method also illustrates the benefits of deep learning techniques in utilizing existing trained parameters to train new windows of features, requiring fewer epochs.

OMGH: Online Manifold-Guided Hashing for Flexible Cross-Modal Retrieval

Cross-modal hashing has recently gained an increasing attention for its efficiency and fast retrieval speed in indexing the multimedia data across different modalities. Nevertheless, the multimedia data points often emerge in a streaming manner, which often makes the offline hashing methods loss their efficiency and flexibility. Besides, existing online methods often lack of learning capacity to preserve the semantic correlations between newly coming data and existing data, while suffering from the limited discriminative capability and less scalability for handling both labeled and unlabeled data. To alleviate these concerns, this paper proposes an Online Manifold-Guided Hashing (OMGH) framework, which can incrementally learn the compact hash code of streaming data while adaptively optimizing the hash function in a streaming manner. To be specific, OMGH first exploits a matrix tri-factorization framework to learn the discriminative hash codes for streaming multi-modal data. Then, an online anchor-based manifold structure is designed to sparsely represent the old data and adaptively guide the hash code learning process, which can well reduce the complexity in preserving the semantic correlation between the old data and streaming data. Meanwhile, such anchor-based manifold embedding is adaptive to the unsupervised and supervised learning strategies in a flexible way. Besides, an online discrete optimization method is efficiently addressed to incrementally update the hash functions and optimize the hash codes on streaming data points. As a result, the derived hash codes are more semantically meaningful for various online cross-modal retrieval tasks. Extensive experiments verify the advantages of the proposed OMGH model, by achieving and improving the state-of-the-art cross-modal retrieval performances on three benchmark datasets.

Nonstationary data stream classification with online active learning and siamese neural networks✩

We have witnessed in recent years an ever-growing volume of information becoming available in a streaming manner in various application areas. As a result, there is an emerging need for online learning methods that train predictive models on-the-fly. A series of open challenges, however, hinder their deployment in practice. These are, learning as data arrive in real-time one-by-one, learning from data with limited ground truth information, learning from nonstationary data, and learning from severely imbalanced data, while occupying a limited amount of memory for data storage. We propose the ActiSiamese algorithm, which addresses these challenges by combining online active learning, siamese networks, and a multi-queue memory. It develops a new density-based active learning strategy which considers similarity in the latent (rather than the input) space. We conduct an extensive study that compares the role of different active learning budgets and strategies, the performance with/without memory, the performance with/without ensembling, in both synthetic and real-world datasets, under different data nonstationarity characteristics and class imbalance levels. ActiSiamese outperforms baseline and state-of-the-art algorithms, and is effective under severe imbalance, even only when a fraction of the arriving instances’ labels is available. We publicly release our code to the community.

Streaming Graph Embeddings via Incremental Neighborhood Sketching

Graph embeddings have become a key paradigm to learn node representations and facilitate downstream graph analysis tasks. Many real-world scenarios such as online social networks and communication networks involve streaming graphs, where edges connecting nodes are continuously received in a streaming manner, making the underlying graph structures evolve over time. Such a streaming graph raises great challenges for graph embedding techniques not only in capturing the structural dynamics of the graph, but also in efficiently accommodating high-speed edge streams. Against this background, we propose SGSketch, a highly-efficient streaming graph embedding technique via incremental neighborhood sketching. SGSketch cannot only generate high-quality node embeddings from a streaming graph by gradually forgetting outdated streaming edges, but also efficiently update the generated node embeddings via an incremental embedding updating mechanism. Our extensive evaluation compares SGSketch against a sizable collection of state-of-the-art techniques using both synthetic and real-world streaming graphs. The results show that SGSketch achieves superior performance on different graph analysis tasks, showing 31.9% and 21.9% improvement on average over the best-performing static and dynamic graph embedding baselines, respectively. Moreover, SGSketch is significantly more efficient in both embedding learning and incremental embedding updating processes, showing 54x-1813x and 118x-1955x speedup over the baseline techniques, respectively.

Lightweight Privacy-Preserving Federated Incremental Decision Trees

Tree-based models are wildly adopted in various real-world scenarios. Recently, there is a growing interest in vertical federated tree-based model learning to build tree-based models by exploiting data from multiple organizations without violating data privacy regulations. However, most existing work focuses on batch learning settings where all training samples are prepared at once. They cannot be applied to scenarios where local samples come in a streaming manner. Additionally, the present federated learning algorithms suffer from inference attacks. In this paper, we present a novel solution that enables different organizations to jointly train a tree-based model in an incremental and privacy-preserving manner. Our solution is based on Very Fast Decision Tree (VFDT) for incrementally building a tree model. Since data statistics exchanged in the training process may implicitly disclose private information, we propose a protection mechanism based on order-preserving encryption. To further improve the efficiency of the solution, we compress the size of statistics by means of regional counting, which not only maintains model accuracy but also enhances privacy. We conduct extensive experiments on various real-world datasets and the results show the superiority of our solution in terms of both efficiency and privacy.

Online Estimation for Functional Data

Functional data analysis has attracted considerable interest and is facing new challenges, one of which is the increasingly available data in a streaming manner. In this article we develop an online nonparametric method to dynamically update the estimates of mean and covariance functions for functional data. The kernel-type estimates can be decomposed into two sufficient statistics depending on the data-driven bandwidths. We propose to approximate the future optimal bandwidths by a sequence of dynamically changing candidates and combine the corresponding statistics across blocks to form the updated estimation. The proposed online method is easy to compute based on the stored sufficient statistics and the current data block. We derive the asymptotic normality and, more importantly, the relative efficiency lower bounds of the online estimates of mean and covariance functions. This provides insight into the relationship between estimation accuracy and computational cost driven by the length of candidate bandwidth sequence. Simulations and real data examples are provided to support such findings. Supplementary materials for this article are available online.

Memory-Efficient Modeling and Slicing of Large-Scale Adaptive Lattice Structures

Abstract Lattice structures have been widely used in various applications of additive manufacturing due to its superior physical properties. If modeled by triangular meshes, a lattice structure with huge number of struts would consume massive memory. This hinders the use of lattice structures in large-scale applications (e.g., to design the interior structure of a solid with spatially graded material properties). To solve this issue, we propose a memory-efficient method for the modeling and slicing of adaptive lattice structures. A lattice structure is represented by a weighted graph where the edge weights store the struts' radii. When slicing the structure, its solid model is locally evaluated through convolution surfaces and in a streaming manner. As such, only limited memory is needed to generate the toolpaths of fabrication. Also, the use of convolution surfaces leads to natural blending at intersections of struts, which can avoid the stress concentration at these regions. We also present a computational framework for optimizing supporting structures and adapting lattice structures with prescribed density distributions. The presented methods have been validated by a series of case studies with large number (up to 100M) of struts to demonstrate its applicability to large-scale lattice structures.

Feature Grouping–based Trajectory Outlier Detection over Distributed Streams

Owing to a wide variety of deployment of GPS -enabled devices, tremendous amounts of trajectories have been generated in distributed stream manner. It opens up new opportunities to track and analyze the moving behaviors of the entities. In this work, we focus on the issue of outlier detection over distributed trajectory streams, where the outliers refer to a few entities whose motion behaviors are significantly different from their local neighbors. In view of skewed distribution property and evolving nature of trajectory data, and on-the-fly detection requirement over distributed streams, we first design a high-efficiency outlier detection solution. It consists of identifying abnormal trajectory fragment and exceptional fragment cluster at the remote sites and then detecting abnormal evolving object at the coordinator site. Further, given that outlier detection accuracy would be damaged due to using inappropriate proximity thresholds or a few trajectory data not having sufficient neighbors at the remote sites, we extract proximity thresholds of different regions and spatial context relationship of each region from historical data to improve the precision. Built upon this is an improved version consisting of off-line modeling phase and on-line detection phase. During the on-line phase, the proximity thresholds that are derived from historical trajectories during the off-line phase are leveraged to assist in detecting abnormal trajectory fragments and exceptional fragment clusters at the remote sites. Additionally, at the coordinator site, the detection results of some remote sites can be refined by incorporating those of other remote sites with neighborhood relationship. Extensive experimental results on real data demonstrate that our proposed methods own high detection validity, less communication cost and linear scalability for online identifying outliers over distributed trajectory streams.

A Systematic Review of Data Models for the Big Data Problem

Nowadays, data are generated in a continuous streaming manner as the inputs of various applications. The sources of such generated data can be wired or wireless sensor networks commonly used in various fields of geographical, traffic, Internet of Things (IoT), financial tickers, Web2 and Web3, e-commerce, social networks, and online communities. The high volume, high variety, and high velocity of data have recently posed the challenge of 3Vs to this field, also known as the Big Data Problem. The 3Vs dimensions of complexities for the big data entails high-speed storage, scalability of database systems, suitable data models, real-time responsiveness and so on. Data model, as the representation schema of data is an essential issue since many others (e.g., DBMS systems’ design, DB languages, etc.) rely on. So, the study of data models is a key and fundamental aspect in structuring, organizing, storing, and manipulating big data. It is also the essence in various areas of cloud migration, web-scale, and so forth. In this paper, we have systematically reviewed different types of data models, the rationale behind them, their applications and support capabilities, and the technologies to switch from one model to another. To address the user needs in various fields, a systematic review method is adopted to classify and present different types and characteristics of data models.

Cloud-based middleware for supporting batch and stream access over smart healthcare wearable device

In IoT-based smart healthcare services, the heterogeneity of connected wearable sensing devices open up a wide opportunity to develop various healthcare services. However, it also poses an interoperability challenge since each sensing device and application may have different communication mechanisms. Considering that challenge, web platform can be seen as a promising candidate for providing an interoperability layer as we can abstract various devices as single representation i.e. web resource. In this paper, we propose the design of middleware for enabling efficient web of things access over healthcare wearable devices. The proposed middleware consists of three components: gateway-to-cloud device, messaging service and data access interface. The gateway-to-cloud device has a role to perform low level sensor data collection from various wearable sensing device through bluetooth low energy (BLE) communication protocol. Collected data are then relayed to the cloud IoT platform using a lightweight MQTT messaging protocol. In order to provide device abstraction along with access to the stored data, the system offers two kind of interfaces: the Restful HTTP identified by unique universal resource locator (URL) for batch access and MQTT websocket interface identified by unique topic to accommodate access on sensing data in near real time stream manner.

Motion Saliency Detection for Surveillance Systems Using Streaming Dynamic Mode Decomposition

Intelligent surveillance systems enable secured visibility features in the smart city era. One of the major models for pre-processing in intelligent surveillance systems is known as saliency detection, which provides facilities for multiple tasks such as object detection, object segmentation, video coding, image re-targeting, image-quality assessment, and image compression. Traditional models focus on improving detection accuracy at the cost of high complexity. However, these models are computationally expensive for real-world systems. To cope with this issue, we propose a fast-motion saliency method for surveillance systems under various background conditions. Our method is derived from streaming dynamic mode decomposition (s-DMD), which is a powerful tool in data science. First, DMD computes a set of modes in a streaming manner to derive spatial–temporal features, and a raw saliency map is generated from the sparse reconstruction process. Second, the final saliency map is refined using a difference-of-Gaussians filter in the frequency domain. The effectiveness of the proposed method is validated on a standard benchmark dataset. The experimental results show that the proposed method achieves competitive accuracy with lower complexity than state-of-the-art methods, which satisfies requirements in real-time applications.

Hadamard Matrix Guided Online Hashing

Online image hashing has attracted increasing research attention recently, which receives large-scale data in a streaming manner to update the hash functions on-the-fly. Its key challenge lies in the difficulty of balancing the learning timeliness and model accuracy. To this end, most works follow a supervised setting, i.e., using class labels to boost the hashing performance, which defects in two aspects: first, strong constraints, e.g., orthogonal or similarity preserving, are used, which however are typically relaxed and lead to large accuracy drops. Second, large amounts of training batches are required to learn the up-to-date hash functions, which largely increase the learning complexity. To handle the above challenges, a novel supervised online hashing scheme termed Hadamard Matrix Guided Online Hashing (HMOH) is proposed in this paper. Our key innovation lies in introducing Hadamard matrix, which is an orthogonal binary matrix built via Sylvester method. In particular, to release the need of strong constraints, we regard each column of Hadamard matrix as the target code for each class label, which by nature satisfies several desired properties of hashing codes. To accelerate the online training, LSH is first adopted to align the lengths of target code and to-be-learned binary code. We then treat the learning of hash functions as a set of binary classification problems to fit the assigned target code. Finally, extensive experiments on four widely-used benchmarks demonstrate the superior accuracy and efficiency of HMOH over various state-of-the-art methods. Codes can be available at https://github.com/lmbxmu/mycode .

Multitask Learning Over Graphs: An Approach for Distributed, Streaming Machine Learning

The problem of learning simultaneously several related tasks has received considerable attention in several domains, especially in machine learning with the so-called multitask learning problem or learning to learn problem [1], [2]. Multitask learning is an approach to inductive transfer learning (using what is learned for one problem to assist in another problem) and helps improve generalization performance relative to learning each task separately by using the domain information contained in the training signals of related tasks as an inductive bias. Several strategies have been derived within this community under the assumption that all data are available beforehand at a fusion center. However, recent years have witnessed an increasing ability to collect data in a distributed and streaming manner. This requires the design of new strategies for learning jointly multiple tasks from streaming data over distributed (or networked) systems. This article provides an overview of multitask strategies for learning and adaptation over networks. The working hypothesis for these strategies is that agents are allowed to cooperate with each other in order to learn distinct, though related tasks. The article shows how cooperation steers the network limiting point and how different cooperation rules allow to promote different task relatedness models. It also explains how and when cooperation over multitask networks outperforms non-cooperative strategies.

A Framework of Virtual War Room and Matrix Sketch-Based Streaming Anomaly Detection for Microservice Systems

Recently, microservice has been a popular architecture to construct cloud-native systems. This novel architecture brings agility and accelerates the software development process significantly. However, it is not easy to manage and operate microservice systems due to their scale and complexity. Many approaches are proposed to automatically operate microservice systems such as anomaly detection. Nevertheless, those methods cannot be sufficiently validated and compared due to a lack of real microservice systems, which leads to the slow process of intelligent operation. These challenges inspire us to build a system named “VWR”, a framework of Virtual War Room for operating microservice applications which allows users to simulate their microservice architectures with low overhead and inject multiple types of faults into the microservice system with chaos engineering. VWR can mimic user requests and record the end-to-end tracing data (i.e., service call chains) for each request in a way consistent with OpenTracing. With easily designed tests and the produced streaming tracing data, the users can validate the performance of their intelligent operation algorithms and improve the algorithms as needed. Besides, based on the streaming tracing data generated by VWR, we introduce a novel unsupervised anomaly detection algorithm based on Matrix Sketch and set it as a default intelligent operation algorithm in VWR. This algorithm can detect anomalies by analyzing high-dimensional performance data collected from a microservice system in a streaming manner. The experimental result in VWR shows that the matrix sketch based method can precisely detect anomalies in microservice systems and outperform some widely used anomaly detection methods such as isolation forest in some scenario. We believe more approaches on the intelligent operation of microservice systems can be constructed based on VWR.