Streaming Data Sets Research Articles

Unified rules of renewable weighted sums for various online updating estimations

This paper establishes unified frameworks of renewable weighted sums (RWS) for various online updating estimations in the models with streaming data sets. The newly defined RWS lays the foundation of online updating likelihood, online updating loss function, online updating estimating equation and so on. The idea of RWS is intuitive and heuristic, and the algorithm is computationally simple. The RWS applies to various types of parametric, nonparametric and semiparametric estimators from likelihood, quasi-likelihood and least squares. The consistency and asymptotic normality of the proposed renewable estimator are established, and the oracle property is obtained. Moreover, these properties are always satisfied, without the familiar constraints on the number of data batches, which means that the new method is adaptive to the situation where streaming data sets arrive perpetually. The performance of the method is further illustrated by various numerical examples from simulation experiments and real data analysis.

Renewable estimation in expectile regression model with streaming data sets

Streaming data continually expands over time, making it challenging to apply traditional expectile regression methods due to memory limitations. Expectile regression is advantageous for studying the entire conditional distribution of the response to a predictor, but handling streaming data requires innovative approaches. This paper proposes an online renewable expectile regression strategy, which updates the estimator using both the current data and summary statistics from historical data. An ADMM algorithm is employed to efficiently compute the proposed estimator. We establish the consistency and asymptotic normality of this renewable estimator under regular conditions. Extensive simulations and real-data experiments demonstrate that our method performs comparably to the oracle estimator while maintaining computational efficiency and low storage requirements.

An empirical analysis of the impacts of live chat social interactions in live streaming commerce: A topic modeling approach

The popularity of live streaming is driving the emergence of a new business model, known as live streaming commerce (LSC). The live chat service serves as a key indicator of viewer engagement in LSC shows, differentiating LSC significantly from traditional e-commerce. In this paper, we collect a rich live streaming dataset and identify two categories of social interactions behind live chat: transaction-oriented and relationship-oriented. Broadcasters and viewers in the transaction-oriented category focus on products and transactions and tend to communicate with non-emotional words in live chats. In contrast, broadcasters and viewers might treat each other as friends and are more likely to post emotional words in the relationship-oriented category. Our econometric model shows a curvilinear association between relationship-oriented social interactions and viewer purchase behaviors. The empirical analysis indicates that to achieve higher sales, LSC broadcasters should make relationship-oriented social interactions in live chat account for 58.64% of the probabilities. We discuss the implications of heterogeneous social interaction strategies across different broadcasters.

Renewable Quantile Regression with Heterogeneous Streaming Datasets

The renewable statistical inference has received much attention since the advent of streaming data collection techniques. However, most existing online updating methods are developed based on a homogeneity assumption and gradients; all data batches are required to be either independent and identically distributed or share the same regression parameters, and objective functions must be smooth concerning parameters. To our best knowledge, the only existing approach that allows some regression parameters to be different for different data batches, was proposed by Luo and Song who required the homogeneous structure to be known, which is difficult to guarantee in actual application. In this article, we develop an online renewable quantile regression method that relies only on the current data and summary statistics of historical data, for both homogeneous and heterogeneous streaming data. The proposed methods are computationally efficient, can automatically detect the unknown potential homogeneous structure, and are robust to heavy-tailed noise and data with outliers. Asymptotic properties show that the proposed renewable estimators can achieve the same statistical efficiency as the oracle estimators based on individual-level data. A numerical simulation and a real data analysis illustrate that the proposed methods perform well. Supplementary materials for this article are available online.

Deep learning based multimodal urban air quality prediction and traffic analytics

Urban activities, particularly vehicle traffic, are contributing significantly to environmental pollution with detrimental effects on public health. The ability to anticipate air quality in advance is critical for public authorities and the general public to plan and manage these activities, which ultimately help in minimizing the adverse impact on the environment and public health effectively. Thanks to recent advancements in Artificial Intelligence and sensor technology, forecasting air quality is possible through the consideration of various environmental factors. This paper presents our novel solution for air quality prediction and its correlation with different environmental factors and urban activities, such as traffic density. To this aim, we propose a multi-modal framework by integrating real-time data from different environmental sensors and traffic density extracted from Closed Circuit Television footage. The framework effectively addresses data inconsistencies arising from sensor and camera malfunctions within a streaming dataset. The dataset exhibits real-world complexities, including abrupt camera or station activations/deactivations, noise interference, and outliers. The proposed system tackles the challenge of predicting air quality at locations having no sensors or experiencing sensor failures by training a joint model on the data obtained from nearby stations/sensors using a Particle Swarm Optimization (PSO)-based merit fusion of the sensor data. The proposed methodology is evaluated using various variants of the LSTM model including Bi-directional LSTM, CNN-LSTM, and Convolutions LSTM (ConvLSTM) obtaining an improvement of 48%, 67%, and 173% for short-term, medium-term, and long-term periods, respectively, over the ARIMA model.

Non‐crossing quantile double‐autoregression for the analysis of streaming time series data

Many financial time series not only have varying structures at different quantile levels and exhibit the phenomenon of conditional heteroscedasticity at the same time but also arrive in the stream. Quantile double‐autoregression is very useful for time series analysis but faces challenges with model fitting of streaming data sets when estimating other quantiles in subsequent batches. This article proposes a renewable estimation method for quantile double‐autoregression analysis of streaming time series data due to its ability to break with storage barrier and computational barrier. Moreover, the proposed flexible parametric structure of the quantile function enables us to predict any interested quantile value without quantile curve crossing problem or keeping the desirable monotone property of the conditional quantile function. The proposed methods are illustrated using current data and the summary statistics of historical data. Theoretically, the proposed statistic is shown to have the same asymptotic distribution as the standard version computed on an entire data stream with the data batches pooled into one data set, without additional condition. Simulation studies and an empirical example are presented to illustrate the finite sample performance of the proposed methods.

Invasion@Ukraine: Providing and Describing a Twitter Streaming Dataset That Captures the Outbreak of War between Russia and Ukraine in 2022

Social media can be a mirror of human interaction, society, and historic disruptions. Their reach enables the global dissemination of information in the shortest possible time and, thus, the individual participation of people worldwide in global events in almost real-time. However, these platforms can be equally efficiently used in information warfare to manipulate human perception and opinion formation. Within this paper, we describe a dataset of raw tweets collected via the Twitter Streaming API in the context of the onset of the war, which Russia started in Ukraine on February 24, 2022. A distinctive feature of the dataset is that it covers the period from one week before to one week after Russia invasion of Ukraine. This paper details the acquisition process and provides first insights into the content of the data stream. In addition, the data has been annotated with availability tags, resulting from rehydration attempts at two points in time: directly after data acquisition and shortly before manuscript submission. This may provide information on Twitter moderation policies. Further, we provide a detailed list of other published dataset covering the same topic. On the content level, we can show that our dataset comprises several distinct topics related to the conflict and conspiracy narratives -- topics that deserve more profound investigation. Therefore, the presented dataset is also made available to the community in an extended version with pseudonymized tweet content upon request.

Online renewable smooth quantile regression

This paper concerns quantile regression for streaming data, where large amounts of data arrive batch by batch. Limited memory and non-smoothness of quantile regression loss all pose challenges in both computation and theoretical development. To address the challenges, we first introduce a convex smooth quantile loss, which is infinitely differentiable and converges to the quantile loss uniformly. Then an online renewable framework is proposed, in which the quantile regression estimator is renewed with current data and summary statistics of historical data. In theory, the estimation consistency and asymptotic normality of the renewable estimator are established without any restriction on the total number of data batches, which leads to the oracle property, and gives theoretical guarantee that the new method is adaptive to the situation where streaming data sets arrive perpetually. Numerical experiments on both synthetic and real data verify the theoretical results and illustrate the good performance of the new method.

EnHAT — Synergy of a tree-based Ensemble with Hoeffding Adaptive Tree for dynamic data streams mining

The goal of this paper is to improve the predictive accuracy of data streaming algorithms without increasing the processing time of the incoming data. We propose the EnHAT (Ensemble Combined with Hoeffding Adaptive Tree) algorithm, which combines the state-of-the-art Hoeffding Adaptive Tree (HAT) algorithm with an ensemble of J48 decision trees induced from sequential chunks of the data stream. The slack time of HAT adaptation to a new window of incoming records is utilized in parallel for building a decision-tree ensemble. In our experiments on 4 benchmark streaming datasets and 4 synthetic datasets with different types of concept drift, EnHAT has reached the highest predictive accuracy in 23 out of 26 cases compared to an ensemble of J48 trees, HAT, and a single J48 model induced from the last sliding window. Thus, we can conclude that the ensemble/HAT synergy yields better prediction results than each one of the two approaches on its own. The higher accuracy does not come at the expense of any additional computational effort beyond the model induction times of the combined algorithms.

High-performance adaptive texture streaming for planetary-scale high-mobility information visualization

Planetary-scale terrain requires streaming of world-scale high-resolution data sets in real time. High-performance streaming of world-scale data sets to achieve high mobility in terms of speed and view such as look-around, side-ways, or backward for high fidelity display systems is challenging. Moreover, streaming data sets without any preprocessing overhead require high-performance decompression and re-projection in real time. This work proposes a high-performance adaptive texture streaming system to capture high-speed scenes with view switching capability over a real-world terrain for high-fidelity display system. The system proposes multi-stage pipeline to stream multi-resolution data sets with a stable high-performance rendering rate. The proposed pipeline stages are asynchronously connected using request queues with a cache hierarchy. Efficient algorithms for resource management, prefetching and caching, prioritization of load requests, and hardware-accelerated processing are proposed for texture streaming without frame-rate stuttering. The proposed algorithms adapt to viewer navigation, display resolution, and dynamic execution environment. The proposed method is evaluated for complex scenes, mobility in terms of speed and view, and high-fidelity display. Our method achieved a ground-speed of 2.4x105,1.2x105,and9.0x104 km/h for HD (1024x720), FHD (1920x1080), and UHD (3840x2160) display resolutions respectively with a stable frame rate of 850, 590, and 420 frames per second.

Renewable quantile regression for streaming data sets

Online updating is an important statistical method for the analysis of big data arriving in streams due to its ability to break the storage barrier and the computational barrier under certain circumstances. The quantile regression, as a widely used regression model in many fields, faces challenges in model fitting and variable selection with big data arriving in streams. Chen et al. (2019, Annals of Statistics) has proposed a quantile regression method for streaming data, but a strong additional condition is required. In this paper, renewable optimized objective functions for regression parameter estimation and variable selection in a quantile regression are proposed. The proposed methods are illustrated using current data and the summary statistics of historical data. Theoretically, the proposed statistics are shown to have the same asymptotic distributions as the standard version computed on an entire data stream with the data batches pooled into one data set, without additional condition. Both simulations and data analysis are conducted to illustrate the finite sample performance of the proposed methods.

Scalable in situ compression of transient simulation data using time-dependent bases

Large-scale simulations of time-dependent problems generate a massive amount of data and with the explosive increase in computational resources the size of the data generated by these simulations has increased significantly. This has imposed severe limitations on the amount of data that can be stored and has elevated the issue of input/output (I/O) into one of the major bottlenecks of high-performance computing. In this work, we present an in situ compression technique to reduce the size of the data storage by orders of magnitude. This methodology is based on time-dependent subspaces and it extracts low-rank structures from multidimensional streaming data by decomposing the data into a set of time-dependent bases and a core tensor. We derive closed-form evolution equations for the core tensor as well as the time-dependent bases. The presented methodology does not require the data history and the computational cost of its extractions scales linearly with the size of data — making it suitable for large-scale streaming datasets. To control the compression error, we present an adaptive strategy to add/remove modes to maintain the reconstruction error below a given threshold. We present four demonstration cases: (i) analytical example, (ii) incompressible unsteady reactive flow, (iii) stochastic turbulent reactive flow, and (iv) three-dimensional turbulent channel flow.

A Visual Analytics Approach for Hardware System Monitoring with Streaming Functional Data Analysis.

Many real-world applications involve analyzing time-dependent phenomena, which are intrinsically functional, consisting of curves varying over a continuum (e.g., time). When analyzing continuous data, functional data analysis (FDA) provides substantial benefits, such as the ability to study the derivatives and to restrict the ordering of data. However, continuous data inherently has infinite dimensions, and for a long time series, FDA methods often suffer from high computational costs. The analysis problem becomes even more challenging when updating the FDA results for continuously arriving data. In this paper, we present a visual analytics approach for monitoring and reviewing time series data streamed from a hardware system with a focus on identifying outliers by using FDA. To perform FDA while addressing the computational problem, we introduce new incremental and progressive algorithms that promptly generate the magnitude-shape (MS) plot, which conveys both the functional magnitude and shape outlyingness of time series data. In addition, by using an MS plot in conjunction with an FDA version of principal component analysis, we enhance the analyst's ability to investigate the visually-identified outliers. We illustrate the effectiveness of our approach with two use scenarios using real-world datasets. The resulting tool is evaluated by industry experts using real-world streaming datasets.

Efficient and error-bounded spatiotemporal quantile monitoring in edge computing environments

Underlying many types of data analytics, a spatiotemporal quantile monitoring (SQM) query continuously returns the quantiles of a dataset observed in a spatiotemporal range. In this paper, we study SQM in an Internet of Things (IoT) based edge computing environment, where concurrent SQM queries share the same infrastructure asynchronously. To minimize query latency while providing result accuracy guarantees, we design a processing framework that virtualizes edge-resident data sketches for quantile computing. In the framework, a coordinator edge node manages edge sketches and synchronizes edge sketch processing and query executions. The co-ordinator also controls the processed data fractions of edge sketches, which helps to achieve the optimal latency with error-bounded results for each single query. To support concurrent queries, we employ a grid to decompose queries into subqueries and process them efficiently using shared edge sketches. We also devise a relaxation algorithm to converge to optimal latencies for those subqueries whose result errors are still bounded. We evaluate our proposals using two high-speed streaming datasets in a simulated IoT setting with edge nodes. The results show that our proposals achieve efficient, scalable, and error-bounded SQM.

High Median Utility Itemset Mining for recovering streaming window transaction using novel Modified Heap‐based Optimization

AbstractHigh utility itemsets (HUIs) are items in the dynamically streaming transaction list that generate a high‐profit margin. Many of the real‐time applications depend on finding HUIs from the transaction list. However, HUI mining is time‐consuming and results in high complexity due to memory requirements, a large search space, and the cost of HUI estimation. To overcome these issues we have proposed a novel High Median Utility Itemset Mining (HMUIM) approach. This approach utilizes the HMUI‐Miner which effectively ignores the unnecessary itemsets, that is, items with less profit, and mines the HUIs from the database. Furthermore, it has the limitation of deleting the transaction list based on the sliding window size used while conducting a dynamic streaming dataset. To tackle this issue we proposed a novel Modified Heap‐based Optimizer (MHBO) algorithm which effectively copies the HUIs transaction list and preserves it for further process. The MHBO is the combination of Heap Based Optimizer and nine fuzzy rules. The fuzzy rules are used to analyze the priority of the transaction list and based on that it ignores the transaction list with low priorities. Experimental analysis is performed for the proposed method on the real‐time dataset and compared with HUI and HAUIM approaches. The proposed method reduces the execution time and memory usage based on the minimum threshold. Meanwhile, the proposed MHBO approach is compared with iMEFIM, REX, and FCHUIM state‐of‐art works based on the sliding window concept. The execution time to store the replicated copies of original data is less for our proposed method.

Efficient quantile tracking using an oracle

Concept drift is a well-known issue that arises when working with data streams. In this paper, we present a procedure that allows a quantile tracking procedure to cope with concept drift. We suggest using expected quantile loss, a popular loss function in quantile regression, to monitor the quantile tracking error, which, in turn, is used to efficiently adapt to concept drift. The suggested procedures adapt efficiently to concept drift, and the tracking performance is close to theoretically optimal. The procedures were further applied to three real-life streaming data sets related to Twitter event detection, activity recognition, and stock trading. The results show that the procedures are efficient at adapting to concept drift, thereby documenting the real-world applicability of the procedures. We further used asymptotic theory from statistics to show the appealing theoretical property that, if the data stream distribution is stationary over time, the procedures converge to the true quantile.

A Twitter Streaming Dataset collected before and after the Onset of the War between Russia and Ukraine in 2022

A Twitter Streaming Dataset collected before and after the Onset of the War between Russia and Ukraine in 2022

DME: An Adaptive and Just-in-Time Weighted Ensemble Learning Method for Classifying Block-Based Concept Drift Steam

This study proposes a novel incremental learning algorithm called distribution matching ensemble (DME) in context of adaptive weighted ensemble learning. In particular, DME estimates the distribution of each received data block by Gaussian mixture model (GMM) and reserves the corresponding distribution information, as well it maintains a group of classifiers in a buffer. When we receive a new data block which is required to be predicted, the similarity between its distribution and each reserved distribution will be calculated by Kullback-Leibler (KL) divergence, and then the similarities can be used to guide the weight assignment of each corresponding classifier to further make adaptive ensemble decision. DME gets rid of the underlying hypothesis that the most recent labeled data block always has the most similar distribution with the current unlabeled data block. In addition, to avoid infinite extension of ensemble buffer during incremental learning, we also develop two dynamic classifier update rules. Experiments results on some synthetic and real-world streaming datasets show that the proposed DME algorithm is able to track and adapt to various types of concept drift just in time. Especially, on data stream with frequent reoccurring drifts, the DME significantly outperforms to several state-of-the-art algorithms, indicating its superiority.

Deep learning framework for handling concept drift and class imbalanced complex decision-making on streaming data

AbstractIn present times, data science become popular to support and improve decision-making process. Due to the accessibility of a wide application perspective of data streaming, class imbalance and concept drifting become crucial learning problems. The advent of deep learning (DL) models finds useful for the classification of concept drift in data streaming applications. This paper presents an effective class imbalance with concept drift detection (CIDD) using Adadelta optimizer-based deep neural networks (ADODNN), named CIDD-ADODNN model for the classification of highly imbalanced streaming data. The presented model involves four processes namely preprocessing, class imbalance handling, concept drift detection, and classification. The proposed model uses adaptive synthetic (ADASYN) technique for handling class imbalance data, which utilizes a weighted distribution for diverse minority class examples based on the level of difficulty in learning. Next, a drift detection technique called adaptive sliding window (ADWIN) is employed to detect the existence of the concept drift. Besides, ADODNN model is utilized for the classification processes. For increasing the classifier performance of the DNN model, ADO-based hyperparameter tuning process takes place to determine the optimal parameters of the DNN model. The performance of the presented model is evaluated using three streaming datasets namely intrusion detection (NSL KDDCup) dataset, Spam dataset, and Chess dataset. A detailed comparative results analysis takes place and the simulation results verified the superior performance of the presented model by obtaining a maximum accuracy of 0.9592, 0.9320, and 0.7646 on the applied KDDCup, Spam, and Chess dataset, respectively.

Technology Focus: Completions (April 2021)

In the 2020 Completions Technology Focus, I stated that digitization will forever change how the most complex problems in our industry are solved. And, despite another severe downturn in the upstream industry, data science continues to provide solutions for complex unconventional well problems. Casing Damage Casing collapse is an ongoing problem and almost always occurs in the heel of the well. It prevents passage of frac plugs and milling tools. Forcing a frac plug through the collapsed section damages the plug, predisposing it to failure, which leads to more casing damage and poor stimulation. One team has developed a machine-learning (ML) model showing a positive correlation between zones with high fracturing gradients and collapsed casing. The objective is a predictive tool that enables a completion design that avoids these zones. Fracture-Driven Interactions (FDIs) Can Be Avoided in Real Time Pressurized fracturing fluids from one well can communicate with fractures in a nearby well or can intersect that well-bore. Such FDIs can occur while fracturing a child well and can negatively affect production in the parent well. FDIs are caused by well spacing, depletion, or completion design but, until recently, were not quickly diagnosed. Analytics and machine learning now are being used to analyze streaming data sets during a frac job to detect FDIs. A recently piloted detection system alerts the operator in real time, which enables avoidance of FDIs on the fly. Data Science Provides the Tools Analyzing casing damage and FDIs is a complex task involving large amounts of data already available or easily acquired. Tools such as ML perform the data analysis and enable decision making. Data science is enabling the unconventional “onion” to be peeled many layers at a time. Recommended additional reading at OnePetro: www.onepetro.org. SPE 199967 - Artificial Intelligence for Real-Time Monitoring of Fracture-Driven Interactions and Simultaneous Completion Optimization by Hayley Stephenson, Baker Hughes, et al. SPE 201615 - Novel Completion Design To Bypass Damage and Increase Reservoir Contact: A Middle Magdalena, Central Colombian Case History by Rosana Polo, Oxy, et al. SPE 202966 - Well Completion Optimization in Canada Tight Gas Fields Using Ensemble Machine Learning by Lulu Liao, Sinopec, et al.