Streaming Data Sources Research Articles

The Graph of Things: A step towards the Live Knowledge Graph of connected things

The Internet of Things (IoT) with billions of connected devices has been generating an enormous amount of data every hour. Connecting every data item generated by IoT to the rest of the digital world to turn this data into meaningful actions will create new capabilities, richer experiences, and unprecedented economic opportunities for businesses, individuals, and countries. However, providing an integrated view for exploring and querying such data at real-time is extremely challenging due to its Big Data natures: big volume, fast real-time update and messy data sources. To address this challenge, we provide a unified integrated and live view for heterogeneous IoT data sources using Linked Data, called the Graph of Things (GoT). GoT is backed by a scalable and elastic software stack to deal with billions of records of historical and static datasets in conjunction with millions of triples being fetched and enriched to connect to GoT per hour in real time. GoT makes approximately a half of million stream data sources queryable via a SPARQL endpoint and a continuous query channel that enable us to create a live explorer of GoT (http://graphofthings.org/) with just HTML and Javascript.

Kalman-Based Stochastic Gradient Method with Stop Condition and Insensitivity to Conditioning

Modern proximal and stochastic gradient descent (SGD) methods are believed to efficiently minimize large composite objective functions, but such methods have two algorithmic challenges: (1) a lack of fast or justified stop conditions, and (2) sensitivity to the objective function's conditioning. In response to the first challenge, modern proximal and SGD methods guarantee convergence only after multiple epochs, but such a guarantee renders proximal and SGD methods infeasible when the number of component functions is very large or infinite. In response to the second challenge, second order SGD methods have been developed, but they are marred by the complexity of their analysis. In this work, we address these challenges on the limited, but important, linear regression problem by introducing and analyzing a second order proximal/SGD method based on Kalman Filtering (kSGD). Through our analysis, we show kSGD is asymptotically optimal, develop a fast algorithm for very large, infinite or streaming data sources with a justified stop condition, prove that kSGD is insensitive to the problem's conditioning, and develop a unique approach for analyzing the complex second order dynamics. Our theoretical results are supported by numerical experiments on three regression problems (linear, nonparametric wavelet, and logistic) using three large publicly available datasets. Moreover, our analysis and experiments lay a foundation for embedding kSGD in multiple epoch algorithms, extending kSGD to other problem classes, and developing parallel and low memory kSGD implementations.

Reactive Vega: A Streaming Dataflow Architecture for Declarative Interactive Visualization.

We present Reactive Vega, a system architecture that provides the first robust and comprehensive treatment of declarative visual and interaction design for data visualization. Starting from a single declarative specification, Reactive Vega constructs a dataflow graph in which input data, scene graph elements, and interaction events are all treated as first-class streaming data sources. To support expressive interactive visualizations that may involve time-varying scalar, relational, or hierarchical data, Reactive Vega's dataflow graph can dynamically re-write itself at runtime by extending or pruning branches in a data-driven fashion. We discuss both compile- and run-time optimizations applied within Reactive Vega, and share the results of benchmark studies that indicate superior interactive performance to both D3 and the original, non-reactive Vega system.

Modeling and Optimization for Big Data Analytics: (Statistical) learning tools for our era of data deluge

With pervasive sensors continuously collecting and storing massive amounts of information, there is no doubt this is an era of data deluge. Learning from these large volumes of data is expected to bring significant science and engineering advances along with improvements in quality of life. However, with such a big blessing come big challenges. Running analytics on voluminous data sets by central processors and storage units seems infeasible, and with the advent of streaming data sources, learning must often be performed in real time, typically without a chance to revisit past entries. Workhorse signal processing (SP) and statistical learning tools have to be re-examined in todays high-dimensional data regimes. This article contributes to the ongoing cross-disciplinary efforts in data science by putting forth encompassing models capturing a wide range of SP-relevant data analytic tasks, such as principal component analysis (PCA), dictionary learning (DL), compressive sampling (CS), and subspace clustering. It offers scalable architectures and optimization algorithms for decentralized and online learning problems, while revealing fundamental insights into the various analytic and implementation tradeoffs involved. Extensions of the encompassing models to timely data-sketching, tensor- and kernel-based learning tasks are also provided. Finally, the close connections of the presented framework with several big data tasks, such as network visualization, decentralized and dynamic estimation, prediction, and imputation of network link load traffic, as well as imputation in tensor-based medical imaging are highlighted.

Real-Time Semiparametric Regression

We develop algorithms for performing semiparametric regression analysis in real time, with data processed as it is collected and made immediately available via modern telecommunications technologies. Our definition of semiparametric regression is quite broad and includes, as special cases, generalized linear mixed models, generalized additive models, geostatistical models, wavelet nonparametric regression models and their various combinations. Fast updating of regression fits is achieved by couching semiparametric regression into a Bayesian hierarchical model or, equivalently, graphical model framework and employing online mean field variational ideas. An Internet site attached to this article, realtime-semiparametric-regression.net, illustrates the methodology for continually arriving stock market, real estate, and airline data. Flexible real-time analyses based on increasingly ubiquitous streaming data sources stand to benefit. This article has online supplementary material.

Subspace clustering of data streams: new algorithms and effective evaluation measures

Nowadays, most streaming data sources are becoming high dimensional. Accordingly, subspace stream clustering, which aims at finding evolving clusters within subgroups of dimensions, has gained a significant importance. However, in spite of the rich literature of subspace and projected clustering algorithms on static data, only three stream projected algorithms are available. Additionally, existing subspace clustering evaluation measures are mainly designed for static data, and cannot reflect the quality of the evolving nature of data streams. On the other hand, available stream clustering evaluation measures care only about the errors of the full-space clustering but not the quality of subspace clustering. In this article we present a method for designing new stream subspace and projected algorithms. We propose also, to the first of our knowledge, the first subspace clustering measure that is designed for streaming data, called SubCMM: Subspace Cluster Mapping Measure. SubCMM is an effective evaluation measure for stream subspace clustering that is able to handle errors caused by emerging, moving, or splitting subspace clusters. Additionally, we propose a novel method for using available offline subspace clustering measures for data streams over the suggested new algorithms within the Subspace MOA framework.

Distributed clustering of ubiquitous data streams

Nowadays information is generated and gathered from distributed streaming data sources, stressing communications and computing infrastructure, making it hard to transmit, compute, and store. Knowledge discovery from ubiquitous data streams has become a major goal for all sorts of applications, mostly based on unsupervised techniques such as clustering. Two subproblems exist: clustering streaming data observations and clustering streaming data sources. The former searches for dense regions of the data space, identifying hot spots where data sources tend to produce data, while the latter finds groups of sources that behave similarly over time. In order to better assess the current status of this topic, this article presents a thorough review on distributed algorithms addressing either of the subproblems. We characterize clustering algorithms for ubiquitous data streams, discussing advantages and disadvantages of distributed procedures. Overall, distributed stream clustering methods improve communication ratios, processing speed, and resources consumption, while achieving similar clustering validity as the centralized counterparts. WIREs Data Mining Knowl Discov 2014, 4:38–54. doi: 10.1002/widm.1109This article is categorized under: Technologies > Machine Learning Technologies > Structure Discovery and Clustering

PS2-42: Developing High-Quality Laboratory Results for the Virtual Data Warehouse: The Importance of Single-site Quality Assessment

2 Kaiser Permanente Northwest Background/Aims: Assessment of data incorporated into the Virtual Data Warehouse (VDW) is crucially important when building a new content area such as laboratory results. At KPCO, we identified two main challenges while constructing tables of laboratory results for the VDW. The first was ensuring that we constructed clinically meaningful laboratory test types. The second was confirming that we identified and classified the correct set of records from source data repositories. The aim of our work was therefore to ensure that these two challenges were successfully met Methods: Initial priority tests were broadly categorized as chemistry, hematology, microbiology, or challenge tests. Many laboratory results could be characterized into a single test type. Other laboratory results required one or more of the following characteristics to differentiate subtypes: assay method; test subtypes such as isoenzymes; result type; fasting status; specimen site; and challenge dose and time since dose. Laboratory results were extracted from two data repositories. The first database includes records from 1999-2004. The second database stores records since 2005 with partial backfill of records from 2004 and earlier. Variable names and values did not follow a common naming convention between the two data sources. Extraction and classification rules were based on Logical Observation Identifiers Names and Codes (LOINCs), test names, result units, and Current Procedural Terminology (CPT) codes. Results: We identified numerous issues while developing the VDW laboratory content area: incomplete clinical information; distinct laboratory types with similar names; missing, incorrect, obsolete, or site-specific component codes; incorrect sample collection times; changes in source data streams; clinical practices that differ from guidelines; and little background knowledge of laboratory tests among those processing the data. Each challenge and the approach to addressing each issue will be illustrated with an example. Conclusions: A systematic approach to collection and processing of laboratory results can yield high-quality, clinically useful data. Key elements of this systematic approach include a logical and flexible taxonomy of laboratory types and careful consideration of identifier codes and supplemental information to categorize laboratory results. Developers

Enabling Query Technologies for the Semantic Sensor Web

Sensor networks are increasingly being deployed in the environment for many different purposes. The observations that they produce are made available with heterogeneous schemas, vocabularies and data formats, making it difficult to share and reuse this data, for other purposes than those for which they were originally set up. The authors propose an ontology-based approach for providing data access and query capabilities to streaming data sources, allowing users to express their needs at a conceptual level, independent of implementation and language-specific details. In this article, the authors describe the theoretical foundations and technologies that enable exposing semantically enriched sensor metadata, and querying sensor observations through SPARQL extensions, using query rewriting and data translation techniques according to mapping languages, and managing both pull and push delivery modes.

An Autonomous Agent Approach to Query Optimization in Stream Grids

Stream grids are wide-area grid computing environments that are fed by a set of stream data sources, and Queries arrive at the grid from users and applications external to the system. The kind of queries considered in this work is long-running continuous (LRC) queries, which are neither short-lived nor infinitely long lived. The queries are “open” from the grid perspective as the grid cannot control or predict the arrival of a query with time, location, required data and query revocations. Query optimization in such an environment has two major challenges, i.e., optimizing in a multi-query environment and continuous optimization, due to new query arrivals and revocations. As generating a globally optimal query plan is an intractable problem, this work explores the idea of emergent optimization where globally optimal query plans emerge as a result of local autonomous decisions taken by the grid nodes. Drawing concepts from evolutionary game theory, grid nodes are modeled as autonomous agents that seek to maximize a self-interest function using one of a set of different strategies. Grid nodes change strategies in response to variations in query arrival and revocation patterns, which is also autonomously decided by each grid node.

SmartCIS

With the increasing adoption of networked sensors, a new class of applications is emerging that combines data from the "digital world" with real-time sensor readings, in order to intelligently manage physical environments and systems (e.g., "smart" buildings, power grids, data centers). This leads to new challenges in providing programmability, performance, extensibility, and multi-purpose heterogeneous data acquisition. The ASPEN project addresses these challenges by extending data integrationtechniques to the distributed stream world, and adding new abstractions for physical phenomena. We describe the architecture and implementation of our ASPEN system and its showcase intelligent building application, SmartCIS, which was demonstrated at SIGMOD 2009. We summarize the new query processing algorithms we have developed for integrating highly distributed stream data sources, both in low-power sensor devices and traditional PCs and servers; describe query optimization techniques for federations of stream processors; and detail new capabilities such as incremental maintenance of recursive views. Our algorithms and techniques generalize across a wide range of data from RFID and light measurements to real-time machine usage monitoring, energy consumption and recursive query computation.

A DICOM-based streaming service for the Digital Operating Room

In the Digital Operating Room there is a need to support data streaming to create advanced integrated surgical assist systems. In this paper we propose a DICOM-based streaming mechanism which leverages the interoperability definitions offered by DICOM to offer a common interface to manage all kinds of streaming data sources, while allowing data and application-specific protocols and infrastructure for the actual data access. We have implemented the proposed solution within the ASTMA project and have shown that thanks to the flexibility in choosing an appropriate streaming protocol we can achieve the necessary streaming quality while transmitting the context information required to create valid DICOM instances. This approach ensures an early integration of streaming data with the rest of the imaging information providing for a simpler data workflow.

Fine‐resolution satellite‐based daily sea surface temperatures over the global ocean

The accuracy and relative merits of two sets of daily global sea surface temperature (SST) analyses are examined and compared. The 1/8° Modular Ocean Data Analysis System (MODAS) of the Naval Research Laboratory (NRL) is based only on infrared satellite retrievals. The 1/2° Real‐Time, Global (RTG) SST analysis of the National Centers for Environmental Prediction (NCEP) supplements infrared satellite observations with ship and buoy data. The accuracy of both products is reported, providing potential users of either data set a common basis to assess the strengths and weaknesses of either product. Differences between the two show the impact of horizontal resolution, inclusion of source data streams, and different assumptions regarding error covariances. The global average of the root‐mean‐square (RMS) SST difference between MODAS and RTG is found to be 0.51°C, with almost no mean bias. A global set of yearlong daily SST time series from moored buoys during 2002–2005 provides extensive validation data for this study. Comparisons at the locations of these 420 yearlong time series give a median RMS SST difference of 0.40°C between MODAS and RTG. RMS error relative to the buoy observations is comparable, 0.38°C for MODAS and 0.36°C for RTG. The seasonal cycle of SST is well produced by both products with respect to the buoys with a median correlation coefficient of 0.94 for both products. Overall, higher resolution is an advantage for MODAS in improving pattern of daily SSTs, while including in situ SSTs is an advantage for RTG.

TimeLine and visualization of multiple-data sets and the visualization querying challenge

Data in its raw form can potentially contain valuable information, but much of that value is lost if it cannot be presented to a user in a way that is useful and meaningful. Data visualization techniques offer a solution to this issue. Such methods are especially useful in spatial data domains such as medical scan data and geophysical data. However, to properly see trends in data or to relate data from multiple sources, multiple-data set visualization techniques must be used. In research with the time-line paradigm, we have integrated multiple streaming data sources into a single visual interface. Data visualization takes place on several levels, from the visualization of query results in a time-line fashion to using multiple visualization techniques to view, analyze, and compare the data from the results. A significant contribution of this research effort is the extension and combination of existing research efforts into the visualization of multiple-data sets to create new and more flexible techniques. We specifically address visualization issues regarding clarity, speed, and interactivity. The developed visualization tools have also led recently to the visualization querying paradigm and challenge highlighted herein.