Internet Of Things Data Streams Research Articles

Real-Time Outlier Detection and Bayesian Classification using Incremental Computations for Efficient and Scalable Stream Analytics for IoT for Manufacturing

As the manufacturing industry progresses towards the Internet of Things (IoT) and Cyber-Physical Systems (CPS), current methods of historical data analytics face difficulties in addressing the new challenges which follow Industry 4.0. Industry 4.0 and IoT technologies facilitate the acquisition of ubiquitous data from machine tools and processes. However, these technologies also lead to the generation of a large number of data that are complex to be analyzed. Due to the streaming nature of the IoT systems, however, stream analytics could be used to extract features as the data are generated and published, which can prevent the need to store the data and perform advanced analytics that require high performance computing. This manuscript aims at demonstrating how traditional historical methods can be modified to be used as stream analytics tools for IoT data streams. Since data analytics is a wide domain, this paper has only focused on the two light-weighted methods that have been popular in the industry: Statistical Process Control Chart (SPCC), and Bayesian classification. This paper has defined, tested, and evaluated the accuracy and latency of the novel variation of these methods. It is concluded that by modifying the traditional methods and defining incremental solutions, methods such as Real-Time Dynamic Statistical Process Control Chart (RTDSPCC) and Incremental Gaussian Naïve Bayes (IGNB) can be formed that are highly beneficial for IoT applications as they are highly scalable, require minimal storage, and can update the models in real-time.

A Survey on Behavioral Pattern Mining From Sensor Data in Internet of Things

The deployment of large-scale wireless sensor networks (WSNs) for the Internet of Things (IoT) applications is increasing day-by-day, especially with the emergence of smart city services. The sensor data streams generated from these applications are largely dynamic, heterogeneous, and often geographically distributed over large areas. For high-value use in business, industry and services, these data streams must be mined to extract insightful knowledge, such as about monitoring (e.g., discovering certain behaviors over a deployed area) or network diagnostics (e.g., predicting faulty sensor nodes). However, due to the inherent constraints of sensor networks and application requirements, traditional data mining techniques cannot be directly used to mine IoT data streams efficiently and accurately in real-time. In the last decade, a number of works have been reported in the literature proposing behavioral pattern mining algorithms for sensor networks. This paper presents the technical challenges that need to be considered for mining sensor data. It then provides a thorough review of the mining techniques proposed in the recent literature to mine behavioral patterns from sensor data in IoT, and their characteristics and differences are highlighted and compared. We also propose a behavioral pattern mining framework for IoT and discuss possible future research directions in this area.

Uncovering data stream behaviour of automated analytical tasks in edge computing

Massive volumes of data streams are expected to be generated by the internet of things (IoT). Due to their dispersed and mobile nature, they need to be processed using automated analytical tasks. The research challenge is to uncover whether the data streams, which are being generated by billions of IoT devices, actually conform to a data flow that is required to perform streaming analytics. In this paper, we propose process discovery and conformance checking techniques of process mining in order to expose the flow dependency of IoT data streams between automated analytical tasks running at the edge of a network. Towards this end, we have developed a Petri Net model to ensure the optimal execution of analytical tasks by finding path deviations, bottlenecks, and parallelism. A real-world scenario in smart transit is used to evaluate the full advantage of our proposed model. Uncovering the actual behaviour of data flows from IoT devices to edge nodes has allowed us to detect discrepancies that have a negative impact on the performance of automated analytical tasks.

High-performance IoT streaming data prediction system using Spark: a case study of air pollution

Internet-of-Things (IoT) devices are becoming prevalent, and some of them, such as sensors, generate continuous time-series data, i.e., streaming data. These IoT streaming data are one of Big Data sources, and they require careful consideration for efficient data processing and analysis. Deep learning is emerging as a solution to IoT streaming data analytics. However, there is a persistent problem in deep learning that it takes a long time to learn neural networks. In this paper, we propose a high-performance IoT streaming data prediction system to improve the learning speed and to predict in real time. We showed the efficacy of the system through a case study of air pollution. The experimental results show that the modified LSTM autoencoder model shows the best performance compared to a generic LSTM model. We noticed that achieving the best performance requires optimizing many parameters, including learning rate, epoch, memory cell size, input timestep size, and the number of features/predictors. In that regard, we show that the high-performance data learning/prediction frameworks (e.g., Spark, Dist-Keras, and Hadoop) are essential to rapidly fine-tune a model for training and testing before real deployment of the model as data accumulate.

Exemplar-based data stream clustering toward Internet of Things

Dealing with dynamic data stream has become one of the most active research fields for Internet of Things (IoT). Specifically, clustering toward dynamic data stream is a necessary foundation for numerous IoT platforms. In this paper, we focus on dynamic exemplar-based clustering models. In terms of the maximum a priori principle, under the probability framework, we first summarize a unified explanation for two typical exemplar-based clustering models, namely enhanced $$\alpha$$-expansion move (EEM) and affinity propagation (AP). Then, a new dynamic exemplar-based data stream clustering algorithm called DSC is proposed accordingly. The distinctive merit of the proposed algorithm DSC is that we can simply utilize the framework of EEM algorithm through modifying the definitions of several variables and do not need to design another optimization mechanism. Moreover, algorithm DSC is capable of dealing to two cases of similarities. In contrast to both AP and EEM, our experimental results indicate the power of algorithm DSC for real-world IoT data streams.

A Multi-Protocol IoT Platform Based on Open-Source Frameworks.

Internet of Things (IoT) technologies have evolved rapidly during the last decade, and many architecture types have been proposed for distributed and interconnected systems. However, most systems are implemented following fragmented approaches for specific application domains, introducing difficulties in providing unified solutions. However, the unification of solutions is an important feature from an IoT perspective. In this paper, we present an IoT platform that supports multiple application layer communication protocols (Representational State Transfer (REST)/HyperText Transfer Protocol (HTTP), Message Queuing Telemetry Transport (MQTT), Advanced Message Queuing Protocol (AMQP), Constrained Application Protocol (CoAP), and Websockets) and that is composed of open-source frameworks (RabbitMQ, Ponte, OM2M, and RDF4J). We have explored a back-end system that interoperates with the various frameworks and offers a single approach for user-access control on IoT data streams and micro-services. The proposed platform is evaluated using its containerized version, being easily deployable on the vast majority of modern computing infrastructures. Its design promotes service reusability and follows a marketplace architecture, so that the creation of interoperable IoT ecosystems with active contributors is enabled. All the platform’s features are analyzed, and we discuss the results of experiments, with the multiple communication protocols being tested when used interchangeably for transferring data. Developing unified solutions using such a platform is of interest to users and developers as they can test and evaluate local instances or even complex applications composed of their own IoT resources before releasing a production version to the marketplace.

An Edge-Fog-Cloud Architecture of Streaming Analytics for Internet of Things Applications.

Exploring Internet of Things (IoT) data streams generated by smart cities means not only transforming data into better business decisions in a timely way but also generating long-term location intelligence for developing new forms of urban governance and organization policies. This paper proposes a new architecture based on the edge-fog-cloud continuum to analyze IoT data streams for delivering data-driven insights in a smart parking scenario.

Probabilistic Top-${k}$ Dominating Query Monitoring Over Multiple Uncertain IoT Data Streams in Edge Computing Environments

Extracting the valuable features and information in Big Data has become one of the important research issues in Data Science. In most Internet of Things (IoT) applications, the collected data are uncertain and imprecise due to sensor device variations or transmission errors. In addition, the sensing data may change as time evolves. We refer an uncertain data stream as a dataset that has velocity, veracity, and volume properties simultaneously. This paper employs the parallelism in edge computing environments to facilitate the top-k dominating query process over multiple uncertain IoT data streams. The challenges of this problem include how to quickly update the result for processing uncertainty and reduce the computation cost as well as provide highly accurate results. By referring to the related existing papers for certain data, we provide an effective probabilistic top-k dominating query process on uncertain data streams, which can be parallelized easily. After discussing the properties of the proposed approach, we validate our methods through the complexity analysis and extensive simulated experiments. In comparison with the existing works, the experimental results indicate that our method can improve almost 60% computation time, reduce nearly 20% communication cost between servers, and provide highly accurate results in most scenarios.

Streaming MASSIF: Cascading Reasoning for Efficient Processing of IoT Data Streams.

In the Internet of Things (IoT), multiple sensors and devices are generating heterogeneous streams of data. To perform meaningful analysis over multiple of these streams, stream processing needs to support expressive reasoning capabilities to infer implicit facts and temporal reasoning to capture temporal dependencies. However, current approaches cannot perform the required reasoning expressivity while detecting time dependencies over high frequency data streams. There is still a mismatch between the complexity of processing and the rate data is produced in volatile domains. Therefore, we introduce Streaming MASSIF, a Cascading Reasoning approach performing expressive reasoning and complex event processing over high velocity streams. Cascading Reasoning is a vision that solves the problem of expressive reasoning over high frequency streams by introducing a hierarchical approach consisting of multiple layers. Each layer minimizes the processed data and increases the complexity of the data processing. Cascading Reasoning is a vision that has not been fully realized. Streaming MASSIF is a layered approach allowing IoT service to subscribe to high-level and temporal dependent concepts in volatile data streams. We show that Streaming MASSIF is able to handle high velocity streams up to hundreds of events per second, in combination with expressive reasoning and complex event processing. Streaming MASSIF realizes the Cascading Reasoning vision and is able to combine high expressive reasoning with high throughput of processing. Furthermore, we formalize semantically how the different layers in our Cascading Reasoning Approach collaborate.

A oneM2M-Based Query Engine for Internet of Things (IoT) Data Streams.

The new standard oneM2M (one machine-to-machine) aims to standardize the architecture and protocols of Internet of Things (IoT) middleware for better interoperability. Although the standard seems promising, it lacks several features for efficiently searching and retrieving IoT data which satisfy users’ intentions. In this paper, we design and develop a oneM2M-based query engine, called OMQ, that provides a real-time processing over IoT data streams. For this purpose, we define a query language which enables users to retrieve IoT data from data sources using JavaScript Object Notation (JSON). We also propose efficient query processing algorithms which utilizes the oneM2M architecture consisting of two nodes: (1) the IoT node and (2) the infrastructure node. IoT nodes of OMQ are mainly sensor devices execute user queries the aggregate, transform and filter operators, whereas the infrastructure node handles the join operator of user queries. Since the query processing algorithms are implemented as the hybrid infrastructure-edge processing, user queries can be executed efficiently in each IoT node rather than only in the infrastructure node. Thus, our OMQ system reduces the query processing time and the network bandwidth. We conducted a comprehensive evaluation of OMQ using a real and a synthetic data set. Experimental results demonstrate the feasibility and efficiency of OMQ system for executing queries and transferring data from each IoT node.

Content-Aware Cognitive Interference Control for Urban IoT Systems

A novel cognitive interference control framework for heterogeneous local access networks supporting computing and data processing in urban Internet of Things (IoT) systems is presented. The notion of cognitive content-aware interference control is introduced, where the transmission pattern of cognitive nodes is dynamically adapted to the state of “protected” IoT data streams. The state describes the performance degradation that interference would cause to algorithms processing the data if the cognitive nodes chose to transmit in the corresponding time period. The framework is instantiated for a case-study scenario, where device-to-device (D2D) and long-term evolution communications coexist on the same channel resource. A city-monitoring application is considered, where the state captures the type of frames within a multimedia video stream processed at the edge of the network to detect and track objects. Numerical results show that the proposed cognitive transmission strategy enables a significant throughput increase of local D2D communications for a target accuracy of the monitoring application.

Using LDA to Uncover the Underlying Structures and Relations in Smart City Data Streams

Recent advancements in sensing, networking technologies, and collecting real-world data on a large scale and from various environments have created an opportunity for new forms of real-world services and applications. This is known under the umbrella term of the Internet of Things (IoT). Physical sensor devices constantly produce very large amounts of data. Methods are needed which give the raw sensor measurements a meaningful interpretation for building automated decision support systems. To extract actionable information from real-world data, we propose a method that uncovers hidden structures and relations between multiple IoT data streams. Our novel solution uses latent Dirichlet allocation (LDA), a topic extraction method that is generally used in text analysis. We apply LDA on meaningful abstractions that describe the numerical data in human understandable terms. We use symbolic aggregate approximation to convert the raw data into string-based patterns and create higher level abstractions based on rules. We finally investigate how heterogeneous sensory data from multiple sources can be processed and analyzed to create near real-time intelligence and how our proposed method provides an efficient way to interpret patterns in the data streams. The proposed method uncovers the correlations and associations between different patterns in IoT data streams. The evaluation results show that the proposed solution is able to identify the correlation with high efficiency with an F -measure up to 90%.

Deep Learning for IoT Big Data and Streaming Analytics: A Survey

In the era of the Internet of Things (IoT), an enormous amount of sensing devices collect and/or generate various sensory data over time for a wide range of fields and applications. Based on the nature of the application, these devices will result in big or fast/real-time data streams. Applying analytics over such data streams to discover new information, predict future insights, and make control decisions is a crucial process that makes IoT a worthy paradigm for businesses and a quality-of-life improving technology. In this paper, we provide a thorough overview on using a class of advanced machine learning techniques, namely deep learning (DL), to facilitate the analytics and learning in the IoT domain. We start by articulating IoT data characteristics and identifying two major treatments for IoT data from a machine learning perspective, namely IoT big data analytics and IoT streaming data analytics. We also discuss why DL is a promising approach to achieve the desired analytics in these types of data and applications. The potential of using emerging DL techniques for IoT data analytics are then discussed, and its promises and challenges are introduced. We present a comprehensive background on different DL architectures and algorithms. We also analyze and summarize major reported research attempts that leveraged DL in the IoT domain. The smart IoT devices that have incorporated DL in their intelligence background are also discussed. DL implementation approaches on the fog and cloud centers in support of IoT applications are also surveyed. Finally, we shed light on some challenges and potential directions for future research. At the end of each section, we highlight the lessons learned based on our experiments and review of the recent literature.

Real-Time Probabilistic Data Fusion for Large-Scale IoT Applications

Internet of Things (IoT) data analytics is underpinning numerous applications, however, the task is still challenging predominantly due to heterogeneous IoT data streams, unreliable networks, and ever increasing size of the data. In this context, we propose a two-layer architecture for analyzing IoT data. The first layer provides a generic interface using a service oriented gateway to ingest data from multiple interfaces and IoT systems, store it in a scalable manner and analyze it in real-time to extract high-level events; whereas second layer is responsible for probabilistic fusion of these high-level events. In the second layer, we extend state-of-the-art event processing using Bayesian networks in order to take uncertainty into account while detecting complex events. We implement our proposed solution using open source components optimized for large-scale applications. We demonstrate our solution on real-world use-case in the domain of intelligent transportation system where we analyzed traffic, weather, and social media data streams from Madrid city in order to predict probability of congestion in real-time. The performance of the system is evaluated qualitatively using a web-interface where traffic administrators can provide the feedback about the quality of predictions and quantitatively using F-measure with an accuracy of over 80%.

RIoTBench: An IoT benchmark for distributed stream processing systems

SummaryThe Internet of Things (IoT) is an emerging technology paradigm where millions of sensors and actuators help monitor and manage physical, environmental, and human systems in real time. The inherent closed‐loop responsiveness and decision making of IoT applications make them ideal candidates for using low latency and scalable stream processing platforms. Distributed stream processing systems (DSPS) hosted in cloud data centers are becoming the vital engine for real‐time data processing and analytics in any IoT software architecture. But the efficacy and performance of contemporary DSPS have not been rigorously studied for IoT applications and data streams. Here, we propose RIoTBench, a real‐time IoT benchmark suite, along with performance metrics, to evaluate DSPS for streaming IoT applications. The benchmark includes 27 common IoT tasks classified across various functional categories and implemented as modular microbenchmarks. Further, we define four IoT application benchmarks composed from these tasks based on common patterns of data preprocessing, statistical summarization, and predictive analytics that are intrinsic to the closed‐loop IoT decision‐making life cycle. These are coupled with four stream workloads sourced from real IoT observations on smart cities and smart health, with peak streams rates that range from 500 to 10 000 messages/second from up to 3 million sensors. We validate the RIoTBench suite for the popular Apache Storm DSPS on the Microsoft Azure public cloud and present empirical observations. This suite can be used by DSPS researchers for performance analysis and resource scheduling, by IoT practitioners to evaluate DSPS platforms, and even reused within IoT solutions.

Predictive Analytics for Complex IoT Data Streams

The requirements of analyzing heterogeneous data streams and detecting complex patterns in near real-time have raised the prospect of complex event processing (CEP) for many Internet of Things (IoT) applications. Although CEP provides a scalable and distributed solution for analyzing complex data streams on the fly, it is designed for reactive applications as CEP acts on near real-time data and does not exploit historical data. In this regard, we propose a proactive architecture which exploits historical data using machine learning for prediction in conjunction with CEP. We propose an adaptive prediction algorithm called adaptive moving window regression for dynamic IoT data and evaluated it using a real-world use case with an accuracy of over 96%. It can perform accurate predictions in near real-time due to reduced complexity and can work along CEP in our architecture. We implemented our proposed architecture using open source components which are optimized for big data applications and validated it on a use-case from intelligent transportation systems. Our proposed architecture is reliable and can be used across different fields in order to predict complex events.

Adaptive Clustering for Dynamic IoT Data Streams

The emergence of the Internet of Things (IoT) has led to the production of huge volumes of real-world streaming data. We need effective techniques to process IoT data streams and to gain insights and actionable information from real-world observations and measurements. Most existing approaches are application or domain dependent. We propose a method which determines how many different clusters can be found in a stream based on the data distribution. After selecting the number of clusters, we use an online clustering mechanism to cluster the incoming data from the streams. Our approach remains adaptive to drifts by adjusting itself as the data changes. We benchmark our approach against state-of-the-art stream clustering algorithms on data streams with data drift. We show how our method can be applied in a use case scenario involving near real-time traffic data. Our results allow to cluster, label, and interpret IoT data streams dynamically according to the data distribution. This enables to adaptively process large volumes of dynamic data online based on the current situation. We show how our method adapts itself to the changes. We demonstrate how the number of clusters in a real-world data stream can be determined by analyzing the data distributions.

A domain-independent methodology to analyze IoT data streams in real-time. A proof of concept implementation for anomaly detection from environmental data

ABSTRACTPushed by the Internet of Things (IoT) paradigm modern sensor networks monitor a wide range of phenomena, in areas such as environmental monitoring, health care, industrial processes, and smart cities. These networks provide a continuous pulse of the almost infinite activities that are happening in the physical space and are thus, key enablers for a Digital Earth Nervous System. Nevertheless, the rapid processing of these sensor data streams still continues to challenge traditional data-handling solutions and new approaches are being requested. We propose a generic answer to this challenge, which has the potential to support any form of distributed real-time analysis. This neutral methodology follows a brokering approach to work with different kinds of data sources and uses web-based standards to achieve interoperability. As a proof of concept, we implemented the methodology to detect anomalies in real-time and applied it to the area of environmental monitoring. The developed system is capable of detecting anomalies, generating notifications, and displaying the recent situation to the user.

Design and Implementation of the Sense Egypt Platform for Real-Time Analysis of IoT Data Streams

Nowadays, we experience an abundance of Internet of Things middleware solutions that make the sensors and the actuators are able to connect to the Internet. These solutions, referred to as platforms to gain a widespread adoption, have to meet the expectations of different players in the IoT ecosystem, including devices [1]. Low cost devices are easily able to connect wirelessly to the Internet, from handhelds to coffee machines, also known as Internet of Things (IoT). This research describes the methodology and the development process of creating an IoT platform. This paper also presents the architecture and implementation for the IoT platform. The goal of this research is to develop an analytics engine which can gather sensor data from different devices and provide the ability to gain meaningful information from IoT data and act on it using machine learning algorithms. The proposed system is introducing the use of a messaging system to improve the overall system performance as well as provide easy scalability.