Internet Of Things Data Streams Research Articles

Revolutionizing Disease Diagnosis: Integrating Artificial Intelligence and IOT In Smart Healthcare Systems

This research paper delves into the transformative role of Artificial Intelligence (AI) and the Internet of Things (IoT) in revolutionizing disease diagnosis within smart healthcare systems. AI, a cornerstone in computer science, is now pivotal in healthcare, offering sophisticated algorithms for analyzing medical data, thus aiding in decision-making and predictions. IoT extends this capability by enabling continuous data collection through web-enabled devices, including wearables and implanted sensors. The integration of AI and IoT in smart healthcare systems not only refines medical procedures but also enhances patient experiences and operational workflows. This paper explores how AI-driven routines in conjunction with IoT data streams facilitate rapid and accurate disease diagnosis, surpassing traditional methods often limited by human biases. The adoption of these technologies in healthcare promises to improve diagnostic accuracy, expedite processes, and broaden access to medical expertise, especially in remote areas. However, challenges such as ensuring data privacy, security, and ethical use of AI remain paramount.

Publish/Subscribe Method for Real-Time Data Processing in Massive IoT Leveraging Blockchain for Secured Storage.

In the Internet of Things (IoT) era, the surge in Machine-Type Devices (MTDs) has introduced Massive IoT (MIoT), opening new horizons in the world of connected devices. However, such proliferation presents challenges, especially in storing and analyzing massive, heterogeneous data streams in real time. In order to manage Massive IoT data streams, we utilize analytical database software such as Apache Druid version 28.0.0 that excels in real-time data processing. Our approach relies on a publish/subscribe mechanism, where device-generated data are relayed to a dedicated broker, effectively functioning as a separate server. This broker enables any application to subscribe to the dataset, promoting a dynamic and responsive data ecosystem. At the core of our data transmission infrastructure lies Apache Kafka version 3.6.1, renowned for its exceptional data flow management performance. Kafka efficiently bridges the gap between MIoT sensors and brokers, enabling parallel clusters of brokers that lead to more scalability. In our pursuit of uninterrupted connectivity, we incorporate a fail-safe mechanism with two Software-Defined Radios (SDR) called Nutaq PicoLTE Release 1.5 within our model. This strategic redundancy enhances data transmission availability, safeguarding against connectivity disruptions. Furthermore, to enhance the data repository security, we utilize blockchain technology, specifically Hyperledger Fabric, known for its high-performance attributes, ensuring data integrity, immutability, and security. Our latency results demonstrate that our platform effectively reduces latency for 100,000 devices, qualifying as an MIoT, to less than 25 milliseconds. Furthermore, our findings on blockchain performance underscore our model as a secure platform, achieving over 800 Transactions Per Second in a dataset comprising 14,000 transactions, thereby demonstrating its high efficiency.

Robust Online Tensor Completion for IoT Streaming Data Recovery.

Reliable data measurement is considered to be one of the critical ingredients for variant Internet of Things (IoT) applications. Gaining full knowledge of measurement data is becoming increasingly crucial to ensure a satisfactory user experience. However, data missing and corruption are inevitable in practical applications, which motivates us to study how to accurately recover the missing IoT measurement data in the presence of outliers. The data recovery problem can be formulated as a tensor completion (TC) problem. Existing TC methods are built on the assumption that the rank of the underlying tensor is fixed, which is not suitable for long data sequences in practice. Consequently, based on the characteristics of IoT streaming data, we assume that the data tensor lies in time-varying subspace, and an accurate estimate of the rank is a prerequisite for filling the missing entries and achieving robustness of the variations in both rank and noise. We built up an updatable framework based on dynamic CANDECOMP/PARAFAC (CP) decomposition. In addition, an efficient algorithm, called temporal multi-aspect streaming (T-MUST), is introduced to solve the optimization problem that originates in our developed model. It is worth noting that the proposed algorithm allows time-varying tensor rank and enables the rank changes could be detected and tracked automatically. Theoretical analysis indicates that T-MUST enjoys a geometric convergence rate. Numerical experiments conducted on various synthetic and real-world datasets empirically validate the superiority of the proposed T-MUST in both efficiency and effectiveness.

Enhancing Cyber Security through Machine Learning-Based Anomaly Detection in IoT Networks

The rapid proliferation of IOT (Internet of Things) networks has brought transformative benefits to industries and everyday life. However, it has also introduced unprecedented cyber security challenges, necessitating advanced techniques for anomaly detection. This research focuses on enhancing cyber security through the application of machine learning-based anomaly detection methods, specifically One-Class Support Vector Machine (SVM) and Isolation Forest, in the context of IOT networks. While Isolation Forest effectively isolates anomalies by building isolation trees, One-Class SVM models the normal data distribution, effectively separating anomalies. To provide a strong security framework for IoT networks, we suggest a comprehensive strategy that combines both algorithms. Our method enables the detection of anomalies in real-time IOT data streams, facilitating prompt responses to new threats. Data collection, preprocessing, and model training are key components. This study helps protect IOT ecosystems and maintain data integrity and privacy in an increasingly connected world by utilizing the benefits of One-Class SVM and Isolation Forest.

Refresh Rate-Based Caching and Prefetching Strategies for Internet of Things Middleware.

One of the research directions in Internet of Things (IoT) is the field of Context Management Platforms (CMPs) which is a specific type of IoT middleware. CMPs provide horizontal connectivity between vertically oriented IoT silos resulting in a noticeable difference in how IoT data streams are processed. As these context data exchanges can be monetised, there is a need to model and predict the context metrics and operational costs of this exchange to provide relevant and timely context in a large-scale IoT ecosystem. In this paper, we argue that caching all transient context information to satisfy this necessity requires large amounts of computational and network resources, resulting in tremendous operational costs. Using Service Level Agreements (SLAs) between the context providers, CMP, and context consumers, where the level of service imperfection is quantified and linked to the associated costs, we show that it is possible to find efficient caching and prefetching strategies to minimize the context management cost. So, this paper proposes a novel method to find the optimal rate of IoT data prefetching and caching. We show the main context caching strategies and the proposed mathematical models, then discuss how a correctly chosen proactive caching strategy and configurations can help to maximise the profit of CMP operation when multiple SLAs are defined. Our model is accurate up to 0.0016 in Root Mean Square Percentage Error against our simulation results when estimating the profits to the system. We also show our model is valid using the t-test value tending to 0 for all the experimental scenarios.

Intrusion detection in the IoT data streams using concept drift localization

<abstract><p>With the widespread application of smart devices, the security of internet of things (IoT) systems faces entirely new challenges. The IoT data stream operates in a non-stationary, dynamic environment, making it prone to concept drift. This paper focused on addressing the issue of concept drift in data streams, with a key emphasis on introducing an innovative drift detection method-ensemble multiple non-parametric concept localization detectors, abbreviated as EMNCD. EMNCD employs an ensemble of non-parametric statistical methods, including the Kolmogorov-Smirnov, Wilcoxon rank sum and Mann-Kendall tests. By comparing sample distributions within a sliding window, EMNCD accurately detects concept drift, achieving precise localization of drift points, and enhancing overall detection reliability. Experimental results demonstrated the superior performance of EMNCD compared to classical methods on artificial datasets. Simultaneously, to enhance the robustness of data stream processing, we presented an online anomaly detection method based on the isolation forest (iForest). Additionally, we proposedwhale optimization algorithm (WOA)-extreme gradient boosting (XGBoost), a drift adaptation model employing XGBoost as a base classifier. This model dynamically updates using drift points detected by EMNCD and fine-tunes parameters through the WOA. Real-world applications on the edge-industrial IoTset (IIoTset) intrusion dataset explore the impact of concept drift on intrusion detection, where IIoT is a subclass of IoT. In summary, this paper focused on EMNCD, introducing innovative approaches for drift detection, anomaly detection, and drift adaptation. The research provided practical and viable solutions to address concept drift in data streams, enhancing security in IoT systems.</p></abstract>

KFIML: Kubernetes-Based Fog Computing IoT Platform for Online Machine Learning

The massive onsite data produced by the Internet of Things (IoT) can bring valuable information and immense potentials, thus empowering a new wave of emerging applications. However, with the rapid increase of onsite IoT data streams, it has become extremely challenging to develop a scalable computing platform and provide a comprehensive workflow for processing IoT data streams with lower latency and more intelligence. To this end, we present a Kubernetes-based scalable fog computing platform (KFIML), integrating big data streaming processing with machine learning (ML)-based applications. We also provide a comprehensive IoT data processing workflow, including data access and transfer, big data processing, online ML, long-term storage, and monitoring. The platform is feasibly validated on a clustered testbed, which comprises a master node, IoT broker servers, worker nodes, and a local database server. By leveraging the lightweight orchestration system, namely Kubernetes, we can readily scale and manage containerized software frameworks on our testbed. The big data processing layer utilizes the advanced data flow frameworks such as Apache Flink, to support both streaming processing and statistical analysis with low latency. In addition, the specified long short-term memory (LSTM)-based ML pipelines are employed on the online ML layer, to enable the real-time predictive analysis of IoT data streams. The experiments on a real-world smart grid use case demonstrate that the container-based KFIML platform can be well-scaled with Kubernetes to efficiently perform big data processing increased onsite IoT data streams with lower latency and conduct ML-based applications.

A Framework for IoT Streaming Data Indexing and Query Optimization

Streaming data are continuously generated by multiple Internet of Things (IoT) sources, e.g., sensors, mobile devices, etc., and sent simultaneously to relevant applications to be processed in real time in a continuous and timely fashion. Existing research has dealt with the integration of IoT streaming data from multiple sources. Some of the earlier research introduced models to facilitate streaming data compression and the subsequent indexing. However, as IoT data come from multiple sensors and differ in terms of variety and velocity, there is still an urgent need to build an efficient indexing mechanism so that we can optimise responses to users’ queries. In this research, we identify a variety of queries from different motivating scenarios and develop a framework to optimise the way to access and retrieve IoT streaming data to respond to the users’ queries. To this end, (i) a streaming data scenario is analysed, and (ii) an optimisation framework with indexing schemes from multiple sources are introduced, and (iii) some common and illustrative queries are presented to explain optimisation. Finally, a set of experiments on five queries is performed to illustrate the ability of the optimisation framework in a wide range of different scenarios. The results prove that the proposed optimisation framework is much better than the existing IoT data integration and indexing frameworks.

Bridging the gap between expressivity and efficiency in stream reasoning: a structural caching approach for IoT streams.

In today’s data landscape, data streams are well represented. This is mainly due to the rise of data-intensive domains such as the Internet of Things (IoT), Smart Industries, Pervasive Health, and Social Media. To extract meaningful insights from these streams, they should be processed in real time, while solving an integration problem as these streams need to be combined with more static data and their domain knowledge. Ontologies are ideal for modeling this domain knowledge and facilitate the integration of heterogeneous data within data-intensive domains such as the IoT. Expressive reasoning techniques, such as OWL2 DL reasoning, are needed to completely interpret the domain knowledge and for the extraction of meaningful decisions. Expressive reasoning techniques have mainly focused on static data environments, as it tends to become slow with growing datasets. There is thus a mismatch between expressive reasoning and the real-time requirements of data-intensive domains. In this paper, we take a first step towards bridging the gap between expressivity and efficiency while reasoning over high-velocity IoT data streams for the task of event enrichment. We present a structural caching technique that eliminates reoccurring reasoning steps by exploiting the characteristics of most IoT streams, i.e., streams typically produce events that are similar in structure and size. Our caching technique speeds up reasoning time up to thousands of times for fully fledged OWL2 DL reasoners and even tenths and hundreds of times for less expressive OWL2 RL and OWL2 EL reasoners.

A Lightweight Concept Drift Detection and Adaptation Framework for IoT Data Streams

In recent years, with the increasing popularity of "Smart Technology", the number of Internet of Things (IoT) devices and systems have surged significantly. Various IoT services and functionalities are based on the analytics of IoT streaming data. However, IoT data analytics faces concept drift challenges due to the dynamic nature of IoT systems and the ever-changing patterns of IoT data streams. In this article, we propose an adaptive IoT streaming data analytics framework for anomaly detection use cases based on optimized LightGBM and concept drift adaptation. A novel drift adaptation method named Optimized Adaptive and Sliding Windowing (OASW) is proposed to adapt to the pattern changes of online IoT data streams. Experiments on two public datasets show the high accuracy and efficiency of our proposed adaptive LightGBM model compared against other state-of-the-art approaches. The proposed adaptive LightGBM model can perform continuous learning and drift adaptation on IoT data streams without human intervention.

Weighted Adaptive Partition for Heterogeneous IoT Data Stream

In the past, many researchers used sliding-window mechanism in Internet of Things (IoT) data stream mining, which can divide the incoming large volume data into small segments and improve the information processing rate. With the emergence of heterogeneous IoT data streams, data forms become richer and more diverse. The fusion analysis of multimodality information can improve the accuracy of information processing and network decision-making, because it can make up for the lack of some information in a single-mode data stream. However, the representation of information in different data sources is different, and the amount of information contained in it is distinct. Thus, a weighted adaptive partition algorithm is proposed in this article, which performs a dynamic weighted fusion of multisource heterogeneous IoT data in sliding windows. The article adopts the Gaussian mixture model (GMM) to adaptively determine the weights of each data source, which is calculated based on the judgement confidence of each modal data to the change in data stream. Besides, the probability distribution change detection mechanism based on multimodality restricted Boltzmann machine (multimodality RBM) is proposed, which measures the joint probability distribution by the weighted data’s energy value. Finally, according to the energy change rate of the data stream, the size of the window is adjusted adaptively to realize the partition for heterogeneous IoT data stream. The algorithm proposed in this article is verified on the CUAVE data set and the analysis of results shows that the data under adaptive weighted fusion has better performance.

Internet‐of‐Things‐Infrastructure‐as‐a‐Service: The democratization of access to public Internet‐of‐Things Infrastructure

SummaryWe propose a new cloud service model called Internet‐of‐Things (IoT)‐Infrastructure‐as‐a‐Service (IoT‐IaaS). Under an IoT‐IaaS, an IaaS cloud service provider can go beyond offering IoT data streams and offer access to IoT infrastructure to users, the same way data center infrastructure is shared in the IaaS model. We developed a reference design for the proposed IoT‐IaaS cloud service model, built from open source components. We demonstrate the proposed reference design in the form of a virtualized‐IoT (vIoT) testbed. The vIoT testbed uses the OpenStack cloud management system and Raspberry Pi embedded computers (standing in for IoT devices) configured as Nova‐compute nodes and LXD containers. We also developed a simple mechanism for shared access to high data rate sensors (a camera) and low data rate sensors (temperature‐humidity) by multiple tenants. We tested the performance of container launch times, memory access (with and without ZRAM optimization kernel module enabled), CPU, and file I/O for applications running inside Linux containers and compared it with the same performance when running on the Raspberry Pi's host OS. We conclude that there is no significant performance penalty for executing the edge component of an IoT application inside a Linux container, rather than directly on the device's host OS.

IoT Architecture for Urban Data-Centric Services and Applications

In this work, we describe an urban Internet of Things (IoT) architecture, grounded in big data patterns and focused on the needs of cities and their key stakeholders. First, the architecture of the dedicated platform USE4IoT (Urban Service Environment for the Internet of Things), which gathers and processes urban big data and extends the Lambda architecture, is proposed. We describe how the platform was used to make IoT an enabling technology for intelligent transport planning. Moreover, key data processing components vital to provide high-quality IoT data streams in a near-real-time manner are defined. Furthermore, tests showing how the IoT platform described in this study provides a low-latency analytical environment for smart cities are included.

IoT streaming data integration from multiple sources

The Internet of Things (IoT) has recently received considerable interest due to the development of smart technologies in today’s interconnected world. With the rapid advancement in Internet technologies and the proliferation of IoT sensors, myriad systems and applications generate data of a massive volume, variety and velocity which traditional databases and systems are unable to manage effectively. Many organizations need to deal with these massive datasets that encounter different types of data (e.g., IoT streaming data, static data) in different formats (e.g., structured, semi-structured) coming from multiple sources. Several data integration mechanisms have been designed to process mostly static data. Unfortunately, these techniques are not able to deal with and integrate IoT streaming datasets from multiple sources. In this paper, we identify the challenges of IoT Streaming Data Integration (ISDI) and present a formal approach for the real-time integration of such IoT streaming datasets. We address one of the important issues of timing conflict/alignment among streaming data coming from multiple sources. A generic window-based ISDI approach is proposed to deal with IoT data in different formats and algorithms are developed to integrate IoT streaming data from multiple sources. In particular, we extend the basic windowing algorithm for real-time data integration and to deal with the timing alignment issue. We also introduce a de-duplication algorithm to deal with data redundancy and to demonstrate the useful fragments of the integrated data. We conduct several sets of experiments and quantify the performance of our proposed window-based approach. In particular, we compare our local experimental results with a real setup for streaming data, using Apache Spark. The results of the experiments, which are performed on several IoT datasets, show the efficiency of our proposed solution in terms of processing time. The results are also used to provide an integrated data view to the users.

Generating Situational Awareness of Pedestrian and Vehicular Movement in Urban Areas Using IoT Data Streams

Humans have continuously endeavored to enhance their sensory perception and awareness of the physical surroundings for the betterment of themselves as individuals as well as communities. Advancing this notion to the present day, the Internet of Things (IoT) provides a unique opportunity to attempt the same in an increasingly digital landscape. The prevalence and pervasiveness of IoT data streams gives us this ability to represent a situation with clarity and nuance, leading to a refined awareness of the environment. However, there are several challenges in managing the scale, velocity, and magnitude of IoT data streams, as well as the cohesive representation of these varied sources in a single frame of reference. In this article, we present a new algorithm, the deep growing self-organizing map (deep GSOM) algorithm that addresses these challenges. Deep GSOM incrementally generates a latent representation of situational awareness from high entropy to low entropy IoT data streams. It utilizes an implementation of the fuzzy integral to define a metric that can be moved across spatial and temporal situations to profile the density of congestion. We have also expanded deep GSOM into an IoT platform that can collate, aggregate, and process an entire network of IoT data streams. We demonstrate the workings of deep GSOM on the real-life scenario of profiling vehicular and pedestrian movement using IoT data streams of two highly urbanized cities. The results of these experiments confirm the validity and effectiveness of the proposed approach for generating situational awareness from multiple IoT data streams.

IoT-Stream: A Lightweight Ontology for Internet of Things Data Streams and Its Use with Data Analytics and Event Detection Services.

With the proliferation of sensors and IoT technologies, stream data are increasingly stored and analysed, but rarely combined, due to the heterogeneity of sources and technologies. Semantics are increasingly used to share sensory data, but not so much for annotating stream data. Semantic models for stream annotation are scarce, as generally, semantics are heavy to process and not ideal for Internet of Things (IoT) environments, where the data are frequently updated. We present a light model to semantically annotate streams, IoT-Stream. It takes advantage of common knowledge sharing of the semantics, but keeping the inferences and queries simple. Furthermore, we present a system architecture to demonstrate the adoption the semantic model, and provide examples of instantiation of the system for different use cases. The system architecture is based on commonly used architectures in the field of IoT, such as web services, microservices and middleware. Our system approach includes the semantic annotations that take place in the pipeline of IoT services and sensory data analytics. It includes modules needed to annotate, consume, and query data annotated with IoT-Stream. In addition to this, we present tools that could be used in conjunction to the IoT-Stream model and facilitate the use of semantics in IoT.

$X-BAND$ : Expiration Band for Anonymizing Varied Data Streams

The Internet of Things (IoT) has formed a whole new layer of the world built on the Internet, reaching every connected device, actuator, and sensor. Many organizations utilize IoT data streams for research and development purposes. To make value out of these data streams, the data handling party must ensure the privacy of the individuals. The most common approach to provide privacy preservation is anonymization. IoT data provide varied data streams due to the nature of the individual's preference and versatile devices pool. The conventional single-tuple expiration-driven sliding window method is not adequate to provide efficient anonymization. Furthermore, the minimization of missingness has to be considered for the varied data stream anonymization. Therefore, we propose the X-BAND algorithm that utilizes the new expiration-band mechanism for handling varied data streams to achieve efficient anonymization, and we introduce weighted distance function for X-BAND to reduce missingness of published data. Our experiment on real data sets shows that X-BAND is effective and efficient compared to the famous conventional anonymization algorithm FADS. X-BAND demonstrated 5%-11% and 1%-3% less information loss on real data sets Adult and PM2.5, respectively, while performing similar on clustering, comparable to reusing suppression and runtime. Also, the new weighted distance function is effective for reducing missingness for anonymization.

A New Pattern Representation Method for Time-Series Data

The rapid growth of Internet of Things (IoT) and sensing technologies has led to an increasing interest in time-series data analysis. In many domains, detecting patterns of IoT data and interpreting these patterns are challenging issues. There are several methods in time-series analysis that deal with issues such as volume and velocity of IoT data streams. However, analysing the content of the data streams and extracting insights from dynamic IoT data is still a challenging task. In this paper, we propose a pattern representation method which represents time-series frames as vectors by first applying Piecewise Aggregate Approximation (PAA) and then applying Lagrangian Multipliers. This method allows representing continuous data as a series of patterns that can be used and processed by various higher-level methods. We introduce a new change point detection method which uses the constructed patterns in its analysis. We evaluate and compare our representation method with Blocks of Eigenvalues Algorithm (BEATS) and Symbolic Aggregate approXimation (SAX) methods to cluster various datasets. We have evaluated our algorithm using UCR time-series datasets and also a healthcare dataset. The evaluation results show significant improvements in analysing time-series data in our proposed method.

Research on Data Sharing Architecture for Ecological Monitoring Using Iot Streaming Data

The rapid development of Internet of Things (IoT) technology and the widespread deployment of various sensors around the world have produced a large number of data streams. Thus, current computing systems face the challenge of quickly receiving and managing these large-scale streaming data. This study builds an efficient distributed database based on Greenplum (GP) and focuses on solving the problem of the low efficiency of structured data queries for observed ecological data collected from fragile areas in Northwest China's desert oasis. First, a distributed database is designed and deployed at the physical storage structure level. A database table structure is then established based on the characteristics of the streaming data. On this basis, the data storage strategy is optimized at the data table level. Additionally, the query efficiency of the distributed database is compared with the query efficiency of traditional standalone databases. The results show that the distributed database significantly improves the data query efficiency. The greater the amount of data stored, the better the improvement in efficiency. Finally, based on the optimized distributed database, we develop a data sharing system for streaming data from ecologically fragile areas in the desert oasis in Northwest China, which provides a new approach for the efficient sharing of massive amounts of IoT streaming data for ecological monitoring. Our storage system is still currently working normally, which is highly important to both data managers and users.

Real-Time Outlier Detection and Bayesian Classification using Incremental Computations for Efficient and Scalable Stream Analytics for IoT for Manufacturing

As the manufacturing industry progresses towards the Internet of Things (IoT) and Cyber-Physical Systems (CPS), current methods of historical data analytics face difficulties in addressing the new challenges which follow Industry 4.0. Industry 4.0 and IoT technologies facilitate the acquisition of ubiquitous data from machine tools and processes. However, these technologies also lead to the generation of a large number of data that are complex to be analyzed. Due to the streaming nature of the IoT systems, however, stream analytics could be used to extract features as the data are generated and published, which can prevent the need to store the data and perform advanced analytics that require high performance computing. This manuscript aims at demonstrating how traditional historical methods can be modified to be used as stream analytics tools for IoT data streams. Since data analytics is a wide domain, this paper has only focused on the two light-weighted methods that have been popular in the industry: Statistical Process Control Chart (SPCC), and Bayesian classification. This paper has defined, tested, and evaluated the accuracy and latency of the novel variation of these methods. It is concluded that by modifying the traditional methods and defining incremental solutions, methods such as Real-Time Dynamic Statistical Process Control Chart (RTDSPCC) and Incremental Gaussian Naïve Bayes (IGNB) can be formed that are highly beneficial for IoT applications as they are highly scalable, require minimal storage, and can update the models in real-time.