Real-time stream processing on IoT data for real-world use cases

Abstract

It is indisputably clear that real-time stream processing of IoT data can result in a series of personal, professional, and social use cases. In this chapter, we have explained how real-time processing of IoT data may lead to a series of real-world premium services and applications across industry verticals including manufacturing.Databases have been an extremely important part of application development, irrespective of how they store data. They follow a paradigm where data is passively stored, waiting for commands from an external part to read or modify the data. Basically, these applications are CRUD based with business logic added on top of a process run by humans through a user interface (UI). A problem with these CRUD style applications is that they commonly lead to an infrastructure with lots of ad hoc solutions using messaging systems, ETL products and other techniques for integrating applications in order to pass data between them. Often code is written for specific integrations, and all this causes a mess of interconnections between applications in an organization.It is better to move away from relying on humans working through a UI to a platform that is able to trigger actions and react on things happening in software. The solution leverages events and event streams. They represent a new paradigm where a system is built to support a flow of data through the business and reacting in real-time to the events occurring. The core idea is that an event stream is a true record of what has happened. Any system or application can read the stream in real time and react on each event. This compares with a central nervous system but for a software defined company, a digital organization needs the software equivalent to a nervous system for connecting all its systems, applications, and processes.For this to work, we have to treat the streams of everything that is happening within an organization as data and enable continuous queries that process this data. In a traditional database, the data sits passively and an application or a user issues queries to retrieve data. In stream processing, this is inverted. Data is an active and continuous stream of events and queries are passive, reacting to and processing the events in the stream as they arrive. Basically, this is a combination of data storage and processing in real time. This is a fundamental change in how applications are built.Other experts however believe that there are benefits to using more types of messages besides events. With only events available, they are also used to indirectly request something to happen in another service. This often increases the coupling between services and can create a very entangled choreography. Therefore, besides events, there is a need for two other types of messages: Commands, which represent an intent to change something, and Queries to fulfill a need for information.