An Efficient Distributed Data Processing Method for Smart Environment

Abstract

Background/Objectives: In current times, huge volume of data at a very high velocity gets generated through social media and various sensors in embedded systems that are connected to the Internet which causes a very Big data problem. These challenging Big data’s need to be processed and stored by traditional Relational Database Management Systems (RDBMS). Due to this reason the need for new software solutions has emerged for managing the Big data in an efficient, scalable and smart way. Methods/Statistical Analysis: In this study, an approach to combine the concept of batch processing and stream processing to an end where we can query the data set which also supports Adhoc Querying with less latency, that can be run on any Large scale Machine Learning Algorithms for recognizing any interest pattern in the streaming data set was employed. The functionalities of Hadoop ecosystem’s tool HIVE can also be used to produce the results to Adhoc queries, User Defined Functions (UDF) similar to writing a SQL Stored Procedures in the Spark System. An interface with SerDes which is Serialization and De-serialization that helps us to talk to the standard stream where we can exactly query the dataset are employed. Findings: : By proposing a new software solution AllJoyn Lambda, in which AllJoyn is integrated in the lambda architecture and the prototype implementation of the architecture is done using Apache Hadoop Yarn over Apache Spark Streaming are presented . This study illuminates the high velocity streaming data set on a database without losing any data from the streaming domain, to support Adhoc Querying from the data set and to provide a mechanism for fast data processing and analytics using Large Scale Machine Learning. This paper highlights the analysis of large scale dataset processing, handling challenges, and its comprehensive systematic review. Applications/Improvements: From this study, we conclude that, building a smart environment by using the big data setup platform improves and enhances the results for the smart environment.