An enhanced distributed framework for real-time performance testing of large scale IoT dataset using big data analytic tools

Abstract

<p>The demand for analyzing enormous IoT datasets is rising in parallel with the popularity of the IoT. There are considerable obstacles to effective processing and analysis due to the amount, velocity, and variety of IoT data. In this research, we present a distributed system that makes use of big data analytic tools like Apache Hive, Spark, and Hadoop to efficiently test the performance of massive IoT datasets. The framework addresses the lack of a comprehensive solution by providing a scalable and fault-tolerant architecture. We discuss the motivation behind real-time performance testing in the context of big data analytics for IoT datasets and highlight the need for a distributed framework. A literature review is conducted to explore existing performance testing frameworks, big data analytic tools, and approaches for performance testing big data analytics. The proposed framework’s key components, including dataset generation, test scenario specification, cluster configuration, performance metrics collection, analysis and visualization modules, and implementation details, including tool choices, are discussed. An experimental evaluation is conducted to validate the framework’s performance, and it is suggested to incorporate blockchain technology. Overall, the proposed framework offers a comprehensive solution for real-time performance testing of large-scale IoT datasets, providing organizations and researchers with a valuable tool to ensure efficient and reliable IoT data processing and analysis.</p>