ML WPStreamCloud: ML-based Workload Prediction and Task Clustering for Efficient Stream Application Ofoading in Heterogeneous Edge and Cloud Environments

Abstract

As the demand for real-time stream processing grows in cloud and edge environments, the ever-changing nature of stream applications (SA) and the unpredictability of resource requirements make it essential to analyze, comprehend, and predict both user-submitted tasks and computing resources within the given distributed and heterogeneous infrastructures. Thus, optimizing resource utilization becomes paramount. This paper presents a novel method, ML_WPStreamCloud, leveraging stream workload analysis, stream task clustering, and resource classification to enhance SA offloading in edge and cloud environments. Aiming to offload tasks to suitable resources Efficiently, ML WPStreamCloud profiles SA workloads based on task profiling using the k-means algorithm. We evaluate the clustering results of state-of-the-art unsupervised clustering algorithms (e.g., K-means, MeanShift, Agglomerative) and find that k-means is more closely aligned with actual task labels, demonstrating superior performance. It outperforms other algorithms with higher silhouette scores (≈ 0.34), lower Davies-Bouldin index (≈ 0.95), and higher Calinski-Harabasz index (≈ 2511), respectively. The proposed method ML WPS treamCloud adeptly overcomes the challenges posed by diverse workloads and resource environments, offering a promising solution for the efficient scheduling of streaming applications. The simulation results with various performance parameters show the effectiveness of ML WPS treamCloud over baseline algorithms.