Design of an Iterative Method for MapReduce Scheduling Using Deep Reinforcement Learning and Anomaly Detection

Abstract

Due to high complexities within distributed computing environments, there's a critical need for advanced scheduling frameworks that are capable of optimizing MapReduce systems. Current approaches have static policies that limit their capability to adapt to changing system dynamics and workload variations for different cloud scenarios. To overcome these issues, this study introduces a robust MapReduce framework empowered by intelligent scheduling algorithms, tailored to enhance system efficiency and resilience levels. The framework introduces three novel scheduling models: Deep Reinforcement Learning for Dynamic Job Scheduling (DRDJS), Anomaly Detection-driven Adaptive Scheduling (ADAS), and Cluster-based Job Categorization and Scheduling (CJCS). DRDJS utilizes deep reinforcement learning to dynamically generate optimal scheduling policies based on multiple metrics that include historical job data, system metrics, and workload characteristics. This adaptive approach leads to significant reductions in job completion times, outperforming static scheduling methods by up to 30%. Next, ADAS leverages anomaly detection to prioritize critical tasks and efficiently allocate resources in response to anomalies, resulting in a notable improve scheduling speed by up to 40%. Finally, CJCS employs clustering algorithms to categorize jobs based on their resource requirements and execution profiles, enabling more accurate resource allocation and reducing average job completion times by up to 25%. By integrating these methods into a unified scheduling framework, the proposed solution addresses the variability in job characteristics and system performance, thereby enhancing the overall throughput and stability of MapReduce operations.