Workload Characteristics Research Articles

Workload Is Associated with Anxiety and Insomnia Symptoms in an Italian Nationally Representative Sample of Public Health Medical Residents: The PHRASI Cross-Sectional Study.

Mental health disorders pose a substantial challenge for healthcare workers, particularly in the post-COVID-19 era. Public health medical residents (PHRs) played a pivotal role during the pandemic and were significantly affected by the heavy workload. This study aims to uncover potential associations between workload characteristics and symptoms of anxiety and insomnia in Italian PHRs based on data collected in 2022 through the Public Health Residents' Anonymous Survey in Italy (PHRASI) study. A total of 379 residents completed the self-administered questionnaire comprising the Generalized Anxiety Disorder-2 (GAD-2), and the Insomnia Severity Index (ISI). While 36% PHRs reported anxiety symptoms (GAD-2 ≥ 3), 12% reported moderate-to-severe insomnia symptoms (ISI ≥ 15). The multivariate logistic regressions showed that a high work-life interference was associated with the presence of anxiety and insomnia symptoms, while attending two or more simultaneous traineeships was associated with insomnia symptoms. A high workload perceived was positively associated with both the mental health outcomes considered, while the perception of work environment manageability was negatively associated with them. These findings underscore the significant role of the workload in influencing the mental health status of PHRs and emphasize the importance of fostering a supportive work environment that prioritizes mental well-being.

Using Deep Reinforcement Learning (DRL) for minimizing power consumption in Video-on-Demand (VoD) storage systems

As video streaming services such as Netflix become popular, resolving the problem of high power consumption arising from both large data size and high bandwidth in video storage systems has become important. However, because various factors, such as the power characteristics of heterogeneous storage devices, variable workloads, and disk array models, influence storage power consumption, reducing power consumption with deterministic policies is ineffective. To address this, we present a new deep reinforcement learning (DRL)-based file placement algorithm for replication-based video storage systems, which aims to minimize overall storage power consumption. We first model the video storage system with time-varying streaming workloads as the DRL environment, in which the agent aims to find power-efficient file placement. We then propose a proximal policy optimization (PPO) algorithm, consisting of (1) an action space that determines the placement of each file; (2) an observation space that allows the agent to learn a power-efficient placement based on the current I/O bandwidth utilization; (3) a reward model that assigns a greater penalty for increased power consumption for each action; and (4) an action masking model that supports effective learning by preventing agents from selecting unnecessary actions. Extensive simulations were performed to evaluate the proposed scheme under various solid-state disk (SSD) models and replication configurations. Results show that our scheme reduces storage power consumption by 5% to 25.8% (average 12%) compared to existing benchmark methods known to be effective for file placement.

T-Smade: A Two-Stage Smart Detector for Evasive Spectre Attacks Under Various Workloads

Evasive Spectre attacks have used additional nop or memory delay instructions to make effective hardware performance counter based detectors with lower attack detection successful rate. Interestingly, the detection performance gets worse under different workloads. For example, the attack detection successful rate is only 59.8% for realistic applications, while it is much lower 27.52% for memory stress test. Therefore, this paper proposes a two-stage smart detector T-Smade designed for evasive Spectre attacks (e.g., evasive Spectre nop and evasive Spectre memory) under various workloads. T-Smade uses the first-stage detector to identify the type of workloads and then selects the appropriate second-stage detector, which uses four hardware performance counter events to characterize the high cache miss rate and low branch miss rate of Spectre attacks. More importantly, the second stage detector adds one dimension of reusing cache miss rate and branch miss rate to exploit the characteristics of various workloads to detect evasive Spectre attacks effectively. Furthermore, to achieve the good generalization for more unseen evasive Spectre attacks, the proposed classification detector T-Smade is trained by the raw data of Spectre attacks and non-attacks in different workloads using simple Multi-Layer Perception models. The comprehensive results demonstrate that T-Smade makes the average attack detection successful rate of evasive Spectre nop under different workload return from 27.52% to 95.42%, and that of evasive Spectre memory from 59.8% up to 100%.

Study on the Driver Visual Workload in High-Density Interchange-Merging Areas Based on a Field Driving Test.

A visual workload model was constructed to determine and evaluate drivers' visual workload characteristics in high-density interchange-merging areas. Five interchanges were selected, and a real-vehicle driving test was conducted with 47 participants. To address the differences in drivers' visual characteristics in the interchange cluster merging areas, the Criteria Importance Through Intercriteria Correlation (CRITIC) objective weighting method was employed. Six visual parameters were selected to establish a comprehensive evaluation model for the visual workload in high-density interchange-merging areas. The results show that the average scanning frequency and average pupil area change rate are most strongly correlated with the visual workload, whereas the average duration of a single gaze has the lowest weight in the visual workload assessment system. Different driver visual workloads were observed depending on the environment of the interchange-merging areas, and based on these, recommendations are proposed to decrease drivers' workload, thereby increasing road safety.

Energy and time-aware scheduling in diverse virtualized cloud computing environments using optimized self-attention progressive generative adversarial network

ABSTRACT The rapid growth of cloud computing has led to the widespread adoption of heterogeneous virtualized environments, offering scalable and flexible resources to meet diverse user demands. However, the increasing complexity and variability in workload characteristics pose significant challenges in optimizing energy consumption. Many scheduling algorithms have been suggested to address this. Therefore, a self-attention-based progressive generative adversarial network optimized with Dwarf Mongoose algorithm adopted Energy and Deadline Aware Scheduling in heterogeneous virtualized cloud computing (SAPGAN-DMA-DAS-HVCC) is proposed in this paper. Here, a self-attention based progressive generative adversarial network (SAPGAN) is proposed to schedule activities in a cloud environment with an objective function of makespan and energy consumption. Then Dwarf Mongoose algorithm is proposed to optimize the weight parameters of SAPGAN. Outcome of proposed approach SAPGAN-DMA-DAS-HVCC contains 32.77%, 34.83% and 35.76% higher right skewed makespan, 31.52%, 33.28% and 29.14% lower cost when analysed to the existing models, like task scheduling in heterogeneous cloud environment utilizing mean grey wolf optimization approach, energy and performance-efficient task scheduling in heterogeneous virtualized Energy and Performance Efficient Task Scheduling Algorithm, energy and make span aware scheduling of deadline sensitive tasks on the cloud environment, respectively.

DuCFF: A Dual-Channel Feature-Fusion Network for Workload Prediction in a Cloud Infrastructure

Cloud infrastructures are designed to provide highly scalable, pay-as-per-use services to meet the performance requirements of users. The workload prediction of the cloud plays a crucial role in proactive auto-scaling and the dynamic management of resources to move toward fine-grained load balancing and job scheduling due to its ability to estimate upcoming workloads. However, due to users’ diverse usage demands, the changing characteristics of workloads have become more and more complex, including not only short-term irregular fluctuation characteristics but also long-term dynamic variations. This prevents existing workload-prediction methods from fully capturing the above characteristics, leading to degradation of prediction accuracy. To deal with the above problems, this paper proposes a framework based on a dual-channel temporal convolutional network and transformer (referred to as DuCFF) to perform workload prediction. Firstly, DuCFF introduces data preprocessing technology to decouple different components implied by workload data and combine the original workload to form new model inputs. Then, in a parallel manner, DuCFF adopts the temporal convolution network (TCN) channel to capture local irregular fluctuations in workload time series and the transformer channel to capture long-term dynamic variations. Finally, the features extracted from the above two channels are further fused, and workload prediction is achieved. The performance of the proposed DuCFF’s was verified on various workload benchmark datasets (i.e., ClarkNet and Google) and compared to its nine competitors. Experimental results show that the proposed DuCFF can achieve average performance improvements of 65.2%, 70%, 64.37%, and 15%, respectively, in terms of Mean Absolute Error (MAE), Root Mean Square Error (RMSE), Mean Absolute Percentage Error (MAPE) and R-squared (R2) compared to the baseline model CNN-LSTM.

Enhancing the performance of extreme learning machine technique using optimization algorithms for embedded workload characterization

Embedded devices are used in many domains, including healthcare, industries, and home automation, all of which entail significant workloads. As a direct consequence, the embedded devices require retrieval and processing of data of large volume, which occupy large memory space in the embedded devices. Compression along with workload characterization is an effective technique for minimizing memory usage and to the improvement of endurance of memory in such devices. This paper investigates the embedded workload characterization using Extreme Learning Machine (ELM) that is particularly suitable for large-scale datasets and real-time applications. Though ELM is single-layer feedforward network, its input weight randomization has a considerable effect on the accuracy of the classification. In this paper, the authors have proposed a hybrid algorithm for the optimization of the randomization of ELM. The Particle Swarm Optimizer (PSO), the Genetic Algorithm (GA), the Ant Colony Optimizer (ACO), and the Whale Optimization Algorithm (WOA) were used in the optimization of the classification process in ELM to increase the accuracy of the results. The input data must be categorized based on the energy consumed by each workload to proceed with further processing according to the system requirements. This paper explores and analyses the performance of hybridized ELM-Genetic algorithm (ELM-GA), ELM-Particle Swarm Optimization (PSO), ELM-Ant Colony Optimizer (ELM-ACO), and ELM- Whale Optimization Algorithm (ELM-WOA) optimization algorithms. The embedded benchmarks Internet of Medical Things (IoMT), Mi Benchmark (MiBench), and Embedded Microprocessor Benchmark Consortium (EEMBC) have been used as a dataset for classification in this paper. The extensive experimental study shows that the hybridized ELM-WOA provides a 98.5 % classification accuracy, 98.1 % specificity, and 98 % sensitivity compared to the other optimization methods discussed in this paper.

Combining Machine Learning techniques and Genetic Algorithm for predicting run times of High Performance Computing jobs

This study proposes a novel approach combining Machine Learning (ML) techniques and Genetic Algorithms (GA) for predicting High-Performance Computing (HPC) job run times. The objective is to create a prediction method universally applicable to any HPC system, irrespective of workload characteristics, application specific parameters, user behavior, or hardware architecture. Since user-supplied run time estimates are often inaccurate, we aim to categorize job runtimes into several classes, allowing users to select appropriate classes for their jobs. A Genetic Algorithm is developed to optimally define these runtime classes, determining both the number of classes and the time intervals represented by them. Four Machine Learning algorithms (K Nearest Neighbours, Support Vector Regression, Extreme Gradient Boosting and Deep Neural Networks) are implemented for run time prediction. A unique set of features extracted from historical job data serves as input to the Machine Learning models. The generalized nature of our method is demonstrated by validating its performance on data from six clusters with distinct configurations, applications and runtime distributions. Our results illustrate the superior performance of Machine Learning models incorporating GA-defined runtime classes for all datasets. Across all six datasets, our method achieves R2 scores exceeding 0.8, and accuracy greater than 0.7.

Revisiting Database Indexing for Parallel and Accelerated Computing: A Comprehensive Study and Novel Approaches

While the importance of indexing strategies for optimizing query performance in database systems is widely acknowledged, the impact of rapidly evolving hardware architectures on indexing techniques has been an underexplored area. As modern computing systems increasingly leverage parallel processing capabilities, multi-core CPUs, and specialized hardware accelerators, traditional indexing approaches may not fully capitalize on these advancements. This comprehensive experimental study investigates the effects of hardware-conscious indexing strategies tailored for contemporary and emerging hardware platforms. Through rigorous experimentation on a real-world database environment using the industry-standard TPC-H benchmark, this research evaluates the performance implications of indexing techniques specifically designed to exploit parallelism, vectorization, and hardware-accelerated operations. By examining approaches such as cache-conscious B-Tree variants, SIMD-optimized hash indexes, and GPU-accelerated spatial indexing, the study provides valuable insights into the potential performance gains and trade-offs associated with these hardware-aware indexing methods. The findings reveal that hardware-conscious indexing strategies can significantly outperform their traditional counterparts, particularly in data-intensive workloads and large-scale database deployments. Our experiments show improvements ranging from 32.4% to 48.6% in query execution time, depending on the specific technique and hardware configuration. However, the study also highlights the complexity of implementing and tuning these techniques, as they often require intricate code optimizations and a deep understanding of the underlying hardware architecture. Additionally, this research explores the potential of machine learning-based indexing approaches, including reinforcement learning for index selection and neural network-based index advisors. While these techniques show promise, with performance improvements of up to 48.6% in certain scenarios, their effectiveness varies across different query types and data distributions. By offering a comprehensive analysis and practical recommendations, this research contributes to the ongoing pursuit of database performance optimization in the era of heterogeneous computing. The findings inform database administrators, developers, and system architects on effective indexing practices tailored for modern hardware, while also paving the way for future research into adaptive indexing techniques that can dynamically leverage hardware capabilities based on workload characteristics and resource availability.

Optimizing Database Performance in Complex Event Processing through Indexing Strategies

Complex event processing (CEP) systems have gained significant importance in various domains, such as finance, logistics, and security, where the real-time analysis of event streams is crucial. However, as the volume and complexity of event data continue to grow, optimizing the performance of CEP systems becomes a critical challenge. This paper investigates the impact of indexing strategies on the performance of databases handling complex event processing. We propose a novel indexing technique, called Hierarchical Temporal Indexing (HTI), specifically designed for the efficient processing of complex event queries. HTI leverages the temporal nature of event data and employs a multi-level indexing approach to optimize query execution. By combining temporal indexing with spatial- and attribute-based indexing, HTI aims to accelerate the retrieval and processing of relevant events, thereby improving overall query performance. In this study, we evaluate the effectiveness of HTI by implementing complex event queries on various CEP systems with different indexing strategies. We conduct a comprehensive performance analysis, measuring the query execution times and resource utilization (CPU, memory, etc.), and analyzing the execution plans and query optimization techniques employed by each system. Our experimental results demonstrate that the proposed HTI indexing strategy outperforms traditional indexing approaches, particularly for complex event queries involving temporal constraints and multi-dimensional event attributes. We provide insights into the strengths and weaknesses of each indexing strategy, identifying the factors that influence performance, such as data volume, query complexity, and event characteristics. Furthermore, we discuss the implications of our findings for the design and optimization of CEP systems, offering recommendations for indexing strategy selection based on the specific requirements and workload characteristics. Finally, we outline the potential limitations of our study and suggest future research directions in this domain.

Design of an Iterative Method for MapReduce Scheduling Using Deep Reinforcement Learning and Anomaly Detection

Due to high complexities within distributed computing environments, there's a critical need for advanced scheduling frameworks that are capable of optimizing MapReduce systems. Current approaches have static policies that limit their capability to adapt to changing system dynamics and workload variations for different cloud scenarios. To overcome these issues, this study introduces a robust MapReduce framework empowered by intelligent scheduling algorithms, tailored to enhance system efficiency and resilience levels. The framework introduces three novel scheduling models: Deep Reinforcement Learning for Dynamic Job Scheduling (DRDJS), Anomaly Detection-driven Adaptive Scheduling (ADAS), and Cluster-based Job Categorization and Scheduling (CJCS). DRDJS utilizes deep reinforcement learning to dynamically generate optimal scheduling policies based on multiple metrics that include historical job data, system metrics, and workload characteristics. This adaptive approach leads to significant reductions in job completion times, outperforming static scheduling methods by up to 30%. Next, ADAS leverages anomaly detection to prioritize critical tasks and efficiently allocate resources in response to anomalies, resulting in a notable improve scheduling speed by up to 40%. Finally, CJCS employs clustering algorithms to categorize jobs based on their resource requirements and execution profiles, enabling more accurate resource allocation and reducing average job completion times by up to 25%. By integrating these methods into a unified scheduling framework, the proposed solution addresses the variability in job characteristics and system performance, thereby enhancing the overall throughput and stability of MapReduce operations.

Resource demands in telco data centers

Telecommunication (telco) cloud services have emerged as crucial components in the modern digital landscape, offering extensive capabilities for data management, connectivity, and service provision. However, research on telco clouds lacks comprehensive data on the characteristics of production workloads, which is fundamental for designing effective resource management systems, such as workload schedulers and power management mechanisms. To this end, this paper addresses a substantial gap in telco cloud research by creating a comprehensive dataset that encapsulates crucial information regarding the pattern demands of applications within telco data centers. In addition, the proposed dataset contributes to the field by enabling strategic network configuration, optimizing data center sizing, facilitating proactive decision-making for data center operations, but its applicability extends beyond these cases. These examples illustrate the practical value of the dataset in enhancing efficiency, reducing operational costs, and ensuring optimal performance within telecommunication data centers.

SpChar: Characterizing the sparse puzzle via decision trees

Sparse matrix computation is crucial in various modern applications, including large-scale graph analytics, deep learning, and recommender systems. The performance of sparse kernels varies greatly depending on the structure of the input matrix, making it difficult to gain a comprehensive understanding of sparse computation and its relationship to inputs, algorithms, and target machine architecture. Despite extensive research on certain sparse kernels, such as Sparse Matrix-Vector Multiplication (SpMV), the overall family of sparse algorithms has yet to be investigated as a whole. This paper introduces SpChar, a workload characterization methodology for general sparse computation. SpChar employs tree-based models to identify the most relevant hardware and input characteristics, starting from hardware and input-related metrics gathered from Performance Monitoring Counters (PMCs) and matrices. Our analysis enables the creation of a characterization loop that facilitates the optimization of sparse computation by mapping the impact of architectural features to inputs and algorithmic choices. We apply SpChar to more than 600 matrices from the SuiteSparse Matrix collection and three state-of-the-art Arm Central Processing Units (CPUs) to determine the critical hardware and software characteristics that affect sparse computation. In our analysis, we determine that the biggest limiting factors for high-performance sparse computation are (1) the latency of the memory system, (2) the pipeline flush overhead resulting from branch misprediction, and (3) the poor reuse of cached elements. Additionally, we propose software and hardware optimizations that designers can implement to create a platform suitable for sparse computation. We then investigate these optimizations using the gem5 simulator to achieve a significant speedup of up to 2.63× compared to a CPU where the optimizations are not applied.

Evaluating the potential of disaggregated memory systems for HPC applications

SummaryDisaggregated memory is a promising approach that addresses the limitations of traditional memory architectures by enabling memory to be decoupled from compute nodes and shared across a data center. Cloud platforms have deployed such systems to improve overall system memory utilization, but performance can vary across workloads. High‐performance computing (HPC) is crucial in scientific and engineering applications, where HPC machines also face the issue of underutilized memory. As a result, improving system memory utilization while understanding workload performance is essential for HPC operators. Therefore, learning the potential of a disaggregated memory system before deployment is a critical step. This paper proposes a methodology for exploring the design space of a disaggregated memory system. It incorporates key metrics that affect performance on disaggregated memory systems: memory capacity, local and remote memory access ratio, injection bandwidth, and bisection bandwidth, providing an intuitive approach to guide machine configurations based on technology trends and workload characteristics. We apply our methodology to analyze thirteen diverse workloads, including AI training, data analysis, genomics, protein, fusion, atomic nuclei, and traditional HPC bookends. Our methodology demonstrates the ability to comprehend the potential and pitfalls of a disaggregated memory system and provides motivation for machine configurations. Our results show that eleven of our thirteen applications can leverage injection bandwidth disaggregated memory without affecting performance, while one pays a rack bisection bandwidth penalty and two pay the system‐wide bisection bandwidth penalty. In addition, we also show that intra‐rack memory disaggregation would meet the application's memory requirement and provide enough remote memory bandwidth.

Efficient Resource Provisioning Approach Using Cloud Computing Environment

Abstract: Cloud computing offers scalability, reliability, and cost-effectiveness, driving its widespread adoption. However, efficient resource management remains a challenge due to virtualization and the pay-as-you-go model (CPU, memory, network, storage, etc.). This paper addresses this by analyzing various job scheduling algorithms for task allocation and execution in the cloud. We propose a set of scheduling rules that aim to minimize completion time, cost, and resource utilization while evaluating their execution time. CloudSim is employed to analyze the overall effectiveness of the proposed scheduling techniques. Furthermore, this study investigates the result of workload characteristics and application requirements on the effectiveness of different scheduling algorithms. By understanding these factors, system designers are able to make deft choices to optimize resource allocation and enhance cloud service efficiency

Cerberus: Triple Mode Acceleration of Sparse Matrix and Vector Multiplication

The multiplication of sparse matrix and vector (SpMV) is one of the most widely used kernels in high-performance computing as well as machine learning acceleration for sparse neural networks. The design space of SpMV accelerators has two axes: algorithm and matrix representation. There have been two widely used algorithms and data representations. Two algorithms, scalar multiplication and dot product, can be combined with two sparse data representations, compressed sparse and bitmap formats for the matrix and vector. Although the prior accelerators adopted one of the possible designs, it is yet to be investigated which design is the best one across different hardware resources and workload characteristics. This paper first investigates the impact of design choices with respect to the algorithm and data representation. Our evaluation shows that no single design always outperforms the others across different workloads, but the two best designs (i.e., compressed sparse format and bitmap format with dot product) have complementary performance with trade-offs incurred by the matrix characteristics. Based on the analysis, this study proposes Cerberus, a triple-mode accelerator supporting two sparse operation modes in addition to the base dense mode. To allow such multi-mode operation, it proposes a prediction model based on matrix characteristics under a given hardware configuration, which statically selects the best mode for a given sparse matrix with its dimension and density information. Our experimental results show that Cerberus provides 12.1× performance improvements from a dense-only accelerator, and 1.5× improvements from a fixed best SpMV design.

Investigation of the Ergonomics and Workload of Farm Women during in Planting and Weeding Operations

The study was done to determine the anthropometric, physical, physiological, and workload characteristics of women operators (N=7) during manual sowing and weeding activities for several medicinal plants. The planting and weeding activities were done using a shovel and a hand hoe. Anthropometric measurements were taken for women aged 20 to 50. The physiological fluctuations in chosen participants' energy and strength during different agricultural operations, as well as the implications in body part discomfort score, were evaluated. The average resting and working heart rates, EER, OCR, and BPDS are all measured to assess physiological characteristics. Female responders reported an average resting heart rate of 78.91 bpm for weeding and 81.46 bpm for planting. Planting and weeding had an average working heart rate of 100.85 and 97.14 (bpm, respectively). The average EER and OCR values for the relevant procedures were 6.35 and 5.31 (kJ/min) and 0.52 and 0.45 (l/min), respectively. The BPDS levels of the chosen participants were 19.5 and 21.4, respectively. The data imply that the workers were more compatible and comfortable when weeding using a hand hoe.

Exploring memory synchronization and performance considerations for FPGA platform using the high-abstracted OpenCL framework: Benchmarks development and analysis.

A key benefit of the Open Computing Language (OpenCL) software framework is its capability to operate across diverse architectures. Field programmable gate arrays (FPGAs) are a high-speed computing architecture used for computation acceleration. This study investigates the impact of memory access time on overall performance in general FPGA computing environments through the creation of eight benchmarks within the OpenCL framework. The developed benchmarks capture a range of memory access behaviors, and they play a crucial role in assessing the performance of spinning and sleeping on FPGA-based architectures. The results obtained guide the formulation of new implementations and contribute to defining an abstraction of FPGAs. This abstraction is then utilized to create tailored implementations of primitives that are well-suited for this platform. While other research endeavors concentrate on creating benchmarks with the Compute Unified Device Architecture (CUDA) to scrutinize the memory systems across diverse GPU architectures and propose recommendations for future generations of GPU computation platforms, this study delves into the memory system analysis for the broader FPGA computing platform. It achieves this by employing the highly abstracted OpenCL framework, exploring various data workload characteristics, and experimentally delineating the appropriate implementation of primitives that can seamlessly integrate into a design tailored for the FPGA computing platform. Additionally, the results underscore the efficacy of employing a task-parallel model to mitigate the need for high-cost synchronization mechanisms in designs constructed on general FPGA computing platforms.

Position-specific workload of professional rugby union players during tactical periodization training.

The positional workload characteristics in rugby union on three acquisition days (i.e. strength, endurance, and speed days) of tactical periodization are still relatively unknown. Therefore, the primary aim of this study was to shed light on the positional external workload variables (10 Hz Global Positioning System and accelerometer microtechnology) and internal workload indicators (the session rating of perceived exertion) of players in a professional rugby union team by utilizing and comparing two tactical periodization models. Twenty-six male players (15 forwards and 11 backs) were recruited from a French second-division rugby club. Data were obtained over 10 weeks of in-season home games: a total of 780 observations were analyzed. Student's t-test observed different external workload profiles between positions among acquisition days. Mean external workload values, except PlayerLoadslow, were significantly higher (p≤0.01; effect size: 0.41-1.93) for backs than forwards for all acquisition days. Moreover, forwards perceived a higher internal workload than backs on the strength day of both models. The findings demonstrate that applying these two tactical periodization models could result in effective rugby union training. Validating external and internal workload characteristics on tactical periodization acquisition days enables extensive analysis of training load monitoring data; these data can be utilized to discover the unique characteristics of each position and design position-specific acquisition days to improve performance.

A novel framework for generic Spark workload characterization and similar pattern recognition using machine learning

Comprehensive workload characterization plays a pivotal role in comprehending Spark applications, as it enables the analysis of diverse aspects and behaviors. This understanding is indispensable for devising downstream tuning objectives, such as performance improvement. To address this pivotal issue, our work introduces a novel and scalable framework for generic Spark workload characterization, complemented by consistent geometric measurements. The presented approach aims to build robust workload descriptors by profiling only quantitative metrics at the application task-level, in a non-intrusive manner. We expand our framework for downstream workload pattern recognition by incorporating unsupervised machine learning techniques: clustering algorithms and feature selection. These techniques significantly improve the process of grouping similar workloads without relying on predefined labels. We effectively recognize 24 representative Spark workloads from diverse domains, including SQL, machine learning, web search, graph, and micro-benchmarks, available in HiBench. Our framework achieves a high accuracy F-Measure score of up to 90.9% and a Normalized Mutual Information of up to 94.5% in similar workload pattern recognition. These scores significantly outperform the results obtained in a comparative analysis with an established workload characterization approach in the literature.