Grid Workloads Research Articles

Energy-efficient cooperative resource allocation and task scheduling for Internet of Things environments

Offloading Internet of Things (IoT) tasks to the cloud for further processing might not always lead to an optimal execution time, particularly in situations such as resource contention, under-provisioning, over-provisioning, and fragmentation. In addition, dynamically optimizing the number of Virtual Machines (VMs) for resource scheduling in order to meet application requirements remains a major research challenge. Further, existing resource scheduling algorithms focus primarily on minimizing operational costs while maximizing resource sharing and utilization. Considering energy utilization as part of the resource allocation and scheduling process as an optimization objective for maintaining load balancing has often been neglected. To address these challenges and more, we propose a cooperative energy-aware resource allocation and scheduling strategy based on a Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS) multi-criteria decision-making method. We used the Grid Workloads Archive dataset to evaluate our proposed TOPSIS-based Resource Allocation (TOPREAL) approach. Experimental results with respect to the allocation of VM resources when considering processing a large segment of tasks indicate that TOPREAL outperforms existing algorithms in terms of energy savings, with an average improvement of 40.25%, while maintaining an average improvement of 16.21% when it comes to execution time. Results also demonstrate that our method can save an average of 78.06 processing hours and 63,215 kJ of energy when compared to existing scheduling algorithms. These results demonstrate the effectiveness of our proposed model and the viability of using multi-criteria decision-making techniques such as TOPSIS to solve the resource allocation and scheduling problem in edge environments.

Consolidating the interactive analysis and Grid infrastructure at DESY

Within WLCG, the DESY site in Hamburg is one of the largest Tier-2 sites with about 18500 CPU cores for Grid workloads. Additionally, about 8000 CPU cores are available for interactive user analyses in the National Analysis Factory [NAF]. After migrating these two batch systems onto a common HTCondor based set-up during the previous four years, we recapitulate the lessons learned during the transition especially since both use cases differ in their workloads. For Grid jobs start-up latencies are negligible and the primary focus is on an optimal utilization of the resources. Complementary, users of the NAF expect a high responsiveness of the batch system as well as the storage for interactive analyses. In this document, we will also give an outlook to future developments and concepts for the DESY high-throughput computing. In the ongoing evolution of the HTC batch system, we are exploring how to integrate anonymous jobs with the batch system as back-end for Function-as-a-Service workflows as well as an option for dynamic expansions to remote computing resources.

Baseline Parametric Survey on Causes of Traffic Congestion in Datacenter Networks

Distributed cloud datacenters are currently used as deployment environment for large-scale web-facing enterprise analytic cloud applications. In the era of cloud computing, these networks allow unprecedented scalability in response to dynamic and unpredictable grid workloads. However, grid applications need to scale beyond the limits of single-tenanted datacenter to multi-tenanted datacenter sites providing regulatory service level compliance, efficient Quality of Service (QoS) and increased fault tolerance at all times. Following the complex hierarchical design structure of emerging cloud networks such as Internet of Things (IoT) and Fog Computing Networks (FCNs), congestion crisis becomes a critical consideration for future expansion of these networks. This paper carried out a Baseline Parametric Survey (BPS) to ascertain major factors of DCN traffic congestion in selected production networks in Nigeria. From the survey, it was observed that the absence of an intelligent load balancer, bandwidth size, network traffic type, and network switch butter capacity, number of users on the network, segmentation and routing devices all contribute to network congestion incidence. The deductions from this work can be used to implement a Fog internet layer for smarter data off-loading in today’s internet of everything.

Query Mechanism for File Correlation in Cooperative Replica Prefetching

副本协作预取是一种网格副本复制策略,旨在通过获取隐性高价值文件进一步降低数据访问延迟.副本协作预取的关键在于隐性高价值文件的确定和获取,因此,提高隐性高价值文件获取的速度能够大幅度提升副本协作预取的性能.利用DHT(distributed hash table)组织网格节点,以快速定位隐性高价值文件查询所必需的文件相关性信息.针对隐性高价值文件的特殊查询模式,提出基于预取规则树的文件相关性信息存储结构及其查询机制,以提高查询效率.基于真实网格作业运行记录的实验,验证了所提出方法的有效性和高效性.;A cooperative replica prefetching mechanism is a new replication strategy aiming to reduce the data access latency in grid by the means of replicating implicit high-value files (IHVFs). The key of this new mechanism is the exploration and location of IHVFs. Locating the IHVFs quickly can improve the performance of the mechanism. This paper employs the DHT (distributed hash table) technology to organize grid nodes to quickly locate the file correlation information which is essential for the query of IHVFs. Considering the specific query pattern, a PTree-based storage structure and corresponding query mechanism for file correlation information are proposed to improve the IHVF query efficiency. Simulation results based on real grid workloads have confirmed the effectiveness and high efficiency of the proposed methods.

Market_based grid resource allocation using new negotiation model

This paper presents a new negotiation model for designing Market- and Behavior-driven Negotiation Agents (MBDNAs) that address computational grid resource allocation problem. To determine the amount of concession for each trading cycle, the MBDNAs are guided by six factors: (1) number of negotiator's trading partners, (2) number of negotiator's competitors, (3) negotiator's time preference, (4) flexibility in negotiator's trading partner's proposal, (5) negotiator's proposal deviation from the average of its trading partners’ proposals, and (6) previous concession behavior of negotiator's trading partner. In our experiments, we compare grid resource consumer (GRC) of type MBDNAs (respectively grid resource owner (GRO) of type MBDNAs) with MDAs (Market Driven Agents) in terms of the following metrics: total tasks complementation and average utility (respectively resource utilization level and average utility). The results show that by taking the proposed factors into account, MBDNAs of both types make a more efficient concession amount than MDAs and are, therefore, considered an appropriate mechanism for grid resource allocation in different grid workloads and market types.

Grid Computing Workloads

In the mid 1990s, the grid computing community promised the "compute power grid," a utility computing infrastructure for scientists and engineers. Since then, a variety of grids have been built worldwide, for academic purposes, specific application domains, and general production work. Understanding grid workloads is important for the design and tuning of future grid resource managers and applications, especially in the recent wake of commercial grids and clouds. This article presents an overview of the most important characteristics of grid workloads in the past seven years (2003-2010). Although grid user populations range from tens to hundreds of individuals, a few users dominate each grid's workload both in terms of consumed resources and the number of jobs submitted to the system. Real grid workloads include very few parallel jobs but many independent single-machine jobs (tasks) grouped into single "bags of tasks."

Scheduling algorithm on grid resources in consideration of local workloads

Performances of two resource scheduling algorithms on grid resources with and without stochastic local workloads were studied. A stochastic local workload model of grid resources was established. Two scheduling algorithms on grid resources, Highest CPU-Rating Available Resource First (HRARF) and Most Suitable CPU-Number Available Resource First (MSNARF), were proposed. Makespans of grid workloads scheduled by the two proposed resource scheduling algorithms with and without stochastic local workloads were simulated. The simulation results show that when the loads of resources are heavy, the relative performance of MSNARF algorithm and HRARF algorithm on grid resources with and without stochastic local workload is reverse. In grid computing, relative performance of two scheduling algorithms on sharing resources and exclusive resources may be different.

Model-based simulation and performance evaluation of grid scheduling strategies

Simulation studies of Grid scheduling strategies require representative workloads to produce dependable results. Real production Grid workloads have shown diverse correlation structures and scaling behavior, which are different to the characteristics of the available supercomputer workloads and cannot be captured by Poisson or simple distribution-based models. We present statistical models that are able to reproduce various autocorrelation structures, including pseudo-periodicity and long range dependence. By conducting model-based simulation, we quantitatively evaluate the performance impacts of workload autocorrelations in Grid scheduling. The results indicate that autocorrelations result in system performance degradation, both at the local and the Grid level. It is shown that realistic workload modeling is not only possible, but also necessary to enable dependable Grid scheduling studies.

The Grid Workloads Archive

While large grids are currently supporting the work of thousands of scientists, very little is known about their actual use. Due to strict organizational permissions, there are few or no traces of grid workloads available to the grid researcher and practitioner. To address this problem, in this work we present the Grid Workloads Archive (GWA), which is at the same time a workload data exchange and a meeting point for the grid community. We define the requirements for building a workload archive, and describe the approach taken to meet these requirements with the GWA. We introduce a format for sharing grid workload information, and tools associated with this format. Using these tools, we collect and analyze data from nine well-known grid environments, with a total content of more than 2000 users submitting more than 7 million jobs over a period of over 13 operational years, and with working environments spanning over 130 sites comprising 10 000 resources. We show evidence that grid workloads are very different from those encountered in other large-scale environments, and in particular from the workloads of parallel production environments: they comprise almost exclusively single-node jobs, and jobs arrive in “bags-of-tasks”. Finally, we present the immediate applications of the GWA and of its content in several critical grid research and practical areas: research in grid resource management, and grid design, operation, and maintenance.

Workload dynamics on clusters and grids

This paper presents a comprehensive statistical analysis of a variety of workloads collected on production clusters and Grids. The applications are mostly computational-intensive and each task requires single CPU for processing data, which dominate the workloads on current production Grid systems. Trace data obtained on a parallel supercomputer is also included for comparison studies. The statistical properties of workloads are investigated at different levels, including the Virtual Organization (VO) and user behavior. The aggregation procedure and scaling analysis are applied to job arrivals, leading to the identifications of several basic patterns, namely pseudo-periodicity, long range dependence (LRD), and multifractals. It is shown that statistical measures based on interarrivals are of limited usefulness and count based measures should be trusted when it comes to correlations. Other job characteristics like run time and memory consumption are also studied. A “bag-of-tasks” behavior is empirically evidenced, strongly indicating temporal locality. The nature of such dynamics in the Grid workloads is discussed. This study has important implications on workload modeling and performance predictions, and points out the need of comprehensive performance evaluation studies given the workload characteristics.

Optorsim: A Grid Simulator for Studying Dynamic Data Replication Strategies

Computational grids process large, computationally intensive problems on small data sets. In contrast, data grids process large computational problems that in turn require evaluating, mining and producing large amounts of data. Replication, creating geographically disparate identical copies of data, is regarded as one of the major optimization techniques for reducing data access costs. In this paper, several replication algorithms are discussed. These algorithms were studied using the Grid simulator: OptorSim. OptorSim provides a modular framework within which optimization strategies can be studied under different Grid configurations. The goal is to explore the stability and transient behaviour of selected optimization techniques. We detail the design and implementation of OptorSim and analyze various replication algorithms based on different Grid workloads.