Multi-cluster Environment Research Articles

A network application approach towards 5G and beyond critical communications use cases

Low latency and high bandwidth heralded with 5G networks will allow transmission of large amounts of Mission-Critical data over a short time period. 5G hence unlocks several capabilities for novel Public Protection and Disaster Relief (PPDR) applications, developed to support first responders in making faster and more accurate decisions during times of crisis. As various research initiatives are giving shape to the Network Application ecosystem as an interaction layer between vertical applications and the network control plane, in this article we explore how this concept can unlock finer network service management capabilities that can be leveraged by PPDR solution developers. In particular, we elaborate on the role of Network Applications as means for developers to assure prioritization of specific emergency flows of data, such as high-definition video transmission from PPDR field users to remote operators. To demonstrate this potential in future PPDR-over-5G services, we delve into the transfer of network-intensive PPDR solutions to the Network Application model. We then explore novelties in Network Application experimentation platforms, aiming to streamline development and deployment of such integrated systems across existing 5G infrastructures, by providing the reliability and multi-cluster environments they require.

Energy hardware and workload aware job scheduling towards interconnected HPC environments

New HPC machines are getting close to the exascale. Power consumption for those machines has been increasing, and researchers are studying ways to reduce it. A second trend is HPC machines' growing complexity, with increasing heterogeneous hardware components and different clusters architectures cooperating in the same machine. We refer to these environments with the term heterogeneous multi-cluster environments. With the aim of optimizing performance and energy consumption in these environments, this paper proposes an Energy-Aware-Multi-Cluster (EAMC) job scheduling policy. EAMC-policy is able to optimize the scheduling and placement of jobs by predicting performance and energy consumption of arriving jobs for different hardware architectures and processor frequencies, reducing workload's energy consumption, makespan, and response time. The policy assigns a different priority to each job-resource combination so that the most efficient ones are favored, while less efficient ones are still considered on a variable degree, reducing response time and increasing cluster utilization. We implemented EAMC-policy in Slurm, and we evaluated a scenario in which two CPU clusters collaborate in the same machine. Simulations of workloads running applications modeled from real-world show a reduction of response time and makespan by up to 25% and 6% while saving up to 20% of total energy consumed when compared to policies minimizing runtime, and by 49%, 26%, and 6% compared to policies minimizing energy.

Efficient Dynamic Resource Allocation in Hadoop Multiclusters for Load- Balancing Problem

Background: ‘Map-Reduce’ is the framework and its processing of data by rationalizing the distributed servers. also its running the various tasks in parallel way. The most important problem in map reduce environment is Resource Allocation in distributed environments and data locality to its corresponding slave nodes. If the applications are not scheduled properly then it leads to load unbalancing problems in the cloud environments. Objective: This Research suggests a new dynamic way of allocating the resources in hadoop multi cluster environment in order to effectively monitor the nodes for faster computation, load balancing and data locality issues. The dynamic slot allocation is explained theoretically in order to address the problems of pre configuration, speculative execution, delay in scheduling and pre slot allocation in hadoop environments. Experiment is done with Hadoop cluster which increases the efficiency of the nodes and solves the load balancing problem. Methods: The Current design of Map Reduce Hadoop systems are affected by underutilization of slots. The reason is the number of maps and reducer is allotted is smaller than the available number of maps and reducers. In Hadoop most of the times its noticed that dynamic slot allocation policy, the mapper or reducers are idle. So we can use those unused map tasks to overloaded reducer tasks in-order to increase the efficiency of MR jobs and vice versa. Results: The real time experiment was implemented with Multinode Hadoop cluster map reduce jobs of file size 1giga bytes to 5 giga bytes and various performance test has been taken. Conclusion: This paper focused on Hadoop map reduce resource allocation management techniques for multi cluster environments. It proposes a novel dynamic slot allocation policy to improve the performance of yarn scheduler and eliminates the load balancing problem. This work proves that dynamic slot allocation is outperforms more than yarn framework. In future it considered to concentrate on CPU bandwidth and processing time.

Resource scheduling for piano teaching system of internet of things based on mobile edge computing

The effective operation of the piano teaching system of the Internet of Things requires the effective support of virtualization technology. In particular, on the basis that the edge computing standards and systems are not yet mature, the resource scheduling problem of edge computing needs to be studied from the actual point of view. In order to improve the effective operation of the piano teaching system of Internet of Things, this study analyzes the resource scheduling of delay-sensitive applications, sets the resource scheduling mode based on the space–time difference of the edge container load in a multi-cluster environment, and proposes a cross-cluster scheduling strategy. Simultaneously, this study uses simulation experiments to analyze the performance of the strategy proposed in this paper. The research results show that the strategy proposed in this paper can perform delay-insensitive application scheduling during system operation, achieve multi-cluster collaborative scheduling goals, and make the load between clusters more balanced.

HYBRID BAT AND CUCKOO BASED META-HEURISTIC ALGORITHM FOR PARALLEL JOB SCHEDULING IN MULTI-CLUSTER ENVIRONMENT

Scheduling in multi cluster environment is NP Hard but yet it is a fundamental to satisfying and improving the performance of parallel processing system. In this paper the parallel job scheduling problem is being addressed in multicluster computing environment using hybrid BAT+Cuckoo Algorithm. Scheduling policy is considered in which BAT algorithm is used to allocate the available resources and Cuckoo algorithm is used to look for constraint resources. Constraint resources may be located within distinct clusters.A mating pool in which resources from distinct clusters are maintained. In case resources are limited, Cuckoo algorithm look for available resources from each cluster and combine them within the mating pool. Parameters considered for evaluation includes Makespan and Flowtime which are substantially reduce using the proposed literature.

Parallel Job Scheduling Using Grey Wolf Optimization Algorithm for Heterogeneous Multi-Cluster Environment

Parallel Job Scheduling Using Grey Wolf Optimization Algorithm for Heterogeneous Multi-Cluster Environment

User-Centric Overlapped Clustering Based on Anchor-Based Precoding in Cellular Networks

Coordinated multipoint (CoMP) is a promising technique for improving the performance of cell edge users, where a number of base stations (BSs) form a cluster to jointly process signals to the users within the cluster. Based on nonoverlapped clustering, dynamic clustering and intercluster co-ordination were proposed to mitigate interference to users at cluster edges. However, to further enhance the performance of cluster-edge users, user-centric overlapped clustering is more effective in providing multiuser MIMO and intercluster CoMP from multiple BSs belonging to different clusters. In this letter, on the basis of an anchor-based precoding scheme, an optimal user-centric clustering scheme is introduced together with a suboptimal clustering algorithm. The throughput performance is evaluated and compared with conventional nonoverlapped clustering in multicluster environments.

Scheduling online mixed-parallel workflows of rigid tasks in heterogeneous multi-cluster environments

Workflow scheduling on parallel systems has long been known to be a NP-complete problem. As modern grid and cloud computing platforms emerge, it becomes indispensable to schedule mixed-parallel workflows in an online manner in a speed-heterogeneous multi-cluster environment. However, most existing scheduling algorithms were not developed for online mixed-parallel workflows of rigid data-parallel tasks and multi-cluster environments, therefore they cannot handle the problem efficiently. In this paper, we propose a scheduling framework, named Mixed-Parallel Online Workflow Scheduling (MOWS), which divides the entire scheduling process into four phases: task prioritizing, waiting queue scheduling, task rearrangement, and task allocation. Based on this framework, we developed four new methods: shortest-workflow-first, priority-based backfilling, preemptive task execution and All-EFT task allocation, for scheduling online mixed-parallel workflows of rigid tasks in speed-heterogeneous multi-cluster environments. To evaluate the proposed scheduling methods, we conducted a series of simulation studies and made comparisons with previously proposed approaches in the literature. The experimental results indicate that each of the four proposed methods outperforms existing approaches significantly and all these approaches in MOWS together can achieve more than 20% performance improvement in terms of average turnaround time.

Multi-criteria genetic algorithm applied to scheduling in multi-cluster environments

Scheduling and resource allocation to optimize performance criteria in multi-cluster heterogeneous environments is known as an NP-hard problem, not only for the resource heterogeneity, but also for the possibility of applying co-allocation to take advantage of idle resources across clusters. A common practice is to use basic heuristics to attempt to optimize some performance criteria by treating the jobs in the waiting queue individually. More recent works proposed new optimization strategies based on Linear Programming techniques dealing with the scheduling of multiple jobs simultaneously. However, the time cost of these techniques makes them impractical for large-scale environments. Population-based meta-heuristics have proved their effectiveness for finding the optimal schedules in large-scale distributed environments with high resource diversification and large numbers of jobs in the batches. The algorithm proposed in the present work packages the jobs in the batch to obtain better optimization opportunities. It includes a multi-objective function to optimize not only the Makespan of the batches but also the Flowtime, thus ensuring a certain level of QoS from the users’ point of view. The algorithm also incorporates heterogeneity and bandwidth awareness issues, and is useful for scheduling jobs in large-scale heterogeneous environments. The proposed meta-heuristic was evaluated with a real workload trace. The results show the effectiveness of the proposed method, providing solutions that improve the performance with respect to other well-known techniques in the literature.

Multiple job co-allocation strategy for heterogeneous multi-cluster systems based on linear programming

Multi-cluster environments are composed of multiple clusters of computers that act collaboratively, and thus allowing computational problems to be treated that require more resources than those available in a single cluster. However, the degree of complexity of the scheduling process is greatly increased by the heterogeneity of resources and co-allocation process, which distributes the tasks of parallel jobs across cluster boundaries.

Minimizing synchronization in IDR (s )

IDR (s) is a family of fast algorithms for iteratively solving large nonsymmetric linear systems. With cluster computing and in particular with Grid computing, the inner product is a bottleneck operation. In this paper, three techniques are investigated for alleviating this bottleneck. First, a recently proposed IDR (s) algorithm that is highly efficient and stable is reformulated in such a way that it has a single global synchronization point per iteration step. Second, the so-called test matrix is chosen so that the work, communication, and storage involving this matrix is minimized in multi-cluster environments. Finally, a methodology is presented for a-priori estimation of the optimal value of s using only problem and machine-based parameters. Numerical experiments applied to a 3D convection–diffusion problem are performed on the DAS-3 Grid computer, demonstrating the effectiveness of our approach. Copyright © 2011 John Wiley & Sons, Ltd.

An Ontological-Fuzzy Approach to Advance Reservation in Multi-Cluster Grids

Advance reservation is an important mechanism for a successful utilization of available resources in distributed multi-cluster environments. This mechanism allows, for example, a user to provide parameters aiming to satisfy requirements related to applications' execution time and temporal dependence. This predictability can lead the system to reach higher levels of QoS. However, the support for advance reservation has been restricted due to the complexity of large scale configurations and also dynamic changes verified in these systems. In this research work it is proposed an advance reservation method, based on a ontology-fuzzy approach. It allows a user to reserve a wide variety of resources and enable large jobs to be reserved among different nodes. In addition, it dynamically verifies the possibility of reservation with the local RMS, avoiding future allocation conflicts. Experimental results of the proposal, through simulation, indicate that the proposed mechanism reached a successful level of flexibility for large jobs and more appropriated distribution of resources in a distributed multi-cluster configuration.

Scheduling DAG-based applications in multicluster environments with background workload using task duplication

Grids or multicluster computing environments are becoming increasingly popular to both scientific and commercial applications. Process scheduling remains a central issue to be effectively resolved in order to exploit the full potential that the grid or multicluster environment can offer. We use a directed acyclic graph (DAG) to model a process or an application where the nodes of the DAG represent the tasks of the process. Prior to the execution of a process in a multicluster environment, the tasks are required to be mapped onto the clusters. In this article, it is shown that the algorithm developed by He et al. [L. He, S.A. Jarvis, D.P. Spooner, D. Bacigalupo, G. Tan, and G.R. Nudd, Mapping DAG-based applications to multiclusters with background workload, Proceedings of the 2005 IEEE International Symposium on Cluster Computing and the Grid, Cardiff, 2005, pp 855–862.] for the multicluster DAG mapping problem can be significantly improved by incorporating the task duplication strategy. The proposed process scheduling algorithm has a time complexity O(| V|2(r+d+1)), where |V| represents the number of tasks; r, the number of clusters; and d, the maximum in-degree of tasks.

Scheduling DAG-based applications in multicluster environments with background workload using task duplication

Grids or multicluster computing environments are becoming increasingly popular to both scientific and commercial applications. Process scheduling remains a central issue to be effectively resolved ...

Guest Editorial: Special Section on Algorithm Design and Scheduling Techniques (Realistic Platform Models) for Heterogeneous Clusters

THE last decade has seen a dramatic increase in the deployment of heterogeneous distributed computing platforms, in particular, those consisting of heterogeneous clusters, and multiple heterogeneous collections of clusters aggregated over wide-area networks into grids. The software infrastructures and mechanisms to deploy such platforms have been well studied and implementations are already used in production, so that heterogeneous platforms represent a significant, and growing, fraction of the computational power delivered by parallel platforms today. In spite of these successes, many research challenges remain, including those pertaining to distributed algorithms and scheduling algorithms, which are critical for ensuring that these platforms are used effectively. In this context, the goal of this special section on “Algorithm Design and Scheduling Techniques (Realistic Platform Models) for Heterogeneous Clusters” is to gather papers that further our understanding of the impact of platform heterogeneity on the design and evaluation of new such algorithms. In the paper entitled “Allocating Non-Real-Time and Soft Real-Time Jobs in Multiclusters,” Ligang He, Stephen A. Jarvis, Daniel P. Spooner, Hong Jiang, Donna N. Dillenberger, and Graham R. Nudd introduce two workload allocation strategies for large-scale heterogeneous platforms. The first strategy achieves an optimized mean response time for jobs having no real-time requirements. The second strategy obtains an optimized mean miss rate for jobs having soft real-time requirements (i.e., a fraction of jobs are permitted to miss the real-time constraints). Both strategies take into account average system behaviors (such as the mean arrival rate of jobs) to calculate the workload proportions for individual clusters, and update on-the-fly the workload allocation when the change in the mean arrival rate reaches a certain threshold. The allocation schemes are combined with two job dispatching strategies (weighted random and weighted round-robin) to generate new job scheduling algorithms for multicluster environments. In their paper “On the Distribution of Sequential Jobs in Random Brokering for Heterogeneous Computational Grids,” Vandy Berten, Joel Goossens, and Emmanuel Jeannot study resource brokering for scheduling sequential jobs onto a grid platform that consists of heterogeneous sets of homogeneous processors, such as a set of clusters. Resources in each cluster are managed by a local scheduler that maintains a job queue. The paper studies a centralized “metascheduler” that uses a randomized strategy to share available resources among competing jobs. This research considers two cases depending on whether the platform is heavily loaded or lightly loaded. For each case, it obtains both analytical and experimental characterizations of the queue lengths at each local scheduler, CPU utilization, and average job slowdowns. Furthermore, the paper presents a discussion of the system’s behavior when it transitions between a heavily loaded state and a lightly loaded one. All presented theoretical results are corroborated by simulations and provide a thorough description of randomized resource brokering. The research in “Multiple Job Scheduling in a Connection-Limited Data Parallel System” presents a new method for scheduling jobs in a distributed system where the critical resource is the bandwidth to access the stored data. The authors, Alessandro Amoroso and Keith Marzullo, describe an approach that supports the master-worker scheme and can be applied to data parallel computation. They consider a typical wide-area data grid that is comprised of a set of sites, where each site has one or more local area networks. The platform model used is based on the Nile data grid. This paper uses a set of synthetic jobs to compare three schedulers: Greedy, Maxfow, and Hybrid. They tested their new approach under various circumstances and measured its performance by means of several metrics. The new Hybrid scheduler is never worse than either of the other two schedulers, and in 20 percent of the simulated runs, it produced runs that were at least 20 percent better. The paper entitled “Capacity-Aware Multicast Algorithms on Heterogeneous Overlay Networks,” coauthored by Zhan Zhang, Shigang Chen, Yibei Ling, and Randy Chow, addresses the problem of multicast for group IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. 17, NO. 2, FEBRUARY 2006 97