Large-scale Grid Environments Research Articles

Greedy Firefly Algorithm for Optimizing Job Scheduling in IoT Grid Computing.

The Internet of Things (IoT) is defined as interconnected digital and mechanical devices with intelligent and interactive data transmission features over a defined network. The ability of the IoT to collect, analyze and mine data into information and knowledge motivates the integration of IoT with grid and cloud computing. New job scheduling techniques are crucial for the effective integration and management of IoT with grid computing as they provide optimal computational solutions. The computational grid is a modern technology that enables distributed computing to take advantage of a organization’s resources in order to handle complex computational problems. However, the scheduling process is considered an NP-hard problem due to the heterogeneity of resources and management systems in the IoT grid. This paper proposed a Greedy Firefly Algorithm (GFA) for jobs scheduling in the grid environment. In the proposed greedy firefly algorithm, a greedy method is utilized as a local search mechanism to enhance the rate of convergence and efficiency of schedules produced by the standard firefly algorithm. Several experiments were conducted using the GridSim toolkit to evaluate the proposed greedy firefly algorithm’s performance. The study measured several sizes of real grid computing workload traces, starting with lightweight traces with only 500 jobs, then typical with 3000 to 7000 jobs, and finally heavy load containing 8000 to 10,000 jobs. The experiment results revealed that the greedy firefly algorithm could insignificantly reduce the makespan makespan and execution times of the IoT grid scheduling process as compared to other evaluated scheduling methods. Furthermore, the proposed greedy firefly algorithm converges on large search spacefaster , making it suitable for large-scale IoT grid environments.

Indoor Emergency Path Planning Based on the Q-Learning Optimization Algorithm

The internal structure of buildings is becoming increasingly complex. Providing a scientific and reasonable evacuation route for trapped persons in a complex indoor environment is important for reducing casualties and property losses. In emergency and disaster relief environments, indoor path planning has great uncertainty and higher safety requirements. Q-learning is a value-based reinforcement learning algorithm that can complete path planning tasks through autonomous learning without establishing mathematical models and environmental maps. Therefore, we propose an indoor emergency path planning method based on the Q-learning optimization algorithm. First, a grid environment model is established. The discount rate of the exploration factor is used to optimize the Q-learning algorithm, and the exploration factor in the ε-greedy strategy is dynamically adjusted before selecting random actions to accelerate the convergence of the Q-learning algorithm in a large-scale grid environment. An indoor emergency path planning experiment based on the Q-learning optimization algorithm was carried out using simulated data and real indoor environment data. The proposed Q-learning optimization algorithm basically converges after 500 iterative learning rounds, which is nearly 2000 rounds higher than the convergence rate of the Q-learning algorithm. The SASRA algorithm has no obvious convergence trend in 5000 iterations of learning. The results show that the proposed Q-learning optimization algorithm is superior to the SARSA algorithm and the classic Q-learning algorithm in terms of solving time and convergence speed when planning the shortest path in a grid environment. The convergence speed of the proposed Q- learning optimization algorithm is approximately five times faster than that of the classic Q- learning algorithm. The proposed Q-learning optimization algorithm in the grid environment can successfully plan the shortest path to avoid obstacle areas in a short time.

Attack-Resilient Optimal PMU Placement via Reinforcement Learning Guided Tree Search in Smart Grids

The operation of smart grids heavily relies on secure and accurate meter measurements provided by phasor measurement units (PMUs). Therefore, the optimal PMU placement (OPP) aiming to achieve the complete system observability of smart grids with as few PMUs as possible has been extensively investigated. Although many existing studies have focused on the OPP, few of them are concerned with the placement order of PMUs. To protect as many buses as possible in smart grids when installing PMUs in stages owing to high cost, this paper proposes the attack-resilient OPP strategy which places PMUs in order by using reinforcement learning guided tree search, where the sequential decision making of reinforcement learning is utilized to explore placement orders. The least-effort attack model is carried out to screen vulnerable buses such that the buses adjacent to these buses can be placed PMUs in advance to reduce the state space and action space of the large-scale smart grid environment. Based on that, the reinforcement learning guided tree search approach is used to explore the key buses which need placing PMUs, where the repeated exploration of the agent is avoided by tree search. Then, a reasonable placement order of PMUs is obtained according to the action sequence the proposed method provides. Finally, the effectiveness of the proposed method is verified on various IEEE standard test systems and the comparison results with existing methods are provided.

A self-adaptive structuring for large-scale P2P Grid environment: design and simulation analysis

Resource management is a key issue for large-scale grid environments. In particular, the resource discovery mechanism used highly impacts on the efficiency of resource sharing and cooperative computing. Meanwhile, the increasing size of resources and users in large-scale distributed systems has lead to a scalability problem. The self-organising, fault tolerance and decentralised nature of P2P technology, which helps to reduce the management cost of grid infrastructure, is a good basis for resolving both aforementioned problems. In this context, we propose a self-adaptive structuring model for large-scale P2P grid environments. The proposed specification, called P2P4GS, is generic, i.e. not linked to a particular P2P architecture. Our structuring approach is completely distributed, and requires only local knowledge about neighbouring nodes to implement node virtual community. Indeed, given a node's connection and based on its neighbours, we dynamically create virtual communities or clusters. A particular node called ISP (Information System Proxy) acts as service directory within each cluster. In addition, to provide an efficient lookup mechanism in our system, we proposed to build a spanning tree constituted by the set of ISPs. An experimental validation, through simulation, shows that our approach ensures a high scalability in terms of clusters distribution and communication cost.

A self-adaptive structuring for large-scale P2P Grid environment: design and simulation analysis

A self-adaptive structuring for large-scale P2P Grid environment: design and simulation analysis

GAMESH: A grid architecture for scalable monitoring and enhanced dependable job scheduling

Grid computing is a largely adopted paradigm to federate geographically distributed data centers. Due to their size and complexity, grid systems are often affected by failures that may hinder the correct and timely execution of jobs, thus causing a non-negligible waste of computing resources. Despite the relevance of the problem, state-of-the-art management solutions for grid systems usually neglect the identification and handling of failures at runtime. Among the primary goals to be considered, we claim the need for novel approaches capable to achieve the objectives of scalable integration with efficient monitoring solutions and of fitting large and geographically distributed systems, where dynamic and configurable tradeoffs between overhead and targeted granularity are necessary. This paper proposes GAMESH, a Grid Architecture for scalable Monitoring and Enhanced dependable job ScHeduling. GAMESH is conceived as a completely distributed and highly efficient management infrastructure, concentrating on two crucial aspects for large-scale and multi-domain grid environments: (i) the scalable dissemination of monitoring data and (ii) the troubleshooting of job execution failures. GAMESH has been implemented and tested in a real deployment encompassing geographically distributed data centers across Europe. Experimental results show that GAMESH (i) enables the collection of measurements of both computing resources and conditions of task scheduling at geographically sparse sites, while imposing a limited overhead on the entire infrastructure, and (ii) provides a failure-aware scheduler able to improve the overall system performance, even in the presence of failures, by coordinating local job schedulers at multiple domains.

Relaxed Dijkstra and A* with linear complexity for robot path planning problems in large-scale grid environments

Although there exist efficient methods to determine an optimal path in a graph, such as Dijkstra and A* algorithms, large instances of the path planning problem need more adequate and efficient techniques to obtain solutions in reasonable time. We propose two new time-linear relaxed versions of Dijkstra (RD) and A* (RA*) algorithms to solve the global path planning problem in large grid environments. The core idea consists in exploiting the grid-map structure to establish an accurate approximation of the optimal path, without visiting any cell more than once. We conducted extensive simulations (1290 runs on 43 maps of various types) for the proposed algorithms, both in four-neighbor and eight-neighbor grid environments, and compared them against original Dijkstra and A* algorithms with different heuristics. We demonstrate that our relaxed versions exhibit a substantial gain in terms of computational time (more than 3 times faster in average), and in most of tested problems an optimal solution (in at least 97 % of cases for RD and 82 % for RA*) or a very close one is reached (at most 9 % of extra length, and less than 2 % in average). Besides, the simulations also show that RA* provides a better trade-off between solution quality and execution time than previous bounded relaxations of A* that exist in the literature.

Performability Evaluation of Grid Environments Using Stochastic Reward Nets

In this paper, performance of grid computing environment is studied in the presence of failure-repair of the resources. To achieve this, in the first step, each of the grid resource is individually modeled using Stochastic Reward Nets (SRNs), and mean response time of the resource for grid tasks is computed as a performance measure. In individual models, three different scheduling schemes called random selection, non-preemptive priority, and preemptive priority are considered to simultaneously schedule local and grid tasks to the processors of a single resource. In the next step, single resource models are combined to shape an entire grid environment. Since the number of the resources in a large-scale grid environment is more than can be handled using such a monolithic SRN, two approximate SRN models using folding and fixed-point techniques are proposed to evaluate the performance of the whole grid environment. Brouwer’s fixed-point theorem is used to theoretically prove the existence of a solution to the fixed-point approximate model. Numerical results indicate an improvement of several orders of magnitude in the model state space reduction without a significant loss of accuracy.

Network and data location aware approach for simultaneous job scheduling and data replication in large-scale data grid environments

Data Grid integrates graphically distributed resources for solving data intensive scientific applications. Effective scheduling in Grid can reduce the amount of data transferred among nodes by submitting a job to a node, where most of the requested data files are available. Scheduling is a traditional problem in parallel and distributed system. However, due to special issues and goals of Grid, traditional approach is not effective in this environment any more. Therefore, it is necessary to propose methods specialized for this kind of parallel and distributed system. Another solution is to use a data replication strategy to create multiple copies of files and store them in convenient locations to shorten file access times. To utilize the above two concepts, in this paper we develop a job scheduling policy, called hierarchical job scheduling strategy (HJSS), and a dynamic data replication strategy, called advanced dynamic hierarchical replication strategy (ADHRS), to improve the data access efficiencies in a hierarchical Data Grid. HJSS uses hierarchical scheduling to reduce the search time for an appropriate computing node. It considers network characteristics, number of jobs waiting in queue, file locations, and disk read speed of storage drive at data sources. Moreover, due to the limited storage capacity, a good replica replacement algorithm is needed. We present a novel replacement strategy which deletes files in two steps when free space is not enough for the new replica: first, it deletes those files with minimum time for transferring. Second, if space is still insufficient then it considers the last time the replica was requested, number of access, size of replica and file transfer time. The simulation results show that our proposed algorithm has better performance in comparison with other algorithms in terms of job execution time, number of intercommunications, number of replications, hit ratio, computing resource usage and storage usage.

C-HLRD: a category-based hierarchical localisation technique for resource discovery in grid environments

Grid resource discovery which is the mechanism of indexing and locating the resources affects significantly the efficiency of grid systems. Some specific characteristics of grid environments such as dynamicity and large-scale make the resource discovery a challenging task. Different approaches for grid resource discovery have been proposed by researchers in recent years. This paper proposes a resource discovery technique highly adapted to large-scale grid environments with the aim of decreasing the message load and response time. It uses a category-based hierarchical structure to index and locate the resources. By this way, the resource query is localised rapidly and efficiently. The indexing structure and processes are implemented in GridSim simulation environment. The results of experiments showed significant decrease in the response time and message load during the search process especially in large scales.

A distributed resource discovery algorithm for P2P grids

Centralized or hierarchical administration of the classical grid resource discovery approaches is unable to efficiently manage the highly dynamic large-scale grid environments. Peer-to-peer (P2P) overlay represents a dynamic, scalable, and decentralized prospect of the grids. Structured P2P methods do not fully support the multi-attribute range queries and unstructured P2P resource discovery methods suffer from the network-wide broadcast storm problem. In this paper, a decentralized learning automata-based resource discovery algorithm is proposed for large-scale P2P grids. The proposed method supports the multi-attribute range queries and forwards the resource queries through the shortest path ending at the grid peers more likely having the requested resource. Several simulation experiments are conducted to show the efficiency of the proposed algorithm. Numerical results reveal the superiority of the proposed model over the other methods in terms of the average hop count, average hit ratio, and control message overhead.

Grid resource management policies for load-balancing and energy-saving by vacation queuing theory

The resource management is the central component of grid system. The analysis of the workload log file of LCG including the job arrival and the resource utilization daily cycle shows that the idle sites in the Grid are the source of load imbalance and energy waste. Here we focus on these two issues: balancing the workload by transferring jobs to idle sites at prime time to minimize the response time and maximize the resource utilization; power management by switch the idle sites to sleeping mode at non-prime time to minimize the energy consume. We form the M/G/1 queue model with server vacations, startup and closedown to analysis the performance metrics to instruct the design of load-balancing and energy-saving policies. We provide our Adaptive Receiver Initiated (ARI) load-balancing strategy and power-management policy for energy-saving. The simulation experiments prove the accuracy of our analysis and the comparisons results indicate our policies are largely suitable for large-scale heterogeneous grid environment.

Optical grid synergy with peer-to-peer

As grid computing continues to gain popularity in the industry and research community, it also attracts more attention from the customer level. The large number of users and high frequency of job requests in the consumer market make this scenario a challenge for grid computing. Clearly, all the current client/server(C/S)-based architectures will become unfeasible for supporting large-scale grid applications due to their poor scalability and poor fault-tolerance. In order to address this issue, a novel C/S and peer-to-peer hybrid architecture to realise a highly scalable and flexible platform for optical grid is proposed here. The anycast algorithms, which are used to search suitable resources for job requests in this hybrid architecture, are also investigated in detail. Simulation and experimental results show that the proposed architecture is suitable and efficient for grid applications, and it outperforms C/S-based architecture in a large-scale grid environment.

Introducing Virtual Private Overlay Network Services in Large Scale Grid Infrastructures

The computational Grid concept is gaining great popularity as the best way to deliver access to a wide range of distributed computing and data resources. But, as Grids move from an experimental phase to real production and their deployment in the Internet significantly increases, controlling the security of a Grid application becomes imperative. The most significant Grid security issue is that the different sites composing the Grid will generally be managed by different organizations each with their own security mechanisms and policies. This makes any communication security arrangement on the entities participating to the Grid generally more difficult than if they were on the same LAN. In this paper, we propose a novel network resource abstraction for delivering dynamic on-demand Virtual Private Overlay connection services, into large-scale Grid environments. Such facility provides to Grid applications an illusion of dedicated layer-2 LAN connections that are fully comparable to a private network in performance, reliability, security and Quality of Service, but also provide topology and control plane virtualization to ensure better isolation also at the protocol and address space level. It may represent a technological breakthrough that can transform the overall connection paradigm in modern Grids, by reducing infrastructure costs, with the elimination of private circuits and long-distance direct connections, and increasing network coverage and flexibility by leveraging the Internet usage. As a proof of concept, the proposed facility has been implemented in a Grid Information Service prototype which was successfully tested on a small dedicated testbed infrastructure.

An End-to-end Workflow Pipeline for Large-scale Grid Computing

In this paper we describe a service-based, software architecture that enables end-to-end, high-level workflow processing in a Grid environment consisting of many heterogeneous resources. Our architecture is essentially a pipeline that extends from the abstract application specification phase to the deployment and execution stages through to returning the results to the user. We envision a large-scale Grid environment that contains heterogeneous resources. Our architecture caters for flexible deployment, performance, reliability and charging for resource usage. These are addressed at the specification level as well as at the realisation (brokering) and execution levels. The proposed architecture is derived from previous work in LeSC that has produced the ICENI pipeline, and our experience with e-Science projects, such as GENIE, e-Protein and RealityGrid from which we derive a set of key requirements.

Self-adaptive task allocation and scheduling of meta-tasks in non-dedicated heterogeneous computing

The efforts to construct a national scale grid computing environment have brought unprecedented computing capacity and complexity. Exploiting this complex infrastructure requires efficient middleware to support the execution of distributed applications, which presents the challenge of how to schedule tasks in shared heterogeneous systems. Most existing scheduling systems are based on predetermined estimation of task completion time and resources availability. They may not provide appropriate scheduling if the underlying computing resources present an abnormal usage pattern during an application execution. For solving long-running applications in a large-scale grid environment, abnormal usage of some resource may not be uncommon. We have proposed the development of the Grid Harvest Service (GHS) performance evaluation and task scheduling system in our previous work. In this study, we present a novel dynamic self-adaptive scheduling algorithm and its implementation under GHS. Scheduling and rescheduling algorithms and mechanisms are carefully investigated. Experimental results show that, equipped with these new scheduling mechanisms, GHS outperforms existing systems considerably in scheduling large applications in a non-dedicated heterogeneous environment.