Data Grid Applications Research Articles

Research on Application and Challenges of Large Data in Smart Grid

In recent years, the research work of smart grid has been carried out in all countries in the world. Smart grid has the basic characteristics of information technology, digitization, automation and interaction, its ultimate goal is to become the construction of power system covering the whole production process, including power generation, transmission, substation, distribution, power dispatching and other aspects. It is a security, self-healing, green, reliable system. To achieve safe and reliable operation of smart grid, we must rely on powerful data collection, analysis and processing technology. With the development of Internet technology, especially in recent years, cloud computing, Internet of things and modern communication technology have been rapidly developing to show the arrival of the era of big data. The development of large data technology has injected great vitality into the smart grid, and the convergence of the two is also the trend of the times. This paper introduces the basic concept and characteristics of big data and smart grid, and the key technology of large data acquisition and data fusion technology, protocol technology, data display technology three aspects of big data technology in smart grid application are discussed, and an overview of big data grid applications can face in intellectual challenges.

Accessibility algorithm based on site availability to enhance replica selection in a data grid environment

A data grid functions as a scalable base for grid services to manage data files and their scattered replicas around the world. The principal objective of grid services is to support various data grid applications (jobs) as well as projects. Replica selection is an essential high-level service that selects a Grid location which verifies the shortest response time for the users' jobs among numerous different locations. In the grid environment, estimating response time precisely is not a simple task. Existing replica selection algorithms consume high response time to retrieve replicas because of miss-estimating replicas transfer times. This paper proposes a novel replica selection algorithm that considers site availability in addition to data transfer time. Site availability has not been addressed in previous efforts in the same context this paper does. Site availability is a new factor that can be utilized to estimate response time more accurately. Selecting an unavailable site or selecting a site with insufficient time will likely lead to disconnection. This in turn will require shifting to another site to resume the download or to start the download from scratch depending on the fault tolerance mechanism. Simulation results demonstrate that the performance of the new algorithm is proved to be better than the existing algorithms mentioned in literature.

Data Placement in P2P Data Grids Considering the Availability, Security, Access Performance and Load Balancing

Data dependability is an important issue in data Grids. Replication schemes have been widely used in distributed systems to ensure availability and improve access performance. Alternatively, data partitioning schemes (secret sharing, erasure coding with encryption) can be used to provide availability and, in addition, to offer confidentiality protection. In peer-to-peer data Grids, such confidentiality protection is essential since the nodes hosting the data shares may not be trustworthy or may be compromised. However, difficulties in generating new shares and potential security concerns for share reallocation make a pure data partitioning scheme not easily adaptable to dynamic user access patterns. In this paper, we consider combining replication and data partitioning to assure data availability, confidentiality, load balance, and efficient access for data Grid applications. Data are partitioned and shares are dispersed. The shares may be replicated to achieve better performance, load balance, and availability. Models for assessing confidentiality, availability, load balance, and communication cost are developed and used as the metrics to guide placement decisions. Due to the nature of contradicting goals, we model the placement decision problem as a multi-objective problem and use a genetic algorithm to determine solutions that are approximate to the Pareto optimal placement solutions.

A PTS-PGATS based approach for data-intensive scheduling in data grids

Grid computing is the combination of computer resources in a loosely coupled, heterogeneous, and geographically dispersed environment. Grid data are the data used in grid computing, which consists of large-scale data-intensive applications, producing and consuming huge amounts of data, distributed across a large number of machines. Data grid computing composes sets of independent tasks each of which require massive distributed data sets that may each be replicated on different resources. To reduce the completion time of the application and improve the performance of the grid, appropriate computing resources should be selected to execute the tasks and appropriate storage resources selected to serve the files required by the tasks. So the problem can be broken into two sub-problems: selection of storage resources and assignment of tasks to computing resources. This paper proposes a scheduler, which is broken into three parts that can run in parallel and uses both parallel tabu search and a parallel genetic algorithm. Finally, the proposed algorithm is evaluated by comparing it with other related algorithms, which target minimizing makespan. Simulation results show that the proposed approach can be a good choice for scheduling large data grid applications.

SIMULATED ANNEALING ALGORITHM FOR SCHEDULING DIVISIBLE LOAD IN LARGE SCALE DATA GRIDS

In many data grid applications, data can be decomposed into multiple independent sub datasets and distributed for parallel execution and analysis. This property has been successfully exploited using Divisible Load Theory (DLT). Many Scheduling approaches have been studied but there is no optimal solution. This paper proposes a novel Simulated Annealing (SA) algorithm for scheduling divisible load in large scale data grids. SA algorithm is integrated with DLT model and compared with the previous approaches. Experimental results show that the proposed model obtains better solution in term of makespan.

QoS-awared replica placement techniques in data grid applications

Quality of service is one of the most important factors for successful grid applications. However, individual QoS constraint (IQC) and QoS-awared replica placement have not yet been well addressed in the existing data grid. Existing QoS-awared replica placement algorithms neither can support multiple QoS attributes, nor have good scalability. In this paper, a three-phrase QoS-awared replica placement algorithm based on matrix, namely TP-GABMAC, is presented to solve these problems. Furthermore, a replica ring mechanism is introduced to maintain replica consistency and manage update conflicting. Experiments and analysis demonstrate that the proposed algorithms are not only stable and scalable, but also work well under different network topologies, access patterns and workloads.

Optimal workload allocation model for scheduling divisible data grid applications

In many data grid applications, data can be decomposed into multiple independent sub-datasets and distributed for parallel execution and analysis. This property has been successfully employed using Divisible Load Theory (DLT), which has been proved a powerful tool for modeling divisible load problems in data-intensive grids. There are some scheduling models that have been studied but no optimal solution has been reached due to the heterogeneity of the grids. This paper proposes a new model called the Iterative DLT (IDLT) for scheduling divisible data grid applications. Recursive numerical closed form solutions are derived to find the optimal workload assigned to the processing nodes. Experimental results show that the proposed IDLT model leads to a better solution than other models (almost optimal) in terms of makespan.

Load Allocation Model for Scheduling Divisible Data Grid Applications

Problem statement: In many data grid applications, data can be decomposed into multiple independent sub-datasets and distributed for parallel execution and analysis. Approach: This property had been successfully employed by using Divisible Load Theory (DLT), which had been proved as a powerful tool for modeling divisible load problems in data-intensive grid. Results: There were some scheduling models had been studied but no optimal solution has been reached due to the heterogeneity of the grids. This study proposed a new optimal load allocation based on DLT model recursive numerical closed form solutions are derived to find the optimal workload assigned to the processing nodes. Conclusion/Recommendations: Experimental results showed that the proposed model obtained better solution than other models (almost optimal) in terms of Makespan.

Closed form Solution for Scheduling Arbitrarily Divisible Load Model in Data Grid Applications: Multiple Sources

Closed form Solution for Scheduling Arbitrarily Divisible Load Model in Data Grid Applications: Multiple Sources

Closed form Solution for Scheduling Arbitrarily Divisible Load Model in Data Grid Applications: Multiple Sources

Scheduling an application in data grid was significantly complex and very challenging because of its heterogeneous in nature of the grid system. When the Divisible Load Theory (DLT) model had emerged as a powerful model for modeling data-intensive grid problem, Task Data Present (TDP) model was proposed based on it. This study presented a new Adaptive TDP (ATDP) for scheduling the intensive grid applications. New closed form solution for obtaining the load allocation was derived while computation speeds and communication links are heterogeneous. Experimental results showed that the proposed model can balance the load efficiently.

Response Time Optimization for Replica Selection Service in Data Grids

Problem Statement: Data Grid architecture provides a scalable infrastructure for grid services in order to manage data files and their corresponding replicas that were distributed across the globe. The grid services are designed to support a variety of data grid applications (jobs) and projects. Replica selection is a high-level service that chooses a replica location from among many distributed replicas with the minimum response time for the users' jobs. Estimating the response time accurately in the grid environment is not an easy task. The current systems expose high response time in selecting the required replicas because the response time is estimated by considering the data transfer time only. Approach: We proposed a replica selection system that selects the best replica location for the users' running jobs in a minimum response time that can be estimated by considering new factors besides the data transfer time, namely, the storage access latency and the replica requests that waiting in the storage queue. Results: The performance of the proposed system was compared with a similar system that exists in the literature namely, SimpleOptimiser. The simulation results demonstrated that our system performed better than the SimpleOptimiser on an average of 6%. Conclusions: The proposed system can select the best replica location in a lesser response time than the SimpleOptimise. The efficiency of the proposed system is 6% higher than the SimpleOptimise. The efficiency level has a high impact on the quality of service that is perceived by grid users in a data grid environment where the data files are relatively big. For example, the data files produced from the scientific applications are of the size hundreds of Terabytes.

Network virtualization under user control

AbstractIn this paper, we present the main challenges for offering end‐to‐end virtual network services under user control. This work is motivated by the growing need for end‐to‐end virtual services to support high‐volume data transfer applications such as large data Center, grid applications and storage area networks over multi‐domain networks. The virtualization of network services has to deal with some important issues, such as dynamic provisioning, sharing control, isolation and security. We have developed and implemented a solution called user control Virtual Service that allows the user to deploy and share different virtual services over a multi‐domain network. UCVS is a network‐driven virtual service solution that enables the coordinated provisioning and reuse of physical and virtualized computing, storage and network resources from shared pools. UCVS ensures that applications are dynamically supported throughout multi‐domain infrastructure. UCVS also achieves cross‐technology orchestration, helping further progress towards a service‐oriented infrastructure. Copyright © 2008 John Wiley & Sons, Ltd.

Data management and transfer in high-performance computational grid environments

An emerging class of data-intensive applications involve the geographically dispersed extraction of complex scientific information from very large collections of measured or computed data. Such applications arise, for example, in experimental physics, where the data in question is generated by accelerators, and in simulation science, where the data is generated by supercomputers. So-called Data Grids provide essential infrastructure for such applications, much as the Internet provides essential services for applications such as e-mail and the Web. We describe here two services that we believe are fundamental to any Data Grid: reliable, high-speed transport and replica management. Our high-speed transport service, GridFTP, extends the popular FTP protocol with new features required for Data Grid applications, such as striping and partial file access. Our replica management service integrates a replica catalog with GridFTP transfers to provide for the creation, registration, location, and management of dataset replicas. We present the design of both services and also preliminary performance results. Our implementations exploit security and other services provided by the Globus Toolkit.