Enabling Distributed Simulations Using Big Data and Clouds

Abstract

Large scientific simulation projects should enable the collaboration of large scientific consortia where members are located in different countries and even continents storing their usually very large data set in different kind of storages. Therefore state-of-the-art simulations should process very large set of data stored in a distributed way in different kind of storages located in all over the world. As the data is big its processing time can be intolerably long. To reduce processing time we have to use large infrastructure that enables the exploitation of parallel processing wherever it is possible in the simulation process. Clouds provide the required large set of computing resources and hence we need simulation environments that enable the easy exploitation of cloud resources. This keynote speech introduces a cloud-oriented simulation platform that enables the exploitation of large cloud resources as well as accessing all the major data storage types. This platform called as WS-PGRADE/gUSE is intensively used in many EU FP7 projects among them in CloudSME where the main target is to enable particularly small and medium-sized manufacturing and engineering companies (SMEs), to use state of the art simulation technology as a Service (SaaS, one-stop-shop, pay-per-use) in the cloud.In this talk we will show the main features of WS-PGRADE/gUSE that enable the use of cloud and large data resources to conduct distributed simulations. First, the workflow creation and execution mechanism will be explained. Then the DCI Bridge service will be shown that enables the exploitation of many independent cloud resources in parallel. Finally, the Data Avenue service that enables the access and transfer of large data among various types of data storages will be described. These services together enable the creation of simulation workflows that are easily portable among different distributed computing and data infrastructures including various types of clouds and cloud storages. At the end of the talk some concrete examples from the CloudSME project (www.cloudsme.eu) will highlight the main advantages of using the platform.