Data-intensive Workflow Applications Research Articles

Research on secure workload execution scheme in heterogeneous cloud environment

<span lang="EN-US">The increasing demand for the hardware, software and infrastructure is playing a big role in the information technology domain towards the need of customer’s specific requirements. Cloud computing is a major backbone for providing such services over the internet. It includes the services such as applications, storage, network, scalability, sharing, virtualization, confidentiality, security, authentication, and integrity. A large number of data intensive workflow applications uses heterogeneous cloud environment for communication and computation operation. An intruder/attacker will utilize these environments for their benefit by flooding malicious links, unwanted information and others. In cloud environment, detecting a malicious device/packet during workflow execution is a critical and challenging task. The various workflow method with security, service level agreement (SLA) and quality of service (QoS) have been modelled in recent time; However, these models are not efficient in detecting malicious users and maintaining high level of QoS or workflow applications. This article focus is on addressing research future direction, issues and challenges of work in meeting secure and efficient workflow execution model for heterogeneous cloud environment.</span>

Multi-Cloud Performance and Security Driven Federated Workflow Management

Federated multi-cloud resource allocation for data-intensive application workflows is generally performed based on performance or quality of service (i.e., QSpecs) considerations. At the same time, end-to-end security requirements of these workflows across multiple domains are considered as an afterthought due to lack of standardized formalization methods. Consequently, diverse/heterogenous domain resource and security policies cause inter-conflicts between application's security and performance requirements that lead to sub-optimal resource allocations. In this paper, we present a joint performance and security-driven federated resource allocation scheme for data-intensive scientific applications. In order to aid joint resource brokering among multi-cloud domains with diverse/heterogenous security postures, we first define and characterize a data-intensive application's security specifications (i.e., SSpecs). Then we describe an alignment technique inspired by Portunes Algebra to homogenize the various domain resource policies (i.e., RSpecs) along an application's workflow lifecycle stages. Using such formalization and alignment, we propose a near optimal cost-aware joint QSpecs-SSpecs-driven, RSpecs-compliant resource allocation algorithm for multi-cloud computing resource domain/location selection as well as network path selection. We implement our security formalization, alignment, and allocation scheme as a framework, viz., “OnTimeURB” and validate it in a multi-cloud environment with exemplar data-intensive application workflows involving distributed computing and remote instrumentation use cases with different performance and security requirements.

A novel data aware task clustering mechanism for scientific workflow applications in cloud

Scientific applications modelled as directed acyclic graphs (DAGs) are composed of complex calculations and a large amount of data transfer. It is very difficult to execute these applications on traditional distributed computing platforms. For such applications, cloud provides a reliable solution due to its unique characteristics, task clustering is performed which combines two or more tasks into a single executable unit. Task clustering can help to reduce the system overheads such as queue delay, engine delay and so on. Existing clustering algorithms in this domain focus more on computational granularity of the tasks without considering the data dependency among the tasks. In this paper, a data aware clustering algorithm has been proposed which combines the tasks depending on the size of data transferred between interdependent tasks. Experiments were conducted to compare the proposed clustering algorithm with the existing baseline and balanced clustering algorithms and it was observed that proposed algorithm gave better makespan and cost for data intensive workflow applications.

A novel data aware task clustering mechanism for scientific workflow applications in cloud

Scientific applications modelled as directed acyclic graphs (DAGs) are composed of complex calculations and a large amount of data transfer. It is very difficult to execute these applications on traditional distributed computing platforms. For such applications, cloud provides a reliable solution due to its unique characteristics, task clustering is performed which combines two or more tasks into a single executable unit. Task clustering can help to reduce the system overheads such as queue delay, engine delay and so on. Existing clustering algorithms in this domain focus more on computational granularity of the tasks without considering the data dependency among the tasks. In this paper, a data aware clustering algorithm has been proposed which combines the tasks depending on the size of data transferred between interdependent tasks. Experiments were conducted to compare the proposed clustering algorithm with the existing baseline and balanced clustering algorithms and it was observed that proposed algorithm gave better makespan and cost for data intensive workflow applications.

Supporting Data-Intensive Workflows in Software-Defined Federated Multi-Clouds

Cloud computing is emerging as a viable platform for scientific exploration. Elastic and on-demand access to resources (and other services), the abstraction of “unlimited” resources, and attractive pricing models provide incentives for scientists to move their workflows into clouds. Generalizing these concepts beyond a single virtualized datacenter, it is possible to create federated marketplaces where different types of resources (e.g., clouds, HPC grids, supercomputers) that may be geographically distributed, are collectively exposed as a single elastic infrastructure. This presents opportunities for optimizing the execution of application workflows with heterogeneous and dynamic requirements, and tackling larger scale problems. In this paper, we introduce a framework to manage the end-to-end execution of data-intensive application workflows in dynamic software-defined resource federation. This framework enables the autonomic execution of workflows by elastically provisioning an appropriate set of resources that meet application requirements, and by adapting this set of resources at runtime as the requirements change. It also allows users to customize scheduling policies that drive the way resources federated and used. To demonstrate the benefits of our approach, we study the execution of two different data-intensive scientific workflows in a multi-cloud federation using different policies and objective functions.

Exploring energy-performance-quality tradeoffs for scientific workflows with in-situ data analyses

Power and energy are critical concerns for high performance computing systems from multiple perspectives, including cost, reliability/resilience and sustainability. At the same time, data locality and the cost of data movement have become dominating concerns in scientific workflows. One potential solution for reducing data movement costs is to use a data analysis pipeline based on in-situ data analysis.However, the energy-performance-quality tradeoffs impact of current optimizations and their overheads can be very hard to assess and understand at the application level.In this paper, we focus on exploring performance and power/energy tradeoffs of different data movement strategies and how to balance these tradeoffs with quality of solution and data speculation. Our experimental evaluation provides an empirical evaluation of different system and application configurations that give insights into the energy-performance-quality tradeoffs space for in-situ data-intensive application workflows. The key contribution of this work is a better understanding of the interactions between different computation, data movement, energy, and quality-of-result optimizations from a power-performance perspective, and a basis for modeling and exploiting these interactions.

Workflow Scheduling Algorithm Based on Reliance Group in Cloud Environments

Cloud computing has attracted more people's attention and been extensive research. In order to provide satisfying service for users during the implement process of data-intensive workflow application in cloud environments, an improved algorithm is proposed which based on reliance group with high degree of dependence. By this measure, users can obtain the ideal acceleration radio of cost and service performance. Furthermore, more comprehensive resource service is provided by service providers.

Optimising chemical information workflows: integrating Reaxys - use cases and applications

Reaxys is an authoritative web-based workflow solution designed for chemistry researchers in drug discovery, chemicals and academic research [1]. Reaxys supports research and fuels discovery by integrating searches for reaction and substance data with synthesis planning and chemical sourcing. With historical coverage dating right back to 1771 and continuing through todays cutting-edge research, Reaxys covers the most important chemistry-related literature and patent sources and provides unparalleled depth of information on reactions, substances and related property data. To make the most effective use of this information Reaxys offers two new entry points into the data: • Chemical space analysis, analogue sourcing, fuelling hit-to-lead programs and IP prior-art searching are possible through the Reaxys Structure Flat File. • Programmatic interaction with the database by custom clients, downloading bulk data from Reaxys using scripts, executing batch queries and integrating Reaxys into server-based data-intensive pipelining or workflow applications are possible through the Reaxys APIs. The Reaxys Structure Flat File and the Reaxys APIs will facilitate chemoinformaticians and information managers to integrate Reaxys into chemical information processing environments. In our poster, we present common use cases and possible applications. In addition, we describe successful implementations, illustrating how additional insights and efficiency can be gained.

DataSpaces: an interaction and coordination framework for coupled simulation workflows

Emerging high-performance distributed computing environments are enabling new end-to-end formulations in science and engineering that involve multiple interacting processes and data-intensive application workflows. For example, current fusion simulation efforts are exploring coupled models and codes that simultaneously simulate separate application processes, such as the core and the edge turbulence. These components run on different high performance computing resources, need to interact at runtime with each other and with services for data monitoring, data analysis and visualization, and data archiving. As a result, they require efficient and scalable support for dynamic and flexible couplings and interactions, which remains a challenge. This paper presents DataSpaces a flexible interaction and coordination substrate that addresses this challenge. DataSpaces essentially implements a semantically specialized virtual shared space abstraction that can be associatively accessed by all components and services in the application workflow. It enables live data to be extracted from running simulation components, indexes this data online, and then allows it to be monitored, queried and accessed by other components and services via the space using semantically meaningful operators. The underlying data transport is asynchronous, low-overhead and largely memory-to-memory. The design, implementation, and experimental evaluation of DataSpaces using a coupled fusion simulation workflow is presented.