Big Data Workflows Research Articles

Integrated workflows and interfaces for data-driven semi-empirical electronic structure calculations.

Modern software engineering of electronic structure codes has seen a paradigm shift from monolithic workflows toward object-based modularity. Software objectivity allows for greater flexibility in the application of electronic structure calculations, with particular benefits when integrated with approaches for data-driven analysis. Here, we discuss different approaches to create deep modular interfaces that connect big-data workflows and electronic structure codes and explore the diversity of use cases that they can enable. We present two such interface approaches for the semi-empirical electronic structure package, DFTB+. In one case, DFTB+ is applied as a library and provides data to an external workflow; in another, DFTB+receives data via external bindings and processes the information subsequently within an internal workflow. We provide a general framework to enable data exchange workflows for embedding new machine-learning-based Hamiltonians within DFTB+ or enabling deep integration of DFTB+ in multiscale embedding workflows. These modular interfaces demonstrate opportunities in emergent software and workflows to accelerate scientific discovery by harnessing existing software capabilities.

Air pollution seasons in urban moderate climate areas through big data analytics

High particulate matter (PM) concentrations have a negative impact on the overall quality of life and health. The annual trends of PM can vary greatly depending on factors such as a country’s energy mix, development level, and climatic zone. In this study, we aimed to understand the annual cycle of PM concentrations in a moderate climate zone using a dense grid of low-cost sensors located in central Europe (Krakow). Over one million unique records of PM, temperature, humidity, pressure and wind speed observations were analyzed to gain a detailed, high-resolution understanding of yearly fluctuations. The comprehensive big-data workflow was presented with the statistical analysis of the meteorological factors. A big data-driven approach revealed the existence of two main PM seasons (warm and cold) in Europe’s moderate climate zone, which do not correspond directly with the traditional four main seasons (Autumn, Winter, Spring, and Summer) with two side periods (early spring and early winter). Our findings also highlighted the importance of high-resolution time and space data for sustainable spatial planning. The observations allowed for distinguishing whether the source of air pollution is related to coal burning for heating in cold period or to agricultural lands burning during the warm period.

Service Capability Aware Big Data Workflow Scheduling Approach in Cloud Datacenter

Service Capability Aware Big Data Workflow Scheduling Approach in Cloud Datacenter

Service capability aware big data workflow scheduling approach in cloud datacentre

Service capability aware big data workflow scheduling approach in cloud datacentre

BIGOWL4DQ: Ontology-driven approach for Big Data quality meta-modelling, selection and reasoning

Context:Data quality should be at the core of many Artificial Intelligence initiatives from the very first moment in which data is required for a successful analysis. Measurement and evaluation of the level of quality are crucial to determining whether data can be used for the tasks at hand. Conscientious of this importance, industry and academia have proposed several data quality measurements and assessment frameworks over the last two decades. Unfortunately, there is no common and shared vocabulary for data quality terms. Thus, it is difficult and time-consuming to integrate data quality analysis within a (Big) Data workflow for performing Artificial Intelligence tasks. One of the main reasons is that, except for a reduced number of proposals, the presented vocabularies are neither machine-readable nor processable, needing human processing to be incorporated. Objective:This paper proposes a unified data quality measurement and assessment information model. This model can be used in different environments and contexts to describe data quality measurement and evaluation concerns. Method:The model has been developed as an ontology to make it interoperable and machine-readable. For better interoperability and applicability, this ontology, BIGOWL4DQ, has been developed as an extension of a previously developed ontology for describing knowledge management in Big Data analytics. Conclusions:This extended ontology provides a data quality measurement and assessment framework required when designing Artificial Intelligence workflows and integrated reasoning capacities. Thus, BIGOWL4DQ can be used to describe Big Data analysis and assess the data quality before the analysis. Result:Our proposal has been validated with two use cases. First, the semantic proposal has been assessed using an academic use case. And second, a real-world case study within an Artificial Intelligence workflow has been conducted to endorse our work.

Big data BPMN workflow resource optimization in the cloud

Cloud computing is one of the critical technologies that meet the demand of various businesses for the high-capacity computational processing power needed to gain knowledge from their ever-growing business data. When utilizing cloud computing resources to deal with Big Data processing, companies face the challenge of determining the optimal use of resources within their business processes. The miscalculation of the necessary resources directly affects their budget and can cause delays in the cycle time of their key processes. This study investigates the simulation of cloud resource optimization for Big Data workflows modeled with the Business Process Modeling Notation (BPMN). To this end, a BPMN performance evaluation framework was developed. The framework’s capabilities were presented using real-world data science workflow and later evaluated on workflows consisting of 13, 52, and 104 tasks. The results show that the developed framework is adequate for estimating the overall run-time distribution and optimizing the cloud resource deployment and that the BPMN can be utilized for Big Data processing workflows. Therefore, this study contributes to BPMN practitioners by providing a tool to apply BPMN for their Big Data workflows and decision-makers by giving them critical insights into their key business processes. The framework source code is available at https://github.com/ntankovic/python-bpmn-engine.

Epigenomics Scientific Big Data Workflow Scheduling for Cancer Diagnosis in Health Care Using Heterogeneous Computing Environment

HIGHLIGHTS Analysis of different scheduling techniques for Epigenomics computation. An efficient algorithm that minimize computation time of Epigenomics Application. Performance analysis in High Performance Computing (HPC) environment. Shows better performance over state-of-art algorithms.

Securing Big Data Scientific Workflows via Trusted Heterogeneous Environments

Big data workflow management systems (BDWMS)s have recently emerged as popular data analytics platforms to conduct large-scale data analytics in the cloud. However, the protection of data confidentiality and secure execution of workflow applications remains an important and challenging problem. Although a few data analytics systems, such as VC3 and Opaque, were developed to address security problems, they are limited to specific domains such as Map-Reduce-style and SQL query workflows. A generic secure framework for BDWMSs is still missing. In this article, we propose SecDATAVIEW, a distributed BDWMS that employs heterogeneous workers, such as Intel SGX and AMD SEV, to protect both workflow and workflow data execution, addressing three major security challenges: (1) Reducing the TCB size of the big data workflow management system in the untrusted cloud by leveraging the hardware-assisted TEE and software attestation; (2) Supporting Java-written workflow tasks to overcome the limitation of SGX’s lack of support for Java programs; and (3) Reducing the adverse impact of SGX enclave memory paging overhead through a “Hybrid” workflow task scheduling system that selectively deploys sensitive tasks to a mix of SGX and SEV worker nodes. Our experimental results show that SecDATAVIEW imposes moderate overhead on the workflow execution time.

How Much of Biodiversity is Represented in Collections: A big data workflow of aggregated occurrence data

Natural history collections play a pivotal role in taxonomy, which in turn supports all of biology, but particularly conservation and biodiversity policy. However, to provide this role, it is necessary to know what specimens are stored where, and how complete the collection is. The biodiversity held within collections globally remains uncertain, with an estimated 1.2 to 2.1 billion (109) specimens (Ariño 2010), of which around 200 million are represented on the Global Biodiversity Information Facility (GBIF). Here we estimate the total biodiversity in collections worldwide by extrapolating from those specimens we know about. Data aggregators such as GBIF provide an ever-changing window into the contents of collections. We use non-parametric estimators that allow for the approximation of the number of classes in an incomplete set, such as the number of species within a collection, but also the proportion of biodiversity preserved on a national or continental level (for example within a taxonomic group, compared to the world or to a continent). Because the contents of data aggregators such as GBIF, are in constant flux, our workflow is made to be repeatable on the monthly snapshots that GBIF provides. The results of the workflow expose data gaps in GBIF, namely that collections from some large geographical regions, such as Asia, are poorly represented, but also taxonomic gaps exist, such as several Coleoptera families where many more species are accepted in the backbone than are represented on GBIF. As more data are published to GBIF the estimates for these taxon groups and geographical regions will improve. The detection of data gaps within data aggregators such as GBIF, and the subsequent mobilisation of missing information remains a priority for both aggregators and researchers (GBIF Secretariat 2022, Collen et al. 2008, Hochkirch et al. 2020). Our workflow will allow for continuous monitoring of collections and groups of collections of their coverage of global biodiversity, and the results can inform their collection development strategy.

Estimating Shear Strength Properties of the Surrounding Soils Based on the Execution Energies of Piles

Historically, empirical relations are the basis of everyday foundation design. These relations, however, rely on specific datasets, which may not represent the true conditions observed in the field. Even in situ tests rely on empirical correlation formulas, which link observed phenomena to soil properties. These correlations should be updated according to the specific design conditions. Big data (BD) workflows enable the use of massive data available to update the correlations and to provide more accurate predictions of the parameters studied. Thus, in this paper, a BD approach is used to study the relation between the drilling process of continuous flight auger piles and the shear strength properties (SSPs) of the surrounding soils. Soil surveys were carried out to identify the soil strata in the site and to validate the estimates of the SSPs. The results show that indirect measurements are in accordance with typical undrained shear strength and friction angles of the materials considered.

Spark-Based Digital Factory Design

Abstract Big data processing often uses the paradigm of parallelism by computing directly on top of the distributed data storage. The existing big data workflows unify the data processing practices to utilize the cloud’s native computational potentials to offer advanced machine learning and BI capabilities. Spark is an open-source massively parallel in-memory data processing framework, the current state-of-the-art. The primary approach is to break down the job into granular-level executed tasks, enabling parallelization. In the discussed case study, through IoT – cloud solutions, the plant data can be converted into an analyzable form to let the farther machine learning modules produce added value. To maximize the efficiency of the processing and accumulation, cloud-based components are introduced. Based on the data insights, the appropriate operative actions can be taken. The cost and performance optimization methods were also discussed in the study. Through achieving higher degree of digitalization, the control over the production increased.

Big Data Workflows: Locality-Aware Orchestration Using Software Containers

The emergence of the edge computing paradigm has shifted data processing from centralised infrastructures to heterogeneous and geographically distributed infrastructures. Therefore, data processing solutions must consider data locality to reduce the performance penalties from data transfers among remote data centres. Existing big data processing solutions provide limited support for handling data locality and are inefficient in processing small and frequent events specific to the edge environments. This article proposes a novel architecture and a proof-of-concept implementation for software container-centric big data workflow orchestration that puts data locality at the forefront. The proposed solution considers the available data locality information, leverages long-lived containers to execute workflow steps, and handles the interaction with different data sources through containers. We compare the proposed solution with Argo workflows and demonstrate a significant performance improvement in the execution speed for processing the same data units. Finally, we carry out experiments with the proposed solution under different configurations and analyze individual aspects affecting the performance of the overall solution.

Empiricism and collaboration on grid data analytics

This paper discusses the need for data-driven tools to manage modern electric grids, where planning and operational decisions increasingly require empirical data on various time scales. The advancement of such tools will hinge on deploying instrumentation to collect faster and more localized measurements, capitalizing on state-of-the-art software solutions to facilitate big-data workflows, and enabling open exchange of data and information with research collaborators.

Conceptualization and scalable execution of big data workflows using domain-specific languages and software containers

Big Data processing, especially with the increasing proliferation of Internet of Things (IoT) technologies and convergence of IoT, edge and cloud computing technologies, involves handling massive and complex data sets on heterogeneous resources and incorporating different tools, frameworks, and processes to help organizations make sense of their data collected from various sources. This set of operations, referred to as Big Data workflows, requires taking advantage of Cloud infrastructures’ elasticity for scalability. In this article, we present the design and prototype implementation of a Big Data workflow approach based on the use of software container technologies, message-oriented middleware (MOM), and a domain-specific language (DSL) to enable highly scalable workflow execution and abstract workflow definition. We demonstrate our system in a use case and a set of experiments that show the practical applicability of the proposed approach for the specification and scalable execution of Big Data workflows. Furthermore, we compare our proposed approach’s scalability with that of Argo Workflows – one of the most prominent tools in the area of Big Data workflows – and provide a qualitative evaluation of the proposed DSL and overall approach with respect to the existing literature.

An energy-efficient big data workflow scheduling algorithm under budget constraints for heterogeneous cloud environment

Infrastructure as a service model of cloud computing provides a tremendous amount of high-performance computing systems for the execution of scientific workflow applications. However, due to explosive growth in energy consumption and high charge cost of using these cloud systems, energy-efficient workflow scheduling under budget constraints becomes the most challenging issue. Very few research works have been done that consider the stated issue. Most of them mainly focus on the minimization of schedule length under user-specified budget constraints or energy consumption constraints. In this article, we propose an energy-efficient workflow scheduling algorithm named reducing energy consumption using fair pre-assignment of available budget (RECFPAB) that reduces energy consumption under client-specified budget constraints. The RECFPAB introduces a flexible mechanism to save energy consumption with the inclusion of energy and cost coefficient factor that enables fair distribution of available budget for unscheduled tasks of the workflow application. In order to compare the performance of the proposed algorithm, an energy-efficient version of the popular existing algorithms such as heterogeneous budget constrained scheduling and minimizing schedule length using budget level are introduced. The experimental evaluation based on Genome, LIGO, and Montage applications shows that RECFPAB gives significant results in comparison with considered algorithms.

An efficient list scheduling algorithm with task duplication for scientific big data workflow in heterogeneous computing environments

SummaryA high‐performance heterogeneous computing environment provides computing resources for efficient computation of scientific Big Data workflow applications. A Big Data workflow application comprises thousands of interdependent tasks with precedence constraints. Basically, the performance of heterogeneous computing systems mainly depends on the workflow scheduling algorithms. These workflow scheduling algorithms are considered an NP‐complete problem. In this article, a List Scheduling with Task Duplication (LSTD) algorithm is proposed that efficiently minimizes the makespan of workflow applications. The LSTD introduces task duplication strategy in the list scheduling algorithm without increasing the overall time complexity. The overall functionality of LSTD mainly consists of three phases. In the first phase, it calculates the rank of the tasks for deciding the scheduling order. The next step is responsible for duplicating the entry task on the processor only if it increases the overall efficiency and avoids processor overloading. Finally, in the last step, the processor is assigned to the tasks based on the popular insertion‐based policy that attempts to insert the task among two earlier assigned tasks on a given processor in earliest idle time. In order to verify the usefulness of the proposed algorithm, several existing well‐known algorithms, such as Heterogeneous Earliest Finish Time (HEFT), Critical Path on a Processor (CPOP), and Predict Earliest Finish Time (PEFT) are considered for comparison. A non‐duplication version of LSTD named List Scheduling without Task Duplication algorithm is also considered for performance evaluation. The experimental analysis based on scientific Big Data workflows (CyberShake, Montage, and LIGO) proves that LSTD significantly surpasses all considered scheduling heuristics concerning schedule length ratio, the percentage of best results, and average running time metrics.

A dynamic VM provisioning and de-provisioning based cost-efficient deadline-aware scheduling algorithm for Big Data workflow applications in a cloud environment

Cloud computing is the fastest growing distributed computing paradigm that provides online IT resources on demand by following a pay-as-you-go billing model. The success of this computing paradigm enables cloud providers to offer an extensive collection of parallel computing resources to deal with Big Data workflow scheduling problems. Although, workflow scheduling has been extensively studied, however, most of them are unable to achieve user-specified deadline constraints at the cheap cost. In this paper, a Dynamic Cost-Efficient Deadline-Aware (DCEDA) heuristic algorithm is proposed for scheduling Big Data workflow that produces the cheapest schedule while achieving the deadline constraints. DCEDA dynamically takes appropriate scheduling decisions for workflow tasks based on the fact that deadline constraint is not violated in the future. Also, it continuously monitors the VM pool for identifying the active idle VMs that incur extra costs and overheads, and subsequently de-provision them. The experimental analysis based on Montage workflow and randomly generated synthetic workflow with various characteristics prove that DCEDA delivers better performance in comparison to the existing algorithms.

Big Data Processing Workflows Oriented Real-Time Scheduling Algorithm using Task-Duplication in Geo-Distributed Clouds

Scheduling big data processing workflows involves both large-scale tasks and transmission of massive intermediate data among tasks, thus optimizing their completion time and monetary cost becomes a challenging issue. Besides, data streams are continuously generated, and dynamically submitted to clouds for real-time or near real-time processing. Naturally, responsive schedules are required to keep pace with such dynamic environments and this further aggravates the difficulty of the workflow scheduling problem. To address these issues, we first derive two theorems to minimize the completion time of a set of parallel workflow tasks and the start time of each workflow task, and then define the latest finish time for workflow tasks, which is also proved its advantage in reducing costs without delaying the completion of workflows. On the basis of these theorems, we propose a novel r eal- t ime s cheduling a lgorithm using t ask- d uplication, RTSATD, such that minimizing both the completion time and monetary cost of processing big data workflows in clouds. The performance of RTSATD is analyzed by using both synthesized and real-world workflows. The experimental results demonstrate the superiority of the proposed algorithm with respect to completion time (up to 28.73 percent) and resource utilization (up to 46.31 percent) over two existing approaches.

Analysis on the influence of big data era on enterprise financial accounting -- a case study of HUA WEI technologies co., LTD

”Big data” is the product of the integration of the highly developed Internet innovation function and various economic fields in today’s society. The development of “big data” is bound to bring significant changes in the economic development of today’s society. Taking HUA WEI technologies co., LTD., financial aspects based on the development of big data, found big data technology in the application process of the impact of the financial accounting, this era of big data work flow for the company in China, the impact of financial decision-making and financial personnel, and the company response to this phenomenon and make a change, and to analyze its causes and solutions. This electronic document is a “live” template. The various components of your paper [title, text, heads, etc.] are already defined on the style sheet, as illustrated by the portions given in this document.

Reducing data transfer in big-data workflows: the computation-flow delegated approach

Existing orchestrated bioinformatics workflow execution approaches necessitate the transfer of datasets from biological data services to the analysis tool (computation) services of the workflow for various data analysis. This model of moving data to computation during workflow execution weakens the performance of the workflow especially when the orchestrated bioinformatics workflow has to handle big-data in it. Since the size of the analysis tools are much smaller than the datasets size in a workflow, in this paper, to minimize the dataflow and improve workflow performance, we propose a novel computation-flow delegated (CFD) approach. The CFD approach lets the tool services of the workflow to dynamically migrate analysis tools towards the datasets to perform computation on data side during workflow execution. We use a set of mobile agents to operate the CFD approach and present a mobile agent-based computation-flow delegation framework (MABCFD) to execute the workflow tasks. We implement the prototype of the MABCFD framework and analyze the performance of the CFD approach empirically by executing in isolation workflow patterns (sequence, fan-out and fan-in) common to bioinformatics applications. Performance analysis shows that the computation-driven CFD approach consistently outperforms the existing data-driven approaches across all patterns and scales favorably with data size.