Complex Scientific Applications Research Articles

Input/output optimization scheduler for cloud-based map reduce framework

Hadoop MapReduce (HMR) provides the most common MapReduce (MR) framework, and it is available as open source. MR is a famous computational framework for evaluating unstructured, and semi-structured big data and executing applications in the past ten years. Memory and input/output (I/O) overhead are just two of the many problems affecting the current HMR scheduler system. This study aims to improve systems resource use including the processing of data in real-time by creating a memory I/O optimized scheduler (MIOOS) for HMR. The disk I/O seek can be reduced by using MIOOS, which analyzes the entire memory management. Additionally, the MIOOS makespan approach is used to reduce the occurrence of problems in intermediary tasks. Both the MIOOS approach and the current approach are assessed by using complex scientific workflow applications with extreme task inter-dependencies. Further, the comparison study demonstrates that the MIOOS framework outdoes the current approach regarding makespan and overall memory usage.

MONWS: Multi-Objective Normalization Workflow Scheduling for Cloud Computing

Cloud computing is a prominent approach for complex scientific and business workflow applications in the pay-as-you-go model. Workflow scheduling poses a challenge in cloud computing due to its widespread applications in physics, astronomy, bioinformatics, and healthcare, etc. Resource allocation for workflow scheduling is problematic due to the computationally intensive nature of the workflow, the interdependence of tasks, and the heterogeneity of cloud resources. During resource allocation, the time and cost of execution are significant issues in the cloud-computing environment, which can potentially degrade the service quality that is provided to end users. This study proposes a method focusing on makespan, average utilization, and cost. The authors propose a task’s dynamic priority for workflow scheduling using MONWS, which uses the min-max algorithm to minimize the finish time and maximize resource utilization by calculating the dynamic threshold value for scheduling tasks on virtual machines. When the experimental results were compared to existing algorithms, MONWS achieved a 35% improvement in makespan, an 8% increase in maximum average cloud utilization, and a 4% decrease in cost.

Developing a Workflow Management System Simulation for Capturing Internal IaaS Behavioural Knowledge

Scientific workflows are becoming increasingly important for complex scientific applications. Conducting real experiments for large-scale workflows is challenging because they are very expensive and time consuming. A simulation is an alternative approach to a real experiment that can help evaluating the performance of workflow management systems (WMS) and optimise workflow management techniques. Although there are several workflow simulators available today, they are often user-oriented and treat the cloud as a black box. Unfortunately, this behaviour prevents the evaluation of the infrastructure level impact of the various decisions made by the WMSs. To address these issues, we have developed a WMS simulator (called DISSECT-CF-WMS) on DISSECT-CF that exposes the internal details of cloud infrastructures. DISSECT-CF-WMS enables better energy awareness by allowing the study of schedulers for physical machines. It also enables dynamic provisioning to meet the resource needs of the workflow application while considering the provisioning delay of a VM in the cloud. We evaluated our simulation extension by running several workflow applications on a given infrastructure. The experimental results show that we can investigate different schedulers for physical machines on different numbers of virtual machines to reduce energy consumption. The experiments also show that DISSECT-CF-WMS is up to 295× faster than WorkflowSim and still provides equivalent results. The experimental results of auto-scaling show that it can optimise makespan, energy consumption and VM utilisation in contrast to static VM provisioning.

MC-DeF

Executing complex scientific applications on Coarse-Grain Reconfigurable Arrays ( CGRAs ) promises improvements in execution time and/or energy consumption compared to optimized software implementations or even fully customized hardware solutions. Typical CGRA architectures contain of multiple instances of the same compute module that consist of simple and general hardware units such as ALUs, simple processors. However, generality in the cell contents, while convenient for serving a wide variety of applications, penalizes performance and energy efficiency. To that end, a few proposed CGRAs use custom logic tailored to a particular application’s specific characteristics in the compute module. This approach, while much more efficient, restricts the versatility of the array. To date, versatility at hardware speeds is only supported with Field programmable gate arrays (FPGAs), that are reconfigurable at a very fine grain. This work proposes MC-DeF, a novel Mixed-CGRA Definition Framework targeting a Mixed-CGRA architecture that leverages the advantages of CGRAs by utilizing a customized cell array, and those of FPGAs by incorporating a separate LUT array used for adaptability. The framework presented aims to develop a complete CGRA architecture. First, a cell structure and functionality definition phase creates highly customized application/domain specific CGRA cells. Then, mapping and routing phases define the CGRA connectivity and cell-LUT array transactions. Finally, an energy and area estimation phase presents the user with area occupancy and energy consumption estimations of the final design. MC-DeF uses novel algorithms and cost functions driven by user defined metrics, threshold values, and area/energy restrictions. The benefits of our framework, besides creating fast and efficient CGRA designs, include design space exploration capabilities offered to the user. The validity of the presented framework is demonstrated by evaluating and creating CGRA designs of nine applications. Additionally, we provide comparisons of MC-DeF with state-of-the-art related works, and show that MC-DeF offers competitive performance (in terms of internal bandwidth and processing throughput) even compared against much larger designs, and requires fewer physical resources to achieve this level of performance. Finally, MC-DeF is able to better utilize the underlying FPGA fabric and achieves the best efficiency (measured in LUT/GOPs).

Cost and makespan aware workflow scheduling in IaaS clouds using hybrid spider monkey optimization

The researcher's predilection towards the concerned infinite resources and the dynamic provisioning on rental premises encourages the scheduling of complex scientific applications in the cloud. The scheduling of workflows in the cloud is constrained to QoS parameters. Many heuristic and meta-heuristic algorithms are widely investigated for the QoS constrained workflow scheduling problem. However, it is still an open area of research, as most of the existing techniques concentrate on minimization of either cost or time and ignores the optimization of multiple QoS constraints simultaneously. To address this problem, in this paper, a Hybrid Spider Monkey Optimization (HSMO) algorithm has been proposed. The proposed algorithm optimizes the makespan and the cost while satisfying the budget and deadline constraints. The proposed algorithm is the hybridization of recently developed SMO and the other popular heuristic BDSD algorithm. BDSD is a budget and deadline constrained algorithm, which guides HSMO in generating a feasible schedule. Moreover, the proposed strategy involves the penalty function to restrict selecting those solutions that fail to satisfy the QoS constraints. Experimental results demonstrate the effectiveness of HSMO over existing ABC, Bi-Criteria PSO, and BDSD algorithms.

Creating Customized CGRAs for Scientific Applications

Executing complex scientific applications on Coarse Grain Reconfigurable Arrays (CGRAs) offers improvements in the execution time and/or energy consumption when compared to optimized software implementations or even fully customized hardware solutions. In this work, we explore the potential of application analysis methods in such customized hardware solutions. We offer analysis metrics from various scientific applications and tailor the results that are to be used by MC-Def, a novel Mixed-CGRA Definition Framework targeting a Mixed-CGRA architecture that leverages the advantages of CGRAs and those of FPGAs by utilizing a customized cell-array along, with a separate LUT array being used for adaptability. Additionally, we present the implementation results regarding the VHDL-created hardware implementations of our CGRA cell concerning various scientific applications.

MR-GAN: Manifold Regularized Generative Adversarial Networks for Scientific Data

Despite the growing interest in applying generative adversarial networks (GANs) in complex scientific applications, training GANs on scientific data remains a challenging problem from both theoreti...

НАУКИ О ЗЕМЛЕ В АСТРАХАНСКОМ ГОСУДАРСТВЕННОМ УНИВЕРСИТЕТЕ

The purpose of the work is a historical review of the development of Earth Sciences at Astrakhan State University (ASU), key research issues in the field of landscape science, geoecology, geology, hydrogeology and oil and gas geology, as well as the consideration of scientific research of candidates and doctors of sciences who have contributed to the development of natural sciences at ASU. Methods. The paper uses: the method of complex scientific analysis, comparative analysis, statistical method, study and application of available literary and scientific materials. Results. The directions of the Earth Sciences, the study of which continues at the present time, are described. Scientific work on the study of various aspects of geology, exploration and development of natural minerals, hydrogeology and geology of oil and gas, landscape studies, urban studies, remote sensing of the earth, technosphere safety and hydrology at Astrakhan State University is active and diverse. Conclusions. New knowledge and discoveries are obtained based on the achieved results and traditions of the university, but at the same timewith the use of the latest technologies and equipment, which allows you to more fully explore many features of the region, especially valuable for improving the efficiency of the scientific industry.

РАЗВИТИЕ И СОВЕРШЕНСТВОВАНИЕ ГЕОЭКОЛОГИЧЕСКОГО ТУРИЗМА В БОГДИНСКО-БАСКУНЧАКСКОМ ПРИРОДНОМ ЗАПОВЕДНИКЕ

The aim of this work is to study the main elements contributing to the development and improvement of ecotourism in the Bogdinsko-Baskunchak nature reserve, as well as to consider and analyze the financing of this specially protected natural area (SPNA) and its impact on the wellbeing of the reserve. Methods. The work uses the method of complex scientific analysis, comparative analysis, statistical method, study and application of available literature and scientific funds . Results. An assessment of the improvement of the infrastructure of the Bogdinsko-Baskunchak nature reserve is given and the main factors contributing to the growth of ecotourism and environmental education are determined. The analysis of the financing of the reserve both at the expense of budgetary and at the expense of extra-budgetary funds and donations to the account of the reserve is given using a comparative analysis by years of the study period. The current tourist routes of the reserve are described. Conclusions. The development of ecotourism is greatly influenced by a complex of internal and external factors. Among them: the need and opportunities of the population to satisfy contact with nature, the epidemiological situation in the country, the presence or absence of certain sources of funding, the availability of opportunities to improve the infrastructure, etc. In addition, one should not forget about the need to develop additional services and tourist routes, as well as introduce new elements of ecotourism activities.

A Generic Performance Analysis Technique Applied to Different CFD Methods for HPC

ABSTRACT For complex engineering and scientific applications, Computational Fluid Dynamics (CFD) simulations require a huge amount of computational power. As such, it is of paramount importance to carefully assess the performance of CFD codes and to study them in depth for enabling optimisation and portability. In this paper, we study three complex CFD codes, OpenFOAM, Alya and CHORUS representing two numerical methods, namely the finite volume and finite-element methods, on both structured and unstructured meshes. To all codes, we apply a generic performance analysis method based on a set of metrics helping the code developer in spotting the critical points that can potentially limit the scalability of a parallel application. We show the root cause of the performance bottlenecks studying the three applications on the MareNostrum4 supercomputer. We conclude providing hints for improving the performance and the scalability of each application.

Research trend of large-scale supercomputers and applications from the TOP500 and Gordon Bell Prize

China is playing an increasingly important role in international supercomputing. In high-performance computing domain, there are two famous awards: The TOP500 list for the fastest 500 supercomputers in the world and the Gordon Bell Prize for the best HPC (high-performance computing) applications. China has been awarded in both TOP500 list and Gordon Bell Prize. In this paper, we review the supercomputers in the latest TOP500 list and seven Gordon Bell Prize applications to show the research trend of the large-scale supercomputers and applications. The first trend we observe is that heterogeneous architectures are widely used in the construction of supercomputing systems. The second trend is that artificial intelligence applications are expected to become one of the main stream applications of supercomputing. The third trend is that applying heterogeneous systems to complex scientific simulation applications will be more difficult.

Fair budget constrained workflow scheduling approach for heterogeneous clouds

The phenomenal advancement of technology paved the way for the execution of complex scientific applications. The emergence of the cloud provides a distributed heterogeneous environment for the execution of large and complex workflows. Due to the dynamic and heterogeneous nature of the cloud, scheduling workflows become a challenging problem. Mapping and assignment of heterogeneous instances for each task while minimizing execution time and cost is a NP-complete problem. For efficient scheduling, it is required to consider various QoS parameters such as time, cost, security, and reliability. Among these, computation time and cost are the two notable parameters. In order to preserve the functionalities of these two parameters in heterogeneous cloud environments, in this paper, a fair budget-constrained workflow scheduling algorithm (FBCWS) is proposed. The novelty of the proposed algorithm is to minimize the makespan while satisfying budget constraints and a fair means of schedule for every task. FBCWS also provides a mechanism to save budget by adjusting the cost-time efficient factor of the minimization problem. The inclusion of a cost-time efficient factor in the algorithm provides flexibility to minimize the makespan or save budget. In order to validate the effectiveness of the proposed approach, several real scientific workflows are simulated, and experimental results are compared with other existing approaches, namely; Heterogeneous Budget Constrained Scheduling (HBCS), Minimizing Schedule Length using Budget Level (MSBL) and Pareto Optimal Scheduling Heuristic (POSH) algorithms. Experimental results prove that the proposed algorithm behaves outstandingly for compute-intensive workflows such as Epigenomic and Sipht. Also, FBCWS outperforms the existing HBCS in most of the cases. Moreover, FBCWS proves to be more time-efficient than POSH and more cost-efficient than MSBL. The effectiveness of the proposed algorithm is illustrated through the popular ANOVA test.

An efficient resource prediction–based scheduling technique for scientific applications in cloud environment

Cloud computing makes scientists to run complex scientific applications. The research community is able to access on-demand compute resources within a short span of time instead of experiencing peak demand bottlenecks. As the demand for cloud resources is dynamic and volatile in nature, this in turn affects the availability of resources during scheduling. In order to allocate sufficient resources for scientific applications with different execution requirements, it is necessary to predict the appropriate set of resources. To attain this objective, a resource prediction–based scheduling technique has been introduced which automates the resource allocation for scientific application in virtualized cloud environment. First, the proposed prediction model is trained on the dataset generated by concurrently deploying tasks of a scientific application on cloud. Then, the resources are scheduled based on the output of proposed prediction technique. The main objective of resource prediction–based scheduling technique is to efficiently handle the resources for virtual machines in order to reduce the execution time, error rate, and improve the accuracy.

Parallelization of Jump Flood-Cut Method in Shared Memory Multicore Systems

Graph cut approach has evidenced to be trenchant strategy for the segmentation of digital images. Primary restraint of existing graph cut method is that of acute arithmetic calculation which use up tremendous processing time to carry into action. This paper aims on parallelizing jump flooding technique and push re-label scheme with NSP and BFS methodology on shared memory multicore system using OpenMP compiler directives. OpenMP API includes a rich set of library routines and environment variables which enables the parallelization of JF-Cut method within a framework where parallelism is achieved by sharing a single common memory among multiple co-processors. OpenMP remains live area of interest for research in segmentation of digital images as it accelerates computational speed of complex scientific applications like image segmentation using multicore architecture in shared memory mode.

Multi-Objective Optimization for scientific workflow task scheduling in IaaS Cloud

The use of scientific applications on cloud networks increases day by day generating volumes of data and consuming large computational power. These scientific applications find its importance in the field of astronomy, geology, genetics and bio-technology etc. Complex and mission critical scientific applications can be modeled as scientific workflows and can be executed in cloud. The tasks of the scientific applications are generally data intensive and compute intensive. Traditional computer networks are not suitable for handling scientific applications and hence ubiquitous distributed networks like cloud are prominent in hosting scientific applications. The cloud hosted scientific applications and the cloud network need to satisfy many objectives to the interest of its users. This paper explores the multi-objective optimization applications in scientific workflow task scheduling in IaaS cloud and the related algorithms employed.

Cluster Programming using the OpenMP Accelerator Model

Computation offloading is a programming model in which program fragments (e.g., hot loops) are annotated so that their execution is performed in dedicated hardware or accelerator devices. Although offloading has been extensively used to move computation to GPUs, through directive-based annotation standards like OpenMP, offloading computation to very large computer clusters can become a complex and cumbersome task. It typically requires mixing programming models (e.g., OpenMP and MPI) and languages (e.g., C/C++ and Scala), dealing with various access control mechanisms from different cloud providers (e.g., AWS and Azure), and integrating all this into a single application. This article introduces computer cluster nodes as simple OpenMP offloading devices that can be used either from a local computer or from the cluster head-node. It proposes a methodology that transforms OpenMP directives to Spark runtime calls with fully integrated communication management, in a way that a cluster appears to the programmer as yet another accelerator device. Experiments using LLVM 3.8, OpenMP 4.5 on well known cloud infrastructures (Microsoft Azure and Amazon EC2) show the viability of the proposed approach, enable a thorough analysis of its performance, and make a comparison with an MPI implementation. The results show that although data transfers can impose overheads, cloud offloading from a local machine can still achieve promising speedups for larger granularity: up to 115× in 256 cores for the2MMbenchmark using 1GB sparse matrices. In addition, the parallel implementation of a complex and relevant scientific application reveals a 80× speedup on a 320 core machine when executed directly from the headnode of the cluster.

Strategy for data-flow synchronizations in stencil parallel computations on multi-/manycore systems

In this paper, an innovative strategy for the data-flow synchronization in shared-memory systems is proposed. This strategy assumes to synchronize only interdependent threads instead of using the barrier approach that—in contrast to our approach—synchronize all threads. We demonstrate the adaptation of the data-flow synchronization strategy to two complex scientific applications based on stencil codes. An algorithm for the data-flow synchronization is developed and successfully used for both applications. The proposed approach is evaluated for various Intel microarchitectures released in the last 5 years, including the newest processors: Skylake and Knights Landing. The important part of this assessment is the performance comparison of the proposed data-flow synchronization with the OpenMP barrier. The experimental results show that the performance of the studied applications can be accelerated up to 1.3 times using the proposed data-flow synchronizations strategy.

Serverless execution of scientific workflows: Experiments with HyperFlow, AWS Lambda and Google Cloud Functions

Scientific workflows consisting of a high number of interdependent tasks represent an important class of complex scientific applications. Recently, a new type of serverless infrastructures has emerged, represented by such services as Google Cloud Functions and AWS Lambda, also referred to as the Function-as-a-Service model. In this paper we take a look at such serverless infrastructures, which are designed mainly for processing background tasks of Web and Internet of Things applications, or event-driven stream processing. We evaluate their applicability to more compute- and data-intensive scientific workflows and discuss possible ways to repurpose serverless architectures for execution of scientific workflows. We have developed prototype workflow executor functions using AWS Lambda and Google Cloud Functions, coupled with the HyperFlow workflow engine. These functions can run workflow tasks in AWS and Google infrastructures, and feature such capabilities as data staging to/from S3 or Google Cloud Storage and execution of custom application binaries. We have successfully deployed and executed the Montage astronomy workflow, often used as a benchmark, and we report on initial results of its performance evaluation. Our findings indicate that the simple mode of operation makes this approach easy to use, although there are costs involved in preparing portable application binaries for execution in a remote environment.While our solution is an early prototype, we find the presented approach highly promising. We also discuss possible future steps related to execution of scientific workflows in serverless infrastructures. Finally, we perform a cost analysis and discuss implications with regard to resource management for scientific applications in general.

A characterization of workflow management systems for extreme-scale applications

Automation of the execution of computational tasks is at the heart of improving scientific productivity. Over the last years, scientific workflows have been established as an important abstraction that captures data processing and computation of large and complex scientific applications. By allowing scientists to model and express entire data processing steps and their dependencies, workflow management systems relieve scientists from the details of an application and manage its execution on a computational infrastructure. As the resource requirements of today’s computational and data science applications that process vast amounts of data keep increasing, there is a compelling case for a new generation of advances in high-performance computing, commonly termed as extreme-scale computing, which will bring forth multiple challenges for the design of workflow applications and management systems. This paper presents a novel characterization of workflow management systems using features commonly associated with extreme-scale computing applications. We classify 15 popular workflow management systems in terms of workflow execution models, heterogeneous computing environments, and data access methods. The paper also surveys workflow applications and identifies gaps for future research on the road to extreme-scale workflows and management systems.

Semantic based Data Integration in Scientific Workflows

Data Integration has become the most prominent aspect of data management applications, especially in scientific domains like ecology, biology, and geosciences. Today’s complex scientific applications and the rise of diverse data generating devices in scientific domains (e.g. sensors) have made data integration a challenging task. In response to these types of challenges, data management applications are providing ground-breaking functionalities which come at the price of high complexity. This paper presents a semantic data integration framework which is based on the exploitation of ontologies. Exploiting a Description Logics formalism and associated reasoning procedures, the framework is able the handle heterogeneous formats and different semantics. Besides an in-depth discussion of the ontology-based integration capability, the paper also discusses a brief overview of the system architecture and its application in a real world scenario taken from ecological research.