Workflow Jobs Research Articles

Batch Job Active-Active Resiliency System and Method

This manuscript presents a robust solution for orchestrating batch job workflows in an active-active resiliency mode across multiple regions. The proposed system abstracts the complexities of job and step/task state management, provides automated and customizable retries, and ensures auditability and observability. The system maintains operational continuity and fault tolerance, leveraging lock marker files and a configurable scheduler, even during regional failures. This approach addresses the challenges of asynchronous workflow management, offering enhanced visibility, control, and efficiency in processing batch jobs.

Research on the Development of Interactive Learning Materials Based on OBE Concepts and Job Workflows

Research on the Development of Interactive Learning Materials Based on OBE Concepts and Job Workflows

Ways to Improve Workflow and Morale in an Ophthalmology Clinic: Survey Advice from Clinic Staff.

We aim to improve job workflow and satisfaction amongst clinic staff at an academic ophthalmology department. We analyzed survey data given over a 2-week period in July 2021. The participants were support staff (N = 18) from an academic ophthalmology department. Paper surveys were distributed to participants and returned anonymously for analysis. Results: The survey contained 9 Likert-style categorical questions, 2 of which were free response options. A total of 22 participants attempted the survey, 18 of these (82%) were complete and included in analysis. About half of the staff were satisfied with the current workflow 10/18 (56%). Staff who were clinical care coordinators had the lowest average satisfaction (2/5 on a 5-point scale) and the nursing team had the highest average (4.75/5). The most common staff suggestion for improving workflow efficiency was to train residents on forwarding and answering messages more effectively. This survey suggests that assigning patient message processing to the nursing staff can improve job satisfaction and workflow. Staff told us that the most exciting part of the job was appreciation from coworkers 9/30 (30%) and from physicians 8/30 (27%). The findings provide advice to physicians for optimizing communication, and staff experience, within their own ophthalmology clinics.

GeoAI Technologies and Their Application Areas in Urban Planning and Development: Concepts, Opportunities and Challenges in Smart City (Kuwait, Study Case)

Artificial intelligence has significantly altered many job workflows, hence expanding earlier notions of limitations, outcomes, size, and prices. GeoAI is a multidisciplinary field that encompasses computer science, engineering, statistics, and spatial science. Because this subject focuses on real-world issues, it has a significant impact on society and the economy. A broad context incorporating fundamental questions of theory, epistemology, and the scientific method is used to bring artificial intelligence (Al) and geography together. This connection has the potential to have far-reaching implications for the geographic study. GeoAI, or the combination of geography with artificial intelligence, offers unique solutions to a variety of smart city issues. This paper provides an overview of GeoAI technology, including the definition of GeoAI and the differences between GeoAI and traditional AI. Key steps to successful geographic data analysis include integrating AI with GIS and using GeoAI tools and technologies. Also shown are key areas of applications and models in GeoAI, likewise challenges to adopt GeoAI methods and technology as well as benefits. This article also included a case study on the use of GeoAI in Kuwait, as well as a number of recommendations.

Dynamic provisioning with structure inspired selection and limitation of VMs based cost-time efficient workflow scheduling in the cloud

Workflow scheduling in cloud computing environments is nowadays a hot topic as scientific workflows application are gradually taking advantage of commercial cloud assets. Common users’ quality of service (QoS) requirements are the respect of defined budget and deadline when executing their workflow job. Since execution cost minimization and completion time minimization are contradictory objectives, addressing such issue through trade-off function approaches have proved to be an efficient way. This paper presents the Cost-Time Trade-off efficient Workflow Scheduling with Dynamic provisioning (CTTWSDP) algorithm. CTTWSDP relies on dynamic VMs provisioning with a limited number of leased VMs, and a cost-time trade-off function over heterogeneous instances to determine the most viable schedule. CTTWSDP also proposed an improved Implicit Requested Instance Types Range (IRITR) evaluation, which is a scheduling concept introduced in our previous work. The IRITR evaluation aims at determining a range of VMs instance types that best suits the workflow execution, in order to avoid overbidding and underbidding that may lead to budget and deadline violation respectively. The results of simulations prove the effectiveness of the proposal. CTTWSDP achieves a 17.09–76.06% higher success rate when compared to four state-of-the-art algorithms. Furthermore, ANOVA along with Tukey–Kramer post-hoc tests have been conducted, revealing the effectiveness of CTTWSDP over three of the baseline algorithm, while for the fourth one the outperformance of CTTWSDP is not statistically significant. An analysis of the standard deviation of the success rate proves that CTTWSDP is more stable in its performance no matter the type and the workload of the workflow. With a standard deviation of 6.73, smaller than the ones obtained by the other algorithms from 18.66 to 34.10.

Scheduling of workflow jobs based on twostep clustering and lowest job weight

AbstractScheduling is an important technique to improve the productivity of workflow applications in a grid system. Serving workflow applications added overhead on the scheduling system since the scheduler should choose the best zero‐dependent job that improves the performance of a grid system. This paper proposes two dependent job scheduling algorithms based on analyzing the effect of categorical and continuous variables of the job, which are used in the calculation of job weight. To find the job weight, twostep clustering is used with 10 groups and the ranking equation. In addition, to verify the ability to apply the proposed algorithms in a real environment, weighted least squares estimation is used. The results showed that the prediction rate is equal to 99.88%, which indicates that the proposed algorithms could be applied in a real grid system with low overhead. Through simulations and after testing the proposed algorithms, the average results showed that the Dependent Job Scheduling (DJS) algorithm outperformed the previous algorithms, in total execution time and an average waiting time with an improvement value of 1.18 and 1.92 times, respectively. While DJS algorithm with weighting factor (DJSJP) outperformed the previous algorithms in total execution time only with an improvement value equal to 1.17 times. The overall results indicated that the proposed algorithms are efficient to be used in a grid system, besides four bits are sufficient to improve the performance of the job scheduling system.

A Heuristics-Based Cost Model for Scientific Workflow Scheduling in Cloud

Scientific Workflow Applications (SWFAs) can deliver collaborative tools useful to researchers in executing large and complex scientific processes. Particularly, Scientific Workflow Scheduling (SWFS) accelerates the computational procedures between the available computational resources and the dependent workflow jobs based on the researchers’ requirements. However, cost optimization is one of the SWFS challenges in handling massive and complicated tasks and requires determining an approximate (near-optimal) solution within polynomial computational time. Motivated by this, current work proposes a novel SWFS cost optimization model effective in solving this challenge. The proposed model contains three main stages: (i) scientific workflow application, (ii) targeted computational environment, and (iii) cost optimization criteria. The model has been used to optimize completion time (makespan) and overall computational cost of SWFS in cloud computing for all considered scenarios in this research context. This will ultimately reduce the cost for service consumers. At the same time, reducing the cost has a positive impact on the profitability of service providers towards utilizing all computational resources to achieve a competitive advantage over other cloud service providers. To evaluate the effectiveness of this proposed model, an empirical comparison was conducted by employing three core types of heuristic approaches, including Single-based (i.e., Genetic Algorithm (GA), Particle Swarm Optimization (PSO), and Invasive Weed Optimization (IWO)), Hybrid-based (i.e., Hybrid-based Heuristics Algorithms (HIWO)), and Hyper-based (i.e., Dynamic Hyper-Heuristic Algorithm (DHHA)). Additionally, a simulation-based implementation was used for SIPHT SWFA by considering three different sizes of datasets. The proposed model provides an efficient platform to optimally schedule workflow tasks by handling data-intensiveness and computational-intensiveness of SWFAs. The results reveal that the proposed cost optimization model attained an optimal Job completion time (makespan) and total computational cost for small and large sizes of the considered dataset. In contrast, hybrid and hyper-based approaches consistently achieved better results for the medium-sized dataset.

Cost‐aware cloud bursting in a fog‐cloud environment with real‐time workflow applications

SummaryCloud bursting is a concept originating from the hybrid cloud computing paradigm. During workload spikes, the local resources of the private cloud are supplemented by resources in the public cloud. This technique could also be applied in a fog computing environment, in order to handle workload fluctuations, by offloading applications to the cloud. Toward this direction, in this article, we propose a strategy for the utilization of supplementary cloud resources, in order to assist in the processing of Internet of Things workflow jobs that arrive dynamically in a fog environment. As the cloud involves higher data transfer latency and monetary cost, our approach takes into account these two factors, in addition to the real‐time constraints of the workload. The proposed scheduling heuristic is based on the tradeoff between performance and monetary cost. During resource selection, different contribution factors of these two parameters are assessed. Furthermore, the proposed scheduling method is compared against a baseline policy that utilizes only the fog resources. The simulation experiments were carried out under different sizes of workflow input data and for workloads with soft and hard deadlines.

Microlearning: A New Learning Model

Changes in the current workplace are creating changes in traditional processes. One such process is the way that the millennial and subsequent generations joining the workplace consume learning. The hospitality industry is an ever-changing service industry in which organizations need to continue to adapt to stay relevant. Trained employees providing excellent guest experiences are a competitive advantage. Some barriers to training include lack of time, lack of resources, and changing learning expectations. The aim of this article was to propose a new learning model that integrates performance workflow and microlearning. Microlearning is an approach that focuses on a single concept, utilizing multisensory and multimodality in a focused short amount of time. It can be easily incorporated to the job workflow. The model, future research, and implications are discussed.

An effective scheduling strategy based on hypergraph partition in geographically distributed datacenters

As the complexity of workflow applications increase, the scheduling and execution of workflow incur more waste of resources. In order to achieve load balancing and reduce the task execution time in the cloud system, an effective scheduling strategy based on hypergraph partition for workflow application in the geographically distributed datacenters is proposed. Firstly, a workflow job scheduling algorithm is proposed, which aims to reduce the response time and considers the cloud state. Besides, the task scheduling algorithm based on the hypergraph partition is designed with the goal of reducing the completion time and energy consumption for the tasks. In addition, the optimal task scheduling strategy can be obtained according to the Dijkstra shortest path algorithm based on Fibonacci heap. Finally, in the experiment, the workflow task scheduling algorithm can optimize the task execution performance and maintain the load balance of the computing nodes in each cloud, so that the average execution time of the tasks and the total energy consumption of the system are minimized.

Load balance based workflow job scheduling algorithm in distributed cloud

As the scale of the geo-distributed cloud increases and the workflow applications become more complex, the system operation is more likely to cause the waste of resources and excessive energy consumption. In this paper, a workflow job scheduling algorithm based on load balancing is proposed to efficiently utilize cloud resources. Firstly, the execution time of the jobs on the cloud is estimated based on the state of the cloud. Then, a queuing model is established for each cloud to minimize the total response time of the system. Finally, the job scheduling problem in a geo-distributed cloud can be transformed into the minimum system response time problem. Moreover, a workflow task scheduling algorithm based on the shortest path algorithm is proposed to minimize all task completion time and energy consumption. Firstly, the directed acyclic graph (DAG) of the tasks can be converted into the hypergraph according to the execution order of the tasks. Then, the k-path hypergraph partition is performed with the balance of the hypergraph. Finally, the Dijkstra shortest path algorithm is used to find the optimal task scheduling strategy which is performed on each hypergraph partition. The experimental results indicate that our proposed workflow scheduling method can effectively utilize cloud resources and reduce system energy consumption. Moreover, the applicability of the proposed effective scheduling strategy is shown in the scenarios of new media live video application.

A Dynamic Data-throttling Approach to Minimize Workflow Imbalance

Scientific workflows enable scientists to undertake analysis on large datasets and perform complex scientific simulations. These workflows are often mapped onto distributed and parallel computational infrastructures to speed up their executions. Prior to its execution, a workflow structure may suffer transformations to accommodate the computing infrastructures, normally involving task clustering and partitioning. However, these transformations may cause workflow imbalance because of the difference between execution task times (runtime imbalance) or because of unconsidered data dependencies that lead to data locality issues (data imbalance). In this article, to mitigate these imbalances, we enhance the workflow lifecycle process in use by introducing a workflow imbalance phase that quantifies workflow imbalance after the transformations. Our technique is based on structural analysis of Petri nets, obtained by model transformation of a data-intensive workflow, and Linear Programming techniques. Our analysis can be used to assist workflow practitioners in finding more efficient ways of transforming and scheduling their workflows. Moreover, based on our analysis, we also propose a technique to mitigate workflow imbalance by data throttling. Our approach is based on autonomic computing principles that determine how data transmission must be throttled throughout workflow jobs. Our autonomic data-throttling approach mainly monitors the execution of the workflow and recompute data-throttling values when certain watchpoints are reached and time derivation is observed. We validate our approach by a formal proof and by simulations along with the Montage workflow. Our findings show that a dynamic data-throttling approach is feasible, does not introduce a significant overhead, and minimizes the usage of input buffers and network bandwidth.

REANA: A System for Reusable Research Data Analyses

The revalidation, reinterpretation and reuse of research data analyses requires having access to the original computing environment, the experimental datasets, the analysis software, and the computational workflow steps which were used by researchers to produce the original scientific results in the first place. REANA (Reusable Analyses) is a nascent platform enabling researchers to structure their research data analyses in view of enabling future reuse. The analysis is described by means of a YAML file that captures sufficient information about the analysis assets, parameters and processes. The REANA platform consists of a set of micro-services allowing to launch and monitor container-based computational workflow jobs on the cloud. The REANA user interface and the command-line client enables researchers to easily rerun analysis workflows with new input parameters. The REANA platform aims at supporting several container technologies (Docker), workflow engines (CWL, Yadage), shared storage systems (Ceph, EOS) and compute cloud infrastructures (Ku-bernetes/OpenStack, HTCondor) used by the community. REANA was developed with the particle physics use case in mind and profits from synergies with general reusable research data analysis patterns in other scientific disciplines, such as bioinformatics and life sciences.

Producing Madgraph5_aMC@NLO gridpacks and using TensorFlow GPU resources in the CMS HTCondor Global Pool

The CMS experiment has an HTCondor Global Pool, composed of more than 200K CPU cores available for Monte Carlo production and the analysis of da.The submission of user jobs to this pool is handled by either CRAB, the standard workflow management tool used by CMS users to submit analysis jobs requiring event processing of large amounts of data, or by CMS Connect, a service focused on final stage condor-like analysis jobs and applications that already have a workflow job manager in place. The latest scenario canbring cases in which workflows need further adjustments in order to efficiently work in a globally distributed pool of resources. For instance, the generation of matrix elements for high energy physics processes via Madgraph5_aMC@NLO and the usage of tools not (yet) fully supported by the CMS software, such as Ten-sorFlow with GPUsupport, are tasks with particular requirements. A special adaption, either at the pool factory level (advertising GPU resources) or at the execute level (e.g: to handle special parameters that describe certain needs for the remote execute nodes during submission) is needed in order to adequately work in the CMS global pool. This contribution describes the challenges and efforts performed towards adaptingsuch workflows so they can properly profit from the Global Pool via CMS Connect.

A fault-tolerant HPC scheduler extension for large and operational ensemble data assimilation: Application to the Red Sea

A fully parallel ensemble data assimilation and forecasting system has been developed for the Red Sea based on the MIT general circulation model (MITgcm) to simulate the Red Sea circulation and the Data Assimilation Research Testbed (DART) ensemble assimilation software. An important limitation of operational ensemble assimilation systems is the risk of ensemble members’ collapse. This could happen in those situations when the filter update step imposes large corrections on one, or more, of the forecasted ensemble members that are not fully consistent with the model physics. Increasing the ensemble size is expected to improve the assimilation system performances, but obviously increases the risk of members’ collapse. Hardware failure or slow numerical convergence encountered for some members should also occur more frequently. In this context, the manual steering of the whole process appears as a real challenge and makes the implementation of the ensemble assimilation procedure uneasy and extremely time consuming.This paper presents our efforts to build an efficient and fault-tolerant MITgcm-DART ensemble assimilation system capable of operationally running thousands of members. Built on top of Decimate, a scheduler extension developed to ease the submission, monitoring and dynamic steering of workflow of dependent jobs in a fault-tolerant environment, we describe the assimilation system implementation and discuss in detail its coupling strategies. Within Decimate, only a few additional lines of Python is needed to define flexible convergence criteria and to implement any necessary actions to the forecast ensemble members, as for instance (i) restarting faulty job in case of job failure, (ii) changing the random seed in case of poor convergence or numerical instability, (iii) adjusting (reducing or increasing) the number of parallel forecasts on the fly, (iv) replacing members on the fly to enrich the ensemble with new members, etc.We demonstrate the efficiency of the system with numerical experiments assimilating real satellites sea surface height and temperature observations in the Red Sea.

Pengaruh Deskripsi Pekerjaan terhadap Kinerja Pegawai Negeri Sipil pada Kantor Camat Tembilahan


 Job description or work flows is an important thing that needed in every company or organization. The purpose of this study is to determine the effect of job description on civil servants performance in Tembilahan subdistrict office .Independent variable of this research is job description, and dependent variable is performance. Population of this research are 30 Civil Servants workers in Tembilahan subdistrict office.Sampling technique used is total sampling or census. Data analysis used is linear regression. Theresult showed that there is significance influence between job description against civil servants performance in Tembilahan Subdistrict office with the number of adjusted R square 72 percent, means the other 28 percent of employees performance influences by another factors. 
 Keywords: Job Descriptions, Performance

Resource optimised workflow scheduling in Hadoop using stochastic hill climbing technique

Hadoop on datacentre is a popular analytical platform for enterprises. Cloud vendors host Hadoop clusters on the datacentre to provide high performance analytical computing facilities to its customers, who demand a parallel programming model to deal with huge data. Effective cost/time management and ingenious resource consumption among the concurrent users, must be the primary concern without which the key aspiration behind high performance cloud computing would suffer. Workflows portray such high performance applications in terms of individual jobs and dependencies between them. Workflows can be scheduled on virtual machines (VMs) in datacentre to make best possible use of resources. In the authors' earlier work, a mechanism to pack and execute the customer jobs as workflows on Hadoop platform was proposed which minimises the VM cost and also executes the workflow jobs within deadline. In this work, the authors try to optimise certain other parameters such as load on cloud, response time for workflows, resource usage effectiveness by applying soft computing methods. Stochastic hill climbing (SCH) is a soft computing approach used to solve many optimisation problems. In this study, they have employed the SHC approach to schedule workflow jobs to VMs and thereby optimise the above mentioned multiple parameters in cloud datacentre.

IceProd 2 Usage Experience

IceProd is a data processing and management framework developed by the IceCube Neutrino Observatory for processing of Monte Carlo simulations, detector data, and data driven analysis. It runs as a separate layer on top of grid and batch systems. This is accomplished by a set of daemons which process job workflow, maintaining configuration and status information on the job before, during, and after processing. IceProd can also manage complex workflow DAGs across distributed computing grids in order to optimize usage of resources.IceProd has recently been rewritten to increase its scaling capabilities, handle user analysis workflows together with simulation production, and facilitate the integration with 3rd party scheduling tools. IceProd 2, the second generation of IceProd, has been running in production for several months now. We share our experience setting up the system and things we’ve learned along the way.

Prior node selection for scheduling workflows in a heterogeneous system

Many workflow scheduling algorithms for heterogeneous systems have been developed to satisfy multiple requirements such as minimizing schedule length while maximizing throughput. In particular, in list-based scheduling approaches, the schedule length depends on the given nodes as well as the task allocation and ordering policies. This is because the scheduling priority is derived by averaging the execution time and communication time of the given nodes. If the set of nodes can be adjusted before the scheduling tasks, a small schedule length can be achieved. In this paper, we propose a prior node selection algorithm, called lower bound based candidate node selection (LBCNS) to select a subset of given nodes to minimize the schedule length while fairly scheduling each job. Our proposal has two approaches: (i) LBCNS_DEFAULT, which considers the job characteristics and each node’s performance, and (ii) priority-based LBCNS, which additionally takes each scheduling priority into account for a dedicated task scheduling algorithm.The experimental results of extensive simulations show that LBCNS_DEFAULT has the best fairness for scheduling multiple workflow jobs, while priority-based LBCNS achieves the minimum schedule length with the highest efficiency for a single workflow job and multiple workflow jobs.

Web-based bioinformatics workflows for end-to-end RNA-seq data computation and analysis in agricultural animal species.

BackgroundRemarkable advances in Next Generation Sequencing (NGS) technologies, bioinformatics algorithms and computational technologies have significantly accelerated genomic research. However, complicated NGS data analysis still remains as a major bottleneck. RNA-seq, as one of the major area in the NGS field, also confronts great challenges in data analysis.ResultsTo address the challenges in RNA-seq data analysis, we developed a web portal that offers three integrated workflows that can perform end-to-end compute and analysis, including sequence quality control, read-mapping, transcriptome assembly, reconstruction and quantification, and differential analysis. The first workflow utilizes Tuxedo (Tophat, Cufflink, Cuffmerge and Cuffdiff suite of tools). The second workflow deploys Trinity for de novo assembly and uses RSEM for transcript quantification and EdgeR for differential analysis. The third combines STAR, RSEM, and EdgeR for data analysis. All these workflows support multiple samples and multiple groups of samples and perform differential analysis between groups in a single workflow job submission. The calculated results are available for download and post-analysis. The supported animal species include chicken, cow, duck, goat, pig, horse, rabbit, sheep, turkey, as well as several other model organisms including yeast, C. elegans, Drosophila, and human, with genomic sequences and annotations obtained from ENSEMBL.The RNA-seq portal is freely available from http://weizhongli-lab.org/RNA-seq.ConclusionsThe web portal offers not only bioinformatics software, workflows, computation and reference data, but also an integrated environment for complex RNA-seq data analysis for agricultural animal species. In this project, our aim is not to develop new RNA-seq tools, but to build web workflows for using popular existing RNA-seq methods and make these tools more accessible to the communities.