Job Execution Research Articles

A fix-and-optimize heuristic for the Unrelated Parallel Machine Scheduling Problem

This paper proposes and evaluates a matheuristic approach for the Unrelated Parallel Machine Scheduling Problem (UPMSP). The UPMSP consists of assigning jobs to unrelated parallel machines considering different processing times for the same job in different machines. Additionally, a setup time is considered between the execution of jobs in the same machine. The problem is addressed by a fix-and-optimize matheuristic that iteratively selects a subset of variables to be fixed to their current values so that the remaining variables will compose a subproblem to be optimized by a mathematical programming solver. In the proposed approach, each subproblem consists of a subset of jobs that are assigned to a subset of machines in the incumbent solution. The subproblems are solved by the state-of-the-art exact algorithm for the UPMSP. In the experiments conducted on benchmark instances, the proposed fix-and-optimize algorithm achieved remarkable results. It outperformed the standalone exact algorithm by a large margin and resulted in competitive solutions when compared to the literature’s best-performing heuristic method for the problem. The proposed algorithm obtained the best solution for 669 out of the 1000 instances addressed in this work. Among them, 338 are new best-known solutions. In general, the proposed approach excels at solving instances with a high number of jobs per machine — it resulted in the best solution for 89% of the instances with a ratio of 10 or more jobs per machine in total.

An online bi-objective scheduling algorithm for service provisioning in cloud computing

To reduce costs and risks, service providers can purchase Infrastructure-as-a-Service (IaaS) instances from public clouds to deploy service applications and deliver job-execution services to users. However, it is challenging to make optimal instance purchasing and job scheduling decisions due to diverse instance billing methods and random service demands. In order to minimize time average service costs while ensuring quality of service (QoS), we consider that service providers purchase combinations of reserved and on-demand instances to meet the workload of their service system. Service providers need to dynamically determine the amount of on-demand and reserved instances to purchase for job executions, and then schedule jobs to run at the appropriate time. In this paper, we first propose a method to dynamically purchase reserved instances based on the historical usage data of on-demand and reserved instances. Then, we use a variable-length cycle scheduling mode for the job processing system and propose an online bi-objective scheduling algorithm based on Lyapunov optimization to schedule jobs. Even without the knowledge about the future, our online algorithm can dynamically purchase on-demand instances in an optimal way under the current number of reserved instances, and schedule jobs in varying time periods according to the arrival of jobs and optimization objectives. Based on both synthetic and real-world data, extensive experimental results show that our algorithm can achieve bi-objective optimization of minimizing time average cost while ensuring QoS.

An improved gene expression programming algorithm for function mining of map-reduce job execution in catenary monitoring systems.

Gene expression programming (GEP) is one of the most prominent algorithms in function mining. In order to obtain a more accurate function model in configuration parameters-execution efficiency (CP-EE) of map-reduce job in the high-speed railway catenary monitoring system, this paper proposes a novel algorithm, called GEP based on multi-strategy (MS-GEP). Compared to traditional GEP, the proposed algorithm can escape premature convergence and jump out of local optimum. First, an adaptive mutation rate is designed according to the evolutionary generations, population diversity, and individual fitness values. A manual intervention strategy is then proposed to determine whether the algorithm enters the dilemma of local optimum based on the generations of population evolutionary stagnation. Finally, the average quality of the population is changed by randomly replacing individuals, and the ancestral population is traced to change the evolutionary direction. The experimental results on the benchmarks of function mining show that the proposed MS-GEP has better solution quality and higher population diversity than other GEP algorithms. Furthermore, the proposed MS-GEP has higher accuracy on the function model of CP-EE of high-speed railway catenary monitoring system than other commonly used algorithms in the field of function mining.

ILIAD: a suite of automated Snakemake workflows for processing genomic data for downstream applications

BackgroundProcessing raw genomic data for downstream applications such as imputation, association studies, and modeling requires numerous third-party bioinformatics software tools. It is highly time-consuming and resource-intensive with computational demands and storage limitations that pose significant challenges that increase cost. The use of software tools independent of one another, in a disjointed stepwise fashion, increases the difficulty and sets forth higher error rates because of fragmented job executions in alignment, variant calling, and/or build conversion complications. As sequencing data availability grows, the ability for biologists to process it using stable, automated, and reproducible workflows is paramount as it significantly reduces the time to generate clean and reliable data.ResultsThe Iliad suite of genomic data workflows was developed to provide users with seamless file transitions from raw genomic data to a quality-controlled variant call format (VCF) file for downstream applications. Iliad benefits from the efficiency of the Snakemake best practices framework coupled with Singularity and Docker containers for repeatability, portability, and ease of installation. This feat is accomplished from the onset with download acquisitions of any raw data type (FASTQ, CRAM, IDAT) straight through to the generation of a clean merged data file that can combine any user-preferred datasets using robust programs such as BWA, Samtools, and BCFtools. Users can customize and direct their workflow with one straightforward configuration file. Iliad is compatible with Linux, MacOS, and Windows platforms and scalable from a local machine to a high-performance computing cluster.ConclusionIliad offers automated workflows with optimized time and resource management that are comparable to other workflows available but generates analysis-ready VCF files from the most common datatypes using a single command. The storage footprint challenge of genomic data is overcome by utilizing temporary intermediate files before the final VCF is generated. This file is ready for use in imputation, genome-wide association study (GWAS) pipelines, high-throughput population genetics studies, select gene candidate studies, and more. Iliad was developed to be portable, compatible, scalable, robust, and repeatable with a simplistic setup, so biologists that are less familiar with programming can manage their own big data with this open-source suite of workflows.

Scheduling on identical machines with preemption and setup times

In this paper, we address the problem of scheduling jobs on identical machines for minimising the maximum completion time (makespan). Each job requires a sequence-independent setup time, which represents the time needed to prepare the machines for job execution. Then, we introduce a dynamic programme to solve the case with two machines, and show that this problem admits a fully polynomial time approximation scheme. For the case of m machines, we propose heuristics and an adapted genetic algorithm. Some numerical experiments are done to evaluate the proposed algorithms.

Assessing the Impact of a Hybrid Work Model on Job Execution, Work-Life Balance, and Employee Satisfaction in a Technology Company

This study delves into the hybrid work model, focusing on the perspectives and experiences of employees within a Quezon City-based Technology Company specializing in industrial software and engineering. It contributes to the organization's goal of being a "Great Place to Work" by adapting to change while valuing well-being. The organization is deeply dedicated to creating a workplace where employee well-being, career growth, and overall happiness are top priorities. The study surveyed 354 employees to understand the impact of the hybrid work paradigm on individual and team deliverables. The study found that 84.4% of respondents believed the hybrid work model was effective in various ways. This includes job execution and teamwork, work-life balance and satisfaction, communication and collaboration, and overall experience. The survey result showed that the model is widely praised for its flexibility, improving work-life balance, and job satisfaction. Respondents agree that the hybrid model balances professional and personal well-being despite managerial expectations and workloads. The study also emphasizes the importance of communication in hybrid work teamwork. The research recognizes the benefits of remote communication tools but stresses the importance of face-to-face interactions, collaborative engagements, and team dynamics. The result of this study can be used to bring in new ideas and potential growth for the company, allowing it to develop innovative strategies and improve employee satisfaction. Embracing the concept of the hybrid work setup is a catalyst for success and ensures the well-being of all stakeholders.

Configuration-Driven No-Code Data Ingestion Engine with RuleBased Processing

The efficient management of batch processes is critical for modern data-driven organizations, especially within the context of complex data ingestion and processing workflows. This abstract explores the essential components and functionalities required to streamline batch job execution within such environments. Firstly, a robust batch management system is essential, providing capabilities for job scheduling, dependency management, and error handling. Such a system orchestrates the execution of batch jobs, ensuring they run in the appropriate sequence and handle dependencies effectively. Additionally, real-time event-driven triggers enable immediate batch job initiation upon file drops, ensuring timely processing of incoming data and swift response to business needs. Conversely, scheduled batch executions provide predictability and automation, allowing organizations to optimize resource utilization and maintain operational efficiency by running batches at predetermined intervals or specific times. Furthermore, the abstract delves into the necessity of a comprehensive configuration setup to support various data formats, transformation requirements, and validation criteria. A rich configuration engine allows for seamless mapping, transformation, and validation of incoming data, enabling flexibility and adaptability to diverse data sources and requirements. It also emphasizes the importance of containerization for batch job execution, ensuring scalability, resource isolation, and efficient utilization of computing resources. Lastly, the provision of job IDs facilitates seamless invocation of batch jobs by external systems, enabling streamlined interaction and integration across different platforms. Together, these components form a cohesive framework for efficient batch job management, empowering organizations to optimize data processing workflows and derive actionable insights from their data assets.

Methodology of evaluating ATLAS computing resources in handling user analysis workflows

The distributed computing infrastructure of the ATLAS Experiment includes over 170 sites and executes up to 3 million computing jobs daily. PanDA (Production and Distributed Analysis) is the Workload Management System responsible for task and job execution; its key components are the broker and job scheduler that define the mapping of computing jobs to the resources. The optimization of this mapping is crucial for handling the expected computational payloads during the HL-LHC era. Considering the heterogeneity and the distributed structure of the Worldwide LHC Computing Grid (WLCG) infrastructure that provides computing resources for analyzing the data, there is a need for specific approaches for evaluating computing resources according to their ability to process different types of workflows. This evaluation can potentially enhance the efficiency of the Grid by optimally distributing different types of payloads in heterogeneous computing environments. To tackle this challenge, this research proposes a method for evaluating WLCG resources regarding their ability to process user analysis payloads. This evaluation is based on leveraging available information about job execution on PanDA queues within the ATLAS computing environment.

Integrated optimization of production scheduling and maintenance planning with dynamic job arrivals and mold constraints

Effective management of production often requires harmonizing maintenance and production activities. However, decsions on production scheduling and machine maintenance are usually made independently, leading to potential overlaps and inefficiencies. This paper joinly investigates the production scheduling and maintenance planning for a production machine subject to different types of jobs that arrive randomly. A unique emphasis is placed on the prerequisite of loading specific molds prior to job execution, an element often overlooked in previous studies. Maintenance considerations employ the reliability/availability framework. The overarching goal is the creation of an integrated schedule that minimizes the weighted sum of total costs and the machine's maximum unavailability. To address this intricate challenge, a novel hybrid differential evolution and genetic algorithm (DE-GA) is proposed, complemented by a solution refinement strategy. Performance evaluations indicate that DE-GA methodology consistently outperforms Gurobi solver and four other prevalent algorithms across different test instances.

Retrofitted Autonomous Inflow Control Devices Optimize Completions Design

_ This article, written by JPT Technology Editor Chris Carpenter, contains highlights of paper OTC 31301, “Optimized Completions Design Using Retrofit Autonomous Inflow Control Devices,” by Ezinne A. Nnebocha, Schlumberger; Akinola Akinbola, NPDC; and Omagbemi G. Kayakor, First Hydrocarbon Nigeria, et al. The paper has not been peer reviewed. Copyright 2021 Offshore Technology Conference. Reproduced by permission. _ Discovered in 1964, the Beta Field in the Niger Delta sedimentary basin consists of 25 stacked hydrocarbon-bearing reservoirs between 5,500 and 12,000 ft true vertical depth subsea (TVDSS). Oil production peaked at 8,900 STB/D shortly after field startup and has been on the decline. Two horizontal wells, Beta 7 and Beta 8, were planned in the field, but a production forecast of the planned wells showed potential early water breakthrough and high water cut. Autonomous inflow control devices (AICDs) were deployed successfully in the Beta Field for the purpose of minimizing water production and maximizing oil recovery. Introduction After history matching, the two new horizontal wells were simulated to target the sweet spots in the reservoir using constraints of oil saturation, porosity and permeability limits, and distance to the oil/water contact (OWC). The history-matched model was then used in forecasting the production profile for the new wells using a target liquid production rate of 2,000 STB/D. The production forecast without inflow control showed water breakthrough in the new wells within the first month of production, reaching to 45% within 3 months of production. Cyclonic AICD completions then were simulated for Beta-7 and Beta-8 horizontal wells, which showed improved oil production and delayed water production. AICD Retrofit Design Cyclonic AICDs were selected for application in the Beta Field to improve production performance. However, the short lead time required for the job execution and border closures caused by the COVID-19 pandemic created restrictions in mobilizing personnel and equipment from outside the country. It did not appear that the equipment could arrive in time for the job. The conventional nozzle-based passive ICD joints were available in country. Therefore, leveraging engineering skills and operational knowledge, a solution to retrofit AICD nozzles in existing ICD screen joints was conceived. The engineering team developed a design of machined components that could be quickly manufactured at the facility of the original equipment manufacturer (OEM) and shipped to the location. Failure-mode and effects-analysis assessment, and subsequent risk-management procedures, became useful to manage the process locally with the help of the remote OEM team. A step-by-step procedure on the retrofit was devised. Fig. 1 shows a flow chart used to guide the retrofitting process and the retrofitted AICD joints after modification. Upon completion of the screen retrofit, the joints were quality-checked and inspected to verify their preparedness for well completion.

Simulation of Job Execution in Distributed Heterogeneous Computing Infrastructures

Simulation of Job Execution in Distributed Heterogeneous Computing Infrastructures

Improving prediction of computational job execution times with machine learning

SummaryPredicting resource consumption and run time of computational workloads is crucial for efficient resource allocation, or cost and energy optimization. In this paper, we evaluate various machine learning techniques to predict the execution time of computational jobs. For experiments we use datasets from two application areas: scientific workflow management and data processing in the ALICE experiment at CERN. We apply a two‐stage prediction method and evaluate its performance. Other evaluated aspects include: (1) comparing performance of global (per‐workflow) versus specialized (per‐job) models; (2) impact of prediction granularity in the first stage of the two‐stage method; (3) using various feature sets, feature selection, and feature importance analysis; (4) applying symbolic regression in addition to classical regressors. Our results provide new valuable insights on using machine learning techniques to predict the runtime behavior of computational jobs.

Analysis of Energy and Network Cost Effectiveness of Scheduling Strategies in Datacentre

Abstract In parallel and distributed computing, cloud computing is progressively replacing the traditional computing paradigm. The cloud is made up of a set of virtualized resources in a data center that can be configured according to users’ needs. In other words, cloud computing faces the problem of a huge number of users requesting unlimited jobs for execution on a limited number of resources, which increases energy consumption and the network cost of the system. This study provides a complete analysis of classic scheduling techniques specifically for handling data-intensive workloads to see the effectiveness of the energy and network costs of the system. The workload is selected from a real-world data center. Moreover, this study offers the pros and cons of several classical heuristics-based job scheduling techniques that take into account the time and cost of transferring data from multiple sources. This study is useful for selecting appropriate scheduling techniques for appropriate environments.

MODELLING THE EFFECT OF CONTAMINATION ON THE RHEOLOGICAL PROPERTIES OF CEMENT SLURRY USING OIL-BASED MUD AS CONTAMINANT, TO IMPROVE CEMENTING JOB DESIGN AND EXECUTION IN THE OIL & GAS INDUSTRY

Cementing job failure often occurs due to cement slurry contamination with drilling mud during primary cementing and remedial cementing jobs in the oil & gas industry. The enormous consequences of cement slurry contamination with drilling mud include drastic reduction in the thickening time of cement slurry and increased rheological properties of the cement slurry due to the much more viscous resultant cement slurry/mud mixture often leading to incomplete cement job, poor mud removal, poor cement bond, gas channeling, gas kick, blow out, downtime, revenue loss and remedial cementing. The rheological properties of cement slurry and drilling mud are strongly dependent on temperature. There are also emergency situations in the oilfield where bottomhole temperature changes to a higher temperature requiring adjustments in the original cementing job design to include the new bottomhole temperature when Laboratory services are not immediately available. The aforesaid and the associated cost of remedial operations prompted this work to investigate the effect of temperature and contamination on the rheological properties of cement slurry using oil-based mud as contaminant to invent a process to better avoid cement slurry contamination and to improve cementing job designs and execution in the oil & gas industry. The methods applied include Laboratory testing and mathematical modeling. Mathematical models were developed using Microsoft Excel to predict the rheological properties of contaminated cement slurries as a function of temperature and contamination with oil-based mud as the contaminant. Results showed that the Plastic viscosity and yield stress of the contaminated cement slurries increased with temperature and extent of contamination with oil-based mud but the yield stress also decreased with extent of contamination at high temperatures. The very high regression coefficient obtained with each of the mathematical models showed that the mathematical models can be confidently applied to predict the rheological properties (Plastic viscosity and yield stress) of the contaminated cement slurries as a function of temperature and extent of contamination with the oil-based mud.

Building a Collaborative Data Analytics System: Opportunities and Challenges

Real-time collaboration has become increasingly important in various applications, from document creation to data analytics. Although collaboration features are prevalent in editing applications, they remain rare in data-analytics applications, where the need for collaboration is even more crucial. This tutorial aims to provide attendees with a comprehensive understanding of the challenges and design decisions associated with supporting real-time collaboration and user interactions in data analytics systems. We will discuss popular conflict resolution technologies, the unique challenges of facilitating collaborative experiences during the workflow construction and execution phases, and the complexities of supporting responsive user interactions during job execution.

Hydra: Deadline-Aware and Efficiency-Oriented Scheduling for Deep Learning Jobs on Heterogeneous GPUs

With the rapid proliferation of deep learning (DL) jobs running on heterogeneous GPUs, scheduling DL jobs to meet various scheduling requirements, such as meeting deadlines and reducing job completion time (JCT), is critical. Unfortunately, existing efficiency-oriented and deadline-aware efforts are still rudimentary. They lack the capability of scheduling jobs to meet deadline requirements while reducing total JCT, especially when the jobs have various execution times on heterogeneous GPUs. Therefore, we present Hydra, a novel quantitative cost comparison approach, to address this scheduling issue. Here, the cost represents the total JCT plus a dynamic penalty calculated from the total tardiness (i.e., the delay time of exceeding the deadline) of all jobs. Hydra adopts a sampling approach that exploits the inherent iterative periodicity of DL jobs to estimate job execution times accurately on heterogeneous GPUs. Then, Hydra considers various combinations of job sequences and GPUs to obtain the minimized cost by leveraging an efficient branch-and-bound algorithm. Finally, the results of evaluation experiments on Alibaba traces show that Hydra can reduce total tardiness by 85.8% while reducing total JCT as much as possible, compared with state-of-the-art efforts.

Reasoning About Expected Job Completion Time in Dynamic Vehicular Clouds

Roughly a decade ago, inspired by the phenomenal success of cloud computing, a group of researchers have defined Vehicular Clouds as a group of vehicles whose sensing, communication, and computing resources can be coordinated and allocated to authorized users. While both conventional and Vehicular Clouds are instances of utility computing, a number of important characteristics set vehicular clouds apart from their conventional counterparts. These characteristics include the mobility of vehicles and the volatility of resources that fluctuate with the arrival and departure of vehicles. As in the conventional version of cloud computing, approximating job completion time is one of the key performance metrics of interest. Unfortunately, estimating job completion time with any degree of accuracy and confidence requires complete knowledge of the distributions of several relevant random variables. Typically, however, these probability distributions are not known. Luckily, in many cases of practical relevance, accumulated empirical evidence allows to estimate the first few moments of these random variables. The main contribution of this paper is to offer an accurate approximation of the expected job completion time in Dynamic Vehicular Clouds built on top of vehicles on a highway. For this purpose, we rely on empirical estimates of the first moment of the user job execution time in the absence of any overhead attributable to the Dynamic Vehicular Cloud itself. Our extensive simulations have confirmed that our approximations of the expected job execution time are very accurate.

Towards General and Efficient Online Tuning for Spark

The distributed data analytic system - Spark is a common choice for processing massive volumes of heterogeneous data, while it is challenging to tune its parameters to achieve high performance. Recent studies try to employ auto-tuning techniques to solve this problem but suffer from three issues: limited functionality, high overhead, and inefficient search. In this paper, we present a general and efficient Spark tuning framework that can deal with the three issues simultaneously. First, we introduce a generalized tuning formulation, which can support multiple tuning goals and constraints conveniently, and a Bayesian optimization (BO) based solution to solve this generalized optimization problem. Second, to avoid high overhead from additional offline evaluations in existing methods, we propose to tune parameters along with the actual periodic executions of each job (i.e., online evaluations). To ensure safety during online job executions, we design a safe configuration acquisition method that models the safe region. Finally, three innovative techniques are leveraged to further accelerate the search process: adaptive sub-space generation, approximate gradient descent, and meta-learning method. We have implemented this framework as an independent cloud service, and applied it to the data platform in Tencent. The empirical results on both public benchmarks and large-scale production tasks demonstrate its superiority in terms of practicality, generality, and efficiency. Notably, this service saves an average of 57.00% memory cost and 34.93% CPU cost on 25K in-production tasks within 20 iterations, respectively.

Juggling During Corporate Change: How Executive Job Demands Affect Corporate Social Irresponsibility

Juggling During Corporate Change: How Executive Job Demands Affect Corporate Social Irresponsibility

Noise Exposure and Job Stress – a Structural Equation Model in Textile Industries

Noise exposure is one of the most important physical agents in the workplace which can induce job stress in several ways. The aim of this study was to model the interactions between independent and mediating variables and job stress using structural equation modeling. In this study, Weinstein’s noise sensitivity scale, noise annoyance questionnaire, Health and Safety Executive (HSE) job stress questionnaire and job satisfaction scale were used. To assess worker’s noise exposure, the 8-hours equivalent continuous A-weighted sound pressure level (LAeq;8 h), was measured based on ISO 9612 (2009). To achieve the aims of study, the structural equation model was run using R software 3.4.1 and Cytoscape software 3.6.0. Based on the results, while there was a direct positive correlation of noise exposure on total job stress, there were also indirect positive effects through job satisfaction and noise sensitivity as mediator variables. Using hearing protective devices negatively affected total job stress through a direct pathway and an indirect pathway when job satisfaction was a mediator variable. Regarding the total effect of noise exposure and using hearing protection devices on job stress subscales, it can be concluded that noise exposure and using hearing protection devices had greatest effect on colleagues support and demand, respectively. It can be concluded that noise exposure and lack of hearing protective devices have a significant positive effect on job stress among workers of a textile industry. In addition to the direct effect, this factor can induce job stress through noise sensitivity, job satisfaction and noise annoyance. Therefore, measures which can decrease any of the mentioned factors, also can alleviate job stress.