Batch Jobs Research Articles

Keep Your Distributed Data Warehouse Consistent at a Minimal Cost

Large data warehouses store interdependent tables that are updated independently in response to business logic changes or late arrival of critical data. To keep the warehouse consistent, changes to upstream tables need to be propagated to downstream tables in a timely fashion. However, a naive change propagation algorithm can cause many unnecessary updates or recalculations of downstream tables, which drives up the cost of data warehouse management. In this paper, we describe our solution that can ensure the eventual consistency of the data warehouse while avoiding unnecessary table updates. We also show that the optimal trade-off between computational cost reduction and meeting data freshness constraints can be found by solving a dynamic programming problem. The proposed solution is currently in production to manage the YouTube Data Warehouse and has reduced update requests by 25% by eliminating non-trivial duplicates. These requests would have been carried out by large batch jobs over big data. Eliminating them has led to a proportionate reduction in computing resources. One key advantage of our approach is that it can be used in a heterogeneous, distributed data warehouse environment where the operator software may not have complete control over the query processors. This is because our approach only relies on having dependency information for tables and can operate on the post-state of data sources.

Coordinated production and delivery scheduling for service-oriented manufacturing systems with deterioration effect

Service-oriented manufacturing (SOM) is a new industrial model that combines manufacturing with service. To enhance the competitiveness of manufacturers in SOM systems, we investigate a coordinated production and delivery scheduling problem considering a third-party logistics (3PL) provider. The considered problem can be divided into two phases, including production phase and delivery phase. In the production phase, jobs are processed by multiple manufacturers located in different geographical zones. Jobs are with deteriorating processing time and set-up time. Several structural properties of the problem in the production phase are proposed, based on which a heuristic algorithm is designed to make decisions on job batching and batch sequencing. After being processed, the finished jobs are immediately delivered to a customer by manufacturers’ own vehicles or the 3PL provider. A dynamic programming algorithm is developed to solve the problem in the delivery phase. The integrated scheduling problem is proved to be NP-hard. Hence, we develop a general variable neighbourhood search (GVNS) to solve the problem in a reasonable time. The basic variable neighbourhood descent (BVND) is used as a local search method in the GVNS. Finally, the computational experiments are conducted and the results verify the effectiveness and stability of the proposed approach.

Versatile software-defined HPC and cloud clusters on Alps supercomputer for diverse workflows

Supercomputers have been driving innovations for performance and scaling benefiting several scientific applications for the past few decades. Yet their ecosystems remain virtually unchanged when it comes to integrating distributed data-driven workflows, primarily due to rather rigid access methods and restricted configuration management options. X-as-a-Service model of cloud has introduced, among other features, a developer-centric DevOps approach empowering developers of infrastructure, platform to software artefacts, which, unfortunately contemporary supercomputers still lack. We introduce vClusters (versatile software-defined clusters), which is based on Infrastructure-as-code (IaC) technology. vClusters approach is a unique fusion of HPC and cloud technologies resulting in a software-defined, multi-tenant cluster on a supercomputing ecosystem, that, together with software-defined storage, enable DevOps for complex, data-driven workflows like grid middleware, alongside a classic HPC platform. IaC has been a commonplace in cloud computing, however, it lacked adoption within multi-Petascale ecosystems due to concerns related to performance and interoperability with classic HPC data centres’ ecosystems. We present an overview of the Swiss National Supercomputing Centre’s flagship Alps ecosystem as an implementation target for vClusters for HPC and data-driven workflows. Alps is based on the Cray-HPE Shasta EX supercomputing platform that includes an IaC compliant, microservices architecture (MSA) management system, which we leverage for demonstrating vClusters usage for our diverse operational workflows. We provide implementation details of two operational vClusters platforms: a classic HPC platform that is used predominantly by hundreds of users running thousands of large-scale numerical simulations batch jobs; and a widely used, data-intensive, Grid computing middleware platform used for CERN Worldwide LHC Computing Grid (WLCG) operations. The resulting solution showcases reuse and reduction of common configuration recipes across vCluster implementations, minimising operational change management overheads while introducing flexibility for managing artefacts for DevOps required by diverse workflows.

Automatic Preprocessing Pipeline for White Matter Functional Analyses of Large-Scale Databases.

Recently, increasing evidence suggests that fMRI signals in white matter (WM), conventionally ignored as nuisance, are robustly detectable using appropriate processing methods and are related to neural activity, while changes in WM with aging and degeneration are also well documented. These findings suggest variations in patterns of BOLD signals in WM should be investigated. However, existing fMRI analysis tools, which were designed for processing gray matter signals, are not well suited for large-scale processing of WM signals in fMRI data. We developed an automatic pipeline for high-performance preprocessing of fMRI images with emphasis on quantifying changes in BOLD signals in WM in an aging population. At the image processing level, the pipeline integrated existing software modules with fine parameter tunings and modifications to better extract weaker WM signals. The preprocessing results primarily included whole-brain time-courses, functional connectivity, maps and tissue masks in a common space. At the job execution level, this pipeline exploited a local XNAT to store datasets and results, while using DAX tool to automatic distribute batch jobs that run on high-performance computing clusters. Through the pipeline, 5,034 fMRI/T1 scans were preprocessed. The intraclass correlation coefficient (ICC) of test-retest experiment based on the preprocessed data is 0.52 - 0.86 (N=1000), indicating a high reliability of our pipeline, comparable to previously reported ICC in gray matter experiments. This preprocessing pipeline highly facilitates our future analyses on WM functional alterations in aging and may be of benefit to a larger community interested in WM fMRI studies.

Exact Algorithm for Batch Scheduling on Unrelated Machine

In this paper, we propose a new linear algorithm to tackle a specific class of unrelated machine scheduling problem, considered as an important real-life situation, which we called Batch Scheduling on Unrelated Machine (BSUM), where we have to schedule a batch of identical and non-preemptive jobs on unrelated parallel machines. The objective is to minimize the makespan (Cmax) of the whole schedule. For this, a mathematical formulation is made and a lower bound is computed based on the potential properties of the problem in order to reduce the search space size and thus accelerate the algorithm. Another property is also deducted to design our algorithm that solves this problem. The latter is considered as a particular case of RmCmax family problems known as strongly NP-hard, therefore, a polynomial reduction should realize a significant efficiency to treat them. As we will show, Batch BSUM is omnipresent in several kind of applications as manufacturing, transportation, logistic and routing. It is of major importance in several company activities. The problem complexity and the optimality of the algorithm are reported, proven and discussed.

Comparison of Real-Time and Batch Job Recommendations

Collaborative filtering recommendation systems are traditionally trained in a batch manner and are designed to produce personalized recommendations for a large number of users at the same time. However, in many industrial use cases, it is reasonable to produce recommendations in real time, taking account of very recent user interactions. In this work, we present the implementation of batch and real-time recommendation systems using the example of the RP3Beta model, a simple scalable graph-based model that outperforms multiple more advanced models. Our approach can be utilized by any recommendation system if user-to-item recommendations can be obtained based on item-to-item recommendations. We show that it covers multiple common recommendation models, especially collaborative filtering approaches where user features are not available. We also provide the results of A/B tests comparing these two approaches in a real-world scenario of a job recommendation task, conducted with almost 200,000 OLX users. We report at least 10% more users applying for recommended job ads when using a real-time instead of a batch approach. We believe that our results can help other organizations to take informed decisions about whether to make the effort of moving from a batch to a real-time recommendation setting.

Modernization of Legacy Batch Processes: An Upgrade to Prolonged Batch Runs with Advanced Computing and Parallel Processing Techniques

In an era dominated by the need for speed and efficiency, long-running batch jobs in legacy systems significantly hinder organizational performance and scalability. This article explores the transformation of these batch jobs through the use of advanced computing and parallel processing techniques. We delve into the challenges posed by outdated batch processing methods and present solutions that leverage modern technology to enhance execution speed, reduce resource consumption, and improve overall system responsiveness.

A pipeline architecture for feature-based unsupervised clustering using multivariate time series from HPC jobs

Time series are key across industrial and research areas for their ability to model behaviour across time, making them ideal for a wide range of use cases such as event monitoring, trend prediction or anomaly detection. This is even more so due to the increasing monitoring capabilities in many areas, with the subsequent massive data generation. But it is also interesting to consider the potential of time series for Machine Learning processing, often fused with Big Data, to search for useful information and solve real-world problems. However, time series can be studied individually, representing a single entity or variable to be analysed, or in a grouped fashion, to study and represent a more complex entity or scenario. In this latter case we are dealing with multivariate time series, which usually imply different approaches when dealt with. In this paper, we present a pipeline architecture to process and cluster multiple groups of multivariate time series. To implement this, we apply a multi-process solution composed by a feature-based extraction stage, followed by a dimension reduction, and finally, several clustering algorithms. The pipeline is also highly configurable in terms of the stage techniques to be used, allowing to perform a search with several combinations for the most promising results. The pipeline has been experimentally applied to batches of HPC jobs from different users of a supercomputer, with the multivariate time series coming from the monitoring of several node resource metrics. The results show how it is possible to apply this multi-process information fusion to create different meaningful clusters from the batches, using only the time series, without any labelling information, thus being an unsupervised scenario. Optionally, the pipeline also supports an outlier detection stage to find and separate jobs that are radically different when compared to others on a dataset. These outliers can be removed for a better clustering, and later reviewed looking for anomalies, or if numerous, fed back to the pipeline to identify possible groupings. The results also include some outliers found in the experiments, as well as scenarios where they are clustered, or ignored and not removed at all. In addition, by leveraging Big Data technologies like Spark, the pipeline is proven to be scalable by working with up to hundreds of jobs and thousands of time series.

QoS-Aware Co-Scheduling for Distributed Long-Running Applications on Shared Clusters

To achieve a high degree of resource utilization, production clusters need to co-schedule diverse workloads – including both batch analytic jobs with short-lived tasks and long-running applications (LRAs) that execute for a long time frame from hours to months – onto the shared resources. Microservice architecture advances the manifestation of distributed LRAs (DLRAs), comprising multiple interconnected microservices that are executed in long-lived distributed containers and serve massive user requests. Detecting and mitigating QoS violation become even more intractable due to the network uncertainties and latency propagation across dependent microservices. However, current resource managers are only responsible for resource allocation among applications/jobs but agnostic to runtime QoS such as latency at application level. The state-of-the-art QoS-aware scheduling approaches are dedicated for monolithic applications, without considering the temporal-spatio performance variability across distributed microservices. In this paper, we present <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Toposch</small> , a new scheduling and execution framework to prioritize the QoS of DLRAs whilst balancing the performance of batch jobs and maintaining high cluster utilization through harvesting idle resources. <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Toposch</small> tracks footprints of every single request across microservices and uses critical path analysis, based on the end-to-end latency graph, to identify microservices that have high risk of QoS violation. Based on microservice and node level risk assessment, we intervene the batch scheduling by adaptively reducing the visible resources to batch tasks and thus delaying their execution to give way to DLRAs. We propose a prediction-based vertical resource auto-scaling mechanism, with the aid of resource-performance modeling and fine-grained resource inference and access control, for prompt recovery of QoS violation. A cost-effective task preemption is leveraged to ensure a low-cost task preemption and resource reclamation during the auto-scaling. <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Toposch</small> is integrated with Apache YARN and experiments show that <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Toposch</small> outperforms other baselines in terms of performance guarantee of DLRAs, at an acceptable cost of batch job slowdown. The tail latency of DLRAs is merely 1.12x of the case of executing alone on average in <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Toposch</small> with a 26% JCT increase of Spark analytic jobs.

An SMDP approach for Reinforcement Learning in HPC cluster schedulers

Deep reinforcement learning applied to computing systems has shown potential for improving system performance, as well as faster discovery of better allocation strategies. In this paper, we map HPC batch job scheduling to the SMDP formalism, and present an online, deep reinforcement learning-based solution that uses a modification of the Proximal Policy Optimization algorithm for minimizing job slowdown with action masking, supporting large action spaces. In our experiments, we assess the effects of noise in run time estimates in our model, evaluating how it behaves in small (64 processors) and large (16384 processors) clusters. We also show our model is robust to changes in workload and in cluster sizes, showing transfer works with changes of cluster size of up to 10×, and changes from synthetic workload generators to supercomputing workload traces. In our experiments, the proposed model outperforms learning models from the literature and classic heuristics, making it a viable modeling approach for robust, transferable, learning scheduling models.

Independent double DQN-based multi-agent reinforcement learning approach for online two-stage hybrid flow shop scheduling with batch machines

Two-stage hybrid flow shop scheduling with batch machines and jobs arriving over time is complex and challenging in various real-world production scenarios. For the online scheduling problem, traditional heuristic rules can quickly respond to dynamically arrived jobs, while with poor and unstable performance. To close the research gap in the problem, this paper proposes an independent double deep-q-network-based multi-agent reinforcement learning (MA-IDDQN) approach to produce an adaptive rule for batch forming and scheduling. Specifically, the online scheduling problem is transformed into a cooperative Markov decision process by defining state space, action space, and reward function for different agents. Then, two agents are constructed and trained via double DQN to address the batch forming task and scheduling task respectively. Meanwhile, multi-agent cooperates through the behavior analysis mechanism among agents. Moreover, we designed a ε-greedy policy considering waiting in batch forming to make a reasonable decision through historical data. To validate the proposed approach, 27 instances with different scales are settled and contrasted. By comparing with frequently-used heuristic rules and other deep reinforcement learning methods, the experimental results demonstrate that the MA-IDDQN can integrate online batch forming and scheduling to minimize the total tardiness time effectively.

Parallel batching with multi-size jobs and incompatible job families

Parallel batch scheduling has many applications in the industrial sector, like in material and chemical treatments, mold manufacturing and so on. The number of jobs that can be processed on a machine mostly depends on the shape and size of the jobs and of the machine. This work investigates the problem of batching jobs with multiple sizes and multiple incompatible families. A flow formulation of the problem is exploited to solve it through two column generation-based heuristics. First, the column generation finds the optimal solution of the continuous relaxation, then two heuristics are proposed to move from the continuous to the integer solution of the problem: one is based on the price-and-branch heuristic, the other on a variable rounding procedure. Experiments with several combinations of parameters are provided to show the impact of the number of sizes and families on computation times and quality of solutions.

TERMS: Task management policies to achieve high performance for mixed workloads using surplus resources

Resource contentions and performance interferences can lead to workload performance degradation in mixed-workload deployment clusters. Previous work guarantees the resource requirements of latency-sensitive tasks and reduces performance losses to batch jobs by reclaiming surplus resources from over-provisioned tasks. While the fragmentation of resources leads to a mismatch between provisioned resources and task requirements, resulting in high operation overheads and losses of task fairness. This paper proposes TERMS, the task management policies based on task relevance, resource distribution, and task fairness to achieve efficient and low-cost task management. TERMS mainly includes three types of management policies. The task scheduling policy can schedule new tasks according to task relevance. Task selection strategies select tasks for resource provisioning and task resumption based on resource requirements and task fairness. If necessary, the node selection strategy can be used to choose befitting target nodes based on task relevance and node resource information for task migration when eliminating straggler tasks. Evaluation results show that TERMS can further improve the performance of latency-sensitive services and batch jobs, reduce management overheads, and avoid operation failures.

Scheduling algorithms for truly heterogeneous hierarchical fog networks

AbstractFog computing has emerged as a viable framework for processing delay sensitive applications. Modern applications consist of latency‐sensitive and latency‐tolerant jobs, leading to fog architectures that are often multi‐tiered/hierarchical. FiFSA (hierarchical first fog scheduling algorithm) and EFSA (hierarchical elected fog scheduling algorithm) are capable of scheduling both online and batch jobs on hierarchical fog‐cloud architectures. We consider heterogeneity in computing capacity—both among fog devices in separate layers, and among fog devices in the same layers. In general, online jobs with modest cpu requirements are scheduled on lower tier fog devices, and batch jobs with significant cpu requirements are scheduled on higher tier‐fog nodes, or the cloud data center (cdc). FiFSA assigns jobs to the first fog device with sufficient spare capacity. EFSA employs a MinMin heuristic that assigns jobs to the fog device that results in minimum completion time, while considering fog load. The performance of the proposed algorithms has been evaluated on a real‐life workload trace, using both simulation scenarios and a prototype testbed. FiFSA and EFSA offer an improvement of 19% to 70% in completion times and an improvement of 42% to 72% in system cost over other comparable algorithms.

Resilient Scheduling of Moldable Parallel Jobs to Cope With Silent Errors

We study the resilient scheduling of moldable parallel jobs on high-performance computing (HPC) platforms. Moldable jobs allow for choosing a processor allocation before execution, and their execution time obeys various speedup models. The objective is to minimize the overall completion time or the makespan, when jobs can fail due to silent errors and hence may need to be re-executed after each failure until successful completion. Our work generalizes the classical scheduling framework for failure-free jobs. To cope with silent errors, we introduce two resilient scheduling algorithms, <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Lpa-List</small> and <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Batch-List</small> , both of which use the <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">List</small> strategy to schedule the jobs. Without knowing a priori how many times each job will fail, <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Lpa-List</small> relies on a local strategy to allocate processors to the jobs, while <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Batch-List</small> schedules the jobs in batches and allows only a restricted number of failures per job in each batch. We prove approximation ratios for the two algorithms under several prominent speedup models (e.g., roofline, communication, Amdahl, power, monotonic, and a mix model). An extensive set of simulations is conducted to evaluate different variants of the two algorithms, and the results show that they consistently outperform some baseline heuristics. Overall, our best algorithm is within a factor of 1.6 of a lower bound on average over the entire set of experiments, and within a factor of 4.2 in the worst case.

ALBERT: An automatic learning based execution and resource management system for optimizing Hadoop workload in clouds

Hadoop is a popular computing framework designed to deliver timely and cost-effective data processing on a large cluster of commodity machines. It relieves the burden of the programmers dealing with distributed programming, and an ecosystem of Big Data solutions has developed around it. However, Hadoop's job execution time can greatly depend on its runtime configurations and resource selections. Given the more than 100 job configuration settings provided by Hadoop, and diverse resource instance options in a cloud or virtualized computing environment, running Hadoop jobs still requires a substantial amount of expertise and experience. To address this challenge, we apply a deep neural network to predict Hadoop's job time based on historical execution data, and propose optimization methods to reduce job execution time and cost. The results show that our prediction method achieves almost 90% time prediction accuracy and clearly outperforms three other state-of-the-art regression-based prediction methods. Based on the time prediction, our proposed configuration search method and job scheduling algorithm successfully shorten the execution time of a single Hadoop job by more than a factor of 2 and reduce the time of processing a batch of Hadoop jobs by 40%∼65%.

Variable Coded Batch Matrix Multiplication

A majority of coded matrix-matrix computation literature has broadly focused in two directions: matrix partitioning for computing a single computation task and batch processing of multiple distinct computation tasks. While these works provide codes with good straggler resilience and fast decoding for their problem spaces, these codes would not be able to take advantage of the natural redundancy of re-using matrices across batch jobs. In this paper, we introduce the Variable Coded Distributed Batch Matrix Multiplication (VCDBMM) problem which tasks a distributed system to perform batch matrix multiplication where matrices are not necessarily distinct among batch jobs. Inspired in part by Cross-Subspace Alignment codes, we develop Flexible Cross-Subspace Alignments (FCSA) codes that are flexible enough to utilize this redundancy. We provide a full characterization of FCSA codes which allow for a wide variety of system complexities including good straggler resilience and fast decoding. We theoretically demonstrate that, under certain practical conditions, FCSA codes are within a factor of 2 of the optimal solution when it comes to straggler resilience. Furthermore, our simulations demonstrate that our codes can achieve even better optimality gaps in practice, even going as low as 1.7.

Improved MILP formulation equipped with valid inequalities for scheduling a batch processing machine with non-identical job sizes

We consider the problem of minimizing makespan on a single batch processing machine with a limited capacity. The machine can simultaneously process a group (batch) of jobs, each of them having different sizes and processing times. The processing time of a batch is the longest processing time of all jobs in the batch. We analyze the strength of the LP-relaxations of two mixed integer linear programming (MILP) formulations: a straightforward formulation and an advanced formulation. We prove that while the LP-relaxation of the straightforward formulation can perform arbitrarily bad in providing an LP bound on the optimal makespan, the LP-relaxation of the advanced formulation have some merit in providing a relatively good bound of makespan. With a detailed analysis of the problem, we prove a number of properties of the optimal solution and derive their respective valid inequalities (VI). These VIs are exploited in a novel MILP formulation for the problem. Extensive computations reveal that, with the help of the VIs, the proposed MILP formulation can solve large instances of the problem.

An efficient system using implicit feedback and lifelong learning approach to improve recommendation

This paper presents a new technique for contextual item-to-item Collaborative Filtering-based Recommender System, an improved version popularised by e-commerce giant Amazon two decades back. The concept is based on items also-viewed under the same browsing session. Users’ browsing patterns, locations, and timestamps are considered as the context and latent factors for each user. The algorithm computes recommendations based on users’ implicit endorsements by clicks. The algorithm does not enforce the user to log in to provide recommendations and is capable of providing accurate recommendations for non-logged-in users and with a setting where the system is unaware of users’ preferences and profile data (non-logged-in users). This research takes the cue from human lifelong incremental learning experience applied to machine learning on a large volume of the data pool. First, all historical data is gathered from collectable sources in a distributed manner through big data tools. Then, a long-running batch job creates the initial model and saves it to Hadoop Distributed File System (HDFS). An ever-running streaming job loads the model from HDFS and builds on top of it in an incremental fashion. At the architectural level, this resembles the big data mix processing Lambda Architecture. The recommendation is computed based on a proposed equation for a weighted sum between near real-time and historical batch data. Real-time and batch processing engines act as autonomous Multi-agent systems in collaboration. We propose an ensemble method for batch-stream the recommendation engine. We introduce a novel Lifelong Learning Model for recommendation through Multi-agent Lambda Architecture. The recommender system incrementally updates its model on streaming datasets to improve over time.

PAS: Performance-Aware Job Scheduling for Big Data Processing Systems

Big data analytics has become increasingly vital in many modern enterprise applications such as user profiling and business process optimization. Today’s big data processing systems, such as Hadoop MapReduce, Spark, and Hive, treat big data applications as a batch of jobs for scheduling. Existing schedulers in production systems often maintain fair allocation without considering application performance and resource utilization simultaneously. It is challenging to perform job scheduling in big data systems to achieve both low turnaround time and high resource utilization due to the high complexity in data processing logics and the dynamic variation in workloads. In this article, we propose a performance-aware scheduler, referred to as PAS, which dynamically schedules big data jobs in Hadoop YARN and Spark and autonomously adjusts scheduling policies to improve application performance and resource utilization. Specifically, PAS schedules multiple concurrent jobs using different policies based on the predicted job completion time and employs a greedy approach and a one-step lookahead strategy to opportunistically maximize the average job performance while still maintaining a satisfactory level of resource utilization. We implement PAS in Hadoop YARN and evaluate its performance with HiBench, a well-known big data processing benchmark. Experimental results show that PAS reduces the average turnaround time by 25% and the makespan by 15% in comparison with four state-of-the-art schedulers.