Multi-stage Job Research Articles

On the Gittins index for multistage jobs

Optimal scheduling in single-server queueing systems is a classic problem in queueing theory. The Gittins index policy is known to be the optimal nonanticipating policy minimizing the mean delay in the M/G/1 queue. While the Gittins index is thoroughly characterized for ordinary jobs whose state is described by the attained service, it is not at all the case with jobs that have more complex structure. Recently, a class of such jobs, multistage jobs, were introduced, and it was shown that the computation of Gittins index of a multistage job decomposes into separable computations for the individual stages. The characterization is, however, indirect in the sense that it relies on the recursion for an auxiliary function (the so-called SJP—single-job profit—function) and not for the Gittins index itself. In this paper, we focus on sequential multistage jobs, which have a fixed sequence of stages, and prove that, for them, it is possible to compute the Gittins index directly by recursively combining the Gittins indices of its individual stages. In addition, we give sufficient conditions for the optimality of the FCFS and SERPT disciplines for scheduling sequential multistage jobs. On the other hand, we demonstrate that, for nonsequential multistage jobs, it is better to compute the Gittins index by utilizing the SJP functions.

Scheduling Coflows With Dependency Graph

Applications in data-parallel computing typically consist of multiple stages. In each stage, a set of intermediate parallel data flows ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Coflow</i> ) is produced and transferred between servers to enable starting of next stage. While there has been much research on scheduling isolated coflows, the dependency between coflows in multi-stage jobs has been largely ignored. In this paper, we consider scheduling coflows of multi-stage jobs represented by general <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">DAG</i> s (Directed Acyclic Graphs) in a shared data center network, so as to minimize the total weighted completion time of jobs. This problem is significantly more challenging than the traditional coflow scheduling, as scheduling even a single multi-stage job to minimize its completion time is shown to be NP-hard. In this paper, we propose a polynomial-time algorithm with approximation ratio of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$O(\mu \log (m)/\log (\log (m)))$ </tex-math></inline-formula> , where <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\mu $ </tex-math></inline-formula> is the maximum number of coflows in a job and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$m$ </tex-math></inline-formula> is the number of servers. For the special case that the jobs’ underlying dependency graphs are <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">rooted trees</i> , we modify the algorithm and improve its approximation ratio. To verify the performance of our algorithms, we present simulation results using real traffic traces that show up to 53% improvement over the prior approach. We conclude the paper by providing a result concerning an optimality gap for scheduling coflows with general DAGs.

Optimizing Internal Overlaps by Self-Adjusting Resource Allocation in Multi-Stage Computing Systems

With the rise of big data, more and more users will launch computing systems to process a large volume of data in various applications. A Scheduling algorithm is crucial to the performance of the processing platforms, especially when they are concurrently executing a batch of jobs. Such jobs usually represent multiple stages. Each stage produces the intermediate data which will be piped to the next stage for further processing. However, the scheduling problem in a big data computing system is different from the traditional multi-stage job scheduling problem as for any two consecutive stages, the later stage usually starts before the former stage is finished to “shuffle” the intermediate data. In this paper, we consider MapReduce/Hadoop as a representative computing system and develop a new strategy named OMO, Optimize MapReduce Overlap with a Good Start (Reduce) and a Good Finish (Map). A MapReduce job contains two consecutive phases: map and reduce. Our general target is to optimize the internal overlap between these two phases. There are two new techniques included in our solution, Lazy start of reduce tasks and Batch finish of map tasks, which aim to approach an effective alignment of the two phases based on the characteristics of the MapReduce process. OMO has been implemented on the Hadoop system with extensive experiments for performance evaluation. The results show that OMO's performance is superior in terms of total completion time (i.e., makespan) of a batch of jobs.

Scheduling coflows of multi-stage jobs under network resource constraints

As an emerging network abstraction, coflow greatly improves the communication performance of data-parallel computing jobs. Many existing studies have focused on the design of coflow scheduling to minimize the completion time of jobs. However, they treat the underlying network as a large non-blocking switch without considering the constraints of network resources, which may increase network bottlenecks, reduce link capacity utilization, and extend job completion time. In this paper, we take network resource constraints into account and study how to schedule coflows in multi-stage jobs with the objective of minimizing the total weighted job completion time. We first formalize this multi-stage job scheduling problem as nonlinear programming and prove its NP-hardness. By introducing a priority order of jobs using a linear programming relaxation-based approach, we propose a polynomial-time algorithm with a performance guarantee, which can achieve a constant approximation ratio in many typical scenarios. Simulation results based on a Facebook trace show that, compared with a state-of-the-art approach, our algorithm can shorten the average weighted job completion time by up to 48.46% and run faster by up to 19.12×.

Research on Construction and Application for the Model of Multistage Job Shop Scheduling Problem

Based on the practical application of an enterprise, we address the multistage job shop scheduling problem with several parallel machines in the first stage (production), a few parallel machines in the second stage (processing and assembly), and one machine in the following stages (including joint debugging, testing, inspection, and packaging). First, we establish the optimization objective model for the first two stages. Then, based on the design of the sequencing algorithm in the first two stages, a correction algorithm is designed between the first stage and the second stage to solve this problem systematically. Finally, we propose two benchmark approaches to verify the performance of our proposed algorithm. Verification of numerical experiments shows that the model and algorithm constructed in this paper effectively improve the production efficiency of the enterprise.

Workload control and order release in two-level multi-stage job shops: an assessment by simulation

Most studies on the performance of workload control (WLC) order release methods assume products have simple structures. But, in practice, products are often complex and consist of a number of sub-assemblies that flow through a ‘level 1’ job shop before converging on several final assembly operations in a ‘level 2’ assembly shop. Evaluating the performance of release methods in this context – referred to as the ‘two-level multi-stage job shop’ – is an important step towards improving the alignment between WLC theory and practice. We use simulation to assess the performance of four of the best-performing WLC order release methods. Results suggest that WLC order release has the potential to limit work-in-process (WIP) while reducing the percentage of tardy jobs. It is also important to consider when and where release should be controlled. Results suggest that: (1) orders should be considered for release to level 2 when the first sub-assembly is complete, rather than only when all of the sub-assemblies that make up an assembly order are complete at level 1; and, (2) exercising control at level 2 (with or without control at level 1) leads to a greater reduction in the percentage of tardy jobs than control at level 1 only.

The performance of Due Date setting rules in assembly and multi-stage job shops: an assessment by simulation

Setting short yet reliable Due Dates (DDs) is an important early production planning and control task. The majority of job-shop research on DD setting assumes simple product structures without assembly operations. However, in practice, product structures are often complex, and multiple final assembly operations may be required. This paper evaluates the performance of DD setting rules in the context of complex product structures, considering two scenarios: two-level assembly job shops, where orders converge on one final assembly operation; and two-level multi-stage job shops, where a series of assembly operations are undertaken. New rules are proposed which are substantially simpler and more suitable for practical use than those in the literature. These rules are only outperformed by a more sophisticated rule from the wider literature, newly introduced into the context of assembly and multi-stage job shops. Which rule to apply in practice depends on whether a manager considers the improvement in performance more important than the loss of simplicity. Future research should investigate how jobs can be planned and controlled effectively when some or all DDs are set externally by customers rather than internally using a DD setting rule.

Nanotechnology for Sale: The Once-Theoretical Becomes Practical

The magic of nanotechnology is on display in the form of a dull silver ball created by Baker Hughes. What looks like solid metal is a composite that is light enough to flow through a well, strong enough to stand up to the impact and pressure associated with opening a sleeve on a multistage fracturing job, and then disappear when it is done. The company is making 1,500 of the balls each week and based on customer calls, it could sell a lot more, said James King, director of applications engineering for sales and operations support at Baker Oil Tools, US Land. The company plans to increase sales, and add new products, but the supply of the material used to make the balls is limited. Producers see value in the balls because that expense is considerably less than the value of the production lost if a ball gets stuck during fracturing, blocking the flow from downhole. What looks like aluminum is a nanomaterial the company calls In-Tallic, which combines magnesium, aluminum, and other alloys. It is one of a class of materials called nano-structured metal composites that offer both strength and the ability to “dissolve” away under certain conditions. The balls are far from the first product engineered to take advantage of the growing understanding of how to take the unique properties observed at nanoscale—generally 100 nanometers (nm) or less—and build those characteristics into products. Tougher coatings using nanomaterials have been around for a while. In-Tallic balls stand out as an example of how nanotechnology can do things that otherwise would not be possible. The balls’ relatively short 18-month development period also shows what has been learned about engineering nanotechnology into products. “We are looking for ways to break that performance envelope,” said Gaurav Agrawal, director of enterprise research at Baker Hughes. But it is not indicative of the time, effort, and barriers to acceptance more often faced by those creating products with nanotechnology for exploration and production (E&amp;P). “New generation materials can sell consumer products, but you cannot sell oilfield tools by saying it is nano,” said Jeff Bahr, chief executive officer (CEO) of NanoComposites, a company that uses carbon nanomaterials to develop polymers for uses such as high performance seals. Oilfield buyers are not averse to nanotechnology, but the focus is on solving difficult problems, not on whether this hard-to-define technology was used. “Customers eyes glaze over when you start talking about how you used nanotechnology,” said Dore Rosenblum, CEO of Sub-One Technology, which makes pipe coatings. One indication of the interest in applying nanotechnology is the list of oil companies doing research and development work. They include ConocoPhillips, Saudi Aramco, and Shell.

Gene expression programming based due date assignment in a simulated job shop

In this paper, a new approach for due date assignment in a multi-stage job shop is proposed and evaluated. The proposed approach is based on a genetic programming technique which is known as gene expression programming (GEP). GEP is a relatively new member of the genetic programming family. The primary objective of this research is to compare the performance of the proposed due date assignment model with several previously proposed conventional due date assignment models. For this purpose, simulation models are developed and comparisons of the due date assignment models are made mainly in terms of the mean absolute percent error (MAPE), mean percent error (MPE) and mean tardiness (MT). Some additional performance measurements are also given. Simulation experiments revealed that for many test conditions the proposed due date assignment method dominates all other compared due date assignment methods.

Finding minimum flow time cyclic schedules for non-identical, multistage jobs

Finding a minimum flow time cyclic schedule for a single, multistage job with a serial, re-entrant routing is known to be NP-hard. This paper addresses the problem of scheduling multiple, non-identical jobs in a cyclic fashion, where the job routings may be arbitrary partial orders as well as re-entrant. Given a fixed cycle length, our goal is to minimize a weighted sum of the job flow times. We present a general schedule construction algorithm for implementing a cyclic version of priority dispatch rules that accepts any user-defined tie-breaking function and naturally yields a feasible cyclic schedule. We also describe a pair of easily solvable subproblems that may be used to tighten existing cyclic schedules, as well as an iterative schedule improvement algorithm based on a technique called compression. A numerical study suggests that our schedule construction algorithm, called Cyclic PDR, outperforms its traditional noncyclic priority dispatch rule counterpart, as well as a previously proposed single-pass algorithm. The Cyclic PDR algorithm is shown to be particularly effective when used in conjunction with a least work remaining tie-breaking function. Taken together, our schedule construction and improvement techniques provide an effective solution approach for producing minimum flow time cyclic schedules.

Makespan Minimization in Job Shops: A Linear Time Approximation Scheme

In this paper we present a linear time approximation scheme for the job shop scheduling problem with a fixed number of machines and fixed number of operations per job. This improves on the previously best $2+\epsilon$, $\epsilon > 0$, approximation algorithm for the problem by Shmoys, Stein, and Wein [SIAM J. Comput., 23 (1994), pp. 617--632]. Our approximation scheme is very general and it can be extended to the case of job shop scheduling problems with release and delivery times, multistage job shops, dag job shops, and preemptive variants of most of these problems.

Sequencing rules, progress milestones and product structure in a multistage job shop

Abstract Manufacturing products which include the assembly of components requires knowledge of both the resources required to manufacture that part, as defined in the product routing sheet, and the relationship of components to be assembled at successive stages as defined in the product structure, or Bill of Materials (BOM). Job shops in which this type of manufacturing is done are generally referred to as multistage assembly job shops. Scheduling a multistage assembly job shop requires the proper coordination of materials flow through the various stages necessary to complete a product. It requires considering the time a product may have to wait for its parallel components before the required assembly operation can begin, as well as the waiting time required for a particular machine. Consequently, multistage assembly job shop scheduling is a much more complex problem than that of scheduling a single-stage job shop. This paper examines several important issues in shop floor control in a multistage assembly job shop, namely: 1. Which sequencing rules are most appropriate for a multistage assembly job shop? 2. Do progress milestones improve the performance of the sequencing rules? 3. Are the various sequencing rules and progress milestones sensitive to product structure? Although an MRP system is not used specifically in this research, questions related to MRP implementation, such as the practice of using component or assembly due dates, are examined. The objectives of the study are to determine the appropriate combinations of sequencing rules and progress milestones for multistage assembly job shops and to evaluate the sensitivity of sequencing rules and progress milestones to the complexity of product structure. This research examines the performance of fourteen due date oriented sequencing rules in a simulated ten-machine multistage job shop having one assembly work center. Due date based sequencing rules are utilized since previous research has indicated that such rules generally resulted in better shop performance than other types of rules tested. Although the focus of this research is on multistage job shops, the sequencing rules tested are inclusive of those due date oriented rules found to be successful in both single-stage and multistage job shops. Each sequencing rule is tested using three methods of setting due date milestones to control the progression of a job toward completion. These methods are: (1) job due dates, (2) assembly due dates, and (3) operation due dates. Again, the use of these three methods of setting progress milestones is based on the findings of previous research. Both the sequencing rules and progress milestones are evaluated with respect to the complexity of the product's structure. Herein, the product structures used in the study represent flat products (many components at few levels) and tall products (few components at many levels). Five separate BOMs are employed for each of the two product structures, to examine the sensitivity of the sequencing rules and type of progress milestones to product structure. Also, the logical extremes of flat and tall BOMs are selected to better assess the impact of product structure on sequencing rule performance. The findings indicate that the practice of using component due dates to sequence jobs as done in many MRP environments may not be appropriate. In fact, the use of progress milestones worsened the performance of the fourteen dispatching rules for all performance criteria collected. Three of the fourteen dispatching rules tested were found to result in consistently poorer performance than the remaining eleven rules. Very little difference was found between the performance of these other eleven dispatching rules. Also, the sequencing rules were found to perform equally well in a shop characterized by tall BOMs as in a shop characterized by flat BOMs.

Due Date Assignment in a Multistage Job Shop

Abstract In a Materials Requirements Planning (MRP) system, component lead times are generally assumed to be known and constant. Since a requirement of a MRP system is that components finish on time, slack is often built into component lead times to insure that actual job flowtimes equal planned job lead times. In this study, job characteristics and shop characteristics are investigated to determine which has a greater impact on predicting job flowtimes. This is accomplished by determining which characteristic most greatly influences the deviation of planned job lead times from actual job flowtimes. After identifying those factors which have the greatest influence on the job's flowtime, due date assignment rules are developed. A simulation study is then made to determine. which of the due date assignment rules perform best for varying product structures, as defined by various Bills of Materials (BOMs), and various shop conditions. Simulation test results are presented and discussed.

Dispatching in a multistage job shop where machine capacities are unbalanced

Abstract In the multistage job shop, several parts are assembled to form sub-assemblies which are then assembled into final products. Job delays may occur due to the lack of capacity and also due to the lack of component parts necessary to form a subassembly. Previous research of similar systems have assumed a balanced capacity. A more realistic assumption is one in which machine capacities are not balanced such that one machine can be identified as a bottleneck. In this paper, the performance of selected dispatching rules in an unbalanced multistage job shop is evaluated to-determine whether the performance is consistent for jobs which are routed through the bottleneck and for jobs which by-pass the bottleneck.

Environments and Diagenesis, Morrow Sandstones, Cimarron County, Oklahoma: ABSTRACT

Detailed investigations of Morrow Sandstones in Cimarron County, Oklahoma, provide a bird's-eye view of problems encountered during regional exploration, production, and completion practices over a wide area from New Mexico to Kansas. The generally poor lateral and vertical control of sand distribution can be improved through a detailed knowledge of environments of deposition. Frequent formation damage because of poor completion procedures can be largely prevented through an understanding of Morrow sandstone diagenesis. Morrow sandstones in Cimarron County form two distinct reservoirs. Type 1 reservoirs are thick (10 to 50 ft or 3 to 15 m), porous (20 to 23%), permeable (47 to 236 md), and very coarse grained (0.83 mm). These reservoirs are fan-shaped, being less than 1 mi (1.6 km) wide at the apex (on the northwest) and 4 mi (6.4 km) wide at the southeastern edge. Sand thickness is greatest at the center of the body. These sands were deposited in the estuarine portions of a braided fluvial system. Type 2 reservoirs are thin (generally less than 20 ft or 6 m), have relatively low porosities (4 to 20%) and permeabilities (3 to 100 md), and are fine grained (0.24 mm). These reservoirs are discontinuous, lenticular, elongate units which trend at approximately right angles (NE-SW) to the Type 1 reservoirs. Maximum width is 1 mi (1.6 km); maximum length is of the order of several miles. These sands were deposited in lower tidal-flat and shallow, offshore-marine environments, as beaches and bars. Once the reservoir has been discovered, it is vital that completion practices be tailored to the specific rock composition. Failure to do this may result in serious formation damage, and the bypassing of potential production. Problems characteristic of these sands include: (1) a migration of fines, (2) extreme acid sensitivity, and (3) possible water sensitivity. The sand composition may often require a multistage acid job with KCl flushes. Hydrofluoric acid should not be used unless the detailed sand composition has been determined by thin-section analysis. End_of_Article - Last_Page 2118------------

Bounds on Multiprocessing Timing Anomalies

Bounds on Multiprocessing Timing Anomalies