Job Queue Research Articles

META-pipe Authorization service

We describe the design, implementation, and use of the META-pipe Authorization service. META-pipe is a complete workflow for the analysis of marine metagenomics data. We will provide META-pipe as a web based data analysis service for ELIXIR users. We have integrated our Authorization service with the ELIXIR Authorization and Authentication Infrastructure (AAI) that allows single sign-on to services across the ELIXIR infrastructure. We use the Authorization service to authorize access to data on the META-pipe storage system and jobs in the META-pipe job queue. Our Authorization server was among the first services that integrated with ELIXIR AAI. The code is open source at: https://gitlab.com/uit-sfb/AuthService2.

The concept of algorithm supporting the process of scheduling production tasks

The purpose of the work was to develop a concept of an algorithm which supports the process of scheduling production tasks. The designed algorithm performs a task queue creation process in six steps: selecting a schedule type, declaring a queue length, selecting orders for queue development, loading data, automatic job analysis, and queue and Gantt chart generation, taking a decision on the acceptance of the designed production program. The designed system will use single-attribute priority rules (defined by experts) and multi-attribute rules that can be created by combining single-attribute rules. The concept of an algorithm that supports the process of scheduling production tasks, developed and tested by conducting simulations, can be the basis for developing a computer application supporting decision-making process in a manufacturing or service company.

Research on elastic resource management for multi-queue under cloud computing environment

As a new approach to manage computing resource, virtualization technology is more and more widely applied in the high-energy physics field. A virtual computing cluster based on Openstack was built at IHEP, using HTCondor as the job queue management system. In a traditional static cluster, a fixed number of virtual machines are pre-allocated to the job queue of different experiments. However this method cannot be well adapted to the volatility of computing resource requirements. To solve this problem, an elastic computing resource management system under cloud computing environment has been designed. This system performs unified management of virtual computing nodes on the basis of job queue in HTCondor based on dual resource thresholds as well as the quota service. A two-stage pool is designed to improve the efficiency of resource pool expansion. This paper will present several use cases of the elastic resource management system in IHEPCloud. The practical run shows virtual computing resource dynamically expanded or shrunk while computing requirements change. Additionally, the CPU utilization ratio of computing resource was significantly increased when compared with traditional resource management. The system also has good performance when there are multiple condor schedulers and multiple job queues.

Integration of Openstack cloud resources in BES III computing cluster

Cloud computing provides a new technical means for data processing of high energy physics experiment. However, the resource of each queue is fixed and the usage of the resource is static in traditional job management system. In order to make it simple and transparent for physicist to use, we developed a virtual cluster system (vpmanager) to integrate IHEPCloud and different batch systems such as Torque and HTCondor. Vpmanager provides dynamic virtual machines scheduling according to the job queue. The BES III use case results show that resource efficiency is greatly improved.

Rationing of Formal Sector Jobs and Informality: The Colombian Case

AbstractWe analyse the factors that affect the decision to find employment in the formal or informal sectors taking into account the fact that there is a rationing of formal sector jobs and hence a job queue to enter that sector. Using data from Colombia, we confirm that formal job rationing does exist and that it affects around 62 per cent of the workers that find themselves in the informal labour market. Further factors determining actively seeking formal employment include personal traits, the characteristics of human capital and of the household and the wage premiums associated with working in the formal sector. Copyright © 2017 John Wiley & Sons, Ltd.

Randomness or stock–flow: which mechanism describes labour market matching in Poland?

ABSTRACTI identify which theoretical model (random, stock–flow, or job queuing) best describes the matching mechanism in the labour market in Poland. The purpose of this work is to formulate policy recommendations aimed at increasing the number of matches. I use monthly registered unemployment data for the period January 1999–June 2013 and econometrically correct for temporal aggregation bias in the data. I extend known solutions and apply them directly to a job queuing model. Job seekers (from the pool) seek work among old and new job posts, but only a small fraction of the newly unemployed individuals find work quickly. Vacancies are the driving force in aggregate hiring, but the inflow is more important than the stock. The random model has greater explanatory power, although the results do not negate the non-random model. Hence, better information and higher inflows (especially of job offers) should facilitate matching.

The impact of participation in job creation schemes in turbulent times

This paper analyzes the impact of participation in job creation schemes (JCSs) on job search outcomes in the context of the turbulent East German labor market in the aftermath of the German reunification. High job destruction characterized the economic environment. JCSs were heavily used in order to cushion this development. Using data from 1990–1999 and building upon the timing-of-events approach, we estimate multivariate discrete time duration models taking selection based on both observed and unobserved heterogeneity into account. Our results indicate that after initial negative effects during the typical program duration of twelve months, probably driven by reduced job search effort during participation resulting in a rearrangement of the job queue, the impact on the job finding probability becomes insignificantly positive. Additional results, however, suggest that female and highly skilled participants leave unemployment quicker than other groups, which results in highly skilled women benefiting from participation. In general, we find no significant impact on post-unemployment employment stability. Our results are robust to allowing for random treatment effects. Also taking into account endogenous participation in training programs, endogenous censoring, or multiple treatment effects do not change the results.

Operation Mode Study in Cloud Manufacturing Ecosystem

With cloud manufacturing and its shared big-data, the life-cycle management of the massive distributed manufacturing resources can be considered as an ecosystem, in which every entity makes their own decisions depend on the enriched information, which will affect the life cycle of the resources and the overall industry states. In this paper, an original operation mode with three extensions are proposed to describe the life cycle vicissitude of each resource. An agent-based model was designed to simulate the ecosystem modes from the very beginning, and the results show that the ecosystem has: 1) shorter job queue length and lower resource idle rate with incubation mode; 2) a little shorter job queue length and fewer amount of registered resource with outsourcing mode; 3) the fewest amount of registered resource but a little higher resource idle rate with metabolism mode.

Efficiency in Spatially Disaggregated Labour Market Matching

We analyse the efficiency in a labour market matching process. We understand efficiency as a share of the mean number of matches (conditional on given covariates) in the number of matches that would occur if search and matching was optimal, bearing in mind that, contrary to the production function, being unemployed or vacant is not freely chosen or changed. We apply a stochastic matching frontier for random, job queuing and stock-flow models. We use data for Poland, a country with a highly regionally diversified unemployment rate. We contribute to the literature by comparing different spatial aggregation levels – NUTS-1 to NUTS-4 in monthly and annual perspectives. We analyse whether and how the efficiency changes over time. We find spatial and temporal heterogeneity in the labour market. Thus, various policy measures should be designed to improve labour market matching efficiency at certain regional levels.

A Novel Multilevel Queue based Performance Analysis of Hadoop Job Schedulers

Objectives: In this paper, we discuss on the importance of multilevel queues in scheduling Hadoopmapreduce jobs. Methods/Statistical Analysis: Modifications are done on HDFS and yarn configuration files to suit the multilevel queues. This work constitutes the performance analysis of various existing job schedulers such as FIFO, Fair and Capacity schedulers. Findings: Significant achievements are achieved which includes performance evaluation metrics for comparative understanding of the proposed and existing techniques. The final outcome of the work demonstrates the need for multilevel queue scheduling with allocation policies and the optimal placement of jobs in queues. Application/Improvements:With the adoption of multilevel queue scheduling, there is a significant improvement in placing jobs in multilevel queues for the jobs submitted by the users.

Enhancing service capability with multiple finite capacity server queues in cloud data centers

Cloud computing took a step forward in the efficient use of hardware through virtualization technology. And as a result, cloud brings evident benefits for both users and providers. While users can acquire computational resources on-demand elastically, cloud vendors can also utilize maximally the investment costs for data centers infrastructure. In the Internet era, the number of appliances and services migrated to cloud environment increases exponentially. This leads to the expansion of data centers, which become bigger and bigger. Not just that these data centers must have the architecture with a high elasticity in order to serve the huge upsurge of tasks and balance the energy consumption. Although in recent times, many research works have dealt with finite capacity for single job queue in data centers, the multiple finite-capacity queues architecture receives less attention. In reality, the multiple queues architecture is widely used in large data centers. In this paper, we propose a novel three-state model for cloud servers. The model is deployed in both single and multiple finite capacity queues. We also bring forward several strategies to control multiple queues at the same time. This approach allows to reduce service waiting time for jobs and managing elastically the service capability for the whole system. We use CloudSim to simulate the cloud environment and to carry out the experiments in order to demonstrate the operability and effectiveness of the proposed method and strategies. The power consumption is also evaluated to provide insights into the system performance in respect of performance-energy trade-off.

Gender Sorting and the Glass Ceiling in High-Tech Firms

With few exceptions, studies have conceived of the glass ceiling as reflecting internal promotion biases. In this article, the authors argue that glass ceiling patterns can also be the result of external recruitment and hiring processes. Using data on people applying by means of the Internet for jobs at 441 small- and medium-sized high-tech firms, they find evidence that the glass ceiling is produced by both internal and external hiring processes. On the supply side, females are sorted into lower-level job queues than males. On the demand side, screening biases against women also are evident, but a series of “what if” simulations suggest that demand-side screening processes play a comparatively minor role in producing the glass ceiling pattern. These results suggest that bias remediation policies designed to equalize gender differences in hiring chances are likely to be less effective than recruitment and outreach policies designed to improve gender disparities in candidate pools.

A Study of a New Dynamic Load Balancing Approach in Cloud Environment

Distributing workloads across multiple computing resources are one of the major challenges in a cloud computing environment. This paper is being discussed over the basic obstacles of load balancing in cloud environment. The paper looks beyond the problems faced by the cloud system to overcome those through probable improvised techniques. This is a paper over solving the problems exist in the present days by logically analyzing and presenting in an algorithmic format. This approach is mainly focused on an effective job queue making strategy which is suitably allocated the various jobs to CPUs based on their priority or without priority. It also deals with some of the major problems of load balancing in cloud environment like, timeout. Finally, it shows how this approach is fitted in famous AWS and GAE cloud architecture partially. This article will provide the readership an overview of various load balancing problems in cloud environment while also simulating further interest to pursue more advanced research in it.

Mean Field Equilibria of Pricing Games in Internet Marketplaces

We model an Internet marketplace using a set of servers that choose prices for performing jobs. Each server has a queue of unfinished jobs, and is penalized for delay by the market maker via a holding cost. A server completes jobs with a low or high "quality", and jobs truthfully report the quality with which they were completed. The best estimate of quality based on these reports is the "reputation" of the server. A server bases its pricing decision on the distribution of its competitors offered prices and reputations. An entering job is given a random sample of servers, and chooses the best one based on a linear combination of price and reputation. We seek to understand how prices would be determined in such a marketplace using the theory of Mean Field Games. We show the existence of a Mean Field Equilibrium and show how reputation plays a role in allowing servers to declare larger prices than their competitors. We illustrate our results by a numerical study of the system via simulation with parameters chosen from data gathered from existing Internet marketplaces.

Scheduling Job Queue on Hadoop using Hybrid Hadoop Fair Sojourn Protocol

Background/Objectives: This paper gives an idea how to avoid starvation and minimize response time using hybrid hadoop fair sojourn protocol. Methods/Statistical analysis: The scheduler assigns the resources to the cluster so that job dimension is conditional whereas the job makes improvement towards its conclusion. Scheduling selection help the approach of computer-generated time and cluster funds are focused on jobs according to their arrangement. This ensures that not any of jobs (small or large jobs) undergo starvation. Findings: The Shortest Remaining Processing Time (SRPT)1, ranks the least amount of work to complete, is the one to minimize response time (sojourn time). The Hybrid Hadoop Fair Sojourn Protocol (hybrid HFSP)2 to pause jobs with higher SRPT and allow other waiting jobs in queue based on First Come First Serve (FCFS). Application/Improvements: Relating to existing method the accuracy is developed in suggested method.

Aggregate matching in Spain. Time series analysis using cointegration techniques

I analyze the matching process in the Spanish labor market from 1994-2005. I use monthly registered unemployment data and refer solely to public employment intermediation. This period reflects an upward movement along a downward sloping Beveridge curve; therefore, major changes in the process efficiency should not be observed. I narrow the considerations to a job queuing model, which is the most relevant description of the labor market matching process in Spain, according to the literature. I apply various quantitative methods to address the problem of non-stationary data. The EngleGranger estimates emphasize the crucial role of the demand in generating the outflows from unemployment to employment. The ECT coefficient confirms that the model efficiently approaches the new equilibrium. These findings confirm that job seekers find themselves on the disadvantaged side of the market and compete for scarce job offers, which, in turn, are ascribed randomly to the workers. The diagnostic tests of the VAR models question the relevance of a multivariate space analysis because the outflow from unemployment to employment appears to be the sole endogenous variable.

A Novel Approach for Dynamic Load Balancing with Effective Bin Packing and VM Reconfiguration in Cloud

Background/Objectives: In cloud computing, the existing job scheduling algorithms were focused either on efficient job scheduling or optimal load balancing among the virtual machines. Methods/Statistical analysis: This paper introduces a novel approach called Dynamic Load Balancing with effective Bin Packing and VM Reconfiguration (DLBPR) in the cloud. In the proposed work, the jobs are initially classified using the deadline based job scheduler and stored in a different job queue based on the expected processing speed of the job. After classification, the jobs in the various queues are prioritized using their attributes. Findings: The proposed approach dynamically splits and coalesces the VMs based on the required processing speed of the job. The VMs in the data center are dynamically clustered based on their processing speed with the support of VM live migration, and the jobs are processed using the VMs in the cluster. Applications/Improvements: The proposed work is experimented in a cloudsim that minimizes the physical machine nearly 22% compared to other existing algorithms.

A job profile oriented scheduling architecture for improving the throughput of industrial printing environments

The Digital Printing industry has become extremely specialized in the past few years. The use of personalized documents has emerged as a consolidated trend in this field. In order to meet this demand, languages to describe templates for personalized documents were proposed along with procedures which allow the correct printing of such documents. One of these procedures, which demands a high computational effort, is the ripping phase performed over a queue of documents in order to convert them into a printable format. An alternative to decrease the ripping phase computational time is to use high performance computing techniques to allow parallel ripping of different documents. However, such strategies present several unsolved issues. One of the most severe issues is the impossibility to assure a fair load balancing for any job queue. In this scenario, this work proposes a job profile oriented scheduling architecture for improving the throughput of industrial printing environments through a more efficient use of the available resources. Our results show a performance gain of up to 10% in average over the previous existing strategies applied on different job queue scenarios.

HEURISTICS BASED MULTI QUEUE JOB SCHEDULING FOR CLOUD COMPUTING ENVIRONMENT

With the influx of large number of users, delivering the services to them and to meet their quality requirements is still challenging. Cloud computing is built upon the advancements of virtualization technology and distributed computing. Cloud computing itself is a grid computing with additional property of elasticity and scalability. Since cloud usually contains a heterogeneous pool of resources, scheduling plays a vital role in cloud computing. Scheduling in cloud computing denotes the execution of current jobs in a given constraint. QoS is inevitably needed to deal with scheduling in cloud computing environment. Efficient scheduling mechanism should meet the QoS parameters and should enhance resource utilization. Traditional scheduling algorithms such as FCFS, SJF, EASY, round robin and backfilling possesses fragmentation problem. The proposed heuristics based multi queue job scheduling method overcomes the problem of fragmentation by providing an on-demand strategy for scheduling. On-demand service provided by introducing priority. Priority is a comprehensive concept, computed based on QoS parameters. The proposed system, by considering the priority concept reduces the problem of fragmentation and also overcomes starving to death problem in scheduling. The proposed method thus can efficiently executes the user requirements and can manifest better performance.

Analysis of Workflow Variability and Its Impacts on Productivity and Performance in Construction of Multistory Buildings

AbstractThe meeting of milestones presents a constant challenge in construction projects. One root cause behind this challenge is the presence of variability in the project workflow. In fact, impacts of variability at both the trade contractor level and project level remain difficult to manage. This paper aims to analyze impacts of variability in the workflow caused by rework and fluctuating work quantities. Toward this end, production data of two residential construction companies were collected, and a mixed method of mathematical and simulation modeling was used to analyze performance and productivity at both the trade and project levels. Performance metrics such as completion times, work-in-process levels, and delays were then measured in different real-life production scenarios. The findings clearly show that the workflow variability significantly downgrades performance metrics by causing inflated completion times, longer queues of uncompleted jobs, and excessive delays, resulting in productivity loss...