Workload Management Tools Research Articles

Integration of the Barcelona Supercomputing Center for CMS computing: Towards large scale production

The CMS experiment is working to integrate an increasing number of High Performance Computing (HPC) resources into its distributed computing infrastructure. The case of the Barcelona Supercomputing Center (BSC) is particularly challenging as severe network restrictions prevent the use of CMS standard computing solutions. The CIEMAT CMS group has performed significant work in order to overcome these constraints and make BSC resources available to CMS. The developments include adapting the workload management tools, replicating the CMS software repository to BSC storage, providing an alternative access to detector conditions data, and setting up a service to transfer produced output data to a nearby storage facility. In this work, we discuss the current status of this integration activity and present recent developments, such as a front-end service to improve slot usage efficiency and an enhanced transfer service that supports the staging of input data for workflows at BSC. Moreover, significant efforts have been devoted to improving the scalability of the deployed solution, automating its operation, and simplifying the matchmaking of CMS workflows that are suitable for execution at BSC.

Enabling CMS Experiment to the utilization of multiple hardware architectures: a Power9 Testbed at CINECA

The CMS software stack (CMSSW) is built on a nightly basis for multiple hardware architectures and compilers, in order to benefit from the diverse platforms. In practice, still, only x86_64 binaries are used in production, and are supported by the workload management tools in charge of production and analysis job delivery to the distributed computing infrastructure. Profiting from an INFN grant at CINECA, a PRACE Tier-0 Center, tests have been carried on using IBM Power9 nodes from the Marconi100 HPC system. A first study on the modifications needed to the standard CMS WMS systems is shown, and very positive proof-of-concept tests have been conducted up to thousands of computing cores, also including an initial utilization of the GPUs which the nodes host. The current status of the tests, including plans to support multi-architecture workflows, are shown and discussed.

New workforce standards for district nursing launched.

The Queen's Nursing Institute has published workforce standards for district nursing services in the UK. The standards are intended to provide top-level guidance to commissioners, service planners and managers about the key considerations when deciding on the number of staff, skill mix, and workload management tools needed to provide healthcare in the community setting. The standards are intended to support the evidence base around the number of suitably qualified and experienced district nurses needed to provide a service to a given population. The standards include 'red flags' that indicate when a service is at increased risk of deferring care due to lack of staff capacity.

Association of the acute:chronic workload ratio and wellness scores in premier league male hockey players.

The relationship between acute:chronic workload ratios (ACWR) and the incidence of injury, as well as the relationship between subjective wellness scores and training load, is unclear in hockey players. This study investigated these relationships to determine if the ACWR is a useful workload management tool for field hockey players. In addition, the study investigated the association between specific subjective wellness scores (fatigue, sleep quality, general muscle soreness, mood and stress level) and the acute:chronic workload ratio and training load. The study also assessed the association between individual players' training loads with the self-reported wellness scores. Fourteen male field hockey players from the men's first team at the University of the Witwatersrand participated in the study over ten weeks during a premier league competition phase. A Google form was completed within thirty minutes of every training session/match. This form consisted of questions that allowed for the calculation of the ACWR. No incidence of injury was reported during the ten-week period. ACWR scores ranged between 0.67 and 1.87. The ACWR was associated with general muscle soreness (p = 0.010) and training load was associated with fatigue (p = 0.002), sleep quality (p = 0.05), general muscle soreness (p = 0.004), and mood (p=0.025). There may be some merit in the use of subjective wellness measures as workload management tools in field hockey. Further research is required to determine if there is an optimal ACWR for this sport.

Telepractice for School-Based Speech and Language Services: A Workload Management Strategy

Purpose This article discusses the combination of onsite and telepractice services as an effective way for professionals working in multiple schools to maximize their time and professional services in an effort to manage their workload. The combined onsite and telepractice services were used by district-wide school-based professionals to deliver direct and indirect services across various locations, thereby reducing time-consuming travel and increasing scheduling efficiency. The possible uses of telepractice as a workload management tool are discussed, as well as some challenges that the team overcame in order to successfully use this method. Conclusions Telepractice, in combination with onsite services, is an effective workload strategy for school-based therapists.

Use of DAGMan in CRAB3 to improve the splitting of CMS user jobs

CRAB3 is a workload management tool used by CMS physicists to analyze data acquired by the Compact Muon Solenoid (CMS) detector at the CERN Large Hadron Collider (LHC). Research in high energy physics often requires the analysis of large collections of files, referred to as datasets. The task is divided into jobs that are distributed among a large collection of worker nodes throughout the Worldwide LHC Computing Grid (WLCG).Splitting a large analysis task into optimally sized jobs is critical to efficient use of distributed computing resources. Jobs that are too big will have excessive runtimes and will not distribute the work across all of the available nodes. However, splitting the project into a large number of very small jobs is also inefficient, as each job creates additional overhead which increases load on infrastructure resources.Currently this splitting is done manually, using parameters provided by the user. However the resources needed for each job are difficult to predict because of frequent variations in the performance of the user code and the content of the input dataset. As a result, dividing a task into jobs by hand is difficult and often suboptimal.In this work we present a new feature called “automatic splitting” which removes the need for users to manually specify job splitting parameters. We discuss how HTCondor DAGMan can be used to build dynamic Directed Acyclic Graphs (DAGs) to optimize the performance of large CMS analysis jobs on the Grid.We use DAGMan to dynamically generate interconnected DAGs that estimate the processing time the user code will require to analyze each event. This is used to calculate an estimate of the total processing time per job, and a set of analysis jobs are run using this estimate as a specified time limit. Some jobs may not finish within the alloted time; they are terminated at the time limit, and the unfinished data is regrouped into smaller jobs and resubmitted.

Evolution of the use of relational and NoSQL databases in the ATLAS experiment

The ATLAS experiment used for many years a large database infrastructure based on Oracle to store several different types of non-event data: time-dependent detector configuration and conditions data, calibrations and alignments, configurations of Grid sites, catalogues for data management tools, job records for distributed workload management tools, run and event metadata. The rapid development of “NoSQL” databases (structured storage services) in the last five years allowed an extended and complementary usage of traditional relational databases and new structured storage tools in order to improve the performance of existing applications and to extend their functionalities using the possibilities offered by the modern storage systems. The trend is towards using the best tool for each kind of data, separating for example the intrinsically relational metadata from payload storage, and records that are frequently updated and benefit from transactions from archived information. Access to all components has to be orchestrated by specialised services that run on front-end machines and shield the user from the complexity of data storage infrastructure. This paper describes this technology evolution in the ATLAS database infrastructure and presents a few examples of large database applications that benefit from it.

Measuring safe staff levels in the community: the ‘DominiC’ workforce management tool

There is significant need to establish and predict what numbers of district nursing staff are needed to provide quality patient care. This is the first in a series of articles sharing the evidence-based and tested solutions being used in clinical practice across the UK to inform caseload and workforce planning. Stockport NHS Foundation Trust's electronic workload management tool 'DominiC' is explored in this article. The tool promotes continuity of patient care, efficient resource allocation and the benchmarking of care. It predicts future service demands and measures these against the staff resources available. Lessons learned from the process of developing and implementing DominiC are shared to help inform clinicians and managers looking for similar workforce planning solutions.

Towards a global monitoring system for CMS computing operations

The operation of the CMS computing system requires a complex monitoring system to cover all its aspects: central services, databases, the distributed computing infrastructure, production and analysis workflows, the global overview of the CMS computing activities and the related historical information. Several tools are available to provide this information, developed both inside and outside of the collaboration and often used in common with other experiments. Despite the fact that the current monitoring allowed CMS to successfully perform its computing operations, an evolution of the system is clearly required, to adapt to the recent changes in the data and workload management tools and models and to address some shortcomings that make its usage less than optimal. Therefore, a recent and ongoing coordinated effort was started in CMS, aiming at improving the entire monitoring system by identifying its weaknesses and the new requirements from the stakeholders, rationalise and streamline existing components and drive future software development. This contribution gives a complete overview of the CMS monitoring system and a description of all the recent activities that have been started with the goal of providing a more integrated, modern and functional global monitoring system for computing operations.

A gLite FTS based solution for managing user output in CMS

The CMS distributed data analysis workflow assumes that jobs run in a different location from where their results are finally stored. Typically the user output must be transferred across the network from one site to another, possibly on a different continent or over links not necessarily validated for high bandwidth/high reliability transfer. This step is named stage-out and in CMS was originally implemented as a synchronous step of the analysis job execution. However, our experience showed the weakness of this approach both in terms of low total job execution efficiency and failure rates, wasting precious CPU resources. The nature of analysis data makes it inappropriate to use PhEDEx, the core data placement system for CMS. As part of the new generation of CMS Workload Management tools, the Asynchronous Stage-Out system (AsyncStageOut) has been developed to enable third party copy of the user output. The AsyncStageOut component manages glite FTS transfers of data from the temporary store at the site where the job ran to the final location of the data on behalf of that data owner. The tool uses python daemons, built using the WMCore framework, and CouchDB, to manage the queue of work and FTS transfers. CouchDB also provides the platform for a dedicated operations monitoring system. In this paper, we present the motivations of the asynchronous stage-out system. We give an insight into the design and the implementation of key features, describing how it is coupled with the CMS workload management system. Finally, we show the results and the commissioning experience.

Workload management: a technology perspective with respect to self-* characteristics

Rapid growth in data, maximum functionality requirements and changing behavior in the database workload tends the workload management to be more complex. Organizations have complex type of workloads that are very difficult to manage by humans and even in some cases this management becomes impossible. Human experts take long time to get sufficient experience so that they can manage the workload efficiently. The versatility in workload due to huge data size and user requirements leads us towards the new challenges. One of the challenges is the identification of the problematic queries and the decision about these, i.e. whether to continue their execution or stop. The other challenge is how to characterize the workload, as the tasks such as configuration, prediction and adoption are fully dependent on the workload characterization. Correct and timely characterization leads managing the workload in an efficient manner and vice versa. In this scenario, our objective is to produce a workload management strategy or framework that is fully adoptive. The paper provides a summary of the structure and achievements of the database tools that exhibit Autonomic Computing or self-* characteristics in workload management. We have categorized the database workload tools to these self-* characteristics and identified their limitations. Finally the paper presents the research done in the database workload management tools with respect to the workload type and Autonomic Computing.

Workload management: A technology perspective with respect to self characteristics

Rapid growth in data, maximum functionality and changing behavior tends the workload to be more complex. Organizations have complex type of workloads that is very difficult to manage by the humans and even in some cases, this management becomes impossible. Human experts take much time to get sufficient experience so that they can manage workload efficiently. The versatility in workload due to huge data size and requests (workload) lead us towards new challenges. One of the challenges is the identification of the problems queries and the decision about these, that is, whether to continue their execution or stop. The other challenge is how to characterize the workload, as good configuration, prediction and adoption is fully dependent on characterization of the workload. Correct and timely characterization leads to managing the workload in an efficient manner and vice versa. In this scenario, our objective is to produce such workload management strategy or framework that is fully autonomic. This paper provides some basis and achievements about the tools that exhibit autonomic computing (AC) in workload management with respect to self-characteristics. We have categorized the workload tools to these self-characteristics and identified their limitations. Finally the paper presents the research done in workload management tools with respect to workload type and autonomic computing. Key words: Autonomic computing, workload, optimization, configuration, prediction, organization, adoption.

Large scale and low latency analysis facilities for the CMS experiment: development and operational aspects

While a majority of CMS data analysis activities rely on the distributed computing infrastructure on the WLCG Grid, dedicated local computing facilities have been deployed to address particular requirements in terms of latency and scale.The CMS CERN Analysis Facility (CAF) was primarily designed to host a large variety of latency-critical workflows. These break down into alignment and calibration, detector commissioning and diagnosis, and high-interest physics analysis requiring fast turnaround. In order to reach the goal for fast turnaround tasks, the Workload Management group has designed a CRABServer based system to fit with two main needs: to provide a simple, familiar interface to the user (as used in the CRAB Analysis Tool[7]) and to allow an easy transition to the Tier-0 system. While the CRABServer component had been initially designed for Grid analysis by CMS end-users, with a few modifications it turned out to be also a very powerful service to manage and monitor local submissions on the CAF. Transition to Tier-0 has been guaranteed by the usage of the WMCore, a library developed by CMS to be the common core of workload management tools, for handing data driven workflow dependencies. This system is now being used with the first use cases, and important experience is being acquired. In addition to the CERN CAF facility, FNAL has CMS dedicated analysis resources at the FNAL LHC Physics Center (LPC). In the first few years of data collection FNAL has been able to accept a large fraction of CMS data. The remote centre is not well suited for the extremely low latency work expected of the CAF, but the presence of substantial analysis resources, a large resident community, and a large fraction of the data make the LPC a strong facility for resource intensive analysis.We present the building, commissioning and operation of these dedicated analysis facilities in the first year of LHC collisions; we also present the specific development to our software needed to allow for the use of these computing facilities in the special use cases of fast turnaround analyses.

Bringing the CMS distributed computing system into scalable operations

Establishing efficient and scalable operations of the CMS distributed computing system critically relies on the proper integration, commissioning and scale testing of the data and workload management tools, the various computing workflows and the underlying computing infrastructure, located at more than 50 computing centres worldwide and interconnected by the Worldwide LHC Computing Grid. Computing challenges periodically undertaken by CMS in the past years with increasing scale and complexity have revealed the need for a sustained effort on computing integration and commissioning activities. The Processing and Data Access (PADA) Task Force was established at the beginning of 2008 within the CMS Computing Program with the mandate of validating the infrastructure for organized processing and user analysis including the sites and the workload and data management tools, validating the distributed production system by performing functionality, reliability and scale tests, helping sites to commission, configure and optimize the networking and storage through scale testing data transfers and data processing, and improving the efficiency of accessing data across the CMS computing system from global transfers to local access. This contribution reports on the tools and procedures developed by CMS for computing commissioning and scale testing as well as the improvements accomplished towards efficient, reliable and scalable computing operations. The activities include the development and operation of load generators for job submission and data transfers with the aim of stressing the experiment and Grid data management and workload management systems, site commissioning procedures and tools to monitor and improve site availability and reliability, as well as activities targeted to the commissioning of the distributed production, user analysis and monitoring systems.

Developing an electronic nursing record based on a philosophy of care and workload management tool: the EOC experience

Developing an electronic nursing record based on a philosophy of care and workload management tool: the EOC experience

Developing an electronic nursing record based on a philosophy of care and workload management tool: the EOC experience

In recent years, the Nursing Area of the Ente Ospedaliero Cantonale (EOC) has reached a consensus on unification of nursing language and patient-oriented documentation. This step was taken before defining future electronically based nursing documentation. In view of the increasing importance of economic aspects (e.g. DRG), a combination between nursing documentation and standardised classification for workload assessment was found in the LEP3 system. This paper discusses the principal goals and design elements of the planned solution within an electronic patient record. In the last few years the Nursing Area of EOC has embarked on a long journey as regards management, registration, planning, assessment and communication of information and nursing activities related to patient care. In this context a working document called DCOP (Patient Oriented Care Document, Documento di Cura Orientato al Paziente) has been created which is based on the principle of focus charting. This has led to unification of nursing language within EOC, including the development of specific documents for the general area, for paediatrics, the neonatal-maternal area, dialysis, intensive care and the emergency unit. The increasing use of this working instrument, concomitantly with the spread of the philosophy of care based relations, will result in a better orientation in the definition of the care processes. Today the DCOP language is used within the whole EOC and represents a solid basis on which to build the future introduction of information technology. Different studies have demonstrated that therapeutic results are influenced by the number of nurses, their mix of competences, their workload and the models applied to nursing care. It is therefore important to have an efficacious tool to measure the nurses’ workload. The current political situation and the next revision of the health insurance system (LaMal, KVG) will increase competition between the public and the private sectors. Showing the actual needs of the personnel, with a direct link to the quality of care, will become of utmost importance in dealing with the missions assigned to EOC. Within EOC there is no homogeneous instrument for this purpose, but we feel that it is absolutely necessary to have a unified system to assess workload. For all these reasons we have decided to adopt the LEP 3 system, which will have to be integrated into the electronic nursing document.

Evaluating the reliability of computational grids from the end user’s point of view

Reliability, in terms of Grid component fault tolerance and minimum quality of service, is an important aspect that has to be addressed to foster Grid technology adoption. Software reliability is critically important in today's integrated and distributed systems, as is often the weak link in system performance. In general, reliability is difficult to measure, and specially in Grid environments, where evaluation methodologies are novel and controversial matters. This paper describes a straightforward procedure to analyze the reliability of computational grids from the viewpoint of an end user. The procedure is illustrated in the evaluation of a research Grid infrastructure based on Globus basic services and the GridWay meta-scheduler. The GridWay support for fault tolerance is also demonstrated in a production-level environment. Results show that GridWay is a reliable workload management tool for dynamic and faulty Grid environments. Transparently to the end user, GridWay is able to detect and recover from any of the Grid element failure, outage and saturation conditions specified by the reliability analysis procedure.

The CMS analysis chain in a distributed environment

The CMS collaboration is undertaking a big effort to define the analysis model and to develop software tools with the purpose of analysing several millions of simulated and real data events by a large number of people in many geographically distributed sites. From the computing point of view, one of the most complex issues when doing remote analysis is the data discovery and access. Some software tools were developed in order to move data, make them available to the full international community and validate them for the subsequent analysis. The batch analysis processing is performed with workload management tools developed on purpose, which are mainly responsible for the job preparation and the job submission. The job monitoring and the output management are implemented as the last part of the analysis chain. Grid tools provided by the LCG project are evaluated to gain access to the data and the resources by providing a user friendly interface to the physicists submitting the analysis jobs. An overview of the current implementation and of the interactions between the previous components of the CMS analysis system is presented in this work.