Abstract
At IHEP (Institute of High Energy Physics, Chinese Academy of Sciences), computing resources are contributed by different experiments including BES, JUNO, DYW, HXMT, etc. The resources were divided into different partitions to satisfy the dedicated experiment data processing requirements. IHEP had a local Torqu&Maui cluster with 50 queues serving for above 10 experiments. The separated resource partitions leaded to imbalance resource load. In a typical situation, BES resource partition was quite busy without free slot but still with lots of jobs in idle, while JUNO resources are free and wasted seriously. After moving resources from Torque&Maui to HTCondor in 2016, job scheduling efficiency has been improved a lot. In order to balance the imbalance resource load, we designed an efficient sharing strategy to improve the overall resourceutilization. We created an unified pool shared by all experiments. For each experiment, resources are divided into two parts: dedicated resource and sharing resource. The slots in dedicated resource only run jobs from its own experiment, and the slots in sharing resource are shared by jobs from all experiments. Default ratio of dedicated resource to sharing resource is 1:4. To maximize the sharing effectiveness, the ratio is dynamically adjusted between 0:5 and 4:1 based on the number of jobs submitted by each experiment. We have developed a central control system to decide how many resources can be allocated to each experiment group. This system is implemented at two sides: server side and client side. A management database is built at server side, which is storing resource, group and experiment information. Once the sharing ratio needs to be adjusted, resource group will be changed and updated into database. The resource group information is published to the server buffer in real-time. The client periodically pulls resource group information from server buffer via https protocol And resource scheduling configuration at client side is changed based on the resource group information. With this method, share ratio can be modified and deployed dynamically. Sharing strategy is implemented with HTCondor. ClassAd mechanism and accounting-group in HTCondor facilitate to utilizethe sharing strategy at IHEP computing cluster. With the sharing strategy, resource usage has been improved dramatically.
Highlights
1.1 BackgroundAt IHEP (Institute of High Energy Physics, Chinese Academy of Sciences), computing resources are mainly used for physics data processing, they are purchased and contributed by different experiments including BES[1], JUNO[2], DYW[3], LHAASO[4], etc
With Torque[5]&Maui[6], IHEP computing center had built a local cluster with 50 queues, served for above 10 experiments for over 10 years[7], but it encountered a bottleneck on scale which is unable to perform well under the high throughput situation
Resource usage can not be improved again when it reached around 80%, that is limited by the separated resource partition
Summary
At IHEP (Institute of High Energy Physics, Chinese Academy of Sciences), computing resources are mainly used for physics data processing, they are purchased and contributed by different experiments including BES[1], JUNO[2], DYW[3], LHAASO[4], etc. All the resources are grouped into several dedicated partitions, each partition is only dedicated to its contributor experiment or specific application without sharing. With Torque[5]&Maui[6], IHEP computing center had built a local cluster with 50 queues, served for above 10 experiments for over 10 years[7], but it encountered a bottleneck on scale which is unable to perform well under the high throughput situation
Sign-in/Register to view highlights & summary Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
