Abstract
IceCube's simulation production relies largely on dynamic, heterogeneous resources spread around the world. Datasets consist of many thousands of job workflow subsets running in parallel as directed acyclic graphs (DAGs) and using varying resources. IceProd is a set of Python daemons which process job workflow and maintain configuration and status information on jobs before, during, and after processing. IceProd manages a complex workflow of DAGs to distribute jobs across all computing grids and optimize resource usage. IceProd2 is a new version of IceProd with substantial increases in security, reliability, scalability, and ease of use. It is undergoing testing and will be deployed this fall.
Highlights
IceProd has served the IceCube collaboration well over the last 8 years, processing Monte Carlo simulations, detector data, and analysis levels [1]
Datasets consist of many thousands of job workflow subsets running in parallel as directed acyclic graphs (DAGs) and using varying resources
IceProd manages a complex workflow of DAGs to distribute jobs across all computing grids and optimize resource usage
Summary
IceProd has served the IceCube collaboration well over the last 8 years, processing Monte Carlo simulations, detector data, and analysis levels [1]. It has run thousands of CPU-core years and stored over 2PB of data. It has experienced its share of issues and the initial software is running into fundamental design limitations. This article describes that history and the development of a second version of IceProd. The new version is a complete rewrite of the code base, fixing long-standing issues and preparing for the increased load of the few years
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
