Abstract
The CernVM File System (CernVM-FS) provides a scalable and reliable software distribution service implemented as a POSIX read-only filesystem in user space (FUSE). It was originally developed at CERN to assist High Energy Physics (HEP) collaborations in deploying software on the worldwide distributed computing infrastructure for data processing applications. Files are stored remotely as content-addressed blocks on standard web servers and are retrieved and cached on-demand through outgoing HTTP connections only. Repository metadata is recorded in SQLite catalogs, which represent implicit Merkle treeencodings of the repository state. For writing, CernVM-FS follows a publish-subscribe pattern with a single source of new content that is propagated to a large number of readers. This paper focuses on the work to move the CernVM-FS architecturein the direction of a responsive data distribution system. A new distributed publication backend allows scaling out large publication tasks across multiple machines, reducing the time to publish. For the faster propagation of new published content, the addition of a notification system allows clients to subscribe to messages about changes in the repository and to request new root catalogs as soon as they become available. These devel-opments make CernVM-FS more responsive and are particularly relevant for use cases where a short propagation delay from repository down to individual clients is important, such as using CernVM-FS as an AFS replacement for distributing software stacks. Additionally, they permit the implementation of more complex workflows, with producer-consumer pipelines, as for example in the ALICE analysis trains system.
Highlights
The CernVM File System (CernVM-FS) is the main software delivery system bringing the high-energy physics (HEP) experiment software stacks to the world-wide LHC computing grid (WLCG) [1]
The CernVM File System (CernVM-FS) provides a scalable and reliable software distribution service implemented as a POSIX read-only filesystem in user space (FUSE)
This paper focuses on the work to move the CernVM-FS architecture in the direction of a responsive data distribution system
Summary
The CernVM File System (CernVM-FS) is the main software delivery system bringing the high-energy physics (HEP) experiment software stacks to the world-wide LHC computing grid (WLCG) [1]. In addition to storing metadata, a catalog references the content-addressed chunks of all the files in the repository. The publisher, called “release manager” in CernVM-FS terminology, is a machine with write access to the authoritative storage of a repository. When the TTL expires, clients download the latest version of the manifest, making the updated content of the repository available. This architecture is robust and efficient, and has been proven to work for the most common use cases [2]. Most CernVM-FS use cases are not sensitive to the propagation delay of changes from repository to clients; in cases where this delay does need to be minimized, the TTL-based approach introduces a lower bound for this delay. This paper describes work done to make the CernVM-FS publication architecture more responsive; removing the limitations listed here allows horizonal scaling to accommodate an increase in the size of publication payloads as well as implementing new use cases previously prohibited by the TTL-based change propagation
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Free) 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
