Abstract
Complex, large-scale distributed systems are frequently used to solve extraordinary computing, storage and other problems. However, the development of these systems usually requires working with several software components, maintaining and improving a large codebase and also providing a collaborative environment for many developers working together. The central role that such complex systems play in mission critical tasks and also in the daily activity of the users means that any software bug affecting the availability of the service has far reaching effects. Providing an easily extensible testing framework is a pre-requisite for building both confidence in the system but also among developers who contribute to the code. The testing framework can address concrete bugs found in the odebase thus avoiding any future regressions and also provides a high degree of confidence for the people contributing new code. Easily incorporating other people's work into the project greatly helps scaling out manpower so that having more developers contributing to the project can actually result in more work being done rather then more bugs added. In this paper we go through the case study of EOS, the CERN disk storage system and introduce the methods and mechanisms of how to achieve all-automatic regression and robustness testing along with continuous integration for such a large-scale, complex and critical system using a container-based environment.
Highlights
EOS [1] is a multi-protocol disk-only storage system developed at CERN since 2010 to store physics data for LHC and non-LHC experiments
EOS is providing the back-end for a sync & share solution developed at CERN, CERNBox, and a Notebook solution, SWAN [2], which inevitably add new requirements towards the system
Given the distributed nature of the system and the growing development team, it is crucial for the future of the project to have ways to perform automatic system, performance and regression tests, possibly linked to every change committed to the code base
Summary
EOS [1] is a multi-protocol disk-only storage system developed at CERN since 2010 to store physics data for LHC and non-LHC experiments. In addition to the already distributed architecture of EOS, starting from 2017 CERN has developed a new key-value pair store, QuarkDB, used as an EOS namespace back-end alternative in order to scale out its performance and capacity. These new additions along with new access patterns contribute to increased complexity of the overall system architecture. Building a new package release based on the latest commits, deploying this new version on a newly spawned test cluster and running the full battery of tests against this new deployment are part of the continuous integration and testing strategy that we have put in place
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
