File-storage cyberinfrastructure for large-scale projects: years before first-light

Abstract

Large ground-based and space-based telescopes are expected to make exciting discoveries in the upcoming decade. These large projects start their construction phase many years before first-light and continue to operate for many years after first-light and usually span multiple countries. The file-storage cyberinfrastructure (file-storage CI) of these largescale projects has to evolve over several years from a conceptual prototype to a highly flexible data distribution network. During this long period the file-storage CI has to transition into multiple stages, starting with a conceptual prototype before first-light, to a large-scale distributed network in production, and finally into a persistent archive once the project is decommissioned. While the project makes these transitions, the file-storage CI has to incorporate several requirements including but not limited to: Technology Evolution, due to changes in Cyberinfrastructure (CI) software or hardware during the lifetime of the project; International Partnerships that are updated during the various phases of the project; and Data Lifecycle that exists in the project. The file-storage and management software's architecture has to be designed with significant consideration of these requirements for these large projects. In this paper, we provide the generic requirements, for file-storage and management cyberinfrastructure in a large project similar to LSST before first-light.