Enterprise IT support for NOAA archives

Abstract

The National Oceanic and Atmospheric Administration (NOAA) is responsible for archiving and disseminating environmental data collected by a variety of ground and space-based observing systems. Integrating data management across NOAA represents a significant change in NOAA business practices that will require evolution of the management structures and approaches used in NOAA for data management. Historically, and dissemination activities have been implemented in dedicated stovepipe systems, each focusing on the needs of a particular data collection and its associated user community. Exponential data volume growth, increasing data diversity, user requirements for common and consistent access methods across data types, and the need for economic efficiency are driving a consolidation of the information technology (IT) systems that support NOAA archives. The Comprehensive Large Array-data Stewardship (CLASS) is a result of this consolidation; CLASS is the enterprise IT solution in support of NOAA archives. Against this backdrop, the meaning of archive is itself changing as the challenges of long-term digital preservation are better understood. It is no longer sufficient to simply preserve bits. Instead, information content is the target of preservation, with bits simply providing a medium for representing information. The ISO standard Reference Model for an Open Archival Information System (OAIS-RM) provides a conceptual framework that describes the responsibilities and activities of organizations attempting to preserve digital information. Applying the OAIS-RM to NOAA's efforts provides the means to identify high-level responsibilities, delegate roles to various NOAA organizational components, and describe the means by which these entities work together to fulfill requirements. At the highest level, NOAA archives maintain and apply information- and user-specific knowledge while CLASS provides IT capabilities in support of activities. Together, the archives and CLASS provide the necessary ingredients to fulfill NOAA's mission. Currently, CLASS holds POES, DMSP, GOES, and MetOp data. Planned future campaigns include NPP, NPOESS, EOS MODIS, NEXRAD, and model and in-situ data. To ensure the preservation of these data, CLASS has implemented a multiple site distributed system that replicates data and metadata holdings automatically. Additionally, CLASS supports the deployment of points of presence that allow the co-location of key CLASS capabilities with high-volume or otherwise critical data producers or consumers to mitigate economic constraints or meet quality-of-service requirements. Among the key challenges facing CLASS are: massive increases in data volumes, the extraordinary diversity of NOAA data, evolving user needs and requirements, and the ever-changing technology environment. CLASS must address all of these challenges - and more - at the same time it ensures the safeguarding and availability of its current holdings. CLASS is evolving toward a service-oriented architecture that provides the flexibility, scalability, and generality necessary to accommodate the vast amount of change in its environment and at the same time provide interoperability with other systems and systems of systems necessary to support NOAA's position in the environmental data arena. Adoption of standards is critical to NOAA's success, and CLASS is working with NOAA's Global Earth Observation Integrated Data Environment (GEO-IDE) project to identify standards that will ensure CLASS'S interoperability with other NOAA systems. The lack of broad, uniform utilization of IT standards that adequately meet NOAA's data integration needs is arguably the most acute factor contributing to the weakness of data integration today. GEO-IDE will ultimately ensure that key NOAA systems, including CLASS, are interoperable within NOAA, with US-Global Earth Observation (US-GEO), and ultimately with US-Global Earth Observation (GEOSS). In addition to its work with GEO-IDE, CLASS is also developing pilots for geospatial capabilities, and dissemination standards to facilitate interoperability and meet user needs. This paper will present the guiding principles and lessons learned while developing CLASS to date. We will discuss our challenges, accomplishments, and future work, as well as how CLASS aligns with NOAA's strategic goals.