The NOAA virtual data system. Improving access to and management of federal environmental data

Abstract

The National Oceanic and Atmospheric Administration's (NOAA) National Environmental Satellite, Data & Information Service (NESDIS) operates three national data centers that are geographically dispersed and provide multi-disciplinary environmental data and information to researchers, government planners, business decision makers, lawyers and other professionals, teachers, students, and the general public. At each of these centers there are multiple formats of data stored on a variety of mass storage devices. Data users must cope with numerous access systems related to the specific type of data and primary customers. An individual familiar with these centers who requires only a single type of data can obtain what he/she needs in a reasonable amount of time. It has become essential for NOAA to change the way it manages these data because of (1) an exponential increase in the volume of data archived, (2) a growing demand for the ability to combine multiple types of environmental data, and (3) customer expectation of instant response to data requests. For many years NOAA's management had a vision of a single unified system to more effectively manage these valuable data and to provide faster and easier customer access. In 1996 funds were appropriated to start development of this system. A new, virtual system is now being implemented to pull these diverse data center operations together. Customers can browse across the distributed data sub-systems and place one order for geophysical, oceanographic, and climatic data, independent of data location, format or storage media. To put these concepts into an operational system with a small budget requires judicious allocation of resources. Numerous storage technologies are being integrated and connected through a scaleable Virtual Private Network (VPN). Other major subsystems of the NOAA Virtual Data System (NVDS) project consist of (1) systems performance monitoring, (2) customer and order management processing, (3) data storage infrastructure, (4) security, (5) metadata management, and (6) customer access mechanisms.