Visualizing and Distributing Decades of Satellite Ocean Remote Sensing Data Products

Abstract

NOAA CoastWatch hosts nearly 2 petabytes of satellite oceanographic data on disk. These freely available data products cover two decades and are primarily stored as standalone NetCDF datasets. Global mapped datasets are processed daily and added to a long chain of previously processed data resulting in a valuable time series of oceanographic data such as sea surface temperature, chlorophyll, sea level anomaly, and sea surface winds. Three decades of experience have taught us that users access data in a myriad of ways. Some download entire collections of global datasets. Others prefer programmatic access using software tools to pull data matching their ‘on premise’ data while others prefer to locate higher level data to integrate into their own processing pipelines. The “NOAA CoastWatch Data Portal” was developed to address the challenges of accommodating our users’ requests. The portal facilitates access to data by providing multiple data-centric services: anonymous FTP/HTTPS service, THREDDS Data Server, and ERDDAP all accessible from a single online map viewer.The NOAA CoastWatch Data Portal map viewer (https://coastwatch.noaa.gov/cw_html/cwViewer.html) serves as the front door to our global data stores by providing visualization, links to product information, and access to datasets in part (i.e., subsets) or whole.In a typical map service, a single dataset is rendered to an image format such as PNG/JPEG for various zoom levels where the number of tiles rendered is 2^(2*zoom level) or a cumulative ~\87,000 images for 8 levels of zoom. The result is a snappy display of data, where the scale and color scheme is determined by the provider allowing little to no user customization. Generating tiles for a multi-decadal dataset results in hundreds of millions of tiles that may or may not be viewed. By contrast, in CoastWatch map tiles are dynamically generated directly from the source NetCDF datafiles using a Web Mapping Service. The result is a slightly slower display but allows user customization and access to pixel-level data. Users can select palettes, data scaling ranges and download the actual data viewed through the map viewer. The map viewer provides simplified access to our backend services.Data selection is performed through a user-controlled calendar and product checkboxes. A change to the date of interest results in a parallel change of that date across multiple products of interest, allowing a user to visualize multiple datasets on a selected date. Products can be promoted to an ‘active layer’ where the data for a selected date is then made persistent so it can be compared to another date. This is useful to identify changes over time. The map viewer has many additional features for data exploration and inter/intra comparison of data products.Data delivery (e.g., the downloading of files to a user) is handled by file services hosted through FTP/HTTPS, THREDDS, and ERDDAP services. FTP and HTTPS allow direct bulk download of entire datafiles or collections. THREDDS and ERDDAP host value-added data services such as OpenDAP allowing users to access partial or reformatted data. Many users access data programmatically using the OpenDAP protocol which facilitates machine-to-machine interoperability. Similarly, services such as ERDDAP’s GRIDDAP allows data subset and reformatting into a variety of formats including JSON, GeoTIFF, CSV, MatLab, etc. Both THREDDS and ERDDAP virtually aggregate data allowing queries to return values for a single pixel over time or download of a NetCDF dataset containing multiple time-levels of spatial data.The CoastWatch Data Portal map viewer will undergo an update over the next year incorporating changes in the underlying JavaScript mapping API. As systems and data move into cloud environments, CoastWatch users may have new expectations from our services. CoastWatch will be called upon to solve new challenges in data distribution. Cloud solutions provide us the opportunity to explore new data and information distribution models.