Not just for the birds: archiving massive data sets

Abstract

Among the 7,000 bird species whose songs are recorded at the Cornell Laboratory of ornithology's Macaulay Library, the prothonotary warbler doesn't seem like a standout. Referring to the sharp way the bird punctuates every high-pitched tweet, the sound is very high-frequency and bursty. Accuracy is paramount for Grotke and his colleagues, who are converting the world's largest animal recording collection from analog to digital to preserve as much information as possible for future scientists. Since 1999, they've digitized one-third of the collection and amassed 4 terabytes (Tbytes) of data. The sound collection dates back to 1929. Macaulay Library engineers are engaged in a particularly dramatic race against time, but they aren't alone in their need to preserve massive amounts of information. From high-energy physics to climate science to biology, new instruments are gathering more experimental data that need to be retained for the long term. Meanwhile, other scientists need to retain results from huge computer simulations. All around the world, data is being lost. To preserve its collections, the Macaulay Library is blazing a trail that others will have to follow