Experimental studies of data transport and data access of earth-science data over networks with high bandwidth delay products

Abstract

Although the amount of earth-science data is growing rapidly, as is the availability of high-performance networks, our ability to access large remote earth-science data sets is still very limited. This is particularly true of networks with high bandwidth-delay products (BDP), such as those between the US and Europe. Recently, several network protocols have emerged that improve the situation and hold the promise of being much more effective than striped TCP (wherein multiple, parallel connections are used to increase data throughput). In this paper, we report on experimental studies using one of these new protocols called UDT, and compare UDT to other approaches. In addition, we consider the effectiveness of these new protocols when reading and writing data to and from disk over high BDP networks. We also consider the problem of accessing remote data by attribute over these same networks. We show that with the appropriate protocol, the speed of accessing data across the Atlantic can be improved significantly. We note that the UDT protocol used here can be deployed as an application library for earth-science applications and neither requires upgrades to existing network infrastructure, such as routers, nor to the Linux kernels on the servers involved.