Flexible bandwidth allocation for big data transfer with deadline constraints

Abstract

Large amount of data is being generated at an alarming rate by various systems and devices such as computing systems, cameras and mobile devices. Owing to the huge volume of this data, its processing and analysis cannot be just limited to the place of origin but require to be done at multiple computing sites. A crucial problem is how to efficiently transfer and handle big data in a network, whose performance is affected by the transfer path and bandwidth allocated to the path. In this paper, we propose a bandwidth allocation scheme that flexibly and adaptively allocates bandwidth to big data transfer requests with an objective to maximize the acceptance ratio of the requests while satisfying the deadline constraints. We first develop an optimization programming formulation and then propose a heuristic algorithm to solve the problem due to its non-linear nature. We evaluate the performance of the proposed algorithm through comprehensive simulations on a realistic network topology for two routing scenarios: a pre-computed path scenario and a load-based routing scenario. In both scenarios, the proposed algorithm outperforms baseline algorithms by reducing the rejection ratio by at least 40% and increasing the data transferred by at least 21 TB in a day.