An Evaluation of a Market Based Resource Trading in a Multi-campus Compute Co-operative (CCC)

Abstract

Computational and data scientists at universities are often limited by the quantity and diversity of the shared resources available at their institution. Access cost for these resources are often uniform, that is, it is not differentiated based on job priority or resource requirements. This flat access policy on shared resources often lead to sub-optimal values for the institutions, and researchers with special requirements (i.e. GPU, large-memory, etc.) often have to wait significantly longer to get their job scheduled. A market-based resource trading in a multi-campus Compute Co-operative can lead to higher aggregated value for the co-operative as well as provide significant benefits for the individual institutions by scheduling jobs opportunistically when resources of one campus are over-subscribed and by placing jobs efficiently based on resource requirements. In this paper, we evaluate a resource allocation scheme in a multi-campus environment, (i.e. CCC [10]) based on job priority and resource cost, with the provision for resource trading between campuses. We collected real data traces from three (3) universities over a month and conducted a simulation to evaluate the effectiveness of our resource trading approach over the existing single institution flat rate allocation policy. Our simulation shows that, with CCC and market-based resource trading, the aggregated institutional value for the co-operative increases by 15% and the average wait time for the jobs reduce by 49%.