Using Broadcast Networks to Create On-demand Extremely Large Scale High-throughput Computing Infrastructures

Abstract

A growing number of very simple parallel applications can benefit from the availability of very large computing resource pools that are exploited in an opportunistic way to considerably speed up the processing of these applications. In particular, many applications can scale out to take advantage of the availability of thousands, or even millions, of processors that can be simultaneously used. Some large-scale distributed computing infrastructures (DCI) have already been successfully assembled to this end. However, these DCI are normally designed to cater to specific kinds of applications, and their assemblage is not only costly, but also time consuming. We propose a novel architecture, named OddCI (On-demand DCI), for the cheap, fast and on-demand instantiation and dismantle of very large DCI. These characteristics allow application users to dynamically commission DCI that can cost-effectively be customized to their particular needs, and disposed after being used. Our approach leverages on broadcast communication as an efficient mechanism to enable aggregation of geographically distributed computing resources, including millions of non-traditional processing devices such as mobile phones, and digital television (DTV) receivers. We discuss how the proposed architecture can be implemented on top of a network of DTV receivers, providing evidences of both the technical, as well as the operational feasibility of the OddCI architecture.