Online Storage and Content Distribution System at a Large Scale: Peer-Assistance and Beyond

Abstract

Summary form only given. Online storage and content distribution systems within the "cloud" of the Internet have provided a wide variety of services ranging from file sharing, software downloads, to stored or live video streaming, to a large community. With the increasing popularity of such systems, however, server and bandwidth costs have become prohibitively expensive, as files are hosted in either content distribution networks or dedicated large data centers. Rapidshare, one of the most well-known one-click hosting systems, deployed a total of 1500 Terabytes of online storage in its data centers, in Asia alone. Skyrocketing bandwidth costs from server-based architectures have made it necessary for all online content distribution systems that remain free of charge to impose certain restrictions, including download bandwidth limits per day, file size limitations, as well as maximum file online available time. Though it seems natural to take advantage of peer bandwidth contributions to mitigate server bandwidth costs, it is nevertheless non-trivial to design a completely new system that utilizes peer bandwidth contributions in a complementary fashion without sacrificing the ease of use, reliability, and performance of one-click hosting. In this talk, we share our experiences in the design, implementation and deployment of FS2You, a large-scale online content distribution system, which provides one-click hosting services with peer bandwidth assistance. FS2You is designed to dramatically mitigate server bandwidth costs, while maintaining the ease of use and performance comparable to the best server-based solutions. In response to a number of fundamental technical challenges, we present the key architectural and protocol design in this system. Further, based on real-world traces involving millions of users over a long period of time, we reveal a number of interesting observations on user behavior, service availability, file characteristics and server involvement, which we believe are of great relevance for the future system design.