Creating Repeatable Computer Science and Networking Experiments on Shared, Public Testbeds

Abstract

There are many compelling reasons to use a shared, public testbed such as GENI, Emulab, or PlanetLab to conduct experiments in computer science and networking. These testbeds support creating experiments with a large and diverse set of resources. Moreover these testbeds are constructed to inherently support the repeatability of experiments as required for scientifically sound research. Finally, the artifacts needed for a researcher to repeat their own experiment can be shared so that others can readily repeat the experiment in the same environment. However using a shared, public testbed is different from conducting experiments on resources either owned by the experimenter or someone the experimenter knows. Experiments on shared, public testbeds are more likely to use large topologies, use scarce resources, and need to be tolerant to outages and maintenances in the testbed. In addition, experimenters may not have access to low-level debugging information. This paper describes a methodology for new experimenters to write and deploy repeatable and sharable experiments which deal with these challenges by: having a clear plan; automating the execution and analysis of an experiment by following best practices from software engineering and system administration; and building scalable experiments. In addition, the paper describes a case study run on the GENI testbed which illustrates the methodology described.