Guest Editorial: Special Section on SC19 Student Cluster Competition

Abstract

Reproducibility is foundational to solid scientific and technical research, and the ability to repeat the research that produced is a key approach for confirming the validity of a new scientific discovery. The IEEE Transactions on Parallel and Distributed Systems (TPDS) is committed to enabling reproducible research through transparency and the availability and potential reuse of code. As part of its Reproducibility Initiative, TPDS is exploring post-publication peer review of code associated with articles published in TPDS. Authors who have published in TPDS can make their published article more reproducible and earn a reproducibility badge by submitting their associated code for post-publication peer review. However, the nature of the parallel and distributed systems research covered by TPDS makes it challenging to evaluate code and data challenging for reproducibility. This is because such an evaluation may require access to specific hardware, system architectures and scales, OS configurations, and so on, which may not be feasible or practical. Consequently, TPDS is exploring an alternate approach where members of the community can submit short, supplemental ‘critique’ papers that present their experiences in reproducing published results using the artifacts and/or evaluations or experiences with published artifacts. These supplemental paper submissions are reviewed and, if accepted, are linked to the original publication and are citable, serving to validate the reproducibility of the original publication. This special section, consisting of a primary paper and 6 critique papers that reproduce the results of the primary paper, is the initial pilot of this approach. The special section builds on the efforts of the SC19 Student Cluster Competition, which was part of the SC19 conference (https://sc19.supercomputing.org/) and represents a step forward in enabling reproducible research publications in parallel and distributed systems.