Tackling the Reproducibility Problem in Systems Research with Declarative Experiment Specifications

TitleTackling the Reproducibility Problem in Systems Research with Declarative Experiment Specifications
Publication TypeConference Paper
Year of Publication2015
AuthorsJimenez, I, Maltzahn, C, Mohror, K, Lofstead, J, Arpaci-Dusseau, A, Arpaci-Dusseau, R
Conference NamePDSW'15: Proceedings of the 10th Workshop on Parallel Data Storage
Date Published11/2015
Conference LocationAustin, TX, USA
Abstract

Validating experimental results in the field of computer systems is a challenging task, mainly due to the many changes in software and hardware that computational environments go through. Determining if an experiment is reproducible entails two separate tasks: re-executing the experiment and validating the results. Existing reproducibility efforts have focused on the former, envisioning techniques and infrastructures that make it easier to re-execute an experiment. In our work, we focus on the latter by analyzing the validation workflow that an experiment re-executioner goes through. We notice that validating results is done on the basis of experiment design and high-level goals, rather than exact quantitative metrics. Based on this insight, we introduce a declarative format for specifying the high-level components of an experiment as well as describing generic, testable conditions that serve as the basis for validation. We present a use case in the area of distributed storage systems to illustrate the usefulness of this approach.