Testing practices for infrastructure as code

Abstract

Infrastructure as code (IaC) helps practitioners to rapidly deploy software services to end-users. Despite reported benefits, IaC scripts are susceptible to defects. Defects in IaC scripts can cause serious consequences, for example, creating large-scale outages similar to the Amazon Web Services (AWS) incident in 2017. The prevalence of defects in IaC scripts necessitates practitioners to implement IaC testing and be aware of IaC testing practices. A synthesis of IaC testing practices can enable practitioners in early mitigation of IaC defects and also help researchers to identify potential research avenues. The goal of this paper is to help practitioners improve the quality of infrastructure as code (IaC) scripts by identifying a set of testing practices for IaC scripts. We apply open coding on 50 Internet artifacts, such as blog posts to derive IaC testing practices. We identify six testing practices that include behavior-focused test coverage, the practice of measuring coverage of IaC test cases in terms of expected behavior. We conclude our paper by discussing how practitioners and researchers can leverage our derived list of testing practices for IaC.