Within-Project Defect Prediction of Infrastructure-as-Code Using Product and Process Metrics

Stefano Dalla Palma,Fabio Palomba,Dario Di Nucci,Damian A. Tamburri

doi:10.1109/tse.2021.3051492

Abstract

Infrastructure-as-code (IaC) is the DevOps practice enabling management and provisioning of infrastructure through the definition of machine-readable files, hereinafter referred to as <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">IaC scripts</i> . Similarly to other source code artefacts, these files may contain defects that can preclude their correct functioning. In this paper, we aim at assessing the role of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">product</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">process</i> metrics when predicting defective IaC scripts. We propose a fully integrated machine-learning framework for IaC Defect Prediction, that allows for repository crawling, metrics collection, model building, and evaluation. To evaluate it, we analyzed 104 projects and employed five machine-learning classifiers to compare their performance in flagging suspicious defective IaC scripts. The key results of the study report <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Random Forest</small> as the best-performing model, with a median AUC-PR of 0.93 and MCC of 0.80. Furthermore, at least for the collected projects, product metrics identify defective IaC scripts more accurately than process metrics. Our findings put a baseline for investigating IaC Defect Prediction and the relationship between the product and process metrics, and IaC scripts’ quality.

Highlights

Modern software relies on the use of automation at both development and operations levels, an engineering strategy known as DevOps [1]; the software code driving such automation is collectively known as Infrastructure-as-Code (IaC)
We propose the RADON FRAMEWORK FOR IAC DEFECT PREDICTION, a fully integrated MachineLearning-based framework that allows for repository crawling, metrics collection, model building, and evaluation
Other 14 metrics in this set have been already studied by Rahman and Williams [8] for predicting defects in Puppet code: as such, we considered them to verify their generalizability when employed for predicting Ansible defects

Summary

Introduction

Modern software relies on the use of automation at both development and operations levels, an engineering strategy known as DevOps [1]; the software code driving such automation is collectively known as Infrastructure-as-Code (IaC). DevOps entails adopting a set of organizational and technical practices, e.g., continuous integration, continuous deployment, blending development, and operation teams, to survive as an organization in the modern digital ecosystem and digital market, which demands fast and early releases, continuous software updates, constant evolution of market needs, and adoption of scalable technologies such as Cloud computing. In this context, IaC is the DevOps practice of describing complex and (usually) Cloud-based deployments using machine-readable code. Ansible is gaining traction in the last years as a simple and agent-less (i.e., no master node) alternative to other more complex IaC technologies such as Chef and Puppet

Objectives

Methods

Results

Conclusion