DevOps Improvements for Reduced Cycle Times with Integrated Test Optimizations for Continuous Integration

Abstract

DevOps, as a growing development practice that aims to enable faster development and efficient deployment of applications without compromising on quality, is often hampered by long cycle times. One contributing factor to long cycle times in DevOps is long build time. Automated testing in continuous integration is one of the build stages that is highly prone to long run-time due to software complexity and evolution, and inefficient due to unoptimized testing approaches. To be cost-effective, testing in continuous integration needs to use only a fast-running set of comprehensive tests that are able to ensure the level of quality needed for deployment to production. Known approaches use time-aware test selection methods to improve time-efficiency of continuous integration testing by providing optimized combinations and order of tests with respect to decreased run-time. However, focusing on time-efficiency as the sole criterion in DevOps often jeopardizes the quality of software deliveries. This paper proposes a technique that integrates fault-based and risk-based test selection and prioritization optimized for low run-time, to improve time-effectiveness of continuous integration testing, and thus reduce long cycle times in DevOps, without compromising on quality. The technique has been evaluated in testing of a large-scale configurable software in continuous integration, and has shown considerable improvement over industry practice with respect to time-efficiency.