Comparing Test and Production Code Quality in a Large Commercial Multicore System

Abstract

A fundamental goal of software engineering practice is to ensure that code quality is maintained throughout its lifetime. Measuring and maintaining the quality of test code should be as important as measuring production (in-the-field) code. However, test code often seems to be a second class citizen compared to production code in terms of its upkeep and general maintenance. Many of the code features we might expect in test code are either absent or, included when they should not be. In this paper, we investigate four releases of an industrial embedded multi-core system from four perspectives and compare results for test code with corresponding production code. The four perspectives we considered as indicators of code quality. Firstly, we looked at whether test and production code conformed to a set of in-house designated design rules. Secondly, we explored whether test code contained a reasonable proportion of comment to code lines ratio relative to production code. Thirdly, we examined test and production code and the number of assertions in that code. Finally we investigated the relationship between faults and code features. In terms of results, test code did not fare well when compared with production code. An interesting and startling result related to the use of assertions, they were used liberally in test and production code. However, their effect, if triggered, was much larger in production code.