Improving developers' confidence in test results of multi-threaded systems

Abstract

Testing multi-threaded systems is challenging, especially due to their inherent non-determinism. As a result, considering the same test, we can have success and failure results for different test executions. This may happen due to bugs that are only revealed on certain thread interleavings, and which are difficult to reproduce. However, this may also be caused by a problem with the test. When the latter case happens, developers may waste a lot of time searching a bug that does not exist, or they may lose trust in the test, maybe not investigating real bugs. With this work, we want to support tests development through an approach to avoid test failures caused by a common problem with multi-threaded systems testing: assertions performed too early or too late. The approach's basic idea is to use thread monitoring and other techniques to avoid such problems during the development of tests and hence increase the developers' confidence in test results.