A framework for data-driven algorithm testing

Abstract

We describe the requirements, design, architecture and implementation of a framework that facilitates the setup, management and realisation of data-driven performance and acceptance tests for algorithms. The framework builds on standard components, supports distributed tests on heterogeneous platforms, is scalable and requires minimum integration efforts for algorithm providers by chaining command line driven applications. We use XML as test specification language, so tests can be set up in a declarative way without any programming effort and the test specification can easily be validated against an XML schema. We consider a test scenario where each test consists of one to many test processes and each process works on a representative set of input data that are accessible as data files. The test process is built up of operations that are executed successively in a predefined sequence. Each operation may be one of the algorithms under test or a supporting functionality (e.g. a file format conversion utility). The test definition and the test results are made persistent in a relational database. We decided to use a J2EE compliant application server as persistence engine, thus the natural choice is to implement the test client as Java application. Java is available for the most important operating systems, provides control of OS-processes, including the input and output channels and has extensive support for XML processing.