Stability evaluation for text localization systems via metamorphic testing

Abstract

The success of learning techniques in solving a variety of hard AI problems promotes the flourish of recognition-based applications. Many state-of-the-art text localization systems, which can detect and report the positions of text segments in an image, are mainly implemented with learning-based techniques. Data-driven learning raises a series of questions on how to verify, validate and evaluate such learning-based systems. In this paper, we propose a methodology to automatically evaluate the stability of text localization systems via metamorphic relations, where a stable system should output consistent results for similar inputs with the same text segments. We introduce six metamorphic relations that should be preserved in a stable text localization system and define the corresponding metrics for stability evaluation. With the defined metamorphic relations, we apply metamorphic testing techniques to compare the inputs and outputs to evaluate system stability, and further diagnose the causes of inconsistency. The extensive experimentation on both academic and commercial text localization systems demonstrates the effectiveness of our method on stability evaluation for such systems.