Chart decoder: Generating textual and numeric information from chart images automatically

Abstract

Charts are commonly used as a graphical representation for visualizing numerical data in digital documents. For many legacy charts or scientific charts, however, underlying data is not available, which hinders the process of redesigning more effective visualizations and further analysis of charts. In response, we present Chart Decoder, a system that implements decoding of visual features and recovers data from chart images. Chart Decoder takes a chart image as input and generates the textual and numeric information of that chart image as output through applying deep learning, computer vision and text recognition techniques. We train a deep learning based classifier to identify chart types of five categories (bar chart, pie chart, line chart, scatter plot and radar chart), which achieves a classification accuracy over 99%. We also complement a textual information extraction pipeline which detects text regions in a chart, recognizes text content and distinguishes their roles. For generating textual and graphical information, we implement automated data recovery from bar charts, one of the most popular chart types. To evaluate the effectiveness of our algorithms, we evaluate our system on two corpora: 1) bar charts collected from the web, 2) charts randomly made by a script. The results demonstrate that our system is able to recover data from bar charts with a high rate of accuracy.