Extracting news content with visual unit of web pages

Abstract

The Document Object Model (DOM) provides a tree structure called DOM tree for representing with objects in HTML. Many researchers have considered using leaf nodes of DOM tree as basic objects in extracting information from web pages. However, web pages are more of information blocks which each have a consistent visual format rather than individual DOM tree nodes. And those information blocks do not necessarily have a direct map to DOM tree nodes. In this paper, we propose a visual oriented extraction method that extracts news content by visual unit (vu, for short). Visual units are identified by a top-down approach based on visual features and text features. After that, page content is extracted according to domain characteristic. In experiments, the proposed approach achieves 94.86% accuracy over 700 news web pages from 7 different news sites. The result demonstrates that our method represents a promising approach for news content extraction with visual units and domain characteristic.