Document Segmentation Research Articles

Automatic generation of scientific papers for data augmentation in document layout analysis

Document layout analysis is an important task to extract information from scientific literature. Deep-learning solutions for document layout analysis require large collections of training data that are not always available. We generate a large number of synthetic pages to subsequently train a neural network to perform document object detection. The proposed pipeline allows users to deal with less common layouts for which it is not easy to find large annotated datasets. High-quality annotations for a small collection of papers are obtained through a semi-automatic approach. Then, a generative model, based on LayoutTransformer, is used to generate plausible layouts that are subsequently populated with random information to perform data augmentation. We evaluate the proposed method considering scientific articles with two different types of layouts: double and single columns. For double-column papers, we improve detection by 1% starting from 385 manually annotated scientific articles. For single-column papers, we improve detection by 49% starting from 218 articles.

An Arabic Manuscript Regions Detection, Recognition and Its Applications for OCRing

The problem of Region of Interest (RoI) in document layout analysis and document recognition has recently become an essential topic in OCRing systems. Arabic manuscript layout analysis and OCRing recognition using language detection, document category, and RoI with Keras and TensorFlow are terms of the state-of-the-art that should be investigated. This article investigates the problem of Arabic manuscript recognition problems with respect to in OCRing-based recognition. A new framework architecture, which integrates Fast Gradient Sign Method (FGSM) using Keras and TensorFlow with adversarial image generation during training procedure is proposed. Also, the article tries to improve the OCRing accuracy of the image enhancement, alignment, layout analysis, and recognition using deep learning in multilingual system. RoIs detections will be performed using a custom trained deep learning model using bounding box regression with Keras and TensorFlow. This topic investigates an extension of Page Segmentation Method (PSM) to enhance OCRing parameter modes and enhances Arabic OCRing system accuracy from reinforcement strategy. Therefore, the article achieves a significant improvement of OCRing results due to the three parameters: language identification, document category, and RoI types (Table, Title, Paragraph, figure, and list). This model is based on “region proposal algorithm” as a basis of CNN object detectors to find the number of the RoIs. Therefore, the proposed framework performs three distinctive tasks: (1) CNN architecture for adversarial training, (2) an implementation of the FGSM with Keras and TensorFlow, and (3) an adversarial training script implementation with the CNN and the FGSM method. The experiments on Arabic manuscript dataset including Arabic text, English/Arabic documents, and Latin digits’ datasets, demonstrate the accuracy of the proposed method. Moreover, the proposed framework performs well and succeeded in defending against adversarial attacks or adversarial images. The experimental results on our collected dataset illustrate the novelty of our proposed framework over the other existing PSM methods to be extended and updated to improve the quality of the OCRing system. The results show that the influence of PSM after expanding using the RoI types, language ID, and document/manuscript category can improve the OCRing accuracy. Also, the experimental results show significant performance by the new framework model with accuracy reached to 99% compared to relative methods.

MFRCNN: Marshalled FRCNN with optimized reading order in XY tree for document layout analysis in scientific research articles

Recently computer vision and NLP based techniques have been employed for document layout analysis where different types of elements in the document and their relative position are identified. This process is trickier as there are blocks which are structurally similar but semantically different such as title, text etc. This works attempts to use region-based CNN architecture (F-RCNN) for determining five different sections in the scientific articles. To improve the performance of detection algorithm, reading order is used as an additional feature and this model is known as MF-RCNN. First, an algorithm is formulated to find the reading order in documents which adopts Manhattan-layout using a color-coding scheme. Secondly, this information is fused with the input image without changing its shape. Experimental results show that MF-RCNN which uses the reading order performs better when compared with F-RCNN when tested on Publaynet dataset.

Semantic Document Layout Analysis of Handwritten Manuscripts

A document layout can be more informative than merely a document’s visual and structural appearance. Thus, document layout analysis (DLA) is considered a necessary prerequisite for advanced processing and detailed document image analysis to be further used in several applications and different objectives. This research extends the traditional approaches of DLA and introduces the concept of semantic document layout analysis (SDLA) by proposing a novel framework for semantic layout analysis and characterization of handwritten manuscripts. The proposed SDLA approach enables the derivation of implicit information and semantic characteristics, which can be effectively utilized in dozens of practical applications for various purposes, in a way bridging the semantic gap and providing more understandable high-level document image analysis and more invariant characterization via absolute and relative labeling. This approach is validated and evaluated on a large dataset of Arabic handwritten manuscripts comprising complex layouts. The experimental work shows promising results in terms of accurate and effective semantic characteristic-based clustering and retrieval of handwritten manuscripts. It also indicates the expected efficacy of using the capabilities of the proposed approach in automating and facilitating many functional, real-life tasks such as effort estimation and pricing of transcription or typing of such complex manuscripts.

HSCA-Net: A Hybrid Spatial-Channel Attention Network in Multi-Scale Feature Pyramid for Document Layout Analysis

Document images often contain various page components and complex logical structures, which makes document layout analysis task challenging. For most deep learning based document layout analysis methods, convolutional neural networks (CNNs) are adopted as the image feature extraction networks. In this paper, a hybrid spatial-channel attention network (HSCA-Net) is proposed to improve feature extraction capability by exerting attention mechanism to explore more salient properties within document pages. The HSCA-Net contains two modules: spatial attention module (SAM) and channel attention module (CAM). They are embedded in the multi-scale feature network by lateral attention connection. SAM extracts contextual information with learning offset in spatial dimension and CAM performs feature recalibration by focusing more on feature channels with important contents. The lateral attention connection is to incorporate SAM and CAM into multi-scale feature pyramid network and retain more of the original feature information. The effectiveness and adaptability of HSCA-Net are evaluated through multiple experiments on publicly available datasets PubLayNet, ICDAR-POD and Article Regions. The mAP on these datasets is as high as 0.940,0.939 and 0.967 respectively, which demonstrate that our HSCA-Net achieves competitive results on document layout analysis task. Text.

Augmentation-based Pseudo-Ground truth Generation for Deep Learning in Historical Document Segmentation for Greater Levels of Archival Description and Access

The successful use of deep learning solutions for document image segmentation typically relies on a large number of manually labeled ground truth examples, which is expensive to obtain for historical document images that have significant noise effects and variation. At the same time, successful applications of deep learning solutions for document image segmentation have rich potential to facilitate greater levels of description in archival collections (e.g., at and below the item-level). These greater levels of description are critical to increasing access and use of archival collections across an array of research domains. In response, this article investigates whether an augmentation-based approach to generating pseudo-ground truth can be effective with a limited number of labeled images in a document segmentation application. The rationale is that if we can decrease the cost of generating ground truth through augmentation-based approaches, we can use these approaches as part of the description and access pipelines for historical library and archival collections. In this initial exploration, we first generate synthetic images and corresponding pseudo-ground truth using a set of existing degradation-based augmentation models from a small number of labeled actual images. When generating synthetic images, we control the visual quality distortion based on OCR word-level confidence to avoid generating images unlikely to be present in the dataset. Then, we perform several investigations to examine the impact of incorporating pseudo-ground truth data in the training of the deep learning network dhSegment and further evaluate the use of multiple combinations of degradation models. We also assess the generalizability of the approach by applying the trained network on a larger dataset. Our investigations primarily use real-world datasets known to have significant noise effects. Results show that augmentation-based pseudo-ground truth generation is capable of improving segmentation performance with the use of the full original dataset and requires only 30% of the original dataset. Results also show that using more than three degradation models is likely to cause overfitting during training. Furthermore, we show that a segmentation network trained on pseudo-ground truth data has generalization capability.

DocBed: A Multi-Stage OCR Solution for Documents with Complex Layouts

Digitization of newspapers is of interest for many reasons including preservation of history, accessibility and search ability, etc. While digitization of documents such as scientific articles and magazines is prevalent in literature, one of the main challenges for digitization of newspaper lies in its complex layout (e.g. articles spanning multiple columns, text interrupted by images) analysis, which is necessary to preserve human read-order. This work provides a major breakthrough in the digitization of newspapers on three fronts: first, releasing a dataset of 3000 fully-annotated, real-world newspaper images from 21 different U.S. states representing an extensive variety of complex layouts for document layout analysis; second, proposing layout segmentation as a precursor to existing optical character recognition (OCR) engines, where multiple state-of-the-art image segmentation models and several post-processing methods are explored for document layout segmentation; third, providing a thorough and structured evaluation protocol for isolated layout segmentation and end-to-end OCR.

IMPROVED OCR QUALITY FOR SMART SCANNED DOCUMENT MANAGEMENT SYSTEM

The quality of the document images is a crucial factor for the performance of an Optical Character Recognition (OCR) model. Various issues from the input data hinder the recognition success such as heterogeneous layouts, skewness and proportional fonts. This paper investigated several algorithms for data pre-processing including image deskewing, table and document layout analysis to improve the accuracy of the OCR model and then built an end-to-end scanned document management system. We verified the algorithms using a well-known OCR software namely Tesseract. The experiments on a real dataset shown that our methods can accurately process document images with arbitrary angles of rotation, and different layouts. As a result, the accuracy by words of Tesseract can boost 23% for documents with complex structures. The quality of the output text allows to build a system to store and search documents efficiently. Index

Reading order detection on handwritten documents

Recent advances in Handwritten Text Recognition and Document Layout Analysis have made it possible to convert digital images of manuscripts into electronic text. However, providing this text with the correct structure and context is still an open problem that needs to be solved to actually enable extracting the relevant information conveyed by the text. The most important structure needed for a set of text elements is their reading order. Most of the studies on the reading order problem are rule-based approaches and focus on printed documents. Much less attention has been paid so far to handwritten text documents, where the problem becomes particularly important—and challenging. In this work, we propose a new approach to automatically determine the reading order of text regions and lines in handwritten text documents. The task is approached as a sorting problem where the order-relation operator is automatically learned from examples. We experimentally demonstrate the effectiveness of our method on three different datasets at different hierarchical levels.

Character Detection and Segmentation of Historical Uchen Tibetan Documents in Complex Situations

Tibetan is a low-resource language, and Tibetan culture carried by historical Tibetan documents is an important part of Chinese civilization. The study of historical Tibetan documents is of great significance to the protection of Tibetan culture and the promotion of Chinese culture. Character segmentation is an important step in image analysis and recognition of historical Tibetan documents. However, the following three challenges prevent solving problems of character segmentation in historical Tibetan documents: 1) the text lines have different degrees of tilt and twist; 2) there are many complex situations such as overlapping, crossing, touching and breaking character strokes; and 3) these documents are written by different people with different stroke styles. To resolve these problems, we propose a character segmentation method based on key feature information for historical Tibetan documents. The proposed method consists of three parts: 1) projection and syllable point location information are used to shorten the text lines of historical Tibetan documents and establish a character block database; 2) the local baseline of the character block is detected by using the location information of syllable points or combined with horizontal projection and straight line detection, and the character block is divided into two areas above and below the baseline, and different segmentation methods are adopted; and 3) in view of the large difference in stroke styles, three stroke attribution distances are used to complete the attribution. The experimental results show that the method proposed in this paper can effectively solve the problem of character segmentation of historical Tibetan documents and achieve a better character segmentation effect, which also provides a reference for the relevant document character segmentation.

Visual domain knowledge-based multimodal zoning for textual region localization in noisy historical document images

Document layout analysis, or zoning, is important for textual content analysis such as optical character recognition. Zoning document images such as digitized historical newspaper pages are challenging due to noise and quality of the document images. Recently, effective data-driven approaches, such as leveraging deep learning, have been proposed, albeit with the concern of requiring larger training data and thus incurring additional cost of ground truthing. We propose a zoning solution by incorporating a knowledge-driven document representation, gravity map, into a multimodal deep learning framework to reduce the amount of time and data required for training. We first generate a gravity map for each image, considering the centroid distance and area between a cell in a Voronoi tessellation and its content to encode visual domain knowledge of a zoning task. Second, we inject the gravity maps into a deep convolution neural network (DCNN) during training, as an additional modality to boost performance. We report on two investigations using two state-of-the-art DCNN architectures and three datasets: two sets of historical newspapers and a set of born-digital contemporary documents. Evaluations show that our solution achieved comparable segmentation accuracy using fewer training epochs and less training data compared to a naïve training scheme.

Document image layout analysis via explicit edge embedding network

Layout analysis from a document image plays an important role in document content understanding and information extraction systems. While many existing methods focus on learning knowledge with convolutional networks directly from color channels, we argue the importance of high-frequency structures in document images, especially edge information. In this paper, we present a novel document layout analysis framework with the Explicit Edge Embedding Network (E3Net). Specifically, the proposed network contains the edge embedding block and dynamic skip connection block to produce detailed features, as well as a lightweight fully convolutional subnet as the backbone for the effectiveness of the framework. The edge embedding block is designed to explicitly incorporate the edge information from the document images. The dynamic skip connection block aims to learn both color and edge representations with learnable weights. In contrast to the previous methods, we harness the model by using a synthetic document approach to overcome data scarcity. The combination of data augmentation and edge embedding is important toward a more compact representation than directly using the training images with only color channels. We conduct experiments using the proposed framework on three document layout analysis benchmarks and demonstrate its superiority in terms of effectiveness and efficiency over previous approaches.

Combining Visual and Textual Features for Semantic Segmentation of Historical Newspapers

The massive amounts of digitized historical documents acquired over the last decades naturally lend themselves to automatic processing and exploration. Research work seeking to automatically process facsimiles and extract information thereby are multiplying with, as a first essential step, document layout analysis. If the identification and categorization of segments of interest in document images have seen significant progress over the last years thanks to deep learning techniques, many challenges remain with, among others, the use of finer-grained segmentation typologies and the consideration of complex, heterogeneous documents such as historical newspapers. Besides, most approaches consider visual features only, ignoring textual signal. In this context, we introduce a multimodal approach for the semantic segmentation of historical newspapers that combines visual and textual features. Based on a series of experiments on diachronic Swiss and Luxembourgish newspapers, we investigate, among others, the predictive power of visual and textual features and their capacity to generalize across time and sources. Results show consistent improvement of multimodal models in comparison to a strong visual baseline, as well as better robustness to high material variance.

Deep Learning in Time-Frequency Domain for Document Layout Analysis

Document layout analysis plays an important role in the area of Document Understanding. It is responsible for identifying and classifying the different components of digital documents. Currently, there is no universal algorithm that fits all types of digital documents. This work presents a novel approach for identifying tables, figures, isolated equations and text regions in scientific papers using deep learning and computer vision techniques. Our proposed approach is a three-stage system: (i) obtaining the spectrograms of the horizontal and vertical intensity histograms of segmented regions of interest; (ii) labeling segmented regions of interest into text, table, and figures using a deep convolutional neural network classifier; and (iii) identifying isolated equations in text regions using Bag of Visual Words (BOVW) with Zernike moments. We built a new dataset composed of 11007 papers to perform the experiments, using two common segmentation metrics to evaluate our model: (1) Adjusted Rand Index (ARI) and (2) Variation of Information (VI). The proposed document layout analysis system reached an overall accuracy of 96.2685%, outperforming prior art with a less computational cost.

Arabic Documents Information Retrieval for Printed, Handwritten, and Calligraphy Image

This paper presents a new computational backend model that supports Arabic document information retrieval (ADIR) as a dataset and OCR services. Therefore, different services that support document analysis, retrieving, processing including dataset preparation, and recognition will be discussed. Consequently, ADIR services provide general functions of the Arabic OCR to compose many other services in the OCR domain. Furthermore, the proposed work can provide accessing different methods of document layout analysis with a platform where they can share and handle such methods (services) without any setup requirements. One of the used datasets composed from 16,800 Arabic letters written by 60 writers. Each writer wrote each letter from Alif to Ya 10 times in two forms. The forms were scanned at 300 DPI resolution and are segmented in two sets: training set with 13,440 letters for 48 images per class label, and testing set with 3,360 letters to 120 images per class label Convolutional neural network (CNN) is used and adapted for Arabic handwritten letters classification. In an experimental test, we showed that our results outperform 100% classification accuracy rate on testing images. Therefore, the ADIR services provide a “service description”, which includes an interface and a server's URL. The interface allows communication process between clients and services. Although, in this article we evaluate IR results and compared them with respect to corrected equivalent.

BINYAS: a complex document layout analysis system

Document layout analysis (DLA) is an irreplaceable pre-requisite for the development of a comprehensive document image processing and analysis system. The main purpose of DLA is to segment an input document image into its constituent and coherent regions and identify their classes. In this paper, we propose a competent DLA system, named as BINYAS, based on the connected component (CC) and pixel analysis based approach. Here, we initially identify the regions and then classify these regions as paragraph, separator, graphic, image, table, chart, and inverted text etc. The proposed system is evaluated on four publicly available standard datasets, namely ICDAR 2009, 2015, 2017 and 2019 page segmentation competition datasets, and the performance is compared with many contemporary methods, which also include some well-known software products and deep learning based methods. Experimental results show that our method performs significantly better than state-of-the-art methods in terms of the evaluation metrics considered by the research community of this domain.

Weakly Supervised Learning-based Table Detection

CNN has given the state-of-the-art results in computer vision and natural language processing (NLP) domain problems. This has motivated researchers to use deep learning-based techniques for document layout analysis. Due to recent advances in communication and in information technology, methods of data storage, extraction and processing are rapidly evolving. In an information space, there is a large volume of digital documents (DD) already available, and more DD is created continuously. The DD can be natural images, scanned documents, mails, books, archives, etc. Processing, extraction and understanding of relevant information from these DD have prime business importance. Information in DD is present in the form of tables, text, figure, images, diagram, etc. Document images (DI) are DD which are present in the form of images. It is especially office, scanned documents. Recent progress in artificial intelligence has created a growing expectation for automation of data extraction from DD. Tables are frequently present in DI. There is a desperate need to handle tabular information present in DI; otherwise, this information will remain unused and there is high possibility that it will be lost. In this work, we present weakly supervised learning-based approach to detect and recognize the location of the table. The novelty of our work is that it did not require bounding box annotations for table detection. Application of our technique on ICDAR 2013 test data has demonstrated encouraging results.

Historical document layout analysis using anisotropic diffusion and geometric features

There are several digital libraries worldwide which maintain valuable historical manuscripts. Usually, digital copies of these manuscripts are offered to researchers and readers in raster-image format. These images carry several document degradations that may hinder automatic information retrieval solutions such as manuscript indexing, categorization, retrieval by content, etc. In this paper, we propose a learning-free and hybrid document layout analysis for handwritten historical manuscripts. It has two main phases: page characterization and segmentation. First, the proposed method locates main-content initially using top-down whitespace analysis. It employs anisotropic diffusion filtering to find whitespaces. Then, it extracts template features representing manuscripts’ authors writing behavior. After that, moving windows are used to scan the manuscript page and define main-content boundaries more precisely. We evaluated the proposed method on two datasets: One set is publicly available with 38 historical manuscript pages, and the other set of 51 historical manuscript pages that are collected from the online Harvard Library. Experiments on both datasets show promising results in terms of segmentation quality of main-content that reaches up to 98.5% success rate.

Использование синтетических данных для тонкой настройки моделей сегментации документов

In this paper, we propose an approach to the document images segmentation in a case of limited set of real data for training. The main idea of our approach is to use artificially created data for training and post-processing. The domain of the paper is PDF documents, such as scanned contracts, commercial proposals and technical specifications without a text layer is considered as data. As part of the task of automatic document analysis, we solve the problem of segmentation of DLA documents (Document Layout Analysis). In the paper we train the known high-level FasterRCNN \cite{ren2015faster} model to segment text blocks, tables, stamps and captions on images of the domain. The aim of the paper is to generate synthetic data similar to real data of the domain. It is necessary because the model needs a large dataset for training and the high labor intensity of their preparation. In the paper, we describe the post-processing stage to eliminate artifacts that are obtained as a result of the segmentation. We tested and compared the quality of a model trained on different datasets (with / without synthetic data, small / large set of real data, with / without post-processing stage). As a result, we show that the generation of synthetic data and the use of post-processing increase the quality of the model with a small real training data.

A Method of Japanese Ancient Text Recognition by Deep Learning

Today, when the importance of the country culture is deeply rooted in the hearts of the people, the protection of ancient books and literature has received more and more attention. In this paper, a method to identify texts in ancient books by deep learning is proposed and ancient book “usonarubesh” is chosen as a dataset to test the performance of the model. In this experiment, the layout of the text is extracted into grayscale image through ARU-Net (a neural pixel labeling machine for historical document layout analysis). At the same time the original image which contains the texts is binarized, which the texts are filled with black, while the backgrounds are filled with white. Each area of text is judged by the density of black pixels and the layouts. The cut texts are then selected as the testing dataset for the trained model of deep learning CNN, AlexNet (the training dataset is ready). Finally, the experimental results are analyzed to draw conclusions and to decide the direction of future work.