Unstructured Documents Research Articles

TableExtractNet: A Model of Automatic Detection and Recognition of Table Structures from Unstructured Documents

This paper presents TableExtractNet, a model that automatically finds and understands tables from scanned documents, tasks that are essential for quick use of information in many fields. This is driven by the growing need for efficient and accurate table interpretation in business documents where tables enhance data communication and aid decision-making. The model uses a mix of two advanced techniques, CornerNet and Faster R-CNN, to accurately locate tables and understand their layout. Tests on standard datasets, IIIT-AR-13K, STDW, SciTSR, and PubTabNet, show that this model performs better than previous ones, making it very good at dealing with tables that have complicated designs or are in documents with a lot of detail. The success of this model marks a step forward in making document analysis more automated. It makes it easier to turn complex scanned documents containing tables into data that are more manipulable by computers.

Unstructured Document Information Extraction Method with Multi-Faceted Domain Knowledge Graph Assistance for M2M Customs Risk Prevention and Screening Application

As a crucial national security defense line, the existing risk prevention and screening system of customs falls short in terms of intelligence and diversity for risk identification factors. Hence, the urgent issues to be addressed in the risk identification system include intelligent extraction technology for key information from Customs Unstructured Accompanying Documents (CUADs) and the reliability of the extraction results. In the customs scenario, OCR is employed for M2M interactions, but current models have difficulty adapting to diverse image qualities and complex customs document content. We propose a hybrid mutual learning knowledge distillation (HMLKD) method for optimizing a pre-trained OCR model’s performance against such challenges. Additionally, current models lack effective incorporation of domain-specific knowledge, resulting in insufficient text recognition accuracy for practical customs risk identification. We propose a customs domain knowledge graph (CDKG) developed using CUAD knowledge and propose an integrated CDKG post-OCR correction method (iCDKG-PostOCR) based on CDKG. The results on real data demonstrate that the accuracies improve for code text fields to 97.70%, for character type fields to 96.55%, and for numerical type fields to 96.00%, with a confidence rate exceeding 99% for each. Furthermore, the Customs Health Certificate Extraction System (CHCES) developed using the proposed method has been implemented and verified at Tianjin Customs in China, where it has showcased outstanding operational performance.

Analysis of Unstructured Document Data Extraction Technology

Analysis of Unstructured Document Data Extraction Technology

Supervised Learning Algorithm on Unstructured Documents for the Classification of Job Offers: Case of Cameroun

Nowadays, in data science, supervised learning algorithms are frequently used to perform text classification. However, African textual data, in general, have been studied very little using these methods. This article notes the particularity of the data and measures the level of precision of predictions of naive Bayes algorithms, decision tree, and SVM (Support Vector Machine) on a corpus of computer jobs taken on the internet. This is due to the data imbalance problem in machine learning. However, this problem essentially focuses on the distribution of the number of documents in each class or subclass. Here, we delve deeper into the problem to the word count distribution in a set of documents. The results are compared with those obtained on a set of French IT offers. It appears that the precision of the classification varies between 88% and 90% for French offers against 67%, at most, for Cameroonian offers. The contribution of this study is twofold. Indeed, it clearly shows that, in a similar job category, job offers on the internet in Cameroon are more unstructured compared to those available in France, for example. Moreover, it makes it possible to emit a strong hypothesis according to which sets of texts having a symmetrical distribution of the number of words obtain better results with supervised learning algorithms.

Tokengrid: Toward More Efficient Data Extraction From Unstructured Documents

Key information extraction from unstructured documents is a practical problem in many industries. Machine learning models aimed at solving this problem should efficiently utilize textual, visual, and 2D spatial layout information of the document. Grid based approaches achieve this by representing the document as a 2D grid and feeding it to a fully convolutional encoder-decoder network that solves a semantic instance segmentation problem. We propose a new method for the instance detection branch of that network for the task of automatic information extraction from invoices. Our approach reduces this problem to 1D region detection. The proposed network has fewer parameters and a shorter inference times. Additionally, we suggest a new metric for evaluating the results.

Efficient Automated Processing of the Unstructured Documents Using Artificial Intelligence: A Systematic Literature Review and Future Directions

The unstructured data impacts 95% of the organizations and costs them millions of dollars annually. If managed well, it can significantly improve business productivity. The traditional information extraction techniques are limited in their functionality, but AI-based techniques can provide a better solution. A thorough investigation of AI-based techniques for automatic information extraction from unstructured documents is missing in the literature. The purpose of this Systematic Literature Review (SLR) is to recognize, and analyze research on the techniques used for automatic information extraction from unstructured documents and to provide directions for future research. The SLR guidelines proposed by Kitchenham and Charters were adhered to conduct a literature search on various databases between 2010 and 2020. We found that: 1. The existing information extraction techniques are template-based or rule-based, 2. The existing methods lack the capability to tackle complex document layouts in real-time situations such as invoices and purchase orders, 3. The datasets available publicly are task-specific and of low quality. Hence, there is a need to develop a new dataset that reflects real-world problems. Our SLR discovered that AI-based approaches have a strong potential to extract useful information from unstructured documents automatically. However, they face certain challenges in processing multiple layouts of the unstructured documents. Our SLR brings out conceptualization of a framework for construction of high-quality unstructured documents dataset with strong data validation techniques for automated information extraction. Our SLR also reveals a need for a close association between the businesses and researchers to handle various challenges of the unstructured data analysis.

Extraction of Sequence of Actions from Unstructured Requirements Specification Document

Extraction of Sequence of Actions from Unstructured Requirements Specification Document

Innovative Model for Student Project Evaluation Based on Text Mining

At present, there is a huge amount of unstructured documentation that is generated by students who study the course project management and evaluation. These unstructured documents cannot be used in direct processing to extract useful information or knowledge. At the same time, make use of them to assess student achievement. In this work, we propose text-mining techniques for the evaluation of the progress of student in a course project management and evaluation. The most straightforward approach is looking for the enrichment of the vocabulary, the casual and condition relations, and the structure of the terms in the different phases of the project. In our paper, we used text mining and the structure of the vocabularies as a tool to measure the success and the progress of the students. It is important to evaluate the effectiveness and efficiency of our approach. The experimental results illustrate an overall performance of project evaluation. By looking into the number of indexing terms, the causal relations and the positive conditional relations, and levels of the hierarchical structure of the terms there is an evidence from this research that the vocabulary enrichment had an impact on the students evaluation and learning ability.

Knowledge Extraction System from Unstructured Documents

In this paper, we design and implement a knowledge extraction system from unstructured documents (unstructured documents are documents where the information is in natural language, and require natural language processing techniques for processing) in HTML format. Basically, the system allows to transform the content of a text that is in natural language, into structured and organized knowledge, semantically described (a Semantic Ontology). Therefore, it is proposed to generate semantic knowledge based on the extraction of entities and relationships, where entities are anything about which something can be said, and relations the interactions between entities. From the generated semantic knowledge model, it is possible to infer new knowledge, such as lexicons, taxonomies and specialized terminological bases. The system can be used by any semantic processing application, in its processes of enriching its information and knowledge

MalayIK: An Ontological Approach to Knowledge Transformation in Malay Unstructured Documents

The number of unstructured documents written in Malay language is enormously available on the web and intranets. However, unstructured documents cannot be queried in simple ways, hence the knowledge contained in such documents can neither be used by automatic systems nor could be understood easily and clearly by humans. This paper proposes a new approach to transform extracted knowledge in Malay unstructured document using ontology by identifying, organizing, and structuring the documents into an interrogative structured form. A Malay knowledge base, the MalayIK corpus is developed and used to test the MalayIK-Ontology against Ontos, an existing data extraction engine. The experimental results from MalayIK-Ontology have shown a significant improvement of knowledge extraction over Ontos implementation. This shows that clear knowledge organization and structuring concept is able to increase understanding, which leads to potential increase in sharable and reusable of concepts among the community.

A Novel Method for Classification of Unstructured Documents by using Wordnet based Semantic Similarity

Now-a-days the impact of Internet and its usage is increased dramatically. The information available all over the world is very high. People are using social media like Facebook, Linkedin, and Twitter etc. Due to information sharing the text documents, audio and video files are gets increased. The Extraction of semantic similar documents is big problem. Constructing a proper classification model under in research. Kmeans algorithm works better for high dimensional data. Our proposed research concentrates on combining Wordnet based semantic similarity with Term Frequency Matrix (TFM).

自然言語で記述された登山計画書の機械可読化に基づく登山計画書共有システムの実装

自然言語で記述された登山計画書の機械可読化に基づく登山計画書共有システムの実装

Application of Predictive and Descriptive Text Mining Techniques for Analysis and Organization of Unstructured Documents

In our presentation, we will report the methodology, as well as the final results of a research project which aimed at validating the hypothesis that predictive and descriptive text mining techniques can successfully assist thematic analysis and organization of unstructured documents.Notre présentation consistera à exposer la méthodologie, ainsi que les résultats finaux d’un projet de recherche qui visait à valider l’hypothèse selon laquelle certaines techniques de forage (fouille) de textes de nature prédictive et descriptive peuvent être efficacement employées afin d’assister les opérations d’analyse et d’organisation thématique de documents textuels non structurés.

Automatic Classification and Categorization: Application for Identifying and Thematic Analysing of Textual Unstructured Documents

Our talk will focus on computer-assisted thematic analysis of unstructured documents. First, we will discuss the main issues raised by this question. Secondly, we will present a methodology based on text mining techniques that can assist thematic analysis of textual documents. Finally, we will expose some results obtained with this methodology.Notre présentation portera sur l’assistance à l'analyse thématique des documents textuels non-structurés. Nous exposerons d’abord les principaux enjeux soulevés par cette problématique. Par la suite, nous présenterons une méthodologie d’analyse fondée sur des techniques de forage de texte permettant d'assister informatiquement l’analyse thématique. Finalement, nous présenterons quelques résultats obtenus en appliquant notre démarche.

Information Extraction from the Un-Structured Document using Grammatical Inference and Alignment Similarity

Huge amount of information is available in un-structured (text) documents. Knowledge discovery in un-structured document has been recognized as promising task in the recent years. Since un-structured document is typically formatted for human viewing, it varies widely from document to document. Frequent changes made to their formatting further causes difficulty in construction of a global schema. So, Discovery of interesting rules form it is complex and tedious process. Most of the existing system uses hand-coded wrappers to extract information, which is monotonous and time consuming. In this paper we propose a novel and hybrid approach of learning (context-free) grammar rules that are based on alignment between texts. Also it automatically discovers the grammar rules using grammatical inference of repeated pattern present in un-structured (text) document. The generated rules can be used to infer the attribute value pairs from the unstructured text document.

A Metadata-Based Approach for Unstructured Document Management in Organizations

Effectively managing documents is a strategic requirement for every organization. Available document management systems (DMSs) often lack effective functions for automatic document management. One reason is that relevant information frequently is conveyed by unstructured documents, whose content cannot be easily accessed and processed by applications. This article proposes a metadata model, the DMSML (Document Management and Sharing Markup Language) to enable and to ease unstructured document management by supporting the design of DMSs. We argue that the extensive use of this metadata language will render organizational information explicit, promoting information reuse and interoperability in a more profitable way than what is guaranteed by proprietary DMSs. We also briefly depict the design and deployment phases of a Web-based DMS prototype based on DMSML. Our overall intent is to increase the awareness of what managers should account for when considering the possibility of adopting a DMS.