Text Processing Tasks Research Articles

VieCap4H Challenge 2021: Vietnamese Image Captioning for Healthcare Domain using Swin Transformer and Attention-based LSTM

This study presents our approach on the automatic Vietnamese image captioning for healthcare domain in text processing tasks of Vietnamese Language and Speech Processing (VLSP) Challenge 2021, as shown in Figure~\ref{fig:example}. In recent years, image captioning often employs a convolutional neural network-based architecture as an encoder and a long short-term memory (LSTM) as a decoder to generate sentences. These models perform remarkably well in different datasets. Our proposed model also has an encoder and a decoder, but we instead use a Swin Transformer in the encoder, and a LSTM combined with an attention module in the decoder. The study presents our training experiments and techniques used during the competition. Our model achieves a BLEU4 score of 0.293 on the vietCap4H dataset, and the score is ranked the 3$^{rd}$ place on the private leaderboard. Our code can be found at \url{https://git.io/JDdJm}.

Generation of training data for named entity recognition of artworks

As machine learning techniques are being increasingly employed for text processing tasks, the need for training data has become a major bottleneck for their application. Manual generation of large scale training datasets tailored to each task is a time consuming and expensive process, which necessitates their automated generation. In this work, we turn our attention towards creation of training datasets for named entity recognition (NER) in the context of the cultural heritage domain. NER plays an important role in many natural language processing systems. Most NER systems are typically limited to a few common named entity types, such as person, location, and organization. However, for cultural heritage resources, such as digitized art archives, the recognition of fine-grained entity types such as titles of artworks is of high importance. Current state of the art tools are unable to adequately identify artwork titles due to unavailability of relevant training datasets. We analyse the particular difficulties presented by this domain and motivate the need for quality annotations to train machine learning models for identification of artwork titles. We present a framework with heuristic based approach to create high-quality training data by leveraging existing cultural heritage resources from knowledge bases such as Wikidata. Experimental evaluation shows significant improvement over the baseline for NER performance for artwork titles when models are trained on the dataset generated using our framework.

Converting Numeral Text in Bulgarian into Digit Number Using GATE

The Internet serves billions of users providing a variety of information resources whereby a lot of the information is presented in natural human language and needs an efficient approach to be processed. Natural language processing (NLP) refers to the ability of computers to analyze and understand the structure of human language. By utilizing NLP this linguistic knowledge is transformed into algorithms for solving specific problems. GATE is widely used, open-source software infrastructure that provides a framework and components for solving NLP tasks. The available GATE tools can be adapted to other languages and text processing tasks. This article will present an approach for converting numeric data, written as words in Bulgarian, into digit numbers. For this case, a relevant configuration file for Bulgarian has been integrated into the general tool set in the open source software for natural language processing GATE. The aim of this survey is to determine the exact numeric value of Bulgarian text numeric data, which can be used as a starting point for producing more complex annotations, such as monetary measurement units, etc.

Improving the robustness and accuracy of biomedical language models through adversarial training

Deep transformer neural network models have improved the predictive accuracy of intelligent text processing systems in the biomedical domain. They have obtained state-of-the-art performance scores on a wide variety of biomedical and clinical Natural Language Processing (NLP) benchmarks. However, the robustness and reliability of these models has been less explored so far. Neural NLP models can be easily fooled by adversarial samples, i.e. minor changes to input that preserve the meaning and understandability of the text but force the NLP system to make erroneous decisions. This raises serious concerns about the security and trust-worthiness of biomedical NLP systems, especially when they are intended to be deployed in real-world use cases. We investigated the robustness of several transformer neural language models, i.e. BioBERT, SciBERT, BioMed-RoBERTa, and Bio-ClinicalBERT, on a wide range of biomedical and clinical text processing tasks. We implemented various adversarial attack methods to test the NLP systems in different attack scenarios. Experimental results showed that the biomedical NLP models are sensitive to adversarial samples; their performance dropped in average by 21 and 18.9 absolute percent on character-level and word-level adversarial noise, respectively, on Micro-F1, Pearson Correlation, and Accuracy measures. Conducting extensive adversarial training experiments, we fine-tuned the NLP models on a mixture of clean samples and adversarial inputs. Results showed that adversarial training is an effective defense mechanism against adversarial noise; the models’ robustness improved in average by 11.3 absolute percent. In addition, the models’ performance on clean data increased in average by 2.4 absolute percent, demonstrating that adversarial training can boost generalization abilities of biomedical NLP systems. This study takes an important step towards revealing vulnerabilities of deep neural language models in biomedical NLP applications. It also provides practical and effective strategies to develop secure, trust-worthy, and accurate intelligent text processing systems in the biomedical domain.

Toward a Period-specific Optimized Neural Network for OCR Error Correction of Historical Hebrew Texts

Over the past few decades, large archives of paper-based historical documents, such as books and newspapers, have been digitized using the Optical Character Recognition (OCR) technology. Unfortunately, this broadly used technology is error-prone, especially when an OCRed document was written hundreds of years ago. Neural networks have shown great success in solving various text processing tasks, including OCR post-correction. The main disadvantage of using neural networks for historical corpora is the lack of sufficiently large training datasets they require to learn from, especially for morphologically rich languages like Hebrew. Moreover, it is not clear what are the optimal structure and values of hyperparameters (predefined parameters) of neural networks for OCR error correction in Hebrew due to its unique features. Furthermore, languages change across genres and periods. These changes may affect the accuracy of OCR post-correction neural network models. To overcome these challenges, we developed a new multi-phase method for generating artificial training datasets with OCR errors and hyperparameters’ optimization for building an effective neural network for OCR post-correction in Hebrew. To evaluate the proposed approach, a series of experiments using several literary Hebrew corpora from various periods and genres were conducted. The obtained results demonstrate that (1) training a network on texts from a similar period dramatically improves the network's ability to fix OCR errors, (2) using the proposed error injection algorithm, based on character-level period-specific errors, minimizes the need for manually corrected data and improves the network accuracy by 9%, (3) the optimized network design improves the accuracy by 3% compared to the state-of-the-art network, and (4) the constructed optimized network outperforms neural machine translation models and industry-leading spellcheckers. The proposed methodology may have practical implications for digital humanities projects that aim to search and analyze OCRed documents in Hebrew and potentially other morphologically rich languages.

Random Kernel Forests

Random forests of axis-parallel decision trees still show competitive accuracy in various tasks; however, they have drawbacks that limit their applicability. Namely, they perform poorly for multidimensional sparse data that is frequently input in many text and image processing tasks. A straightforward solution is to create forests of decision trees with oblique splits; however, most training approaches have low performance or are hard to tune. Besides, those ensembles appear unstable and easily overfit, so they should be regularized to find the trade-off between complexity and generalization. This paper proposes an algorithm to train kernel decision trees and random forests. At the top level, this algorithm follows a common greedy procedure to train binary decision trees; however, it produces quasi-optimal linear and kernel splits at the decision stump level. At each stump, the algorithm finds a quasi-optimal distribution of classes to subtrees and trains this stump via optimization of an SVM-like loss function with a margin re-scaling approach, which helps simultaneously optimize the margin between subtree data and arbitrary impurity criteria. We also try to reveal uniform stability-based generalization bounds for those forests and apply them to select the regularization technique. The obtained bounds explicitly consider primary hyperparameters of forests, trees, and decision stumps. The bounds also show a relationship between outliers in decision tree structure, stability and generalization, and how forests smooth these outliers. We performed an experimental evaluation on several tasks, such as studying the reaction of social media users, image recognition, and bank scoring. The experimental results show that the proposed algorithm trains ensembles, which outperform other oblique or kernel forests in many commonly-recognized datasets. Namely, the proposed algorithm shows 99.1% accuracy on MNIST and 58.1% on CIFAR-10. It has been confirmed that the selected regularization technique helps reduce overfitting on several datasets. Therefore, the proposed algorithm may be considered a small step toward customized and regularized ensembles of oblique or kernel trees that keep reasonable training performance on large datasets.We believe that the proposed algorithmcan be utilized as a construction element of approaches to training context-aware models on complex problems with integer inputs and outputs, such as text mining and image recognition.

Mlphon: A Multifunctional Grapheme-Phoneme Conversion Tool Using Finite State Transducers

In this article we present the design and the development of a knowledge based computational linguistic tool, Mlphon [em.el.foːɳ] for Malayalam language. Mlphon computationally models linguistic rules using finite state transducers and performs multiple functions including grapheme to phoneme (g2p) and phoneme to grapheme (p2g) conversions, syllabification, phonetic feature analysis and script grammar check. This open source software tool, released under MIT license, is developed as a one-stop solution to handle different speech related text processing tasks for automatic speech recognition, text to speech synthesis and non-speech natural language processing tasks including syllable subword based language modeling, phoneme diversity analysis and text sanity check. The tool is evaluated on a manually crafted gold standard lexicon. Mlphon performs orthographic syllabification with 99% accuracy with a syllable error rate of 0.62% on the gold standard lexicon. For grapheme to phoneme conversion task, overall phoneme recognition accuracy of 99% with a phoneme error rate of 0.55% is obtained on gold standard lexicon. Additionally an extrinsic evaluation of Mlphon is performed by employing the pronunciation lexicon created using Mlphon, in Malayalam automatic speech recognition (ASR) task. Performance analysis in terms of the computation time of lexicon creation process and the word error rate (WER) on ASR task are presented along with a comparison over other automated tools for lexicon creation. Pronunciation lexicons with more than 100k commonly used Malayalam words in phonemised and syllabified forms is created and they are published as open language resources along with this work. We also demonstrate the usage of Mlphon on different natural language processing applications - syllable subword ASR, assisted pronunciation learning, phoneme diversity analysis and text sanity check. Being a knowledge based solution with open source code, Mlphon can be adapted to other languages of similar script nature.

The impact of text presentation on translator performance

Abstract Widely used computer-aided translation (CAT) tools divide documents into segments, such as sentences, and arrange them side-by-side in a spreadsheet-like view. We present the first controlled evaluation of these design choices on translator performance, measuring speed and accuracy in three experimental text-processing tasks. We find significant evidence that sentence-by-sentence presentation enables faster text reproduction and within-sentence error identification compared to unsegmented text, and that a top-and-bottom arrangement of source and target sentences enables faster text reproduction compared to a side-by-side arrangement. For revision, on the other hand, we find that presenting unsegmented text results in the highest accuracy and time efficiency. Our findings have direct implications for best practices in designing CAT tools.

Automatic Search-and-Replace From Examples With Coevolutionary Genetic Programming.

We describe the design and implementation of a system for executing search-and-replace text processing tasks automatically, based only on examples of the desired behavior. The examples consist of pairs describing the original string and the desired modified string. Their construction, thus, does not require any specific technical skill. The system constructs a solution to the specified task that can be used unchanged on popular existing software for text processing. The solution consists of a search pattern coupled with a replacement expression: the former is a regular expression which describes both the strings to be replaced and their portions to be reused in the latter, which describes how to build the modified strings. Our proposed system is internally based on genetic programming and implements a form of cooperative coevolution in which two separate populations are evolved independently, one for search patterns and the other for replacement expressions. We assess our proposal on six tasks of realistic complexity obtaining very good results, both in terms of absolute quality of the solutions and with respect to the challenging baselines considered.

Emerging role of artificial intelligence in nuclear medicine.

The role of artificial intelligence is increasing in all branches of medicine. The emerging role of artificial intelligence applications in nuclear medicine is going to improve the nuclear medicine clinical workflow in the coming years. Initial research outcomes are suggestive of increasing role of artificial intelligence in nuclear medicine workflow, particularly where selective automation tasks are of concern. Artificial intelligence-assisted planning, dosimetry and procedure execution appear to be areas for rapid and significant development. The role of artificial intelligence in more directly imaging-related tasks, such as dose optimization, image corrections and image reconstruction, have been particularly strong points of artificial intelligence research in nuclear medicine. Natural Language Processing (NLP)-based text processing task is another area of interest of artificial intelligence implementation in nuclear medicine.

Preliminary Results on Different Text Processing Tasks Using Encoder-Decoder Networks and the Causal Feature Extractor

Deep learning methods are gaining popularity in different application domains, and especially in natural language processing. It is commonly believed that using a large enough dataset and an adequate network architecture, almost any processing problem can be solved. A frequent and widely used typology is the encoder-decoder architecture, where the input data is transformed into an intermediate code by means of an encoder, and then a decoder takes this code to produce its output. Different types of networks can be used in the encoder and the decoder, depending on the problem of interest, such as convolutional neural networks (CNN) or long-short term memories (LSTM). This paper uses for the encoder a method recently proposed, called Causal Feature Extractor (CFE). It is based on causal convolutions (i.e., convolutions that depend only on one direction of the input), dilatation (i.e., increasing the aperture size of the convolutions) and bidirectionality (i.e., independent networks in both directions). Some preliminary results are presented on three different tasks and compared with state-of-the-art methods: bilingual translation, LaTeX decompilation and audio transcription. The proposed method achieves promising results, showing its ubiquity to work with text, audio and images. Moreover, it has a shorter training time, requiring less time per iteration, and a good use of the attention mechanisms based on attention matrices.

Part-of-speech tagging for Arabic tweets using CRF and Bi-LSTM

Over the past few years, Twitter has experienced massive growth and the volume of its online content has increased rapidly. This content has been a rich source for several studies that focused on natural language processing (NLP) research. However, Twitter data pose numerous challenges and obstacles to NLP tasks. For the English language, Twitter has an NLP tool that provides tweet-specific NLP tasks, which present significant opportunities for English NLP research and applications. Part-of-speech (POS) tagging for English tweets is one of the tasks that is offered and facilitated by such a tool. In contrast, only a few attempts have been made to develop POS taggers for Arabic content on Twitter. In this paper, we consider POS tagging, which is one of the NLP tasks that directly affects the performance of other subsequent text processing tasks. We introduce three manually annotated datasets for the POS tagging of Arabic tweets: the ‘Mixed,’ ‘MSA,’ and ‘GLF’ datasets with 3000, 1000, and 1000 Arabic tweets, respectively. In addition, we present an exploratory analysis of the behavior of using hashtags in Arabic tweets, which is a phenomenon that affects the task of POS tagging. We also present two supervised POS taggers that are developed based on two approaches: Conditional Random Fields and Bidirectional Long Short-Term Memory (Bi-LSTM) models. We conclude that the Bi-LSTM-based POS tagger achieves the state-of-the-art results for the ‘Mixed’ dataset with 96.5% accuracy. However, the specific-dialect taggers trained on the ‘MSA’ and ‘GLF’ datasets achieve an accuracy of 95.6% and 95%, respectively. The results for the ‘Mixed’ dataset indicate the effectiveness of developing a joint POS tagger without the need for a dialect-specific POS tagger.

Learning context-dependent word embeddings based on dependency parsing

Word embeddings constitute the basic methods of text representation. Whether they are the inputs to a machine learning algorithm or the features used in a natural language processing application, embeddings have proven helpful in solving various text processing tasks. In natural language texts, contextual information exerts a crucial influence on the semantics of word representations. In current research, most training models are based on shallow textual information and do not fully exploit deep relationships in sentences. To overcome this problem, this paper proposes the dependency-based continuous bag-of-words model which integrates the dependency relationships between words and sentences into the context with weights, thereby increasing the influence of specific contextual information on the prediction of target words. This method increases the abundancy of word context information and enhances the semantics of word embeddings. The experimental results show that the proposed method highlights semantic relations and improves the performance of word representations.

Concurrent phonetic transcription, lexical stress assignment and syllabification with deep neural networks

Abstract This paper evaluates four different sequence-to-sequence deep neural network architectures aimed to jointly solve the tasks of: phonetic transcription, lexical stress assignment and syllabification. These text processing tasks are considered essential components for high quality text-to-speech or automatic speech recognition systems, with the phonetic transcription being the most frequently used in these types of applications. Although each of the tasks has been individually and extensively analyzed in the scientific literature, there are few studies which target a concurrent solution for them. In general, the lexical stress assignment and syllabification are used as augmenting input features to the phonetic transcription model and not considered as target features. The proposed network architectures include recurrent, convolution and attention neural layers and were evaluated on hand-checked English and Romanian datasets. The accuracy of the models was evaluated in terms of accuracy for the concurrent prediction of all three tasks, as well as by discarding the syllabification or lexical stress predictions. The best results were obtained with a combination of convolution and attention layers, where the accuracy of the joint prediction for the three tasks was of 58.96% for English and 86.64% for Romanian. The same model for English obtains an accuracy of 59.70% when syllables are discarded and 64% when the prediction of lexical stress is ignored. With the same best performing model for Romanian an accuracy of 88.83% without syllables and 93.84% without lexical stress is obtained.

Learning context-dependent word embeddings based on dependency parsing

Word embeddings constitute the basic methods of text representation. Whether they are the inputs to a machine learning algorithm or the features used in a natural language processing application, embeddings have proven helpful in solving various text processing tasks. In natural language texts, contextual information exerts a crucial influence on the semantics of word representations. In current research, most training models are based on shallow textual information and do not fully exploit deep relationships in sentences. To overcome this problem, this paper proposes the dependency-based continuous bag-of-words model which integrates the dependency relationships between words and sentences into the context with weights, thereby increasing the influence of specific contextual information on the prediction of target words. This method increases the abundancy of word context information and enhances the semantics of word embeddings. The experimental results show that the proposed method highlights semantic relations and improves the performance of word representations.

Medical Named Entity Recognition Based on Overlapping Neural Networks

Named entity recognition (NER) of medical text is a basic task in electronic medical text processing. In recent years, chinese named entity recognition system, especially in the medical field, has problems with insufficient semantic information and poor coding ability. Aiming at the existing problems, we put forward a overlapping neural network for medical named entity recognition. Compared with the mainstream methods for sequence tagging task in recent years, our proposed method can learn context feature automatically and handle encoding process better. The comparative experiments are carried out on medical NER data set, and the experimental results show that our proposed overlapping neural network model can obtain better performance than the state-of-the-art models.

A New Method for Sentence Vector Normalization Using Word2vec

Word embeddings (WE) have received much attention recently as word to numeric vectors architecture for all text processing approaches and has been a great asset for a large variety of NLP tasks. Most of text processing task tried to convert text components like sentences to numeric matrix to apply their processing algorithms. But the most important problems in all word vector-based text processing approaches are different sentences size and as a result, different dimension of sentences matrices. In this paper, we suggest an efficient but simple statistical method to convert text sentences into equal dimension and normalized matrices Proposed method aims to combines three most efficient methods (averaging based, most likely n-grams, and word’s mover distance) to use their advantages and reduce their constraints. The unique size resulting matrix does not depend on language, Subject and scope of the text and words semantic concepts. Our results demonstrate that normalized matrices capture complementary aspects of most text processing tasks such as coherence evaluation, text summarization, text classification, automatic essay scoring, and question answering.

Practical Application of Ontologies

This article discusses the scope of computer ontologies. A classification of the main problems is given, for the solution of which it is advisable to use the ontological approach. Examples of the use of ontologies in various fields of human activity related to knowledge-based systems are given. The scope of application of computer ontologies is discussed in this article . A classification of the main problems is given, for the solution of which it is advisable to use the ontological approach. The areas of application of ontologies are identified as following:
 
 Designing knowledge-oriented systems, which refers to the creation, development, implementation, interaction with users of such objects as knowledge bases, databases and data warehouses. Besides, the same area includes the integration of existing information repositories, even with a very different data structure.
 Knowledge management (decision making systems).The usage of the ontological approach allows to raise information management technologies to a new level: to introduce the technology of providing explanations to the user about the management process, which can reveal the essence of the situations and their causes, as well as to justify the proposed action scenarios. That is, to provide human-machine dialogue that allows to communicate in a natural language.
 Semantic technology of information search and the ability to build search results using ontological classifiers. This use of ontologies enhances the relevance and continentity of search results. It provides contextual search capabilities, making the most of the natural language vocabulary display.
 Classification tasks: the classification structure (taxonomy) is an integral part of any ontology, we can talk about the presence of ontology elements in special classifications and indexing systems (for example, in library classification codes).
 Support for linguistically oriented information technologies, that is, providing solutions to complex text processing tasks by organizing a system of interactions of various levels of text processing, including the ontological level associated with the construction of a semantically adequate model of the subject area and various linguistic levels associated with linguistic properties and relations of subject objects area.
 Support for transdisciplinarity, namely the formation of common conceptual spaces, providing an adequate understanding of the information exchanged by experts in various fields of human activity.
 
 Examples of using ontologies in various spheres of human activity related to systems based on knowledge, such as education, the creation of the Semantic Web, etc are given.

OCR Evaluation Tools for the 21st Century

We introduce ocreval, a port of the ISRI OCR Evaluation Tools, now with Unicode support. We describe how we upgraded the ISRI OCR Evaluation Tools to support modern text processing tasks. ocreval supports producing character-level and word-level accuracy reports, supporting all characters representable in the UTF-8 character encoding scheme. In addition, we have implemented the Unicode default word boundary specification in order to support word-level accuracy reports for a broad range of writing systems. We argue that character-level and word-level accuracy reports produce confusion matrices that are useful for tasks beyond OCR evaluation—including tasks supporting the study and computational modeling of endangered languages.

Профессиональная подготовка лингвотехнолога в вузе: проект образовательной программы

Современное состояние технологического развития определяются уровнем разработки и общим потенциалом процессов автоматизации в промышленности (Промышленность 4.0), а также способами представления и качеством информации, которая предназначена для автоматизации работ над проектами (Информация 4.0). Актуальные стандарты характера и качества информации во многом определяются относительно новой областью знания – технической коммуникацией. В ее растущем поле наблюдается спрос на профессионалов – «технических коммуникаторов», квалификационные требования к которым растут вместе с увеличением сфер внедрения технической коммуникации. В статье оценивается необходимость подготовки специалистов – лингвотехнологов, готовых решать задачи переработки текстов на этом новом уровне развития техники и технологии, рассматриваются особенности подхода к этой специализации в России и за рубежом, а также концепция разработки специальных направлений обучения для профессиональной подготовки лингвотехнолога на базе классического филологического / лингвистического образования.