Annotated Corpus Research Articles

Building an Annotated L1 Arabic/L2 English Bilingual Writer Corpus: The Qatari Corpus of Argumentative Writing (QCAW)

Abstract The study presents the creation of the Qatari Corpus of Argumentative Writing (QCAW) as an annotated L1 Arabic and L2 English bilingual writer corpus. It comprises 200,000 tokens of argumentative writing by Qatari university students in L1 Arabic and L2 English. The corpus includes 195 essays written by 195 students, 159 females and 36 males. The students were native Arabic speakers proficient in English as a second language. The corpus is divided into Arabic and English sections, accompanied by part-of-speech annotated files in UTF-8 encoded text format. Metadata in CSV format contains information about the students (gender, major, first and second languages) and the essays (text serial numbers, word limits, genre, writing date, time spent, and location). The current study outlines the steps for collecting and analysing the corpus, including details on essay writers, topic selection, pre-analysis text modifications, proficiency level, gender, and major ratings. Statistical analyses were applied to examine the corpus. The QCAW offers a valuable bilingual data source authored by the same students in Arabic and English, with implications for further research.

Detection of violence incitation expressions in Urdu tweets using convolutional neural network

The popularity and widespread use of social media are constantly generating unmonitored data, spreading unwanted content such as hate speech and expressions that incite violence. Automatic detection of violence incitation is a challenging task and to the best of our knowledge, Urdu language has been completely neglected. Therefore, a robust framework is proposed for identifying expressions exhibiting violence incitation in Urdu tweets. The potentials of the semantic, word embeddings, and language models are explored to learn contextualized representations of the violence incitation in Urdu tweets. In addition, the strength of the 1-Dimensional Convolutional Neural Network (1D-CNN) is exploited by tunning its parameters on the newly proposed annotated Urdu corpus. The annotated dataset consists of 4808 tweets manually collected from Pakistani Twitter accounts. The performance of 1D-CNN with word uni-gram, Urdu Bidirectional Encoder Representations from Transformer (Urdu-BERT), and Urdu- Robustly Optimized BERT Approach (Urdu-RoBERTa) models is compared to fine-tuned Urdu-RoBERTa, Bidirectional Long short-term memory (BiLSTM), Convolutional BiLSTM (CBi-LSTM), and six state-of-the-art Machine Learning (ML) models. The results reveal that the 1D-CNN with word uni-gram model shows benchmark performance by demonstrating 89.84% accuracy and 89.80% macro f1-score. Furthermore, it outperforms all comparable models and achieves 89.76% f1-score for the violence class, and 89.84% f1-score for not-violence class identification. The uniqueness of the proposed model is evaluated using MARS shine-through and MARS occlusion metrics and the CNN model outperformed the others. The MARS metrics facilitate evaluation and visualization of the classifier performance in terms of capturing unique true positive samples that are not predicted by other models. The findings of the proposed framework are very supportive for further investigation in this domain.

Identifying and Extracting Rare Diseases and Their Phenotypes with Large Language Models

PurposePhenotyping is critical for informing rare disease diagnosis and treatment, but disease phenotypes are often embedded in unstructured text. While natural language processing (NLP) can automate extraction, a major bottleneck is developing annotated corpora. Recently, prompt learning with large language models (LLMs) has been shown to lead to generalizable results without any (zero-shot) or few annotated samples (few-shot), but none have explored this for rare diseases. Our work is the first to study prompt learning for identifying and extracting rare disease phenotypes in the zero- and few-shot settings.MethodsWe compared the performance of prompt learning with ChatGPT and fine-tuning with BioClinicalBERT. We engineered novel prompts for ChatGPT to identify and extract rare diseases and their phenotypes (e.g., diseases, symptoms, and signs), established a benchmark for evaluating its performance, and conducted an in-depth error analysis.ResultsOverall, fine-tuning BioClinicalBERT resulted in higher performance (F1 of 0.689) than ChatGPT (F1 of 0.472 and 0.610 in the zero- and few-shot settings, respectively). However, ChatGPT achieved higher accuracy for rare diseases and signs in the one-shot setting (F1 of 0.778 and 0.725). Conversational, sentence-based prompts generally achieved higher accuracy than structured lists.ConclusionPrompt learning using ChatGPT has the potential to match or outperform fine-tuning BioClinicalBERT at extracting rare diseases and signs with just one annotated sample. Given its accessibility, ChatGPT could be leveraged to extract these entities without relying on a large, annotated corpus. While LLMs can support rare disease phenotyping, researchers should critically evaluate model outputs to ensure phenotyping accuracy.

CoNECo: a Corpus for Named Entity recognition and normalization of protein Complexes.

Despite significant progress in biomedical information extraction, there is a lack of resources for Named Entity Recognition (NER) and Named Entity Normalization (NEN) of protein-containing complexes. Current resources inadequately address the recognition of protein-containing complex names across different organisms, underscoring the crucial need for a dedicated corpus. We introduce the Complex Named Entity Corpus (CoNECo), an annotated corpus for NER and NEN of complexes. CoNECo comprises 1621 documents with 2052 entities, 1976 of which are normalized to Gene Ontology. We divided the corpus into training, development, and test sets and trained both a transformer-based and dictionary-based tagger on them. Evaluation on the test set demonstrated robust performance, with F-scores of 73.7% and 61.2%, respectively. Subsequently, we applied the best taggers for comprehensive tagging of the entire openly accessible biomedical literature. All resources, including the annotated corpus, training data, and code, are available to the community through Zenodo https://zenodo.org/records/11263147 and GitHub https://zenodo.org/records/10693653.

Označevanje korpusov

Ordered collections of machine-readable texts, corpora, are useful in various branches of linguistics. The present paper focuses on the machine-readable form of corpora, above all on their annotation, i.e. adding interpretative information to the text in the corpus. The annotation presented is based on taking into consideration the international standards from this field, which contributes to better documentation and verifiability, easier use of processing applications, and better interchange and longevity. In the first part, corpus encoding standards, above all XML (eXtended Markup Language) and TEI (Text Encoding Initiative), are discussed, while in the second part some of the more interesting levels of linguistic annotation of corpora are outlined.

Preparing multi-layered visualisations of Old Babylonian cuneiform tablets for a machine learning OCR training model towards automated sign recognition

Abstract In the framework of the CUNE-IIIF-ORM project the aim is to train an Artificial Intelligence Optical Character Recognition (AI-OCR) model that can automatically locate and identify cuneiform signs on photorealistic representations of Old Babylonian texts (c. 2000–1600 B.C.E.). In order to train the model, c. 200 documentary clay tablets have been selected. They are manually annotated by specialist cuneiformists on a set of 12 still raster images generated from interactive Multi-Light Reflectance images. This image set includes visualisations with varying light angles and simplifications based on the dept information on the impressed signs in the surface. In the Cuneur Cuneiform Annotator, a Gitlab-based web application, the identified cuneiform signs are annotated with polygons and enriched with metadata. This methodology builds a qualitative annotated training corpus of approximately 20,000 cropped signs (i.e. 240,000 visualizations), all with their unicode codepoint and conventional sign name. It will act as a multi-layerd core dataset for the further development and fine-tuning of a machine learning OCR training model for the Old Babylonian cuneiform script. This paper discusses how the physical nature of handwritten inscribed Old Babylonian documentary clay tablets challenges the annotation and metadating task, and how these have been addressed within the CUNE-IIIF-ORM project to achieve an effective training corpus to support the training of a machine learning OCR model. ACM CCS Applied computing → Document management and text processing → Document capture → Optical character recognition; Applied computing → Arts and humanities → Language translation.

Corpus-based Technologies in the Translation Study of Tonic Drink Advertisements: An Algorithm of Analysis Procedure

The study is aimed at the procedure used for the analysis of English-language advertising texts for tonic drinks and the peculiarities of their reproduction in Ukrainian using a corpus-based approach. For the research material, 500 advertising texts for the same products of the specified group were selected in English and Ukrainian, since the material of any study involving corpus-based technologies should be chronological and thematically homogeneous. The proposed analysis algorithm involves a number of sequential steps. Firstly, to ensure a multi-channel method of collecting material. Secondly, the creation of a research parallel corpus of English-language advertising texts for tonic drinks and their translations into Ukrainian using the InterText Editor program. Third, the annotation of corpus from the perspective of its composition and lexical and semantic structure, i.e., the compositional components of advertising texts, their obligatory and variable structural elements, as well as speech figures of expression and nationally marked vocabulary were identified with the help of special tags. Fourthly, using corpus managers AntConc, Textanz, and MS Excel, determination of general quantitative characteristics of the corpus and subcorpora, compilation of frequency lists of word usage, lemmatization, and calculation of partial-word distribution in the source and target text sets and structural elements of advertising messages. Fifthly, conducting a systematic inventory of the units under study, explaining the peculiarities of their formation and functioning, and characterizing the verbal content of advertising texts. Sixth, an analysis of the morphological, lexical, semantic, and syntactic levels of the language of tonic drink advertisements in the source and translated texts, as well as stylistic devices peculiar to these texts. Seventh, a systematic consideration of the interaction of verbal, visual, and auditory components, since the transition from one system of signs to another is very important when adapting advertising messages created in one cultural environment to another culture.

Supporting the working life exposome: Annotating occupational exposure for enhanced literature search.

An individual's likelihood of developing non-communicable diseases is often influenced by the types, intensities and duration of exposures at work. Job exposure matrices provide exposure estimates associated with different occupations. However, due to their time-consuming expert curation process, job exposure matrices currently cover only a subset of possible workplace exposures and may not be regularly updated. Scientific literature articles describing exposure studies provide important supporting evidence for developing and updating job exposure matrices, since they report on exposures in a variety of occupational scenarios. However, the constant growth of scientific literature is increasing the challenges of efficiently identifying relevant articles and important content within them. Natural language processing methods emulate the human process of reading and understanding texts, but in a fraction of the time. Such methods can increase the efficiency of both finding relevant documents and pinpointing specific information within them, which could streamline the process of developing and updating job exposure matrices. Named entity recognition is a fundamental natural language processing method for language understanding, which automatically identifies mentions of domain-specific concepts (named entities) in documents, e.g., exposures, occupations and job tasks. State-of-the-art machine learning models typically use evidence from an annotated corpus, i.e., a set of documents in which named entities are manually marked up (annotated) by experts, to learn how to detect named entities automatically in new documents. We have developed a novel annotated corpus of scientific articles to support machine learning based named entity recognition relevant to occupational substance exposures. Through incremental refinements to the annotation process, we demonstrate that expert annotators can attain high levels of agreement, and that the corpus can be used to train high-performance named entity recognition models. The corpus thus constitutes an important foundation for the wider development of natural language processing tools to support the study of occupational exposures.

Лингвокогнитивные основания интеграции поэтического текста в кинодискурс

The article delves into the intricacies of integrating poetic texts into the discourse of au­thorial cinema. It begins by examining the multifaceted interaction between cinema and poet­ry within contemporary artistic culture, framing author cinematography through the lens of the poetic concept. Intermediality and interdiscursivity are explored as intrinsic properties of cinema, serving as tools for shaping the unique style and aesthetics of filmmakers. Two films, “Stalker” by Andrei Tarkovsky and “Ugly Swans” by Konstantin Lopushansky, are selected as primary sources for analysis due to their continuity and innovative qualities. The research employs a comprehensive methodology, including parameterization of linguistic creativity, corpus annotation, and conceptual analysis of verbal and non-verbal elements that contribute to the films’ poetics. Data processing techniques are then applied to interpret the findings. The study uncovers both similarities and differences in the approaches of the two filmmakers to incorporating poetic material into their works. It concludes that the integration of poetic texts into films serves as a mechanism for eliciting a unique psycho-emotional response from viewers, facilitating their engagement with the filmmaker’s ideological concepts.

Improving Amazigh POS tagging using machine learning

Tamazight, Berber, and Amazigh are the multiple names for the same language. It covers a great geographical area including the north of Africa, Sahara Sahel. It is spread principally in Morocco, Algeria, Tunisia, and Mali. In terms of natural language processing, it is considered a low-resource language. This paper presents multiple applications of different machine learning algorithms for part-of-speech tagging Amazigh for the first time. Those algorithms include trigrams 'n' tags (TnT), Brill tagging, hidden Markov model (HMM), Unigram, Bigram, Unigram + Bigram,and conditional random fields (CRF). Also, we present a part-of-speech tagger using CRF with our function of extracting features from the Amazigh language. The importance of finding a performant POS tagger for the Amazigh is to enrich its corpus, which is a main step for other NLP applications. In this research, we used 60000 tokens of annotated Amazigh corpus with 28 tags, and we realized the necessary processing step on it to be in an adequate form for feeding each model. A detailed comparison of the performance results is presented to establish the best one and the results show that our application of CRF model outperforms other techniques.

Functional and Sеmantic Analysis of the Case Markers in Vakh Khanty (a Case Study of the Latest Field Data on LingvoDoc)

In this paper we argue that the category of nominal case in the Vakh Khanty dialect needs reviewing which is due to the availability of the recent field data obtained in 2019. This objective is considered to be topical because the results of the study will equip researches with all necessary data to develop a unified notation system for the case markers to be used in an annotated corpus of Vakh Khanty on the LingvoDoc platform. The declension system of the Eastern Khanty dialect under analysis is heterogeneous and relatively vast. The controversy around the synthetic means of expression of semantic relations concerns the terminology of case markers, their quantity, morphemic status and functional features. The purpose of this paper is to clarify the functional and semantic aspects of the Vakh Khanty case markers drawing on the recently obtained field material. This objective is achieved by means of the tools available on the LingvoDoc platform. The latest field data on the Vakh Khanty dialect was collected in the village of Korliki in 2019. More than 6,000 words of data are posted on the endangered languages documentation platform LingvoDoc. Part of the material used in the analysis is being processed and integrated on the LingvoDoc platform. The underlying opinion of this study draws on the seminal work by N.I. Tereshkin, as well as other prominent researchers of the Khanty language, and their description of the declension system of the Eastern dialect. The latest field data contains texts and questionnaires allowing researches to examine functioning of the morphological markers and compare results with the findings previously presented in the literature. Such approach entails observations leading to the systematization of the case category in this dialect at its current state of development. The examination of the range of meanings conveyed by the Vakh Khanty case markers at present has made it possible to distribute them into groups of semantic and syntactic case markers. As a result, the study confirmed that the nominal declension system encompasses such case markers as the abessive, ablative, allative, distributive, comitative, comparative, lative, locative, nominative, oblicative, and translative cases. The recent field data confirms the presence of the distributive case within the case paradigm and the status of the comparative element niŋit as a postposition. Each case marker of the Vakh Khanty dialect is terminologically specified which is essential for developing a parser for this dialect on the LingvoDoc platform. The development of the parser in the future will speed up the process of processing and analyzing language material and lead to the elimination of errors.

A Technical Review on Emotion Detection from Informal Text for Indian Regional Languages: Approaches and Challenges

Effective communication is a crucial element in the existence of a human being. Emotions are a common component of human communication and can appear in a variety of ways, including spoken words, written messages, and non-verbal indicators like gestures and facial expressions. Textual communication which is the common way of interaction has been rapidly increasing day-by-day. People nowadays are using social media to convey their thoughts and beliefs in the form of comments, posts and stories with associated emotions. As a result, there is a growing need to detect and understand the emotions conveyed in texts. Although the emotions expressed in text are readily understood by the human brain, teaching a machine to do the same is a challenging task. However, when dealing with Indian languages, this process becomes especially difficult. As India has plenty of vernacular languages, people opt to use monolingual or code-mix languages. These types of languages are generally informal way which creates an obstacle to convey the emotions. The shortage of annotated corpus especially for regional languages is also one of the challenges. This paper aims to analyze and review emotion detection from textual data specifically for Indian regional languages and its challenges.

AlgBERT: Automatic Construction of Annotated Corpus for Sentiment Analysis in Algerian Dialect

Nowadays, sentiment analysis is one of the most crucial research fields of Natural Language Processing (NLP), and it is widely applied in a variety of applications such as marketing and politics. However, the Arabic language still lacks sufficient language resources to enable the tasks of opinion and emotion analysis comparing to other language such as English. Additionally, manual annotation requires a lot of effort and time. In this article, we address this problem and propose a novel automated annotation platform for sentiment analysis called AlgBERT by providing annotated corpus and using deep learning technology that includes many automatic natural language processing algorithms, which is the basis for text classification and opinion analysis. We suggest using BERT model as a method; it is the abbreviation of Bidirectional Encoder Representations from Transformers, as it is one of the most effective technologies in terms of results in different world languages. We used around of 54K comments collected from social networking (Twitter, YouTube) written in Arabic and Algerian dialects. Our AlgBERT system obtained excellent results with an accuracy of 91.04%, and this is considered as one of the best results for opinion analysis in Algerian dialect.

Towards understanding policy design through text-as-data approaches: The policy design annotations (POLIANNA) dataset

Despite the importance of ambitious policy action for addressing climate change, large and systematic assessments of public policies and their design are lacking as analysing text manually is labour-intensive and costly. POLIANNA is a dataset of policy texts from the European Union (EU) that are annotated based on theoretical concepts of policy design, which can be used to develop supervised machine learning approaches for scaling policy analysis. The dataset consists of 20,577 annotated spans, drawn from 18 EU climate change mitigation and renewable energy policies. We developed a novel coding scheme translating existing taxonomies of policy design elements to a method for annotating text spans that consist of one or several words. Here, we provide the coding scheme, a description of the annotated corpus, and an analysis of inter-annotator agreement, and discuss potential applications. As understanding policy texts is still difficult for current text-processing algorithms, we envision this database to be used for building tools that help with manual coding of policy texts by automatically proposing paragraphs containing relevant information.

Parallel-Based Corpus Annotation for Malay Health Documents

Named entity recognition (NER) is a crucial component of various natural language processing (NLP) applications, particularly in healthcare. It involves accurately identifying and extracting named entities such as medical terms, diseases, and drug names, and healthcare professionals are essential for tasks like clinical text analysis, electronic health record management, and medical research. However, healthcare NER faces challenges, especially in Malay, in which specialized corpora are limited, and no general corpus is available yet. To address this, the paper proposes a method for constructing an annotated corpus of Malay health documents. The researchers leverage a parallel source that contains annotated entities in English due to the limited tools available for the Malay language, and it is very language-dependent. Additional credible Malay documents are incorporated as sources to enhance the development. The targeted health entities in this research include penyakit (diseases), simptom (symptoms), and rawatan (treatments). The primary objective is to facilitate the development of NER algorithms specifically tailored to the healthcare domain in the Malay language. The methodology encompasses data collection, preprocessing, annotation of text in both English and Malay, and corpus creation. The outcome of this research is the establishment of the Malay Health Document Annotated Corpus, which serves as a valuable resource for training and evaluating NLP models in the Malay language. Future research directions may focus on developing domain-specific NER models, exploring alternative algorithms, and enhancing performance. Overall, this research aims to address the challenges of healthcare NER in the Malay language by constructing an annotated corpus and facilitating the development of tailored NER algorithms for the healthcare domain.

Evaluating rhyme annotations for large corpora

Abstract Recent methods have been proposed to produce automatic rhyme annotators for large rhymed corpora. These methods, such as Baley (2022b) greatly reduce the cost of annotating rhymed material, allowing historical linguists to focus on the analysis of the rhyme patterns. However, evidence for the quality of those annotations has been anecdotal, consisting of a handful of individual poem case studies. This paper proposes to address the issue: first, we discuss previously proposed metrics that evaluate the quality of an annotator’s output against a ground-truth annotation (List, Hill, and Foster; 2019) and we propose an alternative metric that is better suited to the task. Then, sampling from Baley’s published annotated corpus and re-annotating it by hand, we use the sample to demonstrate the lacunae in the original approach and show how to fix them. Finally, the hand-annotated sample and source code are published as additional data, so that other researchers can compare the performance of their own annotators.

The CELI corpus: Design and linguistic annotation of a new online learner corpus

This article introduces the CELI corpus, a new learner corpus of written Italian consisting of ca. 600,000 tokens, evenly distributed among CEFR (Common European Framework of Reference for Languages) proficiency levels B1, B2, C1 and C2. The collected texts derive from the language certification exams administered by the University for Foreigners of Perugia all around the world. The corpus contains rich metadata pertaining to text-related and learner-related variables. It expands the domain of learner corpora by being, among other things, both freely available online to the research community, and by focusing on a target language other than English. The article also presents and evaluates the POS-tagging procedure, thus contributing to best practices in learner corpus annotation.

ClimateBUG : A data-driven framework for analyzing bank reporting through a climate lens

This paper applies computational linguistics learning methods to the banking industry and climate change fields. We introduce our data-driven framework, climateBUG, with the aim of detecting latent information about how banks discuss their activities related to climate change using natural language processing (NLP). This framework consists of an ingestion pipeline, a configurable database, and a set of API’s. In addition, climateBUG offers two standalone components, namely a unique annotated corpus of approximately 1.1M statements from EU banks’ annual and sustainability reporting and a deep learning model adapted to the semantics of the corpus. When benchmarking on classification performance, our model outperforms other models with similar scopes due to its stronger domain relevance. We also provide examples of how the framework can be applied from a user perspective.

Extraction of target structures in learners' corpora

Foreign language (FL) or second language (L2) corpora are sets of productions by non-native speakers, learners of a given language, which contemplate the errors and well-formed structures produced. These serve different research objectives, such as studies on language acquisition (LE and L2), phenomena of linguistic interference or analysis and diagnosis of LE/L2 proficiency levels. In the context of this research, the definition of the learner's proficiency level is often relevant, and this is done, typically, through the analysis of the presence or absence of errors in the learners' productions, based on mappings of typical or expected errors and well-formed structures for a given level of proficiency. However, contrary to the learner's error – which is explicitly marked in the corpus and whose typology and methodology of analysis constitutes a subtopic of investigation on its own –, the well-formed structures, and in particular the target structures (well-formed structures expected in the learners' productions of a given level of proficiency), are not easily identifiable in the corpora. The work presented here aims to fill this gap in COPLE2 – Corpus of Portuguese Foreign/Second Language through the use of expressions in CQL – Corpus Query Language. Based on pre-identified target structures and on the information made available in COPLE2, such as morphosyntactic tagging and different levels of information and annotation (learner production, teacher correction, normalized form, lemma, etc.), we propose query expressions in CQL that easily allow any user to immediately extract examples of target structures by proficiency level. The construction of the query expressions implies the definition and testing of the best strategies for each case and requires the systematization of linguistic rules and patterns of occurrence of the phenomena in question, but also the definition of ways to circumvent the limitations inherent to the corpus annotation, on the one hand, and the query language, on the other.

Adaptive class augmented prototype network for few-shot relation extraction

Relation extraction is one of the most essential tasks of knowledge construction, but it depends on a large amount of annotated data corpus. Few-shot relation extraction is proposed as a new paradigm, which is designed to learn new relationships between entities with merely a small number of annotated instances, effectively mitigating the cost of large-scale annotation and long-tail problems. To generalize to novel classes not included in the training set, existing approaches mainly focus on tuning pre-trained language models with relation instructions and developing class prototypes based on metric learning to extract relations. However, the learned representations are extremely sensitive to discrepancies in intra-class and inter-class relationships and hard to adaptively classify the relations due to biased class features and spurious correlations, such as similar relation classes having closer inter-class prototype representation. In this paper, we introduce an adaptive class augmented prototype network with instance-level and representation-level augmented mechanisms to strengthen the representation space. Specifically, we design the adaptive class augmentation mechanism to expand the representation of classes in instance-level augmentation, and class augmented representation learning with Bernoulli perturbation context attention to enhance the representation of class features in representation-level augmentation and explore adaptive debiased contrastive learning to train the model. Experimental results have been demonstrated on FewRel and NYT-25 under various few-shot settings, and the proposed model has improved accuracy and generalization, especially for cross-domain and different hard tasks.