Large Text Collections Research Articles

"My Very Subjective Human Interpretation": Domain Expert Perspectives on Navigating the Text Analysis Loop for Topic Models

Practitioners dealing with large text collections frequently use topic models such as Latent Dirichlet Allocation (LDA) and Non-negative Matrix Factorization (NMF) in their projects to explore trends. Despite twenty years of accrued advancement in natural language processing tools, these models are found to be slow and challenging to apply to text exploration projects. In our work, we engaged with practitioners (n=15) who use topic modeling to explore trends in large text collections to understand their project workflows and investigate which factors often slow down the processes and how they deal with such errors and interruptions in automated topic modeling. Our findings show that practitioners are required to diagnose and resolve context-specific problems with preparing data and models and need control for these steps, especially for data cleaning and parameter selection. Our major findings resonate with existing work across CSCW, computational social science, machine learning, data science, and digital humanities. They also leave us questioning whether automation is actually a useful goal for tools designed for topic models and text exploration.

Computing MEMs and Relatives on Repetitive Text Collections

We consider the problem of computing the Maximal Exact Matches (MEMs) of a given pattern \(P[1\mathinner{.. }m]\) on a large repetitive text collection \(T[1\mathinner{.. }n]\) over an alphabet of size \(\sigma\) , which is represented as a (hopefully much smaller) run-length context-free grammar of size \(g_{rl}\) . We show that the problem can be solved in time \(O(m^{2}\log^{\epsilon}n)\) , for any constant \(\epsilon\gt0\) , on a data structure of size \(O(g_{rl})\) . Further, on a locally consistent grammar of size \(O(\delta\log\frac{n\log\sigma}{\delta\log n})\) , the time decreases to \(O(m\log m(\log m+\log^{\epsilon}n))\) . The value \(\delta\) is a function of the substring complexity of \(T\) and \(\Omega(\delta\log\frac{n\log\sigma}{\delta\log n})\) is a tight lower bound on the compressibility of repetitive texts \(T\) , so our structure has optimal size in terms of \(n\) , \(\sigma\) , and \(\delta\) . We extend our results to several related problems, such as finding \(k\) -MEMs, MUMs, rare MEMs, and applications. Categories and Subject Descriptors: E.1 [Data structures] ; E.2 [Data storage representations] ; E.4 [Coding and information theory]: Data compaction and compression; F.2.2 [Analysis of algorithms and problem complexity] : Nonnumerical algorithms and problems— Pattern matching, Computations on discrete structures, Sorting and searching

Exploring Qualitative Geographies in Large Volumes of Digital Text: Placing Tourists, Travelers, and Inhabitants in the English Lake District

Thus far, approaches to analyzing geographies in large collections of digital texts have used coordinate-based locations derived from toponyms (place names). Although this approach provides a useful starting point, it ignores both nontoponym references to place and textual structure while stressing Euclidean space. This article addresses these issues. Focusing on a corpus of writing about the English Lake District before 1900, it explores the different geographies associated with tourists, travelers, and inhabitants. Alongside toponyms, we identify the concept of a geo-noun to associate place with geographical features. Sense of place is then identified using the adjectives, nouns, and verbs that writers associate with locations and locales. This methodology allows us to identify and define places described in large volumes of digital text both holistically and textually.

GRAAL: Graph-Based Retrieval for Collecting Related Passages across Multiple Documents

Finding passages related to a sentence over a large collection of text documents is a fundamental task for claim verification and open-domain question answering. For instance, a common approach for verifying a claim is to extract short snippets of relevant text from a collection of reference documents and provide them as input to a natural language inference machine that determines whether the claim can be deduced or refuted. Available approaches struggle when several pieces of evidence from different documents need to be combined to make an inference, as individual documents often have a low relevance with the input and are therefore excluded. We propose GRAAL (GRAph-based retrievAL), a novel graph-based approach that outlines the relevant evidence as a subgraph of a large graph that summarizes the whole corpus. We assess the validity of this approach by building a large graph that represents co-occurring entity mentions on a corpus of Wikipedia pages and using this graph to identify candidate text relevant to a claim across multiple pages. Our experiments on a subset of FEVER, a popular benchmark, show that the proposed approach is effective in identifying short passages related to a claim from multiple documents.

An STS analysis of a digital humanities collaboration: trading zones, boundary objects, and interactional expertise in the DECRYPT project

A widely shared recognition over the past decade is that the methodology and the basic concepts of science and technology studies (STS) can be used to analyze collaborations in the cross-disciplinary field of digital humanities (DH). The concepts of trading zones (Galison, 2010), boundary objects (Star and Griesemer, 1989), and interactional expertise (Collins and Evans, 2007) are particularly fruitful for describing projects in which researchers from massively different epistemic cultures (Knorr Cetina, 1999) are trying to develop a common language. The literature, however, primarily concentrates on examples where only two parties, historians and IT experts, work together. More exciting perspectives open up for analysis when more than two, more nuanced and different epistemic cultures seek a common language and common research goals. In the DECRYPT project funded by the Swedish Research Council, computational linguists, historians, computer scientists and AI experts, cryptologists, computer vision specialists, historical linguists, archivists, and philologists collaborate with strikingly different methodologies, publication patterns, and approaches. They develop and use common resources (including a database and a large collection of European historical texts) and tools (among others a code-breaking software, a hand-written text recognition tool for transcription), researching partly overlapping topics (handwritten historical ciphers and keys) to reach common goals. In this article, we aim to show how the STS concepts are illuminating when describing the mechanisms of the DECRYPT collaboration and shed some light on the best practices and challenges of a truly cross-disciplinary DH project.

Computational linguistics at the crossroads: A comprehensive review of NLP advancements

New NLP breakthroughs have put Computational Linguistics at a crossroads. NLP's past, present, and future are covered. This review explains computational linguistics' creation with a brief history of linguistics and computer science. Early solutions processed and understood natural language using rule-based systems using manually constructed linguistic rules. Over time, these tactics became increasingly problematic as language became more complex and obscure. Statistical approaches transformed operations. Neural network-based machine learning methods are leading the area because they can learn complicated patterns and representations from large text collections. A data-driven model revolution in natural language processing enhanced language modelling, machine translation, and sentiment analysis. Next, NLP improvements for several tasks and applications are evaluated. Language understanding models that capture semantic nuances and contextual relationships use deep learning frameworks. Word embeddings and transformer-based architectures like GPT and BERT perform well on benchmark datasets for text classification, question answering, and named item identification. The paper also shows how NLP interacts with computer vision, voice processing, and other domains to show the merits and cons of cross-disciplinary research. Multimodal techniques that combine text, graphics, and audio may increase natural language processing and interpretation. The review discusses NLP's effects on prejudice, justice, and privacy. Responsible development and implementation are needed when NLP technology becomes widespread due to algorithmic bias and data privacy concerns. NLP research directions and concerns are reviewed. Existing models may meet standards but fail in practice.

Corpus linguistics and the social sciences

Abstract Corpus linguistics, with its methodological orientation towards the empirical analysis of language based on large text collections, has the potential to offer significant tools for addressing real-world problems across various social science domains, including climate change, criminology, healthcare and policy making. Despite this potential, the integration of corpus linguistics into social science disciplines (beyond linguistics) remains hampered by fundamental differences in epistemology, definitions and methodological approaches. This article explores the relationship between corpus linguistics and the social sciences. It is argued that epistemology, or the theory of knowledge, represents a primary barrier to integration, with much corpus linguistics research aligning with positivist and naturalist epistemologies. By contrast, many social science disciplines embrace more interpretive, conventionalist approaches that account for the dynamic nature of social phenomena. Considering the role of naturalism and conventionalism within both corpus linguistics and the social sciences, this article illustrates how these epistemological stances are likely to influence the acceptance and use of corpus methods in social science research. Despite the challenges, areas of convergence (e.g. shared use of data processing tools and the acknowledgement of the central role of language in social processes) provide opportunities for cross-disciplinary collaboration. As means to bridge the epistemological divide, this article advocates for a critical realist approach and concludes by calling on users of corpus linguistic methods to be reflexive and transparent about their epistemological stances when reporting their research.

Topic models with elements of neural networks: investigation of stability, coherence, and determining the optimal number of topics.

Topic modeling is a widely used instrument for the analysis of large text collections. In the last few years, neural topic models and models with word embeddings have been proposed to increase the quality of topic solutions. However, these models were not extensively tested in terms of stability and interpretability. Moreover, the question of selecting the number of topics (a model parameter) remains a challenging task. We aim to partially fill this gap by testing four well-known and available to a wide range of users topic models such as the embedded topic model (ETM), Gaussian Softmax distribution model (GSM), Wasserstein autoencoders with Dirichlet prior (W-LDA), and Wasserstein autoencoders with Gaussian Mixture prior (WTM-GMM). We demonstrate that W-LDA, WTM-GMM, and GSM possess poor stability that complicates their application in practice. ETM model with additionally trained embeddings demonstrates high coherence and rather good stability for large datasets, but the question of the number of topics remains unsolved for this model. We also propose a new topic model based on granulated sampling with word embeddings (GLDAW), demonstrating the highest stability and good coherence compared to other considered models. Moreover, the optimal number of topics in a dataset can be determined for this model.

CIDER: Context-sensitive polarity measurement for short-form text.

Researchers commonly perform sentiment analysis on large collections of short texts like tweets, Reddit posts or newspaper headlines that are all focused on a specific topic, theme or event. Usually, general-purpose sentiment analysis methods are used. These perform well on average but miss the variation in meaning that happens across different contexts, for example, the word "active" has a very different intention and valence in the phrase "active lifestyle" versus "active volcano". This work presents a new approach, CIDER (Context Informed Dictionary and sEmantic Reasoner), which performs context-sensitive linguistic analysis, where the valence of sentiment-laden terms is inferred from the whole corpus before being used to score the individual texts. In this paper, we detail the CIDER algorithm and demonstrate that it outperforms state-of-the-art generalist unsupervised sentiment analysis techniques on a large collection of tweets about the weather. CIDER is also applicable to alternative (non-sentiment) linguistic scales. A case study on gender in the UK is presented, with the identification of highly gendered and sentiment-laden days. We have made our implementation of CIDER available as a Python package: https://pypi.org/project/ciderpolarity/.

КОРПУС-БАЗОВАНИЙ АНАЛІЗ КОНЦЕПТУ ФРАНЦІЯ

The article focuses on a corpus-based analysis of the concept FRANCE. The analysis of concepts through the lens of corpus linguistics allows us to determine the general perception of a particular reality. Given the current political context and the development of diplomatic relationships, the concept FRANCE becomes significant and requires analysis. As the material for our study, we chose the corpus of Ukrainian language GRAK. General Regionally Annotated Corpus of Ukrainian (GRAC) is a large representative collection of texts in Ukrainian accompanied by a program that enables customization of subcorpora, searching words, grammatical forms and their combinations as well as post-processing of the query results. For this analysis, journalistic and literary texts dated from 1991 to 2022 were selected. The lexeme “France”, representing the concept FRANCE, appeared 189,178 times in GRAK between 1991 and 2022 with the majority of occurrences found in journalistic texts. Besides, other lexical representatives of the concept FRANCE were analyzed, such as “French”, “Paris”, “France”. The article pays particular attention to the contexts in which the concept FRANCE is realized. Ten main thematic groups related to the concept FRANCE were identified and analyzed: FRANCE – PRESTIGE; FRANCE – REFUGE; FRANCE – HISTORY; FRANCE – LAW; FRANCE – POLITICS; FRANCE – LANGUAGE; FRANCE – ECONOMY; FRANCE – SPORT; FRANCE – FOOD; FRANCE – STYLE. Key adjectives and verbs that verbalize the concept FRANCE in the corpus were found. These words often evoke images of well-known politicians and the names of European countries. Moreover, crucial collocates were determined. Thirty collocates representing the lexeme France were identified: Germany, Macron (Emmanuel), Francois (Hollande), President, Italy, Britain, Ministry of Foreign Affairs, Spain, Merkel, Sarkozy, Championship, Leaders, Paris, Ambassador, Team, Elections, PSG, Finance, Embassy, Canada, Government, Lady, Great, Match, Ukraine, Protests, Authority, Visit. These collocates predominantly align with themes of politics, international relations and sports. The extensive usage of the concept FRANCE in Ukrainian corpus indicates a strengthening of political relations between Ukraine and France.

Extracting information on virus-human interactions and on antiviral compounds based on automated analysis of large text collections

The development of effective antivirals is of great importance due to the threat associated with the rapid spread of viral infections. The accumulation of data in scientific publications and in databases of biologically active compounds provides an opportunity to extract specific information about interactions between chemicals and their viral and host targets. This information can be used for elucidation of knowledge about potential antiviral activity of chemical compounds, their side effects and toxicities. Our study aims to extract knowledge about virus-host interactions and potential antiviral agents based on the mining of massive amounts of scientific publications. With a set of previously developed algorithms, we have extracted comprehensive information on virus-host interactions and chemical compounds that interact with both viral and host targets. We collected data on the interactions of several viruses, including hepatitis B and C viruses, SARS-CoV-2, influenza A and B, and herpes simplex viruses, with (1) the host (human body), (2) potential antiviral agents, and, also extracted information on the interactions between potential antiviral agents and host proteins. Based on the data analysis performed, we created a freely available knowledge base on the interaction of chemical compounds with viral proteins and their host targets, allowing the exploration of both well-studied and recently discovered novel virus-host-chemical-compound interactions.

Applied corpus linguistics and legal interpretation: A rapidly developing field of interdisciplinary scholarship

This article offers an overview of developments in a newly emerging interdisciplinary research field: legal corpus linguistics. The field brings together corpus research and legal theory by applying corpus-analytic techniques and linguistic concepts to facilitate the interpretation of legal texts. Despite the field's short history, it has already contributed important new insights into the meaning of statutory texts and parts of the U.S. Constitution, insights that may have significant practical implications for the American legal system. Our article provides an overview of relevant developments in legal corpus linguistics, from early success stories to recent and ongoing collaborative work between corpus linguists and legal scholars. It aims to highlight the benefits and illustrate the potential of this type of interdisciplinary work by summarizing three recent case studies, each of which deals with an important topic in American constitutional law. The case studies focus in turn on the following parts of the U.S. Constitution: (1) Article III and the meaning of “cases,” (2) the Appointments Provision in Article II, Section 2 and the meaning of “such inferior officers,” and (3) the Impeachment Provision in Article II, Section 4 and the meaning of “misdemeanors.” All three case studies use corpus analysis to explore phraseological patterns in large collections of Founding Era texts to provide insights into the meanings of the selected words and phrases in context during the time the Constitution was drafted and ratified. The article discusses the practical relevance of results from these case studies and potential implications of this and related work in legal corpus linguistics for contemporary and future litigation.

A Vector Space Approach for Measuring Relationality and Multidimensionality of Meaning in Large Text Collections

In this article, we develop a methodological approach for organizational research regarding the construction of multidimensional and relational similarity measures by using the vector space model in natural language processing (NLP). Our vector space approach draws on the well-established premise in organizational research that texts provide a window into social reality and allow measuring theory-based constructs ( e.g., organizations’ self-representations). Using a vector space approach allows capturing the multidimensionality of these theory-based constructs and computing relational similarities between organizational entities ( e.g., organizations, their members, and subunits) in social spaces and with their environments, such as the organization itself, industries, or countries. Thus, our methodological approach contributes to the recent trend in organizational research to use the potential inherent in big (textual) data by using NLP. In an example, we provide guidance for organizational scholars by illustrating how they can ensure validity when applying our methodological contribution in concrete research practice.

Corpus Analysis with spaCy

This lesson demonstrates how to use the Python library spaCy for analysis of large collections of texts. This lesson details the process of using spaCy to enrich a corpus via lemmatization, part-of-speech tagging, dependency parsing, and named entity recognition. Readers will learn how the linguistic annotations produced by spaCy can be analyzed to help researchers explore meaningful trends in language patterns across a set of texts.

PEDL+: protein-centered relation extraction from PubMed at your fingertip.

Relation extraction (RE) from large text collections is an important tool for database curation, pathway reconstruction, or functional omics data analysis. In practice, RE often is part of a complex data analysis pipeline requiring specific adaptations like restricting the types of relations or the set of proteins to be considered. However, current systems are either non-programmable web sites or research code with fixed functionality. We present PEDL+, a user-friendly tool for extracting protein-protein and protein-chemical associations from PubMed articles. PEDL+ combines state-of-the-art NLP technology with adaptable ranking and filtering options and can easily be integrated into analysis pipelines. We evaluated PEDL+ in two pathway curation projects and found that 59% to 80% of its extractions were helpful. PEDL+ is freely available at https://github.com/leonweber/pedl.

Automatic Genre Identification for Robust Enrichment of Massive Text Collections: Investigation of Classification Methods in the Era of Large Language Models

Massive text collections are the backbone of large language models, the main ingredient of the current significant progress in artificial intelligence. However, as these collections are mostly collected using automatic methods, researchers have few insights into what types of texts they consist of. Automatic genre identification is a text classification task that enriches texts with genre labels, such as promotional and legal, providing meaningful insights into the composition of these large text collections. In this paper, we evaluate machine learning approaches for the genre identification task based on their generalizability across different datasets to assess which model is the most suitable for the downstream task of enriching large web corpora with genre information. We train and test multiple fine-tuned BERT-like Transformer-based models and show that merging different genre-annotated datasets yields superior results. Moreover, we explore the zero-shot capabilities of large GPT Transformer models in this task and discuss the advantages and disadvantages of the zero-shot approach. We also publish the best-performing fine-tuned model that enables automatic genre annotation in multiple languages. In addition, to promote further research in this area, we plan to share, upon request, a new benchmark for automatic genre annotation, ensuring the non-exposure of the latest large language models.

Topic modeling methods for short texts: A survey

In the present day, online users are incentivized to engage in short text-based communication. These short texts harbor a significant amount of implicit information, including opinions, topics, and emotions, which are of notable value for both exploration and analysis. By alleviating the sparsity in short texts, topic models can be used to discover topics from large collections of short texts. While there is a large body of surveys focused on topic modeling, but only a few of them have focused on the short texts. This paper presents a comprehensive overview of topic modeling methods for short texts from a novel perspective. Firstly, it discusses short text probabilistic topic models and outlines the directions in which they can be improved. Secondly, it explores short text neural topic models, which can be categorized into three groups based on their underlying structures. In addition, this paper provides a detailed investigation of embedding methods in topic modeling. Moreover, various applications and corresponding works are surveyed, with a focus on short texts. The commonly used public corpora and evaluation indicators for topic modeling are also summarized. Finally, the advantages and disadvantages of short text topic modeling are discussed in detail, and future research directions are proposed.

Context-aware Transliteration of Romanized South Asian Languages

Abstract While most transliteration research is focused on single tokens such as named entities—for example, transliteration of from the Gujarati script to the Latin script “Ahmedabad” footnoteThe most populous city in the Indian state of Gujarat. the informal romanization prevalent in South Asia and elsewhere often requires transliteration of full sentences. The lack of large parallel text collections of full sentence (as opposed to single word) transliterations necessitates incorporation of contextual information into transliteration via non-parallel resources, such as via mono-script text collections. In this article, we present a number of methods for improving transliteration in context for such a use scenario. Some of these methods in fact improve performance without making use of sentential context, allowing for better quantification of the degree to which contextual information in particular is responsible for system improvements. Our final systems, which ultimately rely upon ensembles including large pretrained language models fine-tuned on simulated parallel data, yield substantial improvements over the best previously reported results for full sentence transliteration from Latin to native script on all 12 languages in the Dakshina dataset (Roark et al. 2020), with an overall 3.3% absolute (18.6% relative) mean word-error rate reduction.

KEBLM: Knowledge-Enhanced Biomedical Language Models

Pretrained language models (PLMs) have demonstrated strong performance on many natural language processing (NLP) tasks. Despite their great success, these PLMs are typically pretrained only on unstructured free texts without leveraging existing structured knowledge bases that are readily available for many domains, especially scientific domains. As a result, these PLMs may not achieve satisfactory performance on knowledge-intensive tasks such as biomedical NLP. Comprehending a complex biomedical document without domain-specific knowledge is challenging, even for humans. Inspired by this observation, we propose a general framework for incorporating various types of domain knowledge from multiple sources into biomedical PLMs.We encode domain knowledge using lightweight adapter modules, bottleneck feed-forward networks that are inserted into different locations of a backbone PLM. For each knowledge source of interest, we pretrain an adapter module to capture the knowledge in a self-supervised way. We design a wide range of self-supervised objectives to accommodate diverse types of knowledge, ranging from entity relations to description sentences.Once a set of pretrained adapters is available, we employ fusion layers to combine the knowledge encoded within these adapters for downstream tasks. Each fusion layer is a parameterized mixer of the available trained adapters that can identify and activate the most useful adapters for a given input. Our method diverges from prior work by including a knowledge consolidation phase, during which we teach the fusion layers to effectively combine knowledge from both the original PLM and newly-acquired external knowledge using a large collection of unannotated texts. After the consolidation phase, the complete knowledge-enhanced model can be fine-tuned for any downstream task of interest to achieve optimal performance.Extensive experiments on many biomedical NLP datasets show that our proposed framework consistently improves the performance of the underlying PLMs on various downstream tasks such as natural language inference, question answering, and entity linking. These results demonstrate the benefits of using multiple sources of external knowledge to enhance PLMs and the effectiveness of the framework for incorporating knowledge into PLMs. While primarily focused on the biomedical domain in this work, our framework is highly adaptable and can be easily applied to other domains, such as the bioenergy sector.

Networks of Migrants’ Narratives: A Post-authentic Approach to Heritage Visualisation

Digital tools, technologies, and infrastructures have increasingly shaped how knowledge is understood, created, managed, maintained, and shared. In sectors such as cultural heritage, for example, curators and scholars now widely apply computer science approaches to their collections for tasks such as enriching content and providing users with visual access. Although there has been an increasing interest in critical digital visual design and cultural heritage, what still seems to be largely missing is a problematised and critical approach to the application of these techniques. Drawing on critical posthumanities, in this article I propose a new critical framework, the post-authentic framework to digital objects and cultural heritage. The framework advocates transparency and openness as well as the active and reflexive participation of the scholar/curator in the process of knowledge production. To exemplify how the post-authentic framework may be applied to digital research, I explore two use cases: critical visualisation design and critical digital heritage research. For the first use case, I examine the design choices for developing the tool DeXTER, an interactive visualisation app that allows users to explore enriched cultural heritage material interactively. I specifically emphasise the challenges facing product design, with a focus on visualising the ambiguities and uncertainties of Network Analysis and Sentiment Analysis. For the second use case, I explore the potential of DeXTER for testing hypotheses on large textual collections, for example by accessing the layers of meaning attached to referential entities. The use case is the Italian American labour movement from 1898 to 1936. The article concludes with key concepts and methods for applying the post-authentic framework to digital heritage practices and knowledge creation more widely.