Large-scale Corpus Research Articles

K-BERT: Enabling Language Representation with Knowledge Graph

Pre-trained language representation models, such as BERT, capture a general language representation from large-scale corpora, but lack domain-specific knowledge. When reading a domain text, experts make inferences with relevant knowledge. For machines to achieve this capability, we propose a knowledge-enabled language representation model (K-BERT) with knowledge graphs (KGs), in which triples are injected into the sentences as domain knowledge. However, too much knowledge incorporation may divert the sentence from its correct meaning, which is called knowledge noise (KN) issue. To overcome KN, K-BERT introduces soft-position and visible matrix to limit the impact of knowledge. K-BERT can easily inject domain knowledge into the models by being equipped with a KG without pre-training by itself because it is capable of loading model parameters from the pre-trained BERT. Our investigation reveals promising results in twelve NLP tasks. Especially in domain-specific tasks (including finance, law, and medicine), K-BERT significantly outperforms BERT, which demonstrates that K-BERT is an excellent choice for solving the knowledge-driven problems that require experts.

Sentiment Classification in Customer Service Dialogue with Topic-Aware Multi-Task Learning

Sentiment analysis in dialogues plays a critical role in dialogue data analysis. However, previous studies on sentiment classification in dialogues largely ignore topic information, which is important for capturing overall information in some types of dialogues. In this study, we focus on the sentiment classification task in an important type of dialogue, namely customer service dialogue, and propose a novel approach which captures overall information to enhance the classification performance. Specifically, we propose a topic-aware multi-task learning (TML) approach which learns topic-enriched utterance representations in customer service dialogue by capturing various kinds of topic information. In the experiment, we propose a large-scale and high-quality annotated corpus for the sentiment classification task in customer service dialogue and empirical studies on the proposed corpus show that our approach significantly outperforms several strong baselines.

Joint Learning of Answer Selection and Answer Summary Generation in Community Question Answering

Community question answering (CQA) gains increasing popularity in both academy and industry recently. However, the redundancy and lengthiness issues of crowdsourced answers limit the performance of answer selection and lead to reading difficulties and misunderstandings for community users. To solve these problems, we tackle the tasks of answer selection and answer summary generation in CQA with a novel joint learning model. Specifically, we design a question-driven pointer-generator network, which exploits the correlation information between question-answer pairs to aid in attending the essential information when generating answer summaries. Meanwhile, we leverage the answer summaries to alleviate noise in original lengthy answers when ranking the relevancy degrees of question-answer pairs. In addition, we construct a new large-scale CQA corpus, WikiHowQA, which contains long answers for answer selection as well as reference summaries for answer summarization. The experimental results show that the joint learning method can effectively address the answer redundancy issue in CQA and achieves state-of-the-art results on both answer selection and text summarization tasks. Furthermore, the proposed model is shown to be of great transferring ability and applicability for resource-poor CQA tasks, which lack of reference answer summaries.

Valency Dictionaries and Chinese Vocabulary Acquisition for Foreign Learners

Valency is a major source of lexical errors in foreign language learning. Accordingly, the research question is how the syntactic and semantic properties of a word can be retrieved from the corpora and represented in a Chinese valency dictionary to facilitate foreign learners' vocabulary acquisition. Within the three aspects of the valency framework — logical-semantic, syntactic and semantic-pragmatic valency — this study examines 60 cases of Chinese lexical misuse extracted from the HSK (Chinese Language Proficiency Test) Dynamic Compositions Corpus. The results suggest that the majority of cases of misuse occur in the dimension of semantic-pragmatic valency and that this semantic-pragmatic misuse can be ascribed to various factors such as semantic collocations, emotive variables, text styles, registers, and other contextual factors. The results are then utilized as syntactic, semantic and pragmatic information to be presented in a Chinese valency dictionary. Specifically, the results obtained from a case study of a misused word by referring to a large-scale native Chinese speaker corpus help retrieve a relatively full list of complementation patterns, based on which the study designs a Chinese valency entry that embodies three basic elements — quantitative valency, qualitative valency and valency patterns.

Writing practice in a society with codified variation: a correspondence analysis of writing practice in New Norwegian/Nynorsk

This paper illustrates the advantages of combining corpus linguistic methods and correspondence analysis when investigating sub-varieties within written languages that have codified variation. Through a study of a large-scale corpus of the written standard New Norwegian/Nynorsk, the paper demonstrates how correspondence analysis is a method that is well-suited to mapping norm clusters consisting of different sub-varieties in actual usage. The study reveals the existence of a norm cluster within the corpus consisting of a moderate sub-variety. Current Norwegian language policy is to base the official norms on developments in observed usage. The paper thus concludes that future standardisation of Nynorsk should be moving towards a narrower, moderate norm in order to be in accordance with the operative norm. The methods presented in this paper may be of value when investigating other written languages with codified variation, such as Irish, or languages without official norms, such as Shona.

Extremely Low-Resource Text Simplification with Pre-trained Transformer Language Model

Inspired by machine translation task, recent text simplification approaches regard a task as a monolingual text-to-text generation, and neural machine translation models have significantly improved the performance of simplification tasks. Although such models require a large-scale parallel corpus, such corpora for text simplification are very few in number and smaller in size compared to machine translation task. Therefore, we have attempted to facilitate the training of simplification rewritings using pre-training from a large-scale monolingual corpus such as Wikipedia articles. In addition, we propose a translation language model to seamlessly conduct a fine-tuning of text simplification from the pre-training of the language model. The experimental results show that the translation language model substantially outperforms a state-of-the-art model under a low-resource setting. In addition, a pre-trained translation language model with only 3000 supervised examples can achieve a performance comparable to that of the state-of-the-art model using 30,000 supervised examples.

“Am besten, du gehst.”Zur nähesprachlichen Verwendung desam-Superlativs im Vor-Vorfeld

AbstractThis paper investigates the use of the Germanam-superlative in colloquial utterances such asam besten, du gehst jetzt. In a first approximation, this construction can be described as elliptically usedam-superlative occurring on the left sentence periphery within an Operator-Skopus-Struktur (Barden/Elstermann/Fiehler 2001). As will be shown in the empirical core part of the study, a qualitative-quantitative analysis, the pattern appears to be characterized by different overlapping and interplaying tendencies in language use (e.g. the selection and frequency of pronouns and sentence mood). The data used for the qualitative-quantitative analysis is taken mainly from a large-scale web corpus (deTenTen13).

GSAM: A deep neural network model for extracting computational representations of Chinese addresses fused with geospatial feature

Addresses are one of the most important geographical reference systems in natural languages. In China, due to the relatively backward address planning, there are a large number of non-standard addresses. This kind of unstructured text makes the management and application of Chinese addresses much more difficult. However, by extracting the computational representations of addresses, it can be structured and its related applications can be extended more conveniently. Therefore, this paper utilizes a deep neural language model from natural language processing (NLP) to automatically extract computational representations through an unsupervised address language model (ALM), which is trained in an unsupervised way and is suitable for a large-scale address corpus. We propose a solution to fuse addresses and geospatial features and construct a geospatial-semantic address model (GSAM) that supports a variety of downstream tasks. Our proposed GSAM constructing process consists of three phases. First, we build an ALM using bidirectional encoder representations from Transformers (BERT) to learn the addresses' semantic representations. Then, the fusion clustering results of the semantic and geospatial information are obtained by a high-dimensional clustering algorithm. Finally, we construct the GSAM based on the fused clustering results using novel fine-tuning techniques. Furthermore, we apply the extracted computational representation from GSAM to the address location prediction task. The experimental results indicate that the target task accuracy of the ALM is 90.79%, and the result of semantic geospatial fusion clustering strongly correlates with fine-grained urban neighbourhood area division. The GSAM can accurately identify clustering labels and the values of evaluation metrics are all above 0.96. We also demonstrate that our model outperforms purely ALM-based and word2vec-based models by address location prediction task.

The pragmatics of possession: A corpus study of English prenominal possessives

We present a large-scale corpus study of English prenominal possessives (e.g., Mary's house) which investigates how their referential interpretations (e.g., ‘the house Mary owns’) are linguistically supported in natural language contexts. The study is set against a long tradition of formal semantic accounts which take possessive interpretations to arise primarily from the lexical semantics of the possessive noun phrase, thereby suggesting that pragmatic reasoning plays a lesser role (Partee, 1983/1997; Barker, 1995, 2000; Vikner & Jensen, 2002, 2003, 2004). One claim of these accounts is that non-lexically derived or ‘pragmatic’ interpretations (e.g., ‘the house Mary dreams of buying’) require explicit contextual support, while this is not needed for ‘lexical’ interpretations (e.g., ‘the house Mary owns’), which are considered default and available in the absence of contextual support. The empirical results from our study suggest a complex picture whereby interpretations of both purported types receive various kinds and degrees of contextual support. We argue that this, coupled with independent evidence that the linguistic co-text usually explicates the relationships between discourse referents and that context-free interpretations are a myth, jeopardizes the cognitive plausibility of interpretation taxonomies which distinguish ‘lexical’ from ‘pragmatic’ interpretations. We sketch a more cognitively-oriented pragmatic approach to possessive interpretations that takes into consideration the multidimensionality of context while attributing an overall greater role to pragmatic reasoning.

First Citations and First Occurrences: How Sensitive to Language Change Were Nineteenth- and Twentieth-Century American Dictionaries?

This article compares the relative effectiveness of large, monolingual American dictionaries and a large-scale corpus of American English (Corpus of Historical American English, or COHA) at documenting new verb senses that arise from nouns through conversion, like backpack and earmark. Seventy-five such denominal verbs, established in English at varying times throughout the nineteenth and twentieth centuries, were selected, and their first record in an American dictionary as well as their first attestation in COHA were tallied. The results show that COHA is slightly better at catching a new use in the empirical record, but dictionaries have been effective as well. For nearly a third of the words, COHA and the dictionaries were essentially tied in documenting newly emerging senses. This suggests that dictionaries have been finely attuned to changes in English throughout the nineteenth and twentieth centuries and that a large-scale corpus like COHA will come close to matching the empirical effectiveness of dictionaries. Since each method uncovered some words much earlier than the other method, current lexicographic practice of including corpora as well as traditional methods like reading programs and citation files seems well justified.

Applying Collocation Analysis to Chinese Discourse: A Case Study of Causal Connectives

AbstractCollocation analysis can be used to extract meaningful linguistic information from large-scale corpus data. This paper reviews the methodological issues one may encounter when performing collocation analysis for discourse studies on Chinese. We propose four crucial aspects to consider in such analyses: (i) the definition of collocates according to various parameters; (ii) the choice of analysis and association measures; (iii) the definition of the search span; and (iv) the selection of corpora for analysis. To illustrate how these aspects can be addressed when applying a Chinese collocation analysis, we conducted a case study of two Chinese causal connectives:yushi‘that is why’ andyin’er‘as a result’. The distinctive collocation analysis shows how these two connectives differ in volitionality, an important dimension of discourse relations. The study also demonstrates that collocation analysis, as an explorative approach based on large-scale data, can provide valuable converging evidence for corpus-based studies that have been conducted with laborious manual analysis on limited datasets.

Mining social influence in science and vice-versa: A topic correlation approach

There is no doubt that scientific discoveries have always brought changes to society. New technologies help solve social problems such as transportation and education, while research brings benefits such as curing diseases and improving food production. Despite the impacts caused by science and society on each other, this relationship is rarely studied and they are often seen as different universes. Previous literature focuses only on a single domain, detecting social demands or research fronts for example, without ever crossing the results for new insights. In this work, we create a system that is able to assess the relationship between social and scholar data using the topics discussed in social networks and research topics. We use the articles as science sensors and humans as social sensors via social networks. Topic modeling algorithms are used to extract and label social subjects and research themes and then topic correlation metrics are used to create links between them if they have a significant relationship. The proposed system is based on topic modeling, labeling and correlation from heterogeneous sources, so it can be used in a variety of scenarios. We make an evaluation of the approach using a large-scale Twitter corpus combined with a PubMed article corpus. In both of them, we work with data of the Zika epidemic in the world, as this scenario provides topics and discussions on both domains. Our work was capable of discovering links between various topics of different domains, which suggests that some of the relationships can be automatically inferred by the sensors. Results can open new opportunities for forecasting social behavior, assess community interest in a scientific subject or directing research to the population welfare.

GO2Vec: transforming GO terms and proteins to vector representations via graph embeddings

BackgroundSemantic similarity between Gene Ontology (GO) terms is a fundamental measure for many bioinformatics applications, such as determining functional similarity between genes or proteins. Most previous research exploited information content to estimate the semantic similarity between GO terms; recently some research exploited word embeddings to learn vector representations for GO terms from a large-scale corpus. In this paper, we proposed a novel method, named GO2Vec, that exploits graph embeddings to learn vector representations for GO terms from GO graph. GO2Vec combines the information from both GO graph and GO annotations, and its learned vectors can be applied to a variety of bioinformatics applications, such as calculating functional similarity between proteins and predicting protein-protein interactions.ResultsWe conducted two kinds of experiments to evaluate the quality of GO2Vec: (1) functional similarity between proteins on the Collaborative Evaluation of GO-based Semantic Similarity Measures (CESSM) dataset and (2) prediction of protein-protein interactions on the Yeast and Human datasets from the STRING database. Experimental results demonstrate the effectiveness of GO2Vec over the information content-based measures and the word embedding-based measures.ConclusionOur experimental results demonstrate the effectiveness of using graph embeddings to learn vector representations from undirected GO and GOA graphs. Our results also demonstrate that GO annotations provide useful information for computing the similarity between GO terms and between proteins.

Refining current insights into the wh-in-situ interrogative construction in French: the case of Contemporary Hexagonal French

Abstract This paper addresses the debate on the existence of specifically syntactic restrictions on one of a larger set of structures available in Modern French for information-requesting questions with a non-subject wh-phrase: the so-called wh-in-situ interrogative construction, in which the wh-phrase occurs in post-verbal position. From a comprehensive investigation of the applicability of such restrictions that is based on a large number of instances of the wh-in-situ interrogative construction from a specially designed large-scale corpus instantiating the present-day French of Metropolitan France, the paper essentially establishes that, in this variety, a fair number of these restrictions do not hold. Additionally, the paper explores the existence of morphological as well as syntactic restrictions not yet thoroughly explored in the literature and, fundamentally, uncovers that, in the variety looked into, various issues pertaining to the verb (tense, mood, passive voice, verb class, non-finite clausal complements) do not bear on the felicity of the wh-in-situ interrogative construction, while a restriction possibly holds that relates to the specificity of the subject.

PAKE: a supervised approach for Persian automatic keyword extraction using statistical features

Keywords are a collection of important words in a document that are the core topic of the discussion. This paper proposes a hybrid method for automatically extracting keywords from Persian documents and web pages. In the proposed method, firstly, based on linguistic knowledge, processing was performed at word and letter levels to optimize of the analysis. Then a new statistical features set is defined and extracted at the word level. At the final stage, keywords are determined using the SVM algorithm. Also, in this paper, due to the lack of a corpus for evaluating the methods of automatic extraction of Persian keywords, a large-scale corpus has been developed and introduced. The achieved F-measure for keywords and non-keywords are 99.89% and 99.99% respectively.

Finding relevant free-text radiology reports at scale with IBM Watson Content Analytics: a feasibility study in the UK NHS

BackgroundSignificant amounts of health data are stored as free-text within clinical reports, letters, discharge summaries and notes. Busy clinicians have limited time to read such large amounts of free-text and are at risk of information overload and consequently missing information vital to patient care. Automatically identifying relevant information at the point of care has the potential to reduce these risks but represents a considerable research challenge. One software solution that has been proposed in industry is the IBM Watson analytics suite which includes rule-based analytics capable of processing large document collections at scale.ResultsIn this paper we present an overview of IBM Watson Content Analytics and a feasibility study using Content Analytics with a large-scale corpus of clinical free-text reports within a UK National Health Service (NHS) context. We created dictionaries and rules for identifying positive incidence of hydronephrosis and brain metastasis from 5.6 m radiology reports and were able to achieve 94% precision, 95% recall and 89% precision, 94% recall respectively on a sample of manually annotated reports. With minor changes for US English we applied the same rule set to an open access corpus of 0.5 m radiology reports from a US hospital and achieved 93% precision, 94% recall and 84% precision, 88% recall respectively.ConclusionsWe were able to implement IBM Watson within a UK NHS context and demonstrate effective results that could provide clinicians with an automatic safety net which highlights clinically important information within free-text documents. Our results suggest that currently available technologies such as IBM Watson Content Analytics already have the potential to address information overload and improve clinical safety and that solutions developed in one hospital and country may be transportable to different hospitals and countries. Our study was limited to exploring technical aspects of the feasibility of one industry solution and we recognise that healthcare text analytics research is a fast-moving field. That said, we believe our study suggests that text analytics is sufficiently advanced to be implemented within industry solutions that can improve clinical safety.

Filtered Pseudo-parallel Corpus Improves Low-resource Neural Machine Translation

Large-scale parallel corpora are essential for training high-quality machine translation systems; however, such corpora are not freely available for many language translation pairs. Previously, training data has been augmented by pseudo-parallel corpora obtained by using machine translation models to translate monolingual corpora into the source language. However, in low-resource language pairs, in which only low-accurate machine translation systems can be used, translation quality degrades when a pseudo-parallel corpus is naively used. To improve machine translation performance with low-resource language pairs, we propose a method to effectively expand the training data via filtering the pseudo-parallel corpus using quality estimation based on sentence-level round-trip translation. For experiments with three language pairs that utilized small, medium, and large size parallel corpora, BLEU scores significantly improved for low-resource language pairs. Additionally, the effects of iterative bootstrapping on translation performance quality is investigated; resultingly, it is confirmed that bootstrapping can further improve the translation performance.

Extracting comprehensive clinical information for breast cancer using deep learning methods

ObjectiveBreast cancer is the most common malignant tumor among women. The diagnosis and treatment information of breast cancer patients is abundant in multiple types of clinical fields, including clinicopathological data, genotype and phenotype information, treatment information, and prognosis information. However, current studies are mainly focused on extracting information from one specific type of clinical field. This study defines a comprehensive information model to represent the whole-course clinical information of patients. Furthermore, deep learning approaches are used to extract the concepts and their attributes from clinical breast cancer documents by fine-tuning pretrained Bidirectional Encoder Representations from Transformers (BERT) language models. Materials and methodsThe clinical corpus that was used in this study was from one 3A cancer hospital in China, consisting of the encounter notes, operation records, pathology notes, radiology notes, progress notes and discharge summaries of 100 breast cancer patients. Our system consists of two components: a named entity recognition (NER) component and a relation recognition component. For each component, we implemented deep learning-based approaches by fine-tuning BERT, which outperformed other state-of-the-art methods on multiple natural language processing (NLP) tasks. A clinical language model is first pretrained using BERT on a large-scale unlabeled corpus of Chinese clinical text. For NER, the context embeddings that were pretrained using BERT were used as the input features of the Bi-LSTM-CRF (Bidirectional long-short-memory-conditional random fields) model and were fine-tuned using the annotated breast cancer notes. Furthermore, we proposed an approach to fine-tune BERT for relation extraction. It was considered to be a classification problem in which the two entities that were mentioned in the input sentence were replaced with their semantic types. ResultsOur best-performing system achieved F1 scores of 93.53% for the NER and 96.73% for the relation extraction. Additional evaluations showed that the deep learning-based approaches that fine-tuned BERT did outperform the traditional Bi-LSTM-CRF and CRF machine learning algorithms in NER and the attention-Bi-LSTM and SVM (support vector machines) algorithms in relation recognition. ConclusionIn this study, we developed a deep learning approach that fine-tuned BERT to extract the breast cancer concepts and their attributes. It demonstrated its superior performance compared to traditional machine learning algorithms, thus supporting its uses in broader NER and relation extraction tasks in the medical domain.

Efficient Corpus Creation Method for NLU Using Interview with Probing Questions

This paper presents an efficient method to build a corpus to train natural language understanding (NLU) modules. Conventional corpus creation methods involve a common cycle: a subject is given a specific situation where the subject operates a device by voice, and then the subject speaks one utterance to execute the task. In these methods, many subjects are required in order to build a large-scale corpus, which causes a problem of increasing lead time and financial cost. To solve this problem, we propose to incorporate a “probing question” into the cycle. Specifically, after a subject speaks one utterance, the subject is asked to think of alternative utterances to execute the same task. In this way, we obtain many utterances from a small number of subjects. An evaluation of the proposed method applied to interview-based corpus creation shows that the proposed method reduces the number of subjects by 41% while maintaining morphological diversity in a corpus and morphological coverage for user utterances spoken to commercial devices. It also shows that the proposed method reduces the total time for interviewing subjects by 36% compared with the conventional method. We conclude that the proposed method can be used to build a useful corpus while reducing lead time and financial cost.

How native language and L2 proficiency affect EFL learners’ capitalisation abilities: a large-scale corpus study

Capitalisation is a salient orthographic feature, which plays an important role in linguistic processing during reading, and in writing assessment. Learners’ second language (L2) capitalisation skills are influenced by their native language (L1), but earlier studies of L1 influence did not focus on learners’ capitalisation, and examined primarily ‘narrow’ samples. This study examines capitalisation error patterns in a large-scale corpus of over 133,000 texts, composed by nearly 38,000 EFL learners, who represent seven different L1s and a wide range of English proficiency levels. The findings show that speakers of all L1s made a large number of capitalisation errors, in terms of errors per word and error proportion (out of all errors), especially at lower L2 proficiency levels. Under-capitalisation was more common than over-capitalisation, though this gap narrowed over time. Interestingly, L1s which share English's Latin script had higher error rates, suggesting that (assumed) perceived similarity between the L1 and the L2 increases interference, though this interference could not be explained only through direct negative transfer. There was also an interaction between L1 influence and L2 proficiency, so that differences between speakers of different L1s became smaller as their L2 proficiency improved.