Large-scale Corpus Research Articles

Improved Distant Supervision Relation Extraction Based on Edge- Reasoning Hybrid Graph Model

Abstract Distant supervision relation extraction (DSRE) trains a classifier by automatically labeling data through aligning triples in the knowledge base (KB) with large-scale corpora. Training data generated by distant supervision may contain many mislabeled instances, which is harmful to the training of the classifier. Some recent methods show that relevant background information in KBs, such as entity type (e.g., Organization and Book), can improve the performance of DSRE. However, there are three main problems with these methods. Firstly, these methods are tailored for a specific type of information. A specific type of information only has a positive effect on a part of instances and will not be beneficial to all cases. Secondly, different background information is embedded independently, and no reasonable interaction is achieved. Thirdly, previous methods do not consider the side effect of the introduced noise of background information. To address these issues, we leverage five types of background information instead of a specific type of information in previous works and propose a novel edge-reasoning hybrid graph (ER-HG) model to realize reasonable interaction between different kinds of information. In addition, we further employ an attention mechanism for the ER-HG model to alleviate the side effect of noise. The ER-HG model integrates all types of information efficiently and is very robust to the noise of information. We conduct experiments on two widely used datasets. The experimental results demonstrate that our model outperforms the state-of-the-art methods significantly in held-out metric and robustness tests.

Wav2KWS: Transfer Learning From Speech Representations for Keyword Spotting

With the expanding development of on-device artificial intelligence, voice-enabled devices such as smart speakers, wearables, and other on-device or edge processing systems have been proposed. However, building or obtaining large training datasets that are essential for robust keyword spotting (KWS) remains cumbersome. To address this problem, we propose a deep neural network that can rapidly establish a high-performance KWS system from arbitrary keyword instruction sets. We use an encoder pretrained with a large-scale speech corpus as the backbone network and then design an effective transfer network for KWS. To demonstrate the feasibility of the proposed network, various experiments were conducted on Google Speech Command Datasets V1 and V2. In addition, to verify the applicability of the network for different languages, we conducted experiments using three different Korean speech command datasets. The proposed network outperforms state-of-the-art deep neural networks in both experiments. Furthermore, the proposed network can understand real human voice even when trained with synthetic text-to-speech data.

GAN-GLS: Generative Lyric Steganography Based on Generative Adversarial Networks

Steganography based on generative adversarial networks (GANs) has become a hot topic among researchers. Due to GANs being unsuitable for text fields with discrete characteristics, researchers have proposed GAN-based steganography methods that are less dependent on text. In this paper, we propose a new method of generative lyrics steganography based on GANs, called GAN-GLS. The proposed method uses the GAN model and the large-scale lyrics corpus to construct and train a lyrics generator. In this method, the GAN uses a previously generated line of a lyric as the input sentence in order to generate the next line of the lyric. Using a strategy based on the penalty mechanism in training, the GAN model generates non-repetitive and diverse lyrics. The secret information is then processed according to the data characteristics of the generated lyrics in order to hide information. Unlike other text generation-based linguistic steganographic methods, our method changes the way that multiple generated candidate items are selected as the candidate groups in order to encode the conditional probability distribution. The experimental results demonstrate that our method can generate high-quality lyrics as stego-texts. Moreover, compared with other similar methods, the proposed method achieves good performance in terms of imperceptibility, embedding rate, effectiveness, extraction success rate and security.

Corpulyzer: A Novel Framework for Building Low Resource Language Corpora

The rapid proliferation of artificial intelligence has led to the development of sophisticated cutting-edge systems in natural language processing and computational linguistics domains. These systems heavily rely on high-quality dataset/corpora for the training of deep-learning algorithms to develop precise models. The preparation of a high-quality gold standard corpus for natural language processing on a large scale is a challenging task due to the need of huge computational resources, accurate language identification models, and precise content parsing tools. This task is further exacerbated in case of regional languages due to the scarcity of web content. In this article, we propose a generic framework of Corpus Analyzer – Corpulyzer – a novel framework for building low resource language corpora. Our framework consists of corpus generation and corpus analyzer module. We demonstrate the efficacy of our framework by creating a high-quality large scale corpus for the Urdu language as a case study. Leveraging dataset from Common Crawl Corpus (CCC), first, we prepare a list of seed URLs by filtering the Urdu language webpages. Next, we use Corpulyzer to crawl the World-Wide-Web (WWW) over a period of four years (2016–2020). We build Urdu web corpus “UrduWeb20” that consists of 8.0 million Urdu webpages crawled from 6,590 websites. In addition, we propose Low-Resource Language (LRL) website scoring algorithm and content-size filter for language-focused crawling to achieve optimal use of computational resources. Moreover, we analyze UrduWeb20 using variety of traditional metrics such as web-traffic-rank, URL depth, duplicate documents, and vocabulary distribution along with our newly defined content-richness metrics. Furthermore, we compare different characteristics of our corpus with three datasets of CCC. In general, we observe that contrary to CCC that focuses on crawling the limited number of webpages from highly ranked Urdu websites, Corpulyzer performs an in-depth crawling of Urdu content-rich websites. Finally, we made available Corpulyzer framework for the research community for corpus building.

Summary of Research Methods on Pre-Training Models of Natural Language Processing

In recent years, deep learning technology has been widely used and developed. In natural language processing tasks, pre-training models have been more widely used. Whether it is sentence extraction or sentiment analysis of text, the pre-training model plays a very important role. The use of a large-scale corpus for unsupervised pre-training of models has proven to be an excellent and effective way to provide models. This article summarizes the existing pre-training models and sorts out the improved models and processing methods of the relatively new pre-training models, and finally summarizes the challenges and prospects of the current pre-training models.

Research on Named Entity Recognition Technology for Chinese Titles

Aiming at the problem that the feature extraction of Chinese title data is insufficient in the process of named entity recognition and the marking data is difficult to obtain, this paper presents a CNN-BiLSTM-CRF model. The model trains large-scale corpus to generate word vectors with semantic information and trains the character vectors in words through CNN to generate character vectors containing character features, the word vector and the character vectors are merged as the input of BiLSTM network. Through the BiLSTM network, the front and back texts features are fully extracted, and CRF is used to restrict the relevance of tags. In addition, this paper proposes an algorithm based on active learning. This algorithm uses the proposed model construction committee to select the samples with large amount of information to mark, which effectively reduces the amount of data annotation.

Building Chinese Request Pattern Graphs for Chinese-Korean Translation

This paper aims to describe diverse request patterns in Chinese for the automatic classification and the Chinese-Korean translation of music-request sentences in AI assistant platforms. These patterns are formalized by Local-Grammar Graphs (LGGs), a finite-state grammar, and represented under directed acyclic graphs based on the main function categories. The UNITEX graph editor has been used to construct Chinese LGGs, and the request pattern tags that are proposed for Korean sentences based on DECO Korean Electronic dictionaries was applied. The result of this study has been examined and evaluated by comparison with Google Translate results at the end of the paper. Experimental results prove that the LGG grammar constructed in this paper can effectively cover and annotate various request patterns. This will provide the possibility to generate a large-scale bilingual annotation corpus in the field of music listening.

Acoustic properties of word and phrasal prominence in Uzbek

Based on a large-scale corpus of experimental data produced by 8 native speakers of Tashkent Uzbek, we assess the presence of canonical word-final stress in real words spoken in three dialogue types: without focus, with contrastive focus, and with new information focus on the target. The first context provides baseline information regarding the manifestation of stress, in the absence of additional focus properties. By comparing the latter two contexts with the former, we are also able to assess the acoustic manifestation of the two types of focus. The most noteworthy properties of the final syllable are its relatively long duration and sharp falling contour, potentially serving as the cues to lexical stress, and enhanced by both types of focus. Due to the word-final position of stress, however, the patterns we observe could also be consistent with boundary properties, a possibility we consider as well. In addition, we briefly compare the prosodic patterns we observe in Uzbek with similarly collected data in Turkish. We find that the prominence patterns in Uzbek, while not particularly strong, are nevertheless stronger than those in Turkish, and also exhibit crucial differences. Implications for Turkic prosody more generally are also suggested.

Ethnic and gender variation in the use of Colloquial Singapore English discourse particles

Discourse particles are among the most commented-upon features of Colloquial Singapore English (CSE). Their use has been shown to vary depending on formality, context, gender and ethnicity, although results differ from one study to another. This study uses theCorpus of Singapore English Messages(CoSEM), a large-scale corpus of texts composed by Singaporeans and sent using electronic messaging services, to investigate gender and ethnic factors as predictors of particle use. The results suggest a strong gender effect as well as several particle-specific ethnic effects. More generally, our study underlines the special nature of the grammatical class of discourse particles in CSE, which is open to new additions as the sociolinguistic and pragmatic need for them develops.

Selection of In-Domain Bilingual Sentence Pairs Based on Topic Information

The performance of a machine translation system (MTS) depends on the quality and size of the training data. How to extend the training dataset for the MTS in specific domains with effective methods to enhance the performance of machine translation needs to be explored. A method for selecting in-domain bilingual sentence pairs based on the topic information is proposed. With the aid of the topic relevance of the bilingual sentence pairs to the target domain, subsets of sentence pairs related to the texts to be translated are selected from a large-scale bilingual corpus to train the translation system in specific domains to improve the translation quality for in-domain texts. Through the test, the bilingual sentence pairs are selected by using the proposed method, and further the MTS is trained. In this way, the translation performance is greatly enhanced.

The linguistic and cultural community of a Slavic village: research project assumptions

The article presents assumptions of a project laying foundations for linguistic and cultural comparative research into the contemporary Slavic village. Relatively complete material obtained from carefully selected thematic micro-fields is intended to give the basis for large-scale corpus research.

Hierarchical state recurrent neural network for social emotion ranking

Text generation with auxiliary attributes, such as topics or sentiments, has made remarkable progress. However, high-quality labeled data is difficult to obtain for the large-scale corpus. Therefore, this paper focuses on social emotion ranking aiming to identify social emotions with different intensities evoked by online documents, which could be potentially beneficial for further controlled text generation. Existing studies often consider each document as an entirety that fail to capture the inner relationship between sentences in a document. In this paper, we propose a novel hierarchical state recurrent neural network for social emotion ranking. A hierarchy mechanism is employed to capture the key hierarchical semantic structure in a document. Moreover, instead of incrementally reading a sequence of words or sentences as in traditional recurrent neural networks, the proposed approach encodes the hidden states of all words or sentences simultaneously at each recurrent step to capture long-range dependencies precisely. Experimental results show that the proposed approach performs remarkably better than the state-of-the-art social emotion ranking approaches and is useful for controlled text generation.

The position of the genitive in Old English prose: Intertextual differences and the role of Latin

AbstractThe present article is a systematic, large-scale corpus study of the varying position of the Old English adnominal genitive. Particular focus is put on intertextual variation and the potential influence of Latin, which is analysed alongside such intra-linguistic factors as syntactic weight and the animacy of the referent. The model of logistic regression adopted helps address a key issue in studies on genitive placement, namely, if and how multiple variables exert combined impact on the choice of the variant. The paper highlights the importance of a bottom-to-top approach in studies of older English syntax, as global tendencies turn out to be the corollary of significantly different contributions on the level of individual texts, whose translation status (original composition or translation) is also of importance to the variation studied.

A hybrid classical-quantum workflow for natural language processing

Natural language processing (NLP) problems are ubiquitous in classical computing, where they often require significant computational resources to infer sentence meanings. With the appearance of quantum computing hardware and simulators, it is worth developing methods to examine such problems on these platforms. In this manuscript we demonstrate the use of quantum computing models to perform NLP tasks, where we represent corpus meanings, and perform comparisons between sentences of a given structure. We develop a hybrid workflow for representing small and large scale corpus data sets to be encoded, processed, and decoded using a quantum circuit model. In addition, we provide our results showing the efficacy of the method, and release our developed toolkit as an open software suite.

A BERT Fine-tuning Model for Targeted Sentiment Analysis of Chinese Online Course Reviews

Accurate analysis of targeted sentiment in online course reviews helps in understanding emotional changes of learners and improving the course quality. In this paper, we propose a fine-tuned bidirectional encoder representation from transformers (BERT) model for targeted sentiment analysis of course reviews. Specifically, it consists of two parts: binding corporate rules — conditional random field (BCR-CRF) target extraction model and a binding corporate rules — double attention (BCR-DA) target sentiment analysis model. Firstly, based on a large-scale Chinese review corpus, intra-domain unsupervised training of a BERT pre-trained model (BCR) is performed. Then, a Conditional Random Field (CRF) layer is introduced to add grammatical constraints to the output sequence of the semantic representation layer in the BCR model. Finally, a BCR-DA model containing double attention layers is constructed to express the sentiment polarity of the course review targets in a classified manner. Experiments are performed on Chinese online course review datasets of China MOOC. The experimental results show that the F1 score of the BCR-CRF model reaches above 92%, and the accuracy of the BCR-DA model reaches above 72%.

Few-Shot Named Entity Recognition via Meta-Learning

Few-shot learning under the <inline-formula><tex-math notation="LaTeX">$N$</tex-math></inline-formula> -way <inline-formula><tex-math notation="LaTeX">$K$</tex-math></inline-formula> -shot setting (i.e., <inline-formula><tex-math notation="LaTeX">$K$</tex-math></inline-formula> annotated samples for each of <inline-formula><tex-math notation="LaTeX">$N$</tex-math></inline-formula> classes) has been widely studied in relation extraction (e.g., FewRel) and image classification (e.g., Mini-ImageNet). Named entity recognition (NER) is typically framed as a sequence labeling problem where the entity classes are inherently entangled together because the entity number and classes in a sentence are not known in advance, leaving the <inline-formula><tex-math notation="LaTeX">$N$</tex-math></inline-formula> -way <inline-formula><tex-math notation="LaTeX">$K$</tex-math></inline-formula> -shot NER problem so far unexplored. In this paper, we first formally define a more suitable <inline-formula><tex-math notation="LaTeX">$N$</tex-math></inline-formula> -way <inline-formula><tex-math notation="LaTeX">$K$</tex-math></inline-formula> -shot setting for NER. Then we propose <small>FewNER</small> , a novel meta-learning approach for few-shot NER. <small>FewNER</small> separates the entire network into a task-independent part and a task-specific part. During training in <small>FewNER</small> , the task-independent part is meta-learned across multiple tasks and the task-specific part is learned for each individual task in a low-dimensional space. At test time, <small>FewNER</small> keeps the task-independent part fixed and adapts to a new task via gradient descent by updating only the task-specific part, resulting in it being less prone to overfitting and more computationally efficient. Compared with pre-trained language models (e.g., BERT and ELMo) which obtain the transferability in an implicit manner (i.e., relying on large-scale corpora), <small>FewNER</small> explicitly optimizes the capability of “learning to adapt quickly” through meta-learning. The results demonstrate that <small>FewNER</small> achieves state-of-the-art performance against nine baseline methods by significant margins on three adaptation experiments (i.e., intra-domain cross-type, cross-domain intra-type and cross-domain cross-type).

Postnominal relative clauses in Chinese

Abstract In contrast to well-studied prenominal relative clauses (RCs) in Chinese, little has been known about postnominal RCs that are non-canonical but existent in spoken Chinese. Focusing on Standard Mandarin, this paper examines in a large-scale spoken corpus the distributional patterns of postnominal RCs. Using distribution patterns of prenominal RCs in existing corpus studies as benchmarks, we show that postnominal RCs in our spoken corpus of Standard Mandarin tend to modify sentential objects more frequently than sentential subjects, and that they are likely to be short, with extremely rare presence of aspect markers. Based on these patterns, we propose that postnominal RCs in Standard Mandarin are mostly afterthoughts, motivated by information structure of spoken languages and word order principles. To better understand their general coverage, we further investigate postnominal RCs in Chinese dialects using available resources, including Yue, Min, Xiang, and Wu, followed by a raw comparison of cross-dialectal similarities and differences. We conclude that postnominal RCs in Chinese are similarly motivated, but their degrees of grammaticalization vary.

Cartolabe: A Web-Based Scalable Visualization of Large Document Collections.

We describe Cartolabe, a web-based multiscale system for visualizing and exploring large textual corpora based on topics, introducing a novel mechanism for the progressive visualization of filtering queries. Initially designed to represent and navigate through scientific publications in different disciplines, Cartolabe has evolved to become a generic framework and accommodate various corpora, ranging from Wikipedia (4.5M entries) to the French National Debate (4.3M entries). Cartolabe is made of two modules: The first relies on natural language processing methods, converting a corpus and its entities (documents, authors, and concepts) into high-dimensional vectors, computing their projection on the two-dimensional plane, and extracting meaningful labels for regions of the plane. The second module is a web-based visualization, displaying tiles computed from the multidimensional projection of the corpus using the Umap projection method. This visualization module aims at enabling users with no expertise in visualization and data analysis to get an overview of their corpus, and to interact with it: exploring, querying, filtering, panning, and zooming on regions of semantic interest. Three use cases are discussed to illustrate Cartolabe's versatility and ability to bring large-scale textual corpus visualization and exploration to a wide audience.

Domain-specific meta-embedding with latent semantic structures

Meta-embedding aims at assembling pre-trained embeddings from various sources and producing more expressively powerful word representations. Many natural language processing (NLP) tasks in a specific domain benefit from meta-embedding, especially when the task suffers from low resources. This paper proposes an unsupervised meta-embedding method that jointly models background knowledge from the source embeddings and domain-specific knowledge from the task domain. Specifically, embeddings from multiple sources for a word are dynamically aggregated to a single meta-embedding by a differentiable attention module. The embeddings derived from pre-training on a large-scale corpus provide complete background knowledge of word usage. Then, the meta-embedding is further enriched by exploring domain-specific knowledge from each task domain in two ways. First, contextual information in the raw corpus is considered to capture the semantics of words. Second, a graph representing domain-specific semantic structures is extracted from the raw corpus to highlight the relationships between salient words, then the graph is modeled by a powerful graph convolution network to effectively capture rich semantic structures among words in the task domain. Experiments conducted on two tasks, i.e., text classification and relation extraction, show that our model outputs more accurate word meta-embeddings for the task domain, compared to other state-of-the-art competitors. .

Formant Frequencies of Adult Speakers of Australian English and Effects of Sex, Age, Geographical Location, and Vowel Quality

The primary aim of this study was to provide normative formant frequency (F) values for male and female speakers of Australian English. The secondary aim was to examine the effects of speaker sex, age, vowel quality, and geographical location on F. The first three monophthong formant frequencies (F1, F2, and F3) for 244 female and 135 male speakers aged 18-60 years from a recent large-scale corpus of Australian English were analysed on a passage reading task. Mixed effects linear regression models suggested that speaker sex, speaker age, and vowel quality significantly predicted F1, F2, and F3 (P=0.000). Effect sizes suggested that speaker sex and vowel quality contributed most to the variations in F1, F2, and F3 whereas speaker age and geographical location contributed a smaller amount. Both clinicians and researchers are provided with normative F data for 18-60 year-old speakers of Australian English. Such data have increased internal and external validity relative to previous literature. F normative data for speakers of Australian English should be considered with reference to speaker sex and vowel but it may not be practically necessary to adjust for speaker age and geographical location.