Monolingual Models Research Articles

Acquiring Knowledge from Pre-Trained Model to Neural Machine Translation

Pre-training and fine-tuning have achieved great success in natural language process field. The standard paradigm of exploiting them includes two steps: first, pre-training a model, e.g. BERT, with a large scale unlabeled monolingual data. Then, fine-tuning the pre-trained model with labeled data from downstream tasks. However, in neural machine translation (NMT), we address the problem that the training objective of the bilingual task is far different from the monolingual pre-trained model. This gap leads that only using fine-tuning in NMT can not fully utilize prior language knowledge. In this paper, we propose an Apt framework for acquiring knowledge from pre-trained model to NMT. The proposed approach includes two modules: 1). a dynamic fusion mechanism to fuse task-specific features adapted from general knowledge into NMT network, 2). a knowledge distillation paradigm to learn language knowledge continuously during the NMT training process. The proposed approach could integrate suitable knowledge from pre-trained models to improve the NMT. Experimental results on WMT English to German, German to English and Chinese to English machine translation tasks show that our model outperforms strong baselines and the fine-tuning counterparts.

Jeffrey Shandler. Holocaust Memory in the Digital Age: Survivors’ Stories and New Media Practices. Stanford, CA: Stanford University Press, 2017. ix + 232 pp.

Reviewed by: Holocaust Memory in the Digital Age: Survivors' Stories and New Media Practices by Jeffrey Shandler Oren Baruch Stier Jeffrey Shandler. Holocaust Memory in the Digital Age: Survivors' Stories and New Media Practices. Stanford, CA: Stanford University Press, 2017. ix + 232 pp. doi:10.1017/S0364009419001120 Jeffrey Shandler's latest book is based on years of online access to the video testimonies collected by the USC Shoah Foundation's Visual History Archive (VHA). The VHA, he notes, arose and developed at a critical temporal juncture in both Holocaust memory and technological dissemination, "as it straddles the temporal boundary marked by the loss of living witnesses to the Holocaust, the VHA also bestrides the transition from the 'video age' to the 'digital age'" (3–4). Shandler asks: What are the implications and impacts of the mix of memory practices employed in and by these testimonies, as well as the archive that houses them? Preceded by a chapter that contextualizes the archive historically, memorially, and technologically, the remainder of this book presents a series of case studies grouped formally according to three categories derived from "Aristotle's Poetics—mythos, lexis, and opsis…. Situat[ing] VHA interviews within the extensive history of performed narrative works, rather than confining them to a particular genre (such as oral history or wartime memoir) or treating them as exceptional phenomena" (5). [End Page 224] The first of these groupings addresses the nature of video-recorded narrative, typically presented by various archiving projects as possessing "an unrivaled immediacy and authority" (43). But, in actuality, the narratives are highly contextual, dependent on social, structural, and institutional frameworks. Even the very nomenclature employed can affect how these narratives are understood: eyewitnesses offering their accounts are referred to as "survivors, which connotes tenacity" (46). Furthermore, "because these narratives are esteemed as providing information to historians, moral guidance to the young, and retorts to Holocaust deniers, survivor videos are often referred to as 'testimonies,' invoking this term's implication of bearing witness in a legal proceeding or making a religious avowal" (47). Of particular interest to Shandler in this chapter are "the VHA recordings in which survivors' life histories are informed by other narratives" (49), including other versions of their own story, because the "tales retold" reflect and inform the very act and craft of storytelling. Shandler focuses on two groups of testimonies that respond to other narratives: accounts that discuss the film Schindler's List (itself embedded in the origin story of the VHA, since the VHA emerged from director Steven Spielberg's discussion with Holocaust survivors in the course of making his award-winning movie) and interviews with celebrity survivors. The second set of case studies considers the complexities of language in the interviews. The VHA, like most collections of a similar nature, works on a monolingual model. Subjects chose which language to speak, which "constrained most survivors' full range of linguistic resources" (90). Shandler, an expert on Yiddish language, literature, and culture, focuses in this chapter on ways Yiddish is engaged in the VHA interviews; 633 of them were conducted fully or partially in the language (92), a choice of the interviewees that reveals the multivalency of Yiddish as well as their own thoughts concerning language and memory and the relationships between the two. In terms of the narration of the past, Shandler examines recordings in Yiddish and English in order to explore the practice of code switching and what it may indicate. He also looks at the significance of Yiddish for survivors who mention it in the course of their testimonies, a "limited, inconstant presence … [that] is itself revealing" (113). Finally, Shandler discusses cases in which Yiddish texts are performed in the interviews: episodes, typically near the end of their testimonies, in which survivors assert a greater degree of agency—moments that accentuate their very survival. Yiddish is not only symbolic in these cases, but it is also quite literally preserved "as a resource for the future" (124). The third and final group of case studies addresses optics. In a chapter titled "Spectacle: Seeing as Believing," Shandler analyzes the uniquely visual components of these interviews, including protocols for videographers and, as others have noted, the domestic setting...

Speech perception in a second language: Hearing is believing-seeing is not.

This study examined the effects of the speaker's face and accent on second language (L2) speech perception. Forty-two Chinese speakers of English immersed in the L2 environment were instructed to perform a cross-modal semantic judgement task. They saw an Asian or Caucasian face on the screen and heard word pairs in L2 in a native English accent or a Chinese accent, and were asked to judge whether the word pairs were related to each other in meaning or not. Results showed that for words presented in the native accent, there was a semantic effect in both reaction time and accuracy, irrespective of the face shown. For words presented in the non-native accent, the RT data showed a semantic effect, whereas the accuracy revealed a reversed semantic effect. The speed-accuracy trade-off suggests a relatively weak semantic effect. These patterns were not modulated by the faces accompanying the word pairs. These results suggest that the cue of accent plays an important role during bilinguals' speech perception in L2, such that non-native accent hampers speech perception, even when it matches bilinguals' first language. In contrast, bilinguals do not seem to depend on the social indexical cue of the face when it is not reliable. The present findings hold implications for the Bilingual Model of Lexical Access (BIMOLA) of bilingual speech perception and the monolingual models of social speech perception.

GILE: A Generalized Input-Label Embedding for Text Classification

Neural text classification models typically treat output labels as categorical variables that lack description and semantics. This forces their parametrization to be dependent on the label set size, and, hence, they are unable to scale to large label sets and generalize to unseen ones. Existing joint input-label text models overcome these issues by exploiting label descriptions, but they are unable to capture complex label relationships, have rigid parametrization, and their gains on unseen labels happen often at the expense of weak performance on the labels seen during training. In this paper, we propose a new input-label model that generalizes over previous such models, addresses their limitations, and does not compromise performance on seen labels. The model consists of a joint nonlinear input-label embedding with controllable capacity and a joint-space-dependent classification unit that is trained with cross-entropy loss to optimize classification performance. We evaluate models on full-resource and low- or zero-resource text classification of multilingual news and biomedical text with a large label set. Our model outperforms monolingual and multilingual models that do not leverage label semantics and previous joint input-label space models in both scenarios.

PARABANK: Monolingual Bitext Generation and Sentential Paraphrasing via Lexically-Constrained Neural Machine Translation

We present PARABANK, a large-scale English paraphrase dataset that surpasses prior work in both quantity and quality. Following the approach of PARANMT (Wieting and Gimpel, 2018), we train a Czech-English neural machine translation (NMT) system to generate novel paraphrases of English reference sentences. By adding lexical constraints to the NMT decoding procedure, however, we are able to produce multiple high-quality sentential paraphrases per source sentence, yielding an English paraphrase resource with more than 4 billion generated tokens and exhibiting greater lexical diversity. Using human judgments, we also demonstrate that PARABANK’s paraphrases improve over PARANMT on both semantic similarity and fluency. Finally, we use PARABANK to train a monolingual NMT model with the same support for lexically-constrained decoding for sentence rewriting tasks.

GlobalTrait: Personality Alignment of Multilingual Word Embeddings

We propose a multilingual model to recognize Big Five Personality traits from text data in four different languages: English, Spanish, Dutch and Italian. Our analysis shows that words having a similar semantic meaning in different languages do not necessarily correspond to the same personality traits. Therefore, we propose a personality alignment method, GlobalTrait, which has a mapping for each trait from the source language to the target language (English), such that words that correlate positively to each trait are close together in the multilingual vector space. Using these aligned embeddings for training, we can transfer personality related training features from high-resource languages such as English to other low-resource languages, and get better multilingual results, when compared to using simple monolingual and unaligned multilingual embeddings. We achieve an average F-score increase (across all three languages except English) from 65 to 73.4 (+8.4), when comparing our monolingual model to multilingual using CNN with personality aligned embeddings. We also show relatively good performance in the regression tasks, and better classification results when evaluating our model on a separate Chinese dataset.

Inference of Biomedical Relations Among Chemicals, Genes, Diseases, and Symptoms Using Knowledge Representation Learning

Knowledge representation learning represents entities and relations of knowledge graph in a continuous low-dimensional semantic space. Recently, various representation learning models have successfully been developed to infer novel relations in general-purpose knowledge bases such as FreeBase and WordNet. However, few studies have used such models for biomedical data for inferring useful relations among biomedical entities such as genes, chemicals, diseases, and symptoms. This study aimed to compare the potential of representation learning models in extracting biomedical relations by using four different types of representation learning models, viz., TransE, PTransE, TransR, and TransH. For training and evaluating the models, we collected and utilized manually curated data from public databases, including relations among chemicals, genes, diseases, and symptoms. Overall, TransE, the most efficient translation-based monolingual knowledge graph embedding model, displayed the best performance with a higher learning speed for large-scale biomedical data. Using TransE, we inferred new relations among chemicals, genes, diseases, and symptoms, and evaluated the reliability of these inferred relations. Furthermore, TransE outperformed an existing statistical method used in the Comparative Toxicogenomics Database for inferring new chemical-disease relations. Together, the present results show that the representation learning model is useful for inferring new biological data from numerous existing biomedical data.

Cross-lingual adaptation of a CTC-based multilingual acoustic model

Multilingual models for Automatic Speech Recognition (ASR) are attractive as they have been shown to benefit from more training data, and better lend themselves to adaptation to under-resourced languages. However, initialisation from monolingual context-dependent models leads to an explosion of context-dependent states. Connectionist Temporal Classification (CTC) is a potential solution to this as it performs well with monophone labels.We investigate multilingual CTC training in the context of adaptation and regularisation techniques that have been shown to be beneficial in more conventional contexts. The multilingual model is trained to model a universal International Phonetic Alphabet (IPA)-based phone set using the CTC loss function. Learning Hidden Unit Contribution (LHUC) is investigated to perform language adaptive training. During cross-lingual adaptation, the idea of extending the multilingual output layer to new phonemes is introduced and investigated. In addition, dropout during multilingual training and cross-lingual adaptation is also studied and tested in order to mitigate the overfitting problem.Experiments show that the performance of the universal phoneme-based CTC system can be improved by applying dropout and LHUC and it is extensible to new phonemes during cross-lingual adaptation. Updating all acoustic model parameters shows consistent improvement on limited data. Applying dropout during adaptation can further improve the system and achieve competitive performance with Deep Neural Network / Hidden Markov Model (DNN/HMM) systems on limited data.

Words only go so far: Linguistic context affects bilingual word processing

In their keynote paper, Dijkstra, Wahl, Buytenhuijs, van Halem, Al-jibouri, de Korte, and Rekké (2018) present a computational model of bilingual word recognition and translation, Multilink, that integrates and further refines the architecture and processing principles of two influential models of bilingual word processing: the Bilingual Activation Model (BIA/BIA+) and the Revised Hierarchical model (RHM). Unlike the earlier models, Multilink has been implemented as a computational model so its design principles and assumptions can be compared with human processing data in simulation studies – which is an important step forward in model development and refinement. But Multilink also leaves behind an important theoretical advancement that was touched upon in extending BIA to BIA+ (Dijkstra & Van Heuven, 2002): how linguistic context influences word processing. In their presentation of BIA+, Dijkstra and Van Heuven (2002) hypothesized that syntactic and semantic aspects of sentence context may affect the word identification system. Theoretically, this was an important step forward, as none of the bilingual word processing models (and few monolingual word processing models, for that matter) had incorporated linguistic context, and at that time only a handful of empirical studies had examined how linguistic context affects bilingual word processing. However, in the past 15 years a significant body of empirical work has been published that examines how semantic and syntactic information in sentences impacts word processing in bilinguals. These important insights are not incorporated in the Multilink model.

Shared-hidden-layer Deep Neural Network for Under-resourced Language the Content

Training speech recognizer with under-resourced language data still proves difficult. Indonesian language is considered under-resourced because the lack of a standard speech corpus, text corpus, and dictionary. In this research, the efficacy of augmenting limited Indonesian speech training data with highly-resourced-language training data, such as English, to train Indonesian speech recognizer was analyzed. The training was performed in form of shared-hidden-layer deep-neural-network (SHL-DNN) training. An SHL-DNN has language-independent hidden layers and can be pre-trained and trained using multilingual training data without any difference with a monolingual deep neural network. The SHL-DNN using Indonesian and English speech training data proved effective for decreasing word error rate (WER) in decoding Indonesian dictated-speech by achieving 3.82% absolute decrease compared to a monolingual Indonesian hidden Markov model using Gaussian mixture model emission (GMM-HMM). The case was confirmed when the SHL-DNN was also employed to decode Indonesian spontaneous-speech by achieving 4.19% absolute WER decrease.

TermFinder: log-likelihood comparison and phrase-based statistical machine translation models for bilingual terminology extraction

Bilingual termbanks are important for many natural language processing applications, especially in translation workflows in industrial settings. In this paper, we apply a log-likelihood comparison method to extract monolingual terminology from the source and target sides of a parallel corpus. The initial candidate terminology list is prepared by taking all arbitrary n-gram word sequences from the corpus. Then, a well-known statistical measure (the Dice coefficient) is employed in order to remove any multi-word terms with weak associations from the candidate term list. Thereafter, the log-likelihood comparison method is applied to rank the phrasal candidate term list. Then, using a phrase-based statistical machine translation model, we create a bilingual terminology with the extracted monolingual term lists. We integrate an external knowledge source—the Wikipedia cross-language link databases—into the terminology extraction (TE) model to assist two processes: (a) the ranking of the extracted terminology list, and (b) the selection of appropriate target terms for a source term. First, we report the performance of our monolingual TE model compared to a number of the state-of-the-art TE models on English-to-Turkish and English-to-Hindi data sets. Then, we evaluate our novel bilingual TE model on an English-to-Turkish data set, and report the automatic evaluation results. We also manually evaluate our novel TE model on English-to-Spanish and English-to-Hindi data sets, and observe excellent performance for all domains.

Reading for the Planet: Toward a GeomethodologyGatekeepers: The Emergence of World Literature and the 1960s

Reading for the Planet: Toward a GeomethodologyGatekeepers: The Emergence of World Literature and the 1960s

Supervised sentiment analysis in multilingual environments

This article tackles the problem of performing multilingual polarity classification on Twitter, comparing three techniques: (1) a multilingual model trained on a multilingual dataset, obtained by fusing existing monolingual resources, that does not need any language recognition step, (2) a dual monolingual model with perfect language detection on monolingual texts and (3) a monolingual model that acts based on the decision provided by a language identification tool. The techniques were evaluated on monolingual, synthetic multilingual and code-switching corpora of English and Spanish tweets. In the latter case we introduce the first code-switching Twitter corpus with sentiment labels. The samples are labelled according to two well-known criteria used for this purpose: the SentiStrength scaleand a trinary scale (positive, neutral and negative categories). The experimental results show the robustness of the multilingual approach (1) and also that it outperforms the monolingual models on some monolingual datasets.

Learning the Multilingual Translation Representations for Question Retrieval in Community Question Answering via Non-Negative Matrix Factorization

Community question answering (CQA) has become an increasingly popular research topic. In this paper, we focus on the problem of question retrieval. Question retrieval in CQA can automatically find the most relevant and recent questions that have been solved by other users. However, the word ambiguity and word mismatch problems bring about new challenges for question retrieval in CQA. State-of-the-art approaches address these issues by implicitly expanding the queried questions with additional words or phrases using monolingual translation models. While useful, the effectiveness of these models is highly dependent on the availability of quality parallel monolingual corpora (e.g., question--answer pairs) in the absence of which they are troubled by noise issues. In this work, we propose an alternative way to address the word ambiguity and word mismatch problems by taking advantage of potentially rich semantic information drawn from other languages. Our proposed method employs statistical machine translation to improve question retrieval and enriches the question representation with the translated words from other languages via non-negative matrix factorization. Experiments conducted on real CQA data sets show that our proposed approach is promising.

Special Education Teachers’ Narratives on Literacy Support for Bilingual Students with Dyslexia in Swedish Compulsory Schools

The aim of the study is to investigate how special education teachers talk about their teaching in relation to bilingual students with dyslexia within Swedish compulsory schools. Data consist of transcripts from in-depth interviews with 15 special education teachers. According to the teacher narratives, the special education services appeared to be biased against bilingual students, as the support provided to bilingual students with dyslexia was revealed to be more or less the same as that provided to monolingual Swedish-speaking students with dyslexia. This bias is discussed in relation to the notion of difference blindness as well as in relation to practical constraints. Nevertheless, the teachers strongly advocated collaborative work with mother tongue teachers in order to facilitate dyslexia identification in bilingual students and to gain a more comprehensive picture of their language and literacy competencies, which is a desire that contrasts and contests a pedagogical monolingual master model within special education services.

Presenting GECO: An eyetracking corpus of monolingual and bilingual sentence reading.

This article introduces GECO, the Ghent Eye-Tracking Corpus, a monolingual and bilingual corpus of the eyetracking data of participants reading a complete novel. English monolinguals and Dutch-English bilinguals read an entire novel, which was presented in paragraphs on the screen. The bilinguals read half of the novel in their first language, and the other half in their second language. In this article, we describe the distributions and descriptive statistics of the most important reading time measures for the two groups of participants. This large eyetracking corpus is perfectly suited for both exploratory purposes and more directed hypothesis testing, and it can guide the formulation of ideas and theories about naturalistic reading processes in a meaningful context. Most importantly, this corpus has the potential to evaluate the generalizability of monolingual and bilingual language theories and models to the reading of long texts and narratives. The corpus is freely available at http://expsy.ugent.be/downloads/geco .

Higher-order Lexical Semantic Models for Non-factoid Answer Reranking

Lexical semantic models provide robust performance for question answering, but, in general, can only capitalize on direct evidence seen during training. For example, monolingual alignment models acquire term alignment probabilities from semi-structured data such as question-answer pairs; neural network language models learn term embeddings from unstructured text. All this knowledge is then used to estimate the semantic similarity between question and answer candidates. We introduce a higher-order formalism that allows all these lexical semantic models to chain direct evidence to construct indirect associations between question and answer texts, by casting the task as the traversal of graphs that encode direct term associations. Using a corpus of 10,000 questions from Yahoo! Answers, we experimentally demonstrate that higher-order methods are broadly applicable to alignment and language models, across both word and syntactic representations. We show that an important criterion for success is controlling for the semantic drift that accumulates during graph traversal. All in all, the proposed higher-order approach improves five out of the six lexical semantic models investigated, with relative gains of up to +13% over their first-order variants.

Conceptualising the potential role of L1 in CLIL

Content and language integrated learning (CLIL) is a rapidly growing area of both research and practice in all parts of the world, especially in Europe and Asia. As a young discipline, CLIL has a good potential of distinguishing itself from monolingual L2 immersion education models by becoming more flexible and balanced about the role of L1 in CLIL lessons. Although recent years have witnessed increasing research on the potential role of L1 in foreign language teaching [e.g. Littlewood, W., & Yu, B. 2009. First language and target language in the foreign language classroom. Language Teacher, 42, 1–14], monolingual immersion ideologies are still dominant in many contexts in the world (especially in Southeast Asia) because of a whole host of ideologies. The beliefs affecting medium of instruction policies and practice have their roots in the traditional tenets (e.g. the maximum input hypothesis) in the discipline of second language acquisition (SLA). Although these tenets are increasingly being countered by recent research in multilingualism [see May, S. (Ed.). (2014). The multilingual turn: Implications for SLA, TESOL and bilingual education. New York: Routledge, for a critique of these tenets], SLA still has an influence on pedagogies in both immersion and CLIL programmes. In this paper, I shall first critically review these deep-rooted monolingual tenets. Then, I shall discuss how we can conceptualise the potential role of L1 in CLIL and by extension in content-based instruction (CBI), as both CLIL and immersion programmes are considered to be key approaches to CBI. I shall conclude with suggestions for future research in CLIL.

Crosslingual and Multilingual Construction of Syntax-Based Vector Space Models

Syntax-based distributional models of lexical semantics provide a flexible and linguistically adequate representation of co-occurrence information. However, their construction requires large, accurately parsed corpora, which are unavailable for most languages. In this paper, we develop a number of methods to overcome this obstacle. We describe (a) a crosslingual approach that constructs a syntax-based model for a new language requiring only an English resource and a translation lexicon; and (b) multilingual approaches that combine crosslingual with monolingual information, subject to availability. We evaluate on two lexical semantic benchmarks in German and Croatian. We find that the models exhibit complementary profiles: crosslingual models yield higher accuracies while monolingual models provide better coverage. In addition, we show that simple multilingual models can successfully combine their strengths.

Effective Bilingual Constraints for Semi-Supervised Learning of Named Entity Recognizers

Most semi-supervised methods in Natural Language Processing capitalize on unannotated resources in a single language; however, information can be gained from using parallel resources in more than one language, since translations of the same utterance in different languages can help to disambiguate each other. We demonstrate a method that makes effective use of vast amounts of bilingual text (a.k.a. bitext) to improve monolingual systems. We propose a factored probabilistic sequence model that encourages both crosslanguage and intra-document consistency. A simple Gibbs sampling algorithm is introduced for performing approximate inference. Experiments on English-Chinese Named Entity Recognition (NER) using the OntoNotes dataset demonstrate that our method is significantly more accurate than state-ofthe- art monolingual CRF models in a bilingual test setting. Our model also improves on previous work by Burkett et al. (2010), achieving a relative error reduction of 10.8% and 4.5% in Chinese and English, respectively. Furthermore, by annotating a moderate amount of unlabeled bi-text with our bilingual model, and using the tagged data for uptraining, we achieve a 9.2% error reduction in Chinese over the state-ofthe- art Stanford monolingual NER system.