Use Of Language Models Research Articles

Negative correlation between word-level surprisal and intersubject neural synchronization during narrative listening

Word-predictability measures, such as surprisal, have been used to show that linguistic prediction is an essential neural mechanism for successful comprehension. However, the neural dynamic of this mechanism may differ across individuals listening to the same narrative, revealing possible idiosyncrasies. Therefore, in this study, we investigated the ability of the surprisal measure to address the linguistic prediction in terms of the intersubject phase synchronization (ISPS) among healthy volunteers listening to a narrative during continuous functional Magnetic Resonance Imaging (fMRI) scanning.The data of twenty-seven participants, acquired in a previous study, were re-analyzed to estimate the ISPS associated with an audiobook listening, played forward and backward. Mean ISPS differences between playing conditions were first analyzed across the whole brain. ISPS time-series during the forward condition was explained with both a lexical-only and a semantics-weighted lexical surprisal model of the narrative word series to detect correlations between ISPS and word surprisal values.Compared to the backward condition, mean ISPS was significantly higher during forward condition in a broad network encompassing frontal, temporal, and parietal areas. The lexical-only model disclosed significant negative ISPS-surprisal correlations in the angular gyrus bilaterally, precuneus, left inferior parietal lobule, left middle frontal gyrus, left cerebellum, and left inferior frontal gyrus. The semantics-weighted surprisal model disclosed significant negative ISPS-surprisal correlations in the right angular gyrus, right precuneus, and left inferior frontal gyrus.Besides engaging language-related areas, narrative processing induces significant ISPS levels beyond the language network. The negative ISPS-surprisal correlations observed would signal divergent neural dynamics among individuals, leading to lower group synchronization, when the words to be integrated are characterized by higher surprisal levels. These results provide further support to the use of language models, such as the surprisal, to explain the neural processes of narrative comprehension and additional information into how the human brain exploits predictions during language comprehension.

Extending brain-computer interface access with a multilingual language model in the P300 speller

ABSTRACT Brain-computer interfaces (BCI) such as the P300 speller have the potential to restore communication to advanced-stage neuromuscular disease patients. Research has improved typing speed and accuracy through innovations including the use of language models. While significant advances have been made, implementations have largely been restricted to a single language, primarily English. It is unclear whether these improvements would extend to other languages that present potential technical hurdles due to different alphabets and grammatical structures. Here, we adapt a language model-based classifier designed for English to two other languages, Spanish and Greek, to demonstrate the generalizability of these methods. Online experimental trials with 30 healthy native English, Spanish, and Greek speakers showed no significant difference between performances using the different versions of the system (66.20 vs. 61.97 vs. 60.89 bits/minute). Extending these methods across languages allows for expanding access to BCI systems to other populations, particularly in the developing world.

The Party Is Over Here: Structure and Content in the 2010 Election

In this work, we study the use of Twitter by House, Senate and gubernatorial candidates during the midterm (2010) elections in the U.S. Our data includes almost 700 candidates and over 690k documents that they produced and cited in the 3.5 years leading to the elections. We utilize graph and text mining techniques to analyze differences between Democrats, Republicans and Tea Party candidates, and suggest a novel use of language modeling for estimating content cohesiveness. Our findings show significant differences in the usage patterns of social media, and suggest conservative candidates used this medium more effectively, conveying a coherent message and maintaining a dense graph of connections. Despite the lack of party leadership, we find Tea Party members display both structural and language-based cohesiveness. Finally, we investigate the relation between network structure, content and election results by creating a proof-of-concept model that predicts candidate victory with an accuracy of 88.0%.

Collaborer avec l'entourage pour mieux implémenter une CAA

Communication partner instruction of indi viduals using augmentative and alternative communication (AAC) is a huge challenge for speech and language therapists. The purpose of this article was to identify the active ingredients of AAC communication partner interventions based on available data. In a first step, we identified targets and implementation strategies that have been sh own as effective in parent implemented communication interventions with young child ren. In a second step, these data were supplemented by studies on the training of communication partners in the context of the use of AAC. Taken together, these studies point out that the most effective targets to improve the interaction style of partners consist in training their responsiveness and use of language models. Pragmatic outcomes (e.g., tu rn-taking), the access to spontaneous communication and a support for understanding have to be added for AAC interventions. To achieve this, the instructional methods employed with communication partners should include various combinations of implementation modalities (e.g., information sharing, modeling, guided practice, verbal rehearsal and feedback). Verbal rehearsal was shown to have a key role for the efficacy of the implementation. Finally, the communication partner's proficiency level with the AAC device should also be considered.

Making the most of MOGUL: Reflections on interlanguage in childhood language disorders

Abstract Interlanguage is a concept that is manifest in any trajectory of language change in a learner: typical first language, second language or language disorder. To understand those trajectories we need a rich psychological model of what creates them. This paper applies one such model–Sharwood Smith and Truscott’s Modular On-Line Growth and Use of Language model (MOGUL)–to childhood developmental language disorders, and suggests that the model’s components of language processing and their interaction shed significant light on why children with a wide range of different language disorders exhibit the language behaviours that are the characteristic of their diagnoses. Following a brief summary of the MOGUL model, the paper explores the impact on language development of differences in sensori-motor input, in the functioning of the various modules and the interfaces between them, and in the storage and activation of memory. Like Relevance Theory, with which the paper makes a direct connection, MOGUL encourages one to take a view of developmental language disorder as emerging from the same set of psychological resources as typical development (first or second) but as being the result of adjustments to, or compensations for, differences in how the various modules function and connect with each other.

A Language Modeling Approach to Information Retrieval

In today's world, there is no shortage of information. However, for a specific information need, only a small subset of all of the available information will be useful. The field of information retrieval (IR) is the study of methods to provide users with that small subset of information relevant to their needs and to do so in a timely fashion. Information sources can take many forms, but this thesis will focus on text based information systems and investigate problems germane to the retrieval of written natural language documents. Central to these problems is the notion of topic. In other words, what are documents about? However, topics depend on the semantics of documents and retrieval systems are not endowed with knowledge of the semantics of natural language. The approach taken in this thesis will be to make use of probabilistic language models to investigate text based information retrieval and related problems. One such problem is the prediction of topic shifts in text, the topic segmentation problem. It will be shown that probabilistic methods can be used to predict topic changes in the context of the task of new event detection. Two complementary sets of features are studied individually and then combined into a single language model. The language modeling approach allows this problem to be approached in a principled way without complex semantic modeling. Next, the problem of document retrieval in response to a user query will be investigated. Models of document indexing and document retrieval have been extensively studied over the past three decades. The integration of these two classes of models has been the goal of several researchers but it is a very difficult problem. Much of the reason for this is that the indexing component requires inferences as to the semantics of documents. Instead, an approach to retrieval based on probabilistic language modeling will be presented. Models are estimated for each document individually. The approach to modeling is non-parametric and integrates the entire retrieval process into a single model. One advantage of this approach is that collection statistics, which are used heuristically for the assignment of concept probabilities in other probabilistic models, are used directly in the estimation of language model probabilities in this approach. The language modeling approach has been implemented and tested empirically and performs very well on standard test collections and query sets. In order to improve retrieval effectiveness, IR systems use additional techniques such as relevance feedback, unsupervised query expansion and structured queries. These and other techniques are discussed in terms of the language modeling approach and empirical results are given for several of the techniques developed. These results provide further proof of concept for the use of language models for retrieval tasks.

HMM word graph based keyword spotting in handwritten document images

Line-level keyword spotting (KWS) is presented on the basis of frame-level word posterior probabilities. These posteriors are obtained using word graphs derived from the recognition process of a full-fledged handwritten text recognizer based on hidden Markov models and N-gram language models. This approach has several advantages. First, since it uses a holistic, segmentation-free technology, it does not require any kind of word or character segmentation. Second, the use of language models allows the context of each spotted word to be taken into account, thereby considerably increasing KWS accuracy. And third, the proposed KWS scores are based on true posterior probabilities, taking into account all (or most) possible word segmentations of the input image. These scores are properly bounded and normalized. This mathematically clean formulation lends itself to smooth, threshold-based keyword queries which, in turn, permit comfortable trade-offs between search precision and recall. Experiments are carried out on several historic collections of handwritten text images, as well as a well-known data set of modern English handwritten text. According to the empirical results, the proposed approach achieves KWS results comparable to those obtained with the recently-introduced “BLSTM neural networks KWS” approach and clearly outperform the popular, state-of-the-art “Filler HMM” KWS method. Overall, the results clearly support all the above-claimed advantages of the proposed approach.

Условное наклонение в “простой мове” Евангелия В. Тяпинского около 1580 г.

This study traces the evolution of conditionals in the Ruthenian literary language basing on V. Tjapinski’s Gospel, printed around the year of 1580 (further on TG). The Ruthenian language evolves from a living language and changes together with it. Here I dwell upon the variation of the conditional. The article represents the reconstructed paradigms with different variants of the use of conditionals in TG. A table shows the use of language models, analogous to Polish, in the Ruthenian language. The verb forms that emerged under the Polish influence are rare in TG and the most distant from the Old Russian language.

Design of language models at various phases of Tamil speech recognition system

This paper describes the use of language models in various phases of Tamil speech recognition system for improving its performance. In this work, the language models are applied at various levels of speech recognition such as segmentation phase, recognition phase and the syllable and word level error correction phase. The speech signals were segmented at phonetic level based on their acoustic characteristics. The wrongly identified segmentation points were detected and corrected using articulatory feature based phoneme language model. The segmented signals were mapped to their phonemes. The ambiguities in the recognized phonemes were reduced by using inter and intra word based language models. The recognized phonemes were grouped together to form syllables and then words. The errors in the syllables and words were detected and corrected by using the syllable and morpheme based language models developed for Tamil language. The performance of the Tamil speech recognition system was improved by using the language models at different phases of speech recognition. Recognition rate of 74.11% was obtained by applying language models at segmentation phase, which was further improved to 84.11% at phoneme recognition phase and finally to 87.1% at syllable level and word level recognition phase. Thus the use of language models has drastically reduced the error rates at various levels and improved the recognition rate of Tamil speech recognition system. Keywords: Language model, articulatory features, morphemes, syllables.

Language Modeling Approaches to Information Retrieval

This article surveys recent research in the area of language modeling (sometimes called statistical language modeling) approaches to information retrieval. Language modeling is a formal probabilistic retrieval framework with roots in speech recognition and natural language processing. The underlying assumption of language modeling is that human language generation is a random process; the goal is to model that process via a generative statistical model. In this article, we discuss current research in the application of language modeling to information retrieval, the role of semantics in the language modeling framework, cluster-based language models, use of language modeling for XML retrieval and future trends.

Information Science in the web era: A term‐based approach to domain mapping

AbstractWe propose a methodology for mapping the research in Information Science (IS) field based on a combined use of symbolic (linguistic) and numeric information. Using the same list of 12 IS journals as in earlier studies on this same topic (White & McCain 1998; Zhao & Strotmann 2008a&b), we mapped the structure of research in IS for two consecutive periods: 1996‐2005 and 2006‐2008. We focused on mapping the content of scientific publications from the title and abstract fields of underlying publications. The labels of clusters were automatically derived from titles and abstracts of scientific publications based on linguistic criteria. The results showed that while Information Retrieval (IR) and Citation studies continued to be the two structuring poles of research in IS, other prominent poles have emerged: webometrics in the first period (1996‐2005) evolved into general web studies in the second period, integrating more aspects of IR research. Hence web studies and IR are more interwoven. There is still persistence of user studies in IS but now dispersed among the web studies and the IR poles. The presence of some recent trends in IR research such as automatic summarization and the use of language models were also highlighted by our method. Theoretic research on “information science” continue to occupy a smaller but persistence place. Citation studies on the other hand remains a monolithic block, isolated from the two other poles (IR and web studies) save for a tenuous link through user studies. Citation studies have also recently evolved internally to accommodate newcomers like “h‐index, Google scholar and the open access model”. All these results were automatically generated by our method without resorting to manual labeling of specialties nor reading the publication titles. Our results show that mapping domain knowledge structures at the term level offers a more detailed and intuitive picture of the field as well as capturing emerging trends.

Dependency Parsing with Lattice Structures for Resource-Poor Languages

In this paper, we present a new dependency parsing method for languages which have very small annotated corpus and for which methods of segmentation and morphological analysis producing a unique (automatically disambiguated) result are very unreliable. Our method works on a morphosyntactic lattice factorizing all possible segmentation and part-of-speech tagging results. The quality of the input to syntactic analysis is hence much better than that of an unreliable unique sequence of lemmatized and tagged words. We propose an adaptation of Eisner's algorithm for finding the k-best dependency trees in a morphosyntactic lattice structure encoding multiple results of morphosyntactic analysis. Moreover, we present how to use Dependency Insertion Grammar in order to adjust the scores and filter out invalid trees, the use of language model to rescore the parse trees and the k-best extension of our parsing model. The highest parsing accuracy reported in this paper is 74.32% which represents a 6.31% improvement compared to the model taking the input from the unreliable morphosyntactic analysis tools.

Offline recognition of unconstrained handwritten texts using HMMs and statistical language models

This paper presents a system for the offline recognition of large vocabulary unconstrained handwritten texts. The only assumption made about the data is that it is written in English. This allows the application of Statistical Language Models in order to improve the performance of our system. Several experiments have been performed using both single and multiple writer data. Lexica of variable size (from 10,000 to 50,000 words) have been used. The use of language models is shown to improve the accuracy of the system (when the lexicon contains 50,000 words, the error rate is reduced by approximately 50 percent for single writer data and by approximately 25 percent for multiple writer data). Our approach is described in detail and compared with other methods presented in the literature to deal with the same problem. An experimental setup to correctly deal with unconstrained text recognition is proposed.

An investigation of continuous speech recognition performance from unconstrained speech utterances

This paper is concerned with small vocabulary speech recognition from conversational utterances over the telephone network. Modeling techniques are investigated for dealing with large numbers of nonvocabulary words and artifacts that arise in these utterances. A hidden Markov model (HMM)-based continuous speech recognition system using a frame synchronous Viterbi beam search decoder is used for recognition. Keyword models compete in the finite state network with ‘‘filler’’ models of nonkeyword speech. Several issues were investigated relating to the quality of acoustic representations and language representations for this task. The first issue that was investigated was the definition of acoustic subword units using allophone clustering procedures. The second issue was the size of the vocabulary used for modeling non-keyword utterances. Finally, the last issue was the use of language models in unconstrained speech tasks. Experimental results will be presented for a 20-keyword recognition task where performance was evaluated on continuous utterances from 22 speakers. The results showed that all of the procedures including decision tree based allophone clustering, better out-of-vocabulary speech representations, and language models contributed to overall recognition performance. The best performing system provided 76% average probability of keyword detection at 5.8 false alarms per keyword per hour.

Acquisition of language models from text

Statistical language models that assign a probability of being generated to each possible string of words are of interest here. Grammatical language models may, moreover, assign probabilities to parses of the given string of words. Language models are used as components of recognition systems, speech and handwriting being current examples of interest. Another use of language models is in text understanding systems and machine translation systems, where grammatical information is very desirable. Because of the richness of language, language models are usually constructed for particular discourse domains, such as medical reports, office correspondence, legal documents, etc. The construction is on the basis of large amounts of sample text, and should be as automatic as possible. For grammatical language models, it is desirable to estimate not only their statistical parameters from the supplied text, but even the grammatical rules themselves. The talk will discuss various methods of language model construction, their relative performance characteristics, and some of the many open problems.