Neural Network Based Language Models Research Articles

A Comparative Study of Khasi Speech Recognition Systems with Recurrent Neural Network-Based Language Model

This paper offers a comparative analysis of Khasi speech recognition systems utilizing a recurrent neural network-based language model (RNN-LM). Develop different acoustic models (AMs) to evaluate the optimal performance. This paper observed that using RNN-LM performed best than traditional other models. The wave surfer performs data processing followed by collecting the recorder based continuous speech database. Moreover, a minimization of word error rate (WER) in 2.83.8% range for major speech data and 2.4-3.5% for minor speech data. Additionally, two acoustic features are used, and from the experimental results, the Mel frequency cepstral coefficient (MFCC) yielded improved performance than the perceptual linear prediction (PLP).

Grammatical versus Spelling Error Correction: An Investigation into the Responsiveness of Transformer-Based Language Models Using BART and MarianMT

Text continues to remain a relevant form of representation for information. Text documents are created either in digital native platforms or through the conversion of other media files such as images and speech. While the digital native text is invariably obtained through physical or virtual keyboards, technologies such as OCR and speech recognition are utilised to transform the images and speech signals into text content. All these variety of mechanisms of text generation also introduce errors into the captured text. This project aims at analysing different kinds of errors that occur in text documents. The work employs two of the advanced deep neural network-based language models, namely, BART and MarianMT, to rectify the anomalies present in the text. Transfer learning of these models with available dataset is performed to finetune their capacity for error correction. A comparative study is conducted to investigate the effectiveness of these models in handling each of the defined error categories. It is observed that while both models can bring down the erroneous sentences by 20+%, BART can handle spelling errors far better (24.6%) than grammatical errors (8.8%).

Exploring Social Biases of Large Language Models in a College Artificial Intelligence Course

Large neural network-based language models play an increasingly important role in contemporary AI. Although these models demonstrate sophisticated text generation capabilities, they have also been shown to reproduce harmful social biases contained in their training data. This paper presents a project that guides students through an exploration of social biases in large language models. As a final project for an intermediate college course in Artificial Intelligence, students developed a bias probe task for a previously-unstudied aspect of sociolinguistic or sociocultural bias they were interested in exploring. Through the process of constructing a dataset and evaluation metric to measure bias, students mastered key technical concepts, including how to run contemporary neural networks for natural language processing tasks; construct datasets and evaluation metrics; and analyze experimental results. Students reported their findings in an in-class presentation and a final report, recounting patterns of predictions that surprised, unsettled, and sparked interest in advocating for technology that reflects a more diverse set of backgrounds and experiences. Through this project, students engage with and even contribute to a growing body of scholarly work on social biases in large language models.

Methodical Systematic Review of Abstractive Summarization and Natural Language Processing Models for Biomedical Health Informatics: Approaches, Metrics and Challenges

Text summarization tasks are primarily very useful for decision support systems and provide a source for useful data for training of bots as they can reduce and retain the useful information from the large corpus. This review article is for studying the literature that already exists in context of abstractive summarization and application of NLP language models in biomedical and associated healthcare applications. In past decade with trends like bigdata, IOT, enormous amount of data is getting processed in all structured, unstructured and semi structured formats. This review provides a comprehensive literature survey in research trends for abstractive summarization, foundations of machine translation and evolution of language models. This review identifies the potential of language model to provide a possible methodology for improving the performance and accuracy of various tasks in summarization. Deep neural network-based language models have now been the widely accepted state of art for various abstractive summarization and there exists an enormous scope to improvise and tune the language models for domain specific use case. This study shows current systems lack in faithfulness to original content and control of degree of hallucination. This review also details on the evaluation criteria and need for automated metrics and attempts to provide guideline for evaluation for abstractive summarization for health informatics.

Developing a Construction Domain–Specific Artificial Intelligence Language Model for NCDOT’s CLEAR Program to Promote Organizational Innovation and Institutional Knowledge

Transportation agency personnel gain valuable knowledge through their work, but such knowledge is lost if it is not documented properly after the worker leaves the organization. The risk of losing institutional knowledge is a current problem at state departments of transportation, including the North Carolina Department of Transportation (NCDOT), due to high personnel turnover. State transportation agencies have implemented knowledge repositories in the form of lessons learned/best practices databases to address this problem. However, motivating end-users to use such databases is challenging. This paper addresses this challenge through novel artificial intelligence technology whereby a neural network–based language model is implemented as part of the NCDOT’s new knowledge management program: Communicate Lessons, Exchange Advice, Record (CLEAR). The CLEAR program encompasses a database of lessons learned/best practices and a website to access and search the database. The developed methodology involves training a language model on transportation construction texts and using that trained model in a novel algorithm enabling users to search the CLEAR database easily. The developed language-processing model provides an easily accessible interface to suggest the most relevant CLEAR data based on the end-user’s searched keywords. The model learns an inference model of construction domain–specific vocabulary extracted from various sources, such as contract documents, textbooks, and specifications, to make meaningful connections between lessons learned/best practices in the CLEAR database and project-specific knowledge. The developed model has been validated by project managers for projects at various life cycle stages. The automation of information retrieval is intended to encourage NCDOT personnel to use and embrace the CLEAR program as part of their routine work to improve project workflow. In the long run, the NCDOT will benefit from consistent usage of the CLEAR program and its high quality content, thereby leading to enhanced institutional knowledge and organizational innovation.

Demystifying GPT and GPT-3: How they can support innovators to develop new digital accessibility solutions and assistive technologies?

GPT (Generative Pre-trained Transformer) is a neural network-based language model developed by OpenAI that has demonstrated impressive capabilities in generating human-like text and performing a wide range of natural language processing (NLP) tasks. GPT-3, the latest version of the model, is currently the largest and most advanced language model available, with 175 billion parameters. GPT and GPT-3 have the potential to support the development of digital accessibility solutions and assistive technologies, including text-to-speech synthesis, language translation, text summarization, and intelligent virtual assistants. In addition to its capabilities as a language model, GPT-3 has also been used as a tool for generating synthetic data and training other machine learning models. Some possible future directions for GPT include increased scale and performance, greater flexibility and adaptability, improved capabilities for unsupervised learning, and integration into more applications and industries.

Do Semantic Vectors Contain Traces of Biophilic Connections Between Nature and Mental Health?

Natural environments have positive effects on mental health, but the nature of this relationship requires further understanding. There may be psychosocial aspects of this relationship that are reflected in the semantic structure of language. Natural language processing methods provide a tool to explore this possibility. In this study, machine learning-derived vector representations of words were provided by a neural network-based language model. This was combined with statistical analyses to test whether nature-related words have particularly strong positive versus negative connections to mental health. Statistically significant associations were indeed found between a range of nature-related words and positive-versus-negative word pairs related to mental health. The results thus confirm a semantic connection between nature and mental health as represented using computational methods trained on language usage. This raises the possibility that semantic associations could play a role in nature's influence on mental health, for instance through appraisal processes. The results provide a proof of principle of a methodological approach that could be used to further probe hypotheses on nature and well-being. Finally, the current results provide information on relationships between specific aspects of nature environments and mental health that may be of use to future research.

Anonymization of German financial documents using neural network-based language models with contextual word representations

The automatization and digitalization of business processes have led to an increase in the need for efficient information extraction from business documents. However, financial and legal documents are often not utilized effectively by text processing or machine learning systems, partly due to the presence of sensitive information in these documents, which restrict their usage beyond authorized parties and purposes. To overcome this limitation, we develop an anonymization method for German financial and legal documents using state-of-the-art natural language processing methods based on recurrent neural nets and transformer architectures. We present a web-based application to anonymize financial documents and a large-scale evaluation of different deep learning techniques.

Sentiment Analysis with Cognitive Attention Supervision

Neural network-based language models such as BERT (Bidirectional Encoder Representations from Transformers) use attention mechanisms to create contextualized representations of inputs, conceptually analogous to humans reading words in context. For the task of classifying the sentiment of texts, we ask whether BERT’s attention can be informed by human cognitive data. During training, we supervise attention with eye-tracking and/or brain imaging data and combine binary sentiment classification loss with these attention losses. We find that attention supervision can be used to manipulate BERT attention to be more similar to the ground truth human data, but that there are no significant differences in sentiment classification accuracy. However, models with cognitive attention supervision more frequently misclassify different samples from the baseline models–they more often make different errors–and the errors from models with supervised attention have a higher ratio of false negatives.

English

English

Neural candidate-aware language models for speech recognition

This paper presents novel neural network based language models that can correct automatic speech recognition (ASR) errors by using speech recognizer outputs as a context. Our proposed models, called neural candidate-aware language models (NCALMs), estimate the generative probability of a target sentence while considering ASR outputs including hypotheses and their posterior probabilities. Recently, neural network language models have achieved great success in ASR field because of their ability to learn long-range contexts and model the word representation in continuous space. However, they estimate a sentence probability without considering other candidates and their posterior probabilities, even though the competing hypotheses are available and include important information to increase the speech recognition accuracy. To overcome this limitation, our idea is to utilize ASR outputs in both the training phase and the inference phase. Our proposed models are conditional generative models consisting of a Transformer encoder and a Transformer decoder. The encoder embeds the candidates as context vectors and the decoder estimates a sentence probability given the context vectors. We evaluate the proposed models in Japanese lecture transcription and English conversational speech recognition tasks. Experimental results show that a NCALM has better ASR performance than a system including a deep neural network-hidden Markov model hybrid system. We further improve ASR performance by using a NCALM and a Transformer language model simultaneously.

HLHLp: Quantized Neural Networks Training for Reaching Flat Minima in Loss Surface

Quantization of deep neural networks is extremely essential for efficient implementations. Low-precision networks are typically designed to represent original floating-point counterparts with high fidelity, and several elaborate quantization algorithms have been developed. We propose a novel training scheme for quantized neural networks to reach flat minima in the loss surface with the aid of quantization noise. The proposed training scheme employs high-low-high-low precision in an alternating manner for network training. The learning rate is also abruptly changed at each stage for coarse- or fine-tuning. With the proposed training technique, we show quite good performance improvements for convolutional neural networks when compared to the previous fine-tuning based quantization scheme. We achieve the state-of-the-art results for recurrent neural network based language modeling with 2-bit weight and activation.

Analysis of Neural Network Based Language Modeling

The fundamental and core process of the natural language processing is the language modelling usually referred as the statistical language modelling. The language modelling is also considered to be vital in the processing the natural languages as the other chores such as the completion of sentences, recognition of speech automatically, translations of the statistical machines, and generation of text and so on. The success of the viable natural language processing totally relies on the quality of the modelling of the language. In the previous spans the research field such as the linguistics, psychology, speech recognition, data compression, neuroscience, machine translation etc. As the neural network are the very good choices for having a quality language modelling the paper presents the analysis of neural networks in the modelling of the language. Utilizing some of the dataset such as the Penn Tree bank, Billion Word Benchmark and the Wiki Test the neural network models are evaluated on the basis of the word error rate, perplexity and the bilingual evaluation under study scores to identify the optimal model.

Analysis of Neural Network Based Language Modeling

The fundamental and core process of the natural language processing is the language modelling usually referred as the statistical language modelling. The language modelling is also considered to be vital in the processing the natural languages as the other chores such as the completion of sentences, recognition of speech automatically, translations of the statistical machines, and generation of text and so on. The success of the viable natural language processing totally relies on the quality of the modelling of the language. In the previous spans the research field such as the linguistics, psychology, speech recognition, data compression, neuroscience, machine translation etc. As the neural network are the very good choices for having a quality language modelling the paper presents the analysis of neural networks in the modelling of the language. Utilizing some of the dataset such as the Penn Tree bank, Billion Word Benchmark and the Wiki Test the neural network models are evaluated on the basis of the word error rate, perplexity and the bilingual evaluation under study scores to identify the optimal model.

Using Morphological Data in Language Modeling for Serbian Large Vocabulary Speech Recognition.

Serbian is in a group of highly inflective and morphologically rich languages that use a lot of different word suffixes to express different grammatical, syntactic, or semantic features. This kind of behaviour usually produces a lot of recognition errors, especially in large vocabulary systems—even when, due to good acoustical matching, the correct lemma is predicted by the automatic speech recognition system, often a wrong word ending occurs, which is nevertheless counted as an error. This effect is larger for contexts not present in the language model training corpus. In this manuscript, an approach which takes into account different morphological categories of words for language modeling is examined, and the benefits in terms of word error rates and perplexities are presented. These categories include word type, word case, grammatical number, and gender, and they were all assigned to words in the system vocabulary, where applicable. These additional word features helped to produce significant improvements in relation to the baseline system, both for n-gram-based and neural network-based language models. The proposed system can help overcome a lot of tedious errors in a large vocabulary system, for example, for dictation, both for Serbian and for other languages with similar characteristics.

User identification via neural network based language models

SummaryIdentifying compromised accounts on online social networks that are used for phishing attacks or sending spam messages is still one of the most challenging problems of cyber security. In this research, the authors explore an artificial neural network‐based language model to differentiate the writing styles of different users on short text messages. In doing so, the aim is to be able to identify compromised user accounts. The results obtained indicate that one can learn the language model on one dataset and can generalize it to different datasets to identify users with high accuracy and low false alarm rates without any modification to the language model.

Enhancing recurrent neural network-based language models by word tokenization

Different approaches have been used to estimate language models from a given corpus. Recently, researchers have used different neural network architectures to estimate the language models from a given corpus using unsupervised learning neural networks capabilities. Generally, neural networks have demonstrated success compared to conventional n-gram language models. With languages that have a rich morphological system and a huge number of vocabulary words, the major trade-off with neural network language models is the size of the network. This paper presents a recurrent neural network language model based on the tokenization of words into three parts: the prefix, the stem, and the suffix. The proposed model is tested with the English AMI speech recognition dataset and outperforms the baseline n-gram model, the basic recurrent neural network language models (RNNLM) and the GPU-based recurrent neural network language models (CUED-RNNLM) in perplexity and word error rate. The automatic spelling correction accuracy was enhanced by approximately 3.5% for Arabic language misspelling mistakes dataset.

An End-to-End Deep Learning Approach to Simultaneous Speech Dereverberation and Acoustic Modeling for Robust Speech Recognition

We propose an integrated end-to-end automatic speech recognition (ASR) paradigm by joint learning of the front-end speech signal processing and back-end acoustic modeling. We believe that “only good signal processing can lead to top ASR performance” in challenging acoustic environments. This notion leads to a unified deep neural network (DNN) framework for distant speech processing that can achieve both high-quality enhanced speech and high-accuracy ASR simultaneously. Our goal is accomplished by two techniques, namely: (i) a reverberation-time-aware DNN based speech dereverberation architecture that can handle a wide range of reverberation times to enhance speech quality of reverberant and noisy speech, followed by (ii) DNN-based multicondition training that takes both clean-condition and multicondition speech into consideration, leveraging upon an exploitation of the data acquired and processed with multichannel microphone arrays, to improve ASR performance. The final end-to-end system is established by a joint optimization of the speech enhancement and recognition DNNs. The recent REverberant Voice Enhancement and Recognition Benchmark (REVERB) Challenge task is used as a test bed for evaluating our proposed framework. We first report on superior objective measures in enhanced speech to those listed in the 2014 REVERB Challenge Workshop on the simulated data test set. Moreover, we obtain the best single-system word error rate (WER) of 13.28% on the 1-channel REVERB simulated data with the proposed DNN-based pre-processing algorithm and clean-condition training. Leveraging upon joint training with more discriminative ASR features and improved neural network based language models, a low single-system WER of 4.46% is attained. Next, a new multi-channel-condition joint learning and testing scheme delivers a state-of-the-art WER of 3.76% on the 8-channel simulated data with a single ASR system. Finally, we also report on a preliminary yet promising experimentation with the REVERB real test data.

Automatic estimation of extra-linguistic information in speech and its integration into recurrent neural network-based language models for speech recognition

When one talks with others, he/she often changes the way of lexical choice and the speaking style depending on various contextual factors. For example in Japanese, we often use gender-dependent expressions and, when we speak to elderly people, we use polite expressions. It can be said that in any language, formality or informality of speaking changes drastically depending on the situation where speakers are involved. The aim of automatic speech recognition (ASR) is to convert speech signals into a sequence of words and the above fact indicates that the performance of ASR can be improved by taking the various contextual factors into account in the ASR modules. In this study, at first, we attempt to estimate those contextual factors as para-linguistic or extra-linguistic information and then, to integrate the results into language models based on Recurrent Neural Network (RNN) for speech recognition. In experiments, from an input utterance, i-vector and openSMILE features were extracted to represent speaker identity and speaking style. These acoustically driven features were integrated into the reranking process of RNN-based language models. Reductions of perplexity of the language models were shown to be 1 to 2% relative.

워드 임베딩과 품사 태깅을 이용한 클래스 언어모델 연구

Recurrent neural network based language models (RNN LM) have shown improved results in language model researches. The RNN LMs are limited to post processing sessions, such as the N-best rescoring step of the wFST based speech recognition. However, it has considerable vocabulary problems that require large computing powers for the LM training. In this paper, we try to find the 1st pass N-gram model using word embedding, which is the simplified deep neural network. The class based language model (LM) can be a way to approach to this issue. We have built class based vocabulary through word embedding, by combining the class LM with word N-gram LM to evaluate the performance of LMs. In addition, we propose that part-of-speech (POS) tagging based LM shows an improvement of perplexity in all types of the LM tests.