Word Sequences Research Articles

What should be encoded by position embedding for neural network language models?

AbstractWord order is one of the most important grammatical devices and the basis for language understanding. However, as one of the most popular NLP architectures, Transformer does not explicitly encode word order. A solution to this problem is to incorporate position information by means of position encoding/embedding (PE). Although a variety of methods of incorporating position information have been proposed, the NLP community is still in want of detailed statistical researches on position information in real-life language. In order to understand the influence of position information on the correlation between words in more detail, we investigated the factors that affect the frequency of words and word sequences in large corpora. Our results show that absolute position, relative position, being at one of the two ends of a sentence and sentence length all significantly affect the frequency of words and word sequences. Besides, we observed that the frequency distribution of word sequences over relative position carries valuable grammatical information. Our study suggests that in order to accurately capture word–word correlations, it is not enough to focus merely on absolute and relative position. Transformers should have access to more types of position-related information which may require improvements to the current architecture.

Semantic reconstruction of continuous language from non-invasive brain recordings.

A brain-computer interface that decodes continuous language from non-invasive recordings would have many scientific and practical applications. Currently, however, non-invasive language decoders can only identify stimuli from among a small set of words or phrases. Here we introduce a non-invasive decoder that reconstructs continuous language from cortical semantic representations recorded using functional magnetic resonance imaging (fMRI). Given novel brain recordings, this decoder generates intelligible word sequences that recover the meaning of perceived speech, imagined speech and even silent videos, demonstrating that a single decoder can be applied to a range of tasks. We tested the decoder across cortex and found that continuous language can be separately decoded from multiple regions. As brain-computer interfaces should respect mental privacy, we tested whether successful decoding requires subject cooperation and found that subject cooperation is required both to train and to apply the decoder. Our findings demonstrate the viability of non-invasive language brain-computer interfaces.

Generative models for sequential dynamics in active inference

AbstractA central theme of theoretical neurobiology is that most of our cognitive operations require processing of discrete sequences of items. This processing in turn emerges from continuous neuronal dynamics. Notable examples are sequences of words during linguistic communication or sequences of locations during navigation. In this perspective, we address the problem of sequential brain processing from the perspective of active inference, which inherits from a Helmholtzian view of the predictive (Bayesian) brain. Underneath the active inference lies a generative model; namely, a probabilistic description of how (observable) consequences are generated by (unobservable) causes. We show that one can account for many aspects of sequential brain processing by assuming the brain entails a generative model of the sensed world that comprises central pattern generators, narratives, or well-defined sequences. We provide examples in the domains of motor control (e.g., handwriting), perception (e.g., birdsong recognition) through to planning and understanding (e.g., language). The solutions to these problems include the use of sequences of attracting points to direct complex movements—and the move from continuous representations of auditory speech signals to the discrete words that generate those signals.

Domain Adaptation Speech-to-Text for Low-Resource European Portuguese Using Deep Learning

Automatic speech recognition (ASR), commonly known as speech-to-text, is the process of transcribing audio recordings into text, i.e., transforming speech into the respective sequence of words. This paper presents a deep learning ASR system optimization and evaluation for the European Portuguese language. We present a pipeline composed of several stages for data acquisition, analysis, pre-processing, model creation, and evaluation. A transfer learning approach is proposed considering an English language-optimized model as starting point; a target composed of European Portuguese; and the contribution to the transfer process by a source from a different domain consisting of a multiple-variant Portuguese language dataset, essentially composed of Brazilian Portuguese. A domain adaptation was investigated between European Portuguese and mixed (mostly Brazilian) Portuguese. The proposed optimization evaluation used the NVIDIA NeMo framework implementing the QuartzNet15×5 architecture based on 1D time-channel separable convolutions. Following this transfer learning data-centric approach, the model was optimized, achieving a state-of-the-art word error rate (WER) of 0.0503.

GINGER WRITER AS AN ONLINE REPHRASE TOOL WITH AI-POWERED SUGGESTIONS OF ALTERNATIVE SENTENCES IN ENGLISH WRITING

This is a descriptive-qualitative study analyzes the use of Ginger Writer as an online rephrase tool based on AI-powered suggestions in English writing. The result of analysis shows that the free edition of the Ginger Writer application is used for the implementation. Some techniques of rephrasing applied b the tools are: (1) rephrasing by using equations/synonyms of words. (2) rephrasing by changing/rearranging the sequence of words based on the relevant rules. (3) rephrasing by altering the word's form, (4) rephrasing by using the active or passive form, (5) rephrasing by changing the word class, and (6) rephrasing word into phrase, phrase into clause, and vice versa. Ginger Writer makes writing high-quality work effortless for users. Not only adds missing words, completes fragmented phrases, and corrects spelling problems related to the context of sentences, Ginger Writer also adds missing words, completes fragmented sentences, and corrects spelling errors. This online rephrasing tool is considerably fast but teachers and students should double-check their work. To some extent, manual paraphrasing is considered better and, even the best in quality, compared to the automatic ones.

Word Frequencies in Linguistic Articles Published in SINTA Indexed Journals

Multiword sequences are a language pattern that occurs when a bunch of words emerge in a similar register. In research papers conducted by lecturers and students, different topic areas and indexes has created various characteristics of lexical bundles. The method of this research is qualitative, combining corpus design to identify the sequence of words within the text. The corpus data were generated from five different indexing journals, yet the topic is linguistics. Initially, the whole papers were converted to text format to deal with readability in the program used. The program used was Orange Apps version 3.27 by applying the textable, data table, and text mining menus. The sources of the data are emphasized as being academic research indexed in SINTA 5, published in 2020. The main theory of used in this research is that of Biber’s (2007) which discusses the main characteristics and number of criteria for defining word strings. This observation resulted in 207.896 characters and 33.636 words. There were 4,273 words based on the pre-processing analysis result, which included transformation, tokenization, and PoS-tagging. From a total of 4,273 words, virus, deixis, and slang were the most frequently occurring. Based on these results, it can be concluded that the majority of journal articles are about viruses and slang. They pertain to the prevalent topic of pandemics at the time the journals were published. When the process of writing a journal article is in progress, this information may aid the authors in identifying the journal’s keywords and most frequent words.

A Novel Neural Machine Translation Approach for low-resource Sanskrit-Hindi Language pair

Sanskrit is one of the earliest native languages and is correctly described as "the gods' language" because of its wide use in Indian religious literature from the past. However, it is becoming less popular in modern India. Due in significant part to the need for more materials for translation both in and out of Sanskrit, it is no longer commonly utilized. This study explores the feasibility of using machine translation (MT) to provide a link between Sanskrit and, one of the earliest native languages, and its contemporary descendant Hindi. A study was conducted between existing modelling methodologies, notably Statistical machine translation (SMT), and the proposed novel deep learning-based Machine translation strategy using a manually created parallel corpus for the Sanskrit-Hindi language pair. While SMT creates interpretations by mapping phrases from the languages of the source and destination, statistical models, and bilingual text corpora for learning parameters, neural machine translation (NMT) frequently models entire phrases in a single integrated model, using a convolutional neural network to calculate the probability of a word sequence. The proposed NMT model is implemented using an encoder-decoder with an attention mechanism paradigm and the inclusion of gated recurrent units. Our approach involved development of a novel model for Sanskrit-Hindi machine translation using deep learning and the creation of parallel corpora for the Sanskrit-Hindi language pair. The proposed model is evaluated on automated and human-based metrics, and results show that our proposed deep learning-based model outperforms statistical modelling techniques on Moses, surpassing them both with a BLEU score of 53.8% compared to 34.56%. This article examines the undiscovered area of machine translation from Sanskrit to Hindi and discusses the main benefits and drawbacks of statistical and neural machine translation while providing a fresh viewpoint on the subject.

Facial Expression Recognition With Visual Transformers and Attentional Selective Fusion

Facial Expression Recognition (FER) in the wild is extremely challenging due to occlusions, variant head poses, face deformation and motion blur under unconstrained conditions. Although substantial progresses have been made in automatic FER in the past few decades, previous studies were mainly designed for lab-controlled FER. Real-world occlusions, variant head poses and other issues definitely increase the difficulty of FER on account of these information-deficient regions and complex backgrounds. Different from previous pure CNNs based methods, we argue that it is feasible and practical to translate facial images into sequences of visual words and perform expression recognition from a global perspective. Therefore, we propose the Visual Transformers with Feature Fusion (VTFF) to tackle FER in the wild by two main steps. First, we propose the attentional selective fusion (ASF) for leveraging two kinds of feature maps generated by two-branch CNNs. The ASF captures discriminative information by fusing multiple features with the global-local attention. The fused feature maps are then flattened and projected into sequences of visual words. Second, inspired by the success of Transformers in natural language processing, we propose to model relationships between these visual words with the global self-attention. The proposed method is evaluated on three public in-the-wild facial expression datasets (RAF-DB, FERPlus and AffectNet). Under the same settings, extensive experiments demonstrate that our method shows superior performance over other methods, setting new state of the art on RAF-DB with 88.14%, FERPlus with 88.81% and AffectNet with 61.85%. The cross-dataset evaluation on CK+ shows the promising generalization capability of the proposed method.

Actualization of the German Language Status in the Absence of the State Language Category in the German Legislation

This article addresses the issue of the representation of the German language status in legislative texts of the Federal Republic of Germany in the absence of the category «state language». The research methodology is based on the E. S. Kubryakova's understanding of the nominative aspect of the speech activity, who distinguishes two ways of designating objects: both through nomination and predication. The aim of this paper is to determine main functions of linguistic expressions, which regulate the role of the German language in legislation (nomination and / or predication), and language means of their representation. Research objectives: 1) to make a selection of fragments of normative legal acts on the federal level related to language legislation; 2) to select the nominations and / or predications used to actualize the leading role of the German language; 3) to study the structure of selected nominations and / or predications. The material of this paper is presented by excerpts from the Fiscal code, Authentication law, Court constitution law, Law on the protection of brands and other marks, Law on alternative dispute resolution in consumer matters, Law on the residence, employment and integration of foreigners in Germany, Law on the supervision of insurance companies, Trading law, Disability equality act, Commercial Code. In the theoretical part of the work, definitions of the categories «state language», «official language», «nomination» and «predication» are presented. The practical part of the paper addresses the analysis of excerpts from federal legislation. In conclusion, the results of the issue are presented: the leading role of the German language in the legislation of the Federal Republic of Germany is updated through nominations and predications; nominations are represented both by separate compound words and by sequences of words syntactically related to each other; predications are represented by semantic verbs in the form Präsens Indikativ (in conditional clauses) as well as several types of modal constructions.

Improving the Robustness of Loanword Identification in Social Media Texts

As a potential bilingual resource, loanwords play a very important role in many natural language processing tasks. If loanwords in a low-resource language can be identified effectively, the generated donor-receipt word pairs will benefit many cross-lingual natural language processing tasks. However, most studies on loanword identification mainly focus on formal texts such as news and government documents. Loanword identification in social media texts is still an under-studied field. Since it faces many challenges and can be widely used in several downstream tasks, more efforts should be put on loanword identification in social media texts. In this study, we present a multi-task learning architecture with deep bi-directional recurrent neural networks for loanword identification in social media texts, where different task supervision can happen at different layers. The multi-task neural network architecture learns higher-order feature representations from word and character sequences along with basic spell error checking, part-of-speech tagging, and named entity recognition information. Experimental results on Uyghur loanword identification in social media texts in five donor languages (Chinese, Arabic, Russian, Turkish, and Farsi) show that our method achieves the best performance compared with several strong baseline systems. We also combine the loanword detection results into the training data of neural machine translation for low-resource language pairs. Experiments show that models trained on the extended datasets achieve significant improvements compared with the baseline models in all language pairs.

A model for representing the semantics of MWEs: From lexical semantics to the semantic annotation of complex predicates.

Multiword expressions (MWEs) are sequences of words that pose a challenge to the computational processing of human languages due to their idiosyncrasies and the mismatch between their phrasal structure and their semantics. These idiosyncrasies are of lexical, morphosyntactic and semantic 11 nature, namely: non-compositionality, i.e., the meaning of the expression cannot be computed from the meanings of its constituents; discontinuity, i.e., alien elements may intervene; non-13 substitutability, i.e., at least one of the expression constituents is lexicalized and therefore, does not enter in alternations at the paradigmatic axis; and non-modifiability, in that they enter in syntactically 15 rigid structures, posing further constraints over modification, transformations, etc. The paper presents a model for representing MWEs at the level of semantics by taking into account all these inherent idiosyncrasies. The model assumes the form of a linguistic ontology and is applied to Greek verbal multi-word expressions (VMWEs); moreover, the semantics of the lexical entries under scrutiny is also represented via the semantics of their arguments based on corpus evidence. In this regard, modeling the semantics of VMWEs is placed in the lexicon-corpus interface.

Do People With Apraxia of Speech and Aphasia Improve or Worsen Across Repeated Sequential Word Trials?

During motor speech examinations for suspected apraxia of speech (AOS), clients are routinely asked to repeat words several times sequentially. The purpose of this study was to understand the task in terms of the relationship among consecutive attempts. We asked to what extent phonemic accuracy changes across trials and whether the change is predicted by AOS diagnosis and sound production severity. One hundred thirty-three participants were assigned to four diagnostic groups based on quantitative metrics (aphasia plus AOS, aphasia-only, and aphasia with two borderline speech profiles). Each participant produced four multisyllabic words 5 times consecutively. These productions were audio-recorded and transcribed phonetically and then summarized as the proportion of target phonemes that was produced accurately. Nonparametric statistics were used to analyze percent change in accuracy from the first to the last production based on diagnostic group and a broad measure of speech sound accuracy. Accuracy on the repeated words deteriorated across trials for all groups, showing reduced accuracy from the first to the last repetition for 62% of participants. Although diagnostic groups differed on the broad measure of speech sound accuracy, severity classification based on this measure did not determine degree of deterioration on the repeated words task. Responding to a request to say multisyllabic words 5 times sequentially is challenging for people with aphasia with and without AOS, and as such, performance is prone to errors even with mild impairment. For most, the task does not encourage self-correction. Instead, it promotes errors, regardless of diagnosis, and is, therefore, useful for screening purposes.

ParsingPhrase: Parsing-based automated quality phrase mining

Phrases represent independent semantics in natural language but usually have indeterminate lengths and different combinations. So, extracting meaningful phrases from unstructured texts will substantially reduce semantic ambiguity and lay the foundation for downstream natural language tasks. Most existing research obtains candidate phrases by N-grams, which includes meaningless word sequences and degrades algorithm performance. In this paper, we propose a novel phrase-mining algorithm, called ParsingPhrase, which effectively extracts combination phrases from text and improves phrase quality by syntactic features. It consists of three stages. Firstly, all sentences in texts are represented as parsing trees by PCFG (Probabilistic Context-Free Grammar). We propose PBMP (Parsing-Based Phrase mining) to obtain candidate phrases from those parsing trees. Then, we introduce a new phrase evaluation indicator, called Significance, that relies on the role of phrases to measure their importance. We integrate the Significance with conventional evaluation indexes for a more reasonable phrase evaluation. Finally, we optimize the phrase quality again by exploiting the optimal phrase composition features of sentences. To the best of our knowledge, it is the first work to employ parsing for combination phrase mining and evaluation, meanwhile offering a solution for syntactic disambiguation. Experiments on three real corpora demonstrate that the ParsingPhrase exceeds state-of-the-art baselines, is 7% higher in the candidate phrase conversion rate, and is 6% better in terms of Precision.

Effects of alternating letter case on processing sequences of written words.

In three grammatical decision experiments, we examined the impact of alternating letter case on sentence reading to determine the locus of case-alternation effects. Experiments 1 and 2 compared grammatical decision responses ("Is this a grammatically correct sequence of words or not?") in three different conditions: (1) SAME CASE/same case; (2) alternating CASE between WORDS; and (3) aLterNaTing cAsE wItHin WoRdS. For the grammatically correct sequences, we observed significantly faster responses in the same-case conditions compared with the between-word case manipulation, as well as a significant advantage for the between-word condition compared with within-word alternating case. These results confirm that case-alternation deteriorates sentence reading, but more so at the level of single word processing (within-word alternation) than at the sentence level (between-word alternation). Experiment 3 demonstrated that between-word case-alternation facilitates sentence processing compared with an all-lowercase condition when betweenWORDspacesAREremoved. Therefore, in the absence of between-word spacing, case changes across words facilitate sentence processing, possibly by guiding readers' eyes to optimal locations for word identification.

Effects of age and hearing loss on talker identification and talker change detection

While the detrimental effects of age and hearing loss on word recognition and speech comprehension are well-established, surprisingly little is known about how these factors affect the ability to extract talker-related information from speech. Here, we describe two studies that used different behavioral tasks to address this question. In the first study (Talker Identification), listeners were trained to identify talkers by the sound of their voice, and then tested on their ability to identify those talkers in quiet or in the presence of background noise or competing speech. In the second study (Talker Change Detection), explicit talker identification was not required, but instead listeners detected if a change in talker had occurred within a sequence of words. In both studies, listeners were recruited in four groups to include younger/older ages and normal/impaired hearing. In both studies, we found that age and hearing loss had independent effects on performance. We argue that a reduced ability to make use of talker-related information in speech may contribute to the difficulty encountered by older listeners and listeners with hearing loss, when communicating in group situations.

CNN-OLSTM: Convolutional Neural Network with Optimized Long Short-Term Memory Model for Twitter based Sentiment Analysis

Social Networking continues the growth of web users, with people sharing their ideas and opinions daily in the form of texts, pictures, videos, and speech. Text classification is still an important issue because these large texts were derived from diverse sources and different thinking people. The shared concept must be incomplete, random, noisy, and in the form of different languages. To solve this issue, in this paper, Convolutional Neural Network with Optimized Long Short Term Memory Model (CNN-OLSTM) based sentimental analysis is proposed. The presented approach contains pre-processing, word2vec conversion, and prediction. Initially, data are pre-processed by using tokenization, stop word removal, and stemming. After the pre-processing, the skip-gram model (SGM) based word2vec conversion is performed. Then, the extracted vectors are given to the CNN-OLSTM classifier to classify a tweet as positive or negative polarity. In this, the CNN model effectively reduces the dimension of the input vector using max-pooling layers and convolutional layers. Also, the LSTM model is capable of catching long-term dependencies between word sequences. To enhance the LSTM performance, the rain optimization algorithm (ROA) is effectively used. The results show that the suggested CNN-OLSTM approach outperforms conventional deep neural networks in terms of accuracy.

Consumer sentiment analysis with aspect fusion and GAN‐BERT aided adversarial learning

AbstractThe sentiment analysis (SA) from the user‐generated reviews is the latest research topic in natural language processing. Nowadays, the extraction of consumer sentiment from the content of the consumer reviews is getting much attention because of its importance in understanding consumers' experiences regarding services or products. Consumer sentiment may be helpful for both consumers and organizations; a consumer can refer to previous consumers' feedback while making their purchase decisions, and organizations can use it in service improvements. For the consumer SA, this article proposed the BERT‐GAN model with review aspect fusion, which improves the fine‐tuning performance of the BERT model by introducing semi‐supervised adversarial learning. For our objective, we extracted various service aspects from consumer reviews and fused them with the word sequences before feeding them into the model. That helps in incorporating aspect representation as well as position information in context with the sentences. Our results analysis and their demonstration show the contribution of the presented model in terms of accuracy compared with the existing models found in the previous work.

Digital Health Transformers and Opportunities for Artificial Intelligence-Enabled Nephrology.

Digital Health Transformers and Opportunities for Artificial Intelligence-Enabled Nephrology.

Bye-Bye Bunny: Place and Manner Sequences in Children's C1VC2V-Shaped Words.

Place and manner of articulation in American English-learning children's salient consonant-vowel-consonant-vowel (C1VC2V) target words (e.g., baby, bunny, and cookie) were compared with their actual productions of these words. We hypothesized that target words with repeated place and/or manner characteristics in C1VC2V consonant sequences would be matched in children's actual productions more frequently than target words with variegated place and manner sequences. This hypothesis was based on a proposal that children use available production system capacities to produce salient word forms derived from perceptual input on those word forms. Place and manner sequences were analyzed in 2,092 tokens of C1VC2V target words produced by 18 typically developing children in the single-word period. All data were from these children's spontaneous functional speech in a familiar speaking context. Both target word forms and actual child productions of those targets were analyzed. Results indicated that C1VC2V word target sequences predominantly consisted of repetitions in both place and manner of articulation (e.g., labial-labial, stop-stop). Targets with repetitions of consonant sequences were matched more frequently than targets with variegated sequences for both place and manner (e.g., labial-coronal, stop-nasal) in these children's actual productions. Results also indicated that C1VC2V target words beginning with a labial consonant (e.g., baby, bunny, and piggy) were matched in children's actual productions more frequently than words with coronal or dorsal consonant onsets (e.g., daddy and cookie). In C1VC2V word forms, occurring in early output, children's actual productions matched their target word sequences when the words consisted of repeated sequences in both place and manner. These results suggest that salient target words with repeated sequences may help children support increases in their repertoire of meaningful vocalizations during the transition from babbling to meaningful speech.

Assessing children’s writing products using curriculum-based measures of writing (CBM-W)

AimWriting is key to academic success and job opportunities later in life. Nevertheless, every year children leave primary school struggling to write and few assessments enable schools to reliably identify and target weaknesses in children’s writing. This study explored curriculum-based measures of writing (CBM-W) for assessing writing in primary schools in the United Kingdom (UK).MethodTwenty-seven children from a Year Five class in an English primary school responded to a narrative prompt administered for five minutes once a fortnight for a 10-week period. Children completed a standardised assessment of writing quality (the WIAT) prior to the first CBM-W assessment. CBM-W measures included number of words written (WW), words spelled correctly (WSC), correct word sequences (CWS; any two adjacent words used correctly in context) and correct minus incorrect word sequences (C-IWS).FindingsAll four CBM-W measures demonstrated concurrent validity in relation to the WIAT and captured significant differences in writing performance between children with and without a SEN. Measures also captured pupil growth, demonstrating the utility of CBM-W both as a tool for screening and progress monitoring.LimitationsThe small sample placed limits on the generalisability of findings. Additionally, piloting the equivalence of the writing prompts used in this study may have reduced the risk of measurement error.ConclusionFindings demonstrate the technical adequacy of CBM-W and promising evidence of its potential use by teachers, SENCOs and EPs to assess the writing of primary school children in the UK.