Word Sequences Research Articles

Phase-locking of Neural Activity to the Envelope of Speech in the Delta Frequency Band Reflects Differences between Word Lists and Sentences.

The envelope of a speech signal is tracked by neural activity in the cerebral cortex. The cortical tracking occurs mainly in two frequency bands, theta (4-8 Hz) and delta (1-4 Hz). Tracking in the faster theta band has been mostly associated with lower-level acoustic processing, such as the parsing of syllables, whereas the slower tracking in the delta band relates to higher-level linguistic information of words and word sequences. However, much regarding the more specific association between cortical tracking and acoustic as well as linguistic processing remains to be uncovered. Here, we recorded EEG responses to both meaningful sentences and random word lists in different levels of signal-to-noise ratios (SNRs) that lead to different levels of speech comprehension as well as listening effort. We then related the neural signals to the acoustic stimuli by computing the phase-locking value (PLV) between the EEG recordings and the speech envelope. We found that the PLV in the delta band increases with increasing SNR for sentences but not for the random word lists, showing that the PLV in this frequency band reflects linguistic information. When attempting to disentangle the effects of SNR, speech comprehension, and listening effort, we observed a trend that the PLV in the delta band might reflect listening effort rather than the other two variables, although the effect was not statistically significant. In summary, our study shows that the PLV in the delta band reflects linguistic information and might be related to listening effort.

The impact of atypical text presentation on transposed-word effects.

When asked to decide if an ungrammatical sequence of words is grammatically correct or not, readers find it more difficult to do so (longer response times (RTs) and more errors) if the ungrammatical sequence is created by transposing two words from a correct sentence (e.g., the white was cat big) compared with matched ungrammatical sequences where transposing two words does not produce a correct sentence (e.g., the white was cat slowly). Here, we provide a further exploration of transposed-word effects when reading unspaced text in Experiment 1, and when reading from right-to-left ("backwards" reading) in Experiment 2. We found significant transposed-word effects in error rates but not in RTs, a pattern previously found in studies using a one-word-at-a-time sequential presentation. We conclude that the absence of transposed-word effects in RTs in the present study and prior work is due to that atypical nature of the way that text was presented. Under the hypothesis that transposed-word effects at least partly reflect a certain amount of parallel word processing during reading, we further suggest that the ability to process words in parallel would require years of exposure to text in its regular format.

A Pragmatic Analysis of Machine Translation Techniques for Preserving the Authenticity of the Sanskrit Language

Machine Translation has been a field of study for over six decades, but it has acquired substantial prominence in the last decade as processing capacity in personal computers has increased. The purpose of this paper is to discuss the usage of Sanskrit as a source, target, or supporting language in various Machine Translation systems. To investigate Machine Translation, researchers use a variety of strategies, including corpus-based, direct, and rule-based approaches. The primary goal of employing Sanskrit in Machine Translation is to evaluate its appropriateness, lexicon, and performance when proper Machine Translation methods are used. The research examines various modelling strategies for developing a machine translation system, specifically Statistical and Neural Machine Translation, in order to bridge the gap between Sanskrit and its current successor, Hindi. Interpretations are formed in Statistical Machine Translation by matching words from the source and target languages with statistical models and bilingual text corpora to learn parameters. Neural Machine Translation, on the other hand, uses an artificial neural network to predict the likelihood of a word sequence, frequently modelling entire phrases within a single integrated model. Neural Machine Translation is implemented using an encoder-decoder architecture with an attention mechanism. One of the most significant contributions of this paper is the use of different data sources, data collecting, and scraping to create a complete dataset. According to the study's findings, Neural Machine Translation outperforms the Statistical Machine Translation modelling technique. Furthermore, the paper examines the distinctive qualities of the Sanskrit language as well as the difficulties encountered by researchers in digesting Sanskrit while constructing the machine translation system. This study investigates the use of Sanskrit in Machine Translation and analyses several modelling methods, such as Statistical and Neural Machine Translation. The paper emphasizes the advantages of Neural Machine Translation and discusses the unique characteristics and challenges of the Sanskrit language in machine translation development.

Fine-tuning language models to recognize semantic relations

Transformer-based pre-trained Language Models (PLMs) have emerged as the foundations for the current state-of-the-art algorithms in most natural language processing tasks, in particular when applied to context rich data such as sentences or paragraphs. However, their impact on the tasks defined in terms of abstract individual word properties, not necessary tied to their specific use in a particular sentence, has been inadequately explored, which is a notable research gap. Addressing this gap is crucial for advancing our understanding of natural language processing. To fill this void, we concentrate on classification of semantic relations: given a pair of concepts (words or word sequences) the aim is to identify the semantic label to describe their relationship. E.g. in the case of the pair green/colour, “is a” is a suitable relation while “part of”, “property of”, and “opposite of” are not suitable. This classification is independent of a particular sentence in which these concepts might have been used. We are first to incorporate a language model into both existing approaches to this task, namely path-based and distribution-based methods. Our transformer-based approaches exhibit significant improvements over the state-of-the-art and come remarkably close to achieving human-level performance on rigorous benchmarks. We are also first to provide evidence that the standard datasets over-state the performance due to the effect of “lexical memorisation.” We reduce this effect by applying lexical separation. On the new benchmark datasets, the algorithmic performance remains significantly below human-level, highlighting that the task of semantic relation classification is still unresolved, particularly for language models of the sizes commonly used at the time of our study. We also identify additional challenges that PLM-based approaches face and conduct extensive ablation studies and other experiments to investigate the sensitivity of our findings to specific modelling and implementation choices. Furthermore, we examine the specific relations that pose greater challenges and discuss the trade-offs between accuracy and processing time.

Text Algorithms in Economics

This article provides an overview of the methods used for algorithmic text analysis in economics, with a focus on three key contributions. First, we introduce methods for representing documents as high-dimensional count vectors over vocabulary terms, for representing words as vectors, and for representing word sequences as embedding vectors. Second, we define four core empirical tasks that encompass most text-as-data research in economics and enumerate the various approaches that have been taken so far to accomplish these tasks. Finally, we flag limitations in the current literature, with a focus on the challenge of validating algorithmic output.

Defending against Backdoor Attacks in Natural Language Generation

The frustratingly fragile nature of neural network models make current natural language generation (NLG) systems prone to backdoor attacks and generate malicious sequences that could be sexist or offensive. Unfortunately, little effort has been invested to how backdoor attacks can affect current NLG models and how to defend against these attacks. In this work, by giving a formal definition of backdoor attack and defense, we investigate this problem on two important NLG tasks, machine translation and dialog generation. Tailored to the inherent nature of NLG models (e.g., producing a sequence of coherent words given contexts), we design defending strategies against attacks. We find that testing the backward probability of generating sources given targets yields effective defense performance against all different types of attacks, and is able to handle the one-to-many issue in many NLG tasks such as dialog generation. We hope that this work can raise the awareness of backdoor risks concealed in deep NLG systems and inspire more future work (both attack and defense) towards this direction.

Sequence Generation with Label Augmentation for Relation Extraction

Sequence generation demonstrates promising performance in recent information extraction efforts, by incorporating large-scale pre-trained Seq2Seq models. This paper investigates the merits of employing sequence generation in relation extraction, finding that with relation names or synonyms as generation targets, their textual semantics and the correlation (in terms of word sequence pattern) among them affect model performance. We then propose Relation Extraction with Label Augmentation (RELA), a Seq2Seq model with automatic label augmentation for RE. By saying label augmentation, we mean prod semantically synonyms for each relation name as the generation target. Besides, we present an in-depth analysis of the Seq2Seq model's behavior when dealing with RE. Experimental results show that RELA achieves competitive results compared with previous methods on four RE datasets.

Deep learning and gradient-based extraction of bug report features related to bug fixing time

Bug reports typically contain detailed descriptions of failures, hints at the location of the corresponding defects, and discussions. Developers usually resolve bugs using comments in descriptions and discussions. The time to fix a bug varies greatly. Previous studies have investigated bug reports, but the influence of comments on bug fixing time is not well understood. This study adopts a convolutional neural network (CNN) and gradient-based visualization approach called Grad-cam to elucidate the impact of comments on bug fixing time and extract features. A feature represents an observed characteristic in a bug report when processing via deep learning. Specifically, CNN classifies bug reports, and then Grad-cam visualizes the decision basis of CNN by identifying the top 10 word sequences used in the prediction. Here, the features are major word sequences extracted by Grad-cam. In an experiment, the proposed method classified more than 36,000 actual bug reports from Bugzilla with an accuracy of 75%–80%. Additionally, the visualization highlighted differences in the stack trace and word abstraction by bug fixing time. Bug reports with short bug fixing times are concrete, whereas those with a long bug fixing time are abstract.

Heart Rate Variability Biofeedback for Mild Traumatic Brain Injury: A Randomized-Controlled Study

To determine whether heart rate variability biofeedback (HRV-BF) training, compared to a psychoeducation control condition can strengthen the integration of the central and autonomic nervous systems as measured by neuropsychological measures in patients with mild traumatic brain injury (mTBI). Participants were recruited from two university hospitals in Taipei, Taiwan. A total of 49 participants with mTBI were recruited for this study. Forty-one participants completed the study, 21 in the psychoeducation group and 20 in the HRV-BF group. Randomized controlled study. The Taiwanese Frontal Assessment Battery, the Semantic Association of Verbal Fluency Test, the Taiwanese version of the Word Sequence Learning Test, the Paced Auditory Serial Addition Test-Revised, and the Trail Making Test were used as performance-based neuropsychological functioning measures. The Checklist of Post-concussion Symptoms, the Taiwanese version of the Dysexecutive Questionnaire, the Beck Anxiety Inventory, the Beck Depression Inventory, and the National Taiwan University Irritability Scale were used as self-report neuropsychological functioning measures. Furthermore, heart rate variability pre- vs. post-training was used to measure autonomic nervous system functioning. Executive, information processing, verbal memory, emotional neuropsychological functioning, and heart rate variability (HRV) were improved significantly in the HRV-BF group at the posttest whereas the psychoeducation group showed no change. HRV biofeedback is a feasible technique following mild TBI that can improve neuropsychological and autonomic nervous system functioning. HRV-BF may be clinically feasible for the rehabilitation of patients with mTBI.

Do Japanese word-embedded representations obtained in the academic corpus retain the medical concepts of “infarction”?

ObjectiveThe pathophysiological concepts of diseases are encapsulated in patients' medical histories. Whether information on the pathophysiology or anatomy of “infarction” can be preserved and objectively expressed in the distributed representation obtained from a corpus of scientific Japanese medical texts in the “infarction” domain is currently unknown. Word2Vec was used to obtain distributed representations, meanings, and word analogies of word vectors, and this process was verified mathematically. Materials & methodsThe texts were abstracts that were obtained by searching for “infarction,” “abstract,” and “case report” in the Japan Medical Journal Association's Ichushi Data Base. The abstracted text was morphologically analyzed to produce word sequences converted into their standard form. MeCab was used for morphological analysis and mecab-ipadic-NEologd and ComeJisyo were used as dictionaries. The accuracy of the known tasks for medical terms was evaluated using a word analogy task specific to the “infarction” domain. ResultsOnly 33 % of the word analogy tasks for medical terminology were correct. However, 52 % of the new original tasks, which were specific to the “infarction” domain, were correct, especially those regarding anatomical differences. DiscussionDocuments related to “infarction” were collected from a corpus of Japanese medical documents and word-embedded expressions were obtained using Word2Vec. Terminology that had similar meanings to “infarction” included words such as “cavity” and “ischemia,” which suggest the pathology of an infarction. ConclusionThe pathophysiological and anatomical features of an “infarction” may be retained in a distributed representation.

Quantum Language Model With Entanglement Embedding for Question Answering.

Quantum language models (QLMs) in which words are modeled as a quantum superposition of sememes have demonstrated a high level of model transparency and good post-hoc interpretability. Nevertheless, in the current literature, word sequences are basically modeled as a classical mixture of word states, which cannot fully exploit the potential of a quantum probabilistic description. A quantum-inspired neural network (NN) module is yet to be developed to explicitly capture the nonclassical correlations within the word sequences. We propose a NN model with a novel entanglement embedding (EE) module, whose function is to transform the word sequence into an entangled pure state representation. Strong quantum entanglement, which is the central concept of quantum information and an indication of parallelized correlations among the words, is observed within the word sequences. The proposed QLM with EE (QLM-EE) is proposed to implement on classical computing devices with a quantum-inspired NN structure, and numerical experiments show that QLM-EE achieves superior performance compared with the classical deep NN models and other QLMs on question answering (QA) datasets. In addition, the post-hoc interpretability of the model can be improved by quantifying the degree of entanglement among the word states.

Feature-based approach and sequential pattern mining to enhance quality of Indonesian automatic text summarization

Indonesian automatic text summarization research is developed rapidly. The quality, especially readability aspect, of text summary can be reached if the meaning of the text can be maintained properly. Therefore, this research aims to enhance the quality of extractive Indonesian automatic text summarization with considering the quality of structured representation of text. This research uses sequential pattern mining (SPM) to produce This research use SPM to produce sequence of words (SoW) as structured text representation using PrefixSpan algorithm. Then, SPM is combined with feature-based approach using sentence scoring method to produce summary. The experiment result using IndoSum dataset shows that even though the combination of SPM and sentence scoring can increase the precision value of recall-oriented understudy for gisting evaluation (ROUGE)-1, ROUGE-2, and ROUGE-L, from 0.68 to 0.76, 0.54 to 0.69, and 0.51 to 0.72. Especially, combination of SPM and Sentence Scoring can enhance precision, recall, and f-measure of ROUGE-L that consider the order of word occurance in measurement. SPM increases ROUGE-L f-measure value of sentence scoring from 0.32 to 0.36. Moreover, combination of sentence scoring and SPM is better than SumBasic that used as feature-based approach in the previous Indonesian text summarization research.

A fusion recommendation model based on mutual information and attention learning in heterogeneous social networks

Heterogeneous social networks generate vast amounts of data due to their large user base. In response, recommendation systems have been developed as an effective tool to mitigate the problem of information overload. Nonetheless, standalone recommendation methods are often limited by their own inherent deficiencies. Consequently, deep learning has emerged as a powerful approach to address these issues. By integrating diverse recommendation methods, the shortcomings of individual methods can be partially compensated for and improved recommendation outcomes can be achieved. This paper presents a novel user interest fusion recommendation model called MIAL, which leverages mutual information and attention-based learning mechanisms. Specifically, a user’s posts are first transformed into word sequences via a Convolution Neural Network (CNN) based interest recommendation model for pre-processing. To enhance the multi-label prediction model, this study adopts a label selection method based on the label maximum mutual information spanning tree (LMMIST) to determine the final recommended label set. To identify reliable user similarity, the concept of pseudo-similar friends is introduced to construct heterogeneous networks. During the network embedding phase, a new random walk algorithm termed MRWalk and a pseudo-similar propagation mechanism are proposed. Experimental results using the Twitter dataset and public Amazon product dataset demonstrate that the proposed MIAL algorithm outperforms various state-of-the-art algorithms. The comparative analysis shows that MIAL has significantly improved performance.

Optimized experimental designs to best detect spatial positional association of response codes in working memory.

The SPoARC (Spatial Positional Association of Response Codes) effect refers to spatialization of information in working memory. Among the potential factors that could influence how order is mapped onto a mental space during the recognition process, we selected the following two factors: i) the type of stimuli, in particular their verbal vs. visual aspects and ii) the number of probes. In this study, 137 participants memorized sequences of either words or pictures and subsequently performed a recognition test for which they responded using lateralized keys. For half of the participants, only one probe was presented after each sequence, whereas the other half was administered several probes. A significantly greater number of participants presented a SPoARC using a single probe. We discuss that spatialization is best detected when the sequence is scanned only once. Results also showed no difference between the two types of stimuli (i.e., verbal vs. visual). This finding raises the question of the respective roles of verbalization and visualization in the SPoARC.

Linear Dependency Segments in Foreign Language Acquisition: Syntactic Complexity Analysis in Czech Learners’ Texts

Abstract The paper discusses a new way to measure syntactic complexity in foreign language acquisition. It is based on a recently proposed syntactic unit called linear dependency segment (LDS), the longest possible sequence of words belonging to the same clause where all linear neighbours are also syntactic neighbours. The dataset comprises 5,721 Czech texts from the CzeSL-SGT learner corpus covering five CEFR proficiency levels (A1–C1). The study covers two analyses. First, the development of the average clause length in terms of LDS and the average LDS length in the number of words across the latter language proficiency levels. Second, we consider the differences between Slavic and non-Slavic speakers. The results show an increasing tendency of the average clause length measured in LDS while the average clause length measured in words is decreasing. Results also show statistically significant differences between Slavic and non-Slavic speakers in most cases. Our results indicate that using LDS may be a useful unit of syntactic complexity measure in foreign language acquisition research.

Captioning Image Using Deep Learning Approach

Abstract: Captioning image using deep learning is a technology that aims to generate descriptive and accurate textual descriptions for images. By using the power of deep neural networks, this approach enables computers to understand and interpret visual content bridging the gap between the visual and textual domains. The process of image captioning involves two main components: an image encoder and a language decoder. The image encoder is basically a Convolutional Neural Network (CNN) that processes the input image extracting high-level features and representations. These features capture the visual content and generating meaningful captions. The language decoder on the other hand is usually a Recurrent Neural Network (RNN), such as a long short-term memory (LSTM) network. It takes the encoded image features as input and generates a sequence of words to form the caption

Combinatorial structure of Sturmian words and continued fraction expansion of Sturmian numbers

Let θ=[0;a 1 ,a 2 ,⋯] be the continued fraction expansion of an irrational real number θ∈(0,1). It is well-known that the characteristic Sturmian word of slope θ is the limit of a sequence of finite words (M k ) k≥0 , with M k of length q k (the denominator of the k-th convergent to θ) being a suitable concatenation of a k copies of M k-1 and one copy of M k-2 . Our first result extends this to any Sturmian word s. Let b≥2 be an integer. Our second result gives the continued fraction expansion of any real number ξ whose b-ary expansion is a Sturmian word s over the alphabet {0,b-1}. This extends a classical result of Böhmer who considered only the case where s is characteristic. As a consequence, we obtain a formula for the irrationality exponent of ξ in terms of the slope and the intercept of s.

A Virtual Simulation-Pilot Agent for Training of Air Traffic Controllers

In this paper we propose a novel virtual simulation-pilot engine for speeding up air traffic controller (ATCo) training by integrating different state-of-the-art artificial intelligence (AI)-based tools. The virtual simulation-pilot engine receives spoken communications from ATCo trainees, and it performs automatic speech recognition and understanding. Thus, it goes beyond only transcribing the communication and can also understand its meaning. The output is subsequently sent to a response generator system, which resembles the spoken read-back that pilots give to the ATCo trainees. The overall pipeline is composed of the following submodules: (i) an automatic speech recognition (ASR) system that transforms audio into a sequence of words; (ii) a high-level air traffic control (ATC)-related entity parser that understands the transcribed voice communication; and (iii) a text-to-speech submodule that generates a spoken utterance that resembles a pilot based on the situation of the dialogue. Our system employs state-of-the-art AI-based tools such as Wav2Vec 2.0, Conformer, BERT and Tacotron models. To the best of our knowledge, this is the first work fully based on open-source ATC resources and AI tools. In addition, we develop a robust and modular system with optional submodules that can enhance the system’s performance by incorporating real-time surveillance data, metadata related to exercises (such as sectors or runways), or even a deliberate read-back error to train ATCo trainees to identify them. Our ASR system can reach as low as 5.5% and 15.9% absolute word error rates (WER) on high- and low-quality ATC audio. We also demonstrate that adding surveillance data into the ASR can yield a callsign detection accuracy of more than 96%.

Sound-meaning mapping: Verbal imitation of Super Mario music by Yorùbá gamers

An aspect of gaming culture among Yorùbá millennials is verbally interpreting certain musical motifs of the popular videogame called Super Mario Bros. The themes of the verbal interpretations are comparable to those of music texts at traditional Yorùbá competitions. Drawing on the Yorùbá music tradition, the account in this work is that, to the gamers, the background music of the videogame performs a similar function as the music at traditional Yorùbá competitions. Semantically, the choice of words in the linguistic interpretation is conditioned by the situational contexts or scenes where the music is heard in the videogame. The results of an acoustic analysis show that the pitch contours of the linguistic interpretations resemble the pitch trajectories of the corresponding music motifs. Thus, the sequence of words in each linguistic interpretation is determined by vocal imitation. This study suggests that the linguistic processing of music does not only involve phonetic iconicity but includes contextual inference and social expectation. The interpretive moves clearly point to strong parallels between sound-meaning mapping in spoken language and music.

Eyettention: An Attention-based Dual-Sequence Model for Predicting Human Scanpaths during Reading

Eye movements during reading offer insights into both the reader's cognitive processes and the characteristics of the text that is being read. Hence, the analysis of scanpaths in reading have attracted increasing attention across fields, ranging from cognitive science over linguistics to computer science. In particular, eye-tracking-while-reading data has been argued to bear the potential to make machine-learning-based language models exhibit a more human-like linguistic behavior. However, one of the main challenges in modeling human scanpaths in reading is their dual-sequence nature: the words are ordered following the grammatical rules of the language, whereas the fixations are chronologically ordered. As humans do not strictly read from left-to-right, but rather skip or refixate words and regress to previous words, the alignment of the linguistic and the temporal sequence is non-trivial. In this paper, we develop Eyettention, the first dual-sequence model that simultaneously processes the sequence of words and the chronological sequence of fixations. The alignment of the two sequences is achieved by a cross-sequence attention mechanism. We show that Eyettention outperforms state-of-the-art models in predicting scanpaths. We provide an extensive within- and across-data set evaluation on different languages. An ablation study and qualitative analysis support an in-depth understanding of the model's behavior.