Sub-word Level Research Articles

Valence without meaning: Investigating form and semantic components in pseudowords valence

Valence is a dominant semantic dimension, and it is fundamentally linked to basic approach-avoidance behavior within a broad range of contexts. Previous studies have shown that it is possible to approximate the valence of existing words based on several surface-level and semantic components of the stimuli. Parallelly, recent studies have shown that even completely novel and (apparently) meaningless stimuli, like pseudowords, can be informative of meaning based on the information that they carry at the subword level. Here, we aimed to further extend this evidence by investigating whether humans can reliably assign valence to pseudowords and, additionally, to identify the factors explaining such valence judgments. In Experiment 1, we trained several models to predict valence judgments for existing words from their combined form and meaning information. Then, in Experiment 2 and Experiment 3, we extended the results by predicting participants’ valence judgments for pseudowords, using a set of models indexing different (possible) sources of valence and selected the best performing model in a completely data-driven procedure. Results showed that the model including basic surface-level (i.e., letters composing the pseudoword) and orthographic neighbors information performed best, thus tracing back pseudoword valence to these components. These findings support perspectives on the nonarbitrariness of language and provide insights regarding how humans process the valence of novel stimuli.

A Language Model Optimization Method for Turkish Automatic Speech Recognition System

The current Automatic Speech Recognition (ASR) modeling strategy still suffers from huge performance degradation when faced with languages with limited resources such as Turkish. Especially when the Language Model (LM) does not support the Acoustic Model (AM) sufficiently, the Word Error Rate (WER) increases. Therefore, a robust LM makes a strong contribution to improving ASR performance by generating word relations from the existing corpus. However, developing a robust language model is a challenging task due to the agglutinative nature of Turkish. Therefore, within the scope of the study, a sentence-level LM optimization method is proposed to improve the WER performance of Turkish ASR. In the proposed method, instead of a fixed word sequence obtained from the Markov assumptions, the probability of the word sequence forming a sentence was calculated. A method with n-gram and skip-gram properties is presented to obtain the word sequence probability. The proposed method has been tested on both statistical and Artificial Neural Network (ANN) based LMs. In the experiments carried out using, not only words but also sub-word level, two Turkish corpora (METU and Bogazici) shared via Linguistic Data Consortium (LDC) and a separate corpus, which we separate corpus that we specially created as HS was used. According to the experimental results obtained from statistical-based LM, 0.5% WER increases for the METU corpus, 1.6% WER decreases for the Bogazici corpus, and a 2.5% WER decrease for the HS corpus were observed. In the Feedforward Neural Networks (FNN) based LM, WER decreases were observed 0.2% for the METU corpus, 0.8% for the Bogazici corpus, and 1.6% for the HS corpus. Also, in the Recurrent Neural Network (RNN)-Long Short Term Memory (LSTM) based LM, WER decreases were observed 0.6% for METU corpus, 1.1% for the Bogazici corpus and 1.5% for the HS corpus. As a result, when the proposed method was applied to the LMs required for ASR, WER decreased, and the total performance of ASR increased.

Enhancements to Language Modeling Techniques for Adaptable Log Message Classification

Minimizing the resolution time of service-impacting incidents is a fundamental objective of Information Technology (IT) operations. Efficient root cause analysis, adaptable to diverse service environments, is key to meeting this objective. One method that provides additional insight into an incident, and hence allows enhanced root cause analysis, is categorisation of the events and log messages that characterize an incident into pre-defined operational groups. Well established natural language processing techniques that utilize pre-trained language models and word embeddings can be leveraged for this task. The adaptability of pre-trained models to classify log messages, containing large quantities of domain-specific language, remains unknown. The current contribution investigates multiple ways of addressing this deficiency. We demonstrate increased granularity of word embeddings by using character decompositions and sub-word level representations, and also explore the augmentation of word embeddings using features derived from convolutional operations. After observing that the performance of high-specificity models decreases as the number of previously unseen words increases, we explore the circumstances in which we can use a model trained with a low-specificity corpus to correctly classify log messages. Through the application of fine-tuning techniques, we can adapt our pre-trained classifier to classify log messages from service environments not encountered during pre-training in a time, and memory efficient manner. We conclude that we can effectively adapt pre-trained classifiers for impromptu service environments.

GramBeddings: A New Neural Network for URL Based Identification of Phishing Web Pages Through N-gram Embeddings

There has been ever-growing use of Internet and progress within many communication channels such as social media and this escalates the need for rapid and low source demanding phishing detection mechanisms. In this very study, we propose a new deep neural model for phishing URL identification so-called GramBeddings introducing some distinguishing novelties by (1) proposing the use of n-gram embeddings, computed on the fly, requiring no pre-training stage, (2) removing the necessity of word and sub-word level information, (3) providing a smart and efficient n-gram selection pipeline, and benefiting from attention mechanism. Other than that, we share a publicly available, large-scale and novel dataset22https://web.cs.hacettepe.edu.tr/~selman/grambeddings-dataset/ including 800K real-world phishing and legitimate URLs. Our scheme suggests an adjustable and automated n-gram selection and filtering mechanism along with a new neural network architecture concatenating four-channel information flow through cascading CNN, LSTM, and attention layers. With that, discriminative multi-level character patterns can be discovered without any hand-crafted operation and are enabled to contribute to prediction. As a result, the proposed system provides the following features in the problem domain: (i) real-time, end-to-end and high performance inference, (ii) language-agnostic prediction, and (iii) removal of the necessity of any third-party service or hand-crafted feature. These experiments show that our approach outperforms the other models in the literature with an accuracy of 98.27%. Moreover, the comparative study conducted with several datasets clearly verifies the superiority of our model in all tests. We also examine the robustness of our model against a real-world adversarial attack and discuss the methods of overcoming such an attack. Our codebase33The code and our supplementary material will be made available at https://www.github.com/fcdalgic/GramBeddings is shared with the community to be used for benchmarking purposes in the future.

Textline alignment on the image domain

Editing and publishing a historical manuscript involves a research phase to recover the original manuscript and reconstruct the transmission of its text based on the relations between its surviving copies. Manuscript alignment, which aims to locate the shared and the different text among a set of copies of the same manuscript, is essential for this phase. In this paper, we present an alignment algorithm for historical handwritten documents that works directly on the image domain due to the absence of an accurate handwritten text recognition (HTR) system for handwritten historical documents and the necessity to visualize the original manuscripts in parallel to examine features beyond the transcribed text. Our approach extracts subwords, estimates the similarity among these subwords, and establishes an alignment among them. We extract subwords from textlines images and convert them into sequences of subword images. It estimates the similarity between two subwords using a Siamese network model and applies Longest Common Subsequence (LCS) to establish the alignment between two image sequences. We have implemented our algorithm, trained the Siamese model, and evaluate its performance using textline images from historical documents. Our algorithm outperformed the state-of-the-art by large margins. Unlike the state-of-the-art, the framework builds the alignment from scratch without requiring any prior knowledge concern subwords boundaries. In addition, we build a new dataset for textline alignment for historical documents, which include ten pairs of pages taken from two copies of two Arabic manuscripts and annotated at the subword level.

Authorship Attribution in Bangla Literature (AABL) via Transfer Learning using ULMFiT

Authorship Attribution is the task of creating an appropriate characterization of text that captures the authors’ writing style to identify the original author of a given piece of text. With increased anonymity on the internet, this task has become increasingly crucial in various security and plagiarism detection fields. Despite significant advancements in other languages such as English, Spanish, and Chinese, Bangla lacks comprehensive research in this field due to its complex linguistic feature and sentence structure. Moreover, existing systems are not scalable with the increasing number of authors, and performance drops with the small number of samples per author. In this paper, we propose the use of Average-Stochastic Gradient Descent Weight-Dropped Long Short-Term Memory (AWD-LSTM) architecture and an effective transfer learning approach that addresses the problem of complex linguistic features extraction and scalability for authorship attribution in Bangla Literature (AABL). We analyze the effect of different tokenization, such as word, sub-word, and character level tokenization, and demonstrate the effectiveness of these tokenizations in the proposed model. Moreover, we introduce the publicly available Bangla Authorship Attribution Dataset of 16 authors (BAAD16) containing 17,966 sample texts and 13.4+ million words to solve the standard dataset scarcity problem and release six variations of pre-trained language models for use in any Bangla NLP downstream task. For evaluation, we used our developed BAAD16 dataset as well as other publicly available datasets. Empirically, our proposed model outperformed state-of-the-art models and achieved 99.8% accuracy in the BAAD16 dataset. Furthermore, we showed that the proposed system scales much better with the increasing number of authors, and performance remains steady even with few training samples.

Fine-grained time course of verb aspect processing.

Sentence processing is known to be highly incremental. Speakers make incremental commitments as the sentence unfolds, dynamically updating their representations based on the smallest pieces of information from the incoming speech stream. Less is known about linguistic processing on the sub-word level, especially with regard to abstract grammatical information. This study employs the Visual World Paradigm to investigate the processing of grammatical aspect by Russian-speaking adults (n = 124). Aspectual information is encoded relatively early within the Russian verb which makes this an ideal testing ground to investigate the incrementality of grammatical processing on the sub-word level. Participants showed preference for pictures of ongoing events when they heard sentences involving Imperfective verbs, and for pictures of completed events when they heard sentences involving Perfective verbs. Crucially, the analysis of the participants’ eye-movements showed that they exhibited preference for the target picture already before they heard the end of the verb. Moreover, the latency of this effect depended on where the aspectual information was encoded within the verb. These results indicate that the processing and integration of grammatical aspect information can happen rapidly and incrementally on a fine-grained word-internal level. Methodologically, the study draws together a set of analytical techniques which can be fruitfully applied to the analysis of effect latencies in a wide range of studies within the Visual World eye-tracking paradigm.

Tri-training for Dependency Parsing Domain Adaptation

In recent years, the research on dependency parsing focuses on improving the accuracy of the domain-specific (in-domain) test datasets and has made remarkable progress. However, there are innumerable scenarios in the real world that are not covered by the dataset, namely, the out-of-domain dataset. As a result, parsers that perform well on the in-domain data usually suffer from significant performance degradation on the out-of-domain data. Therefore, to adapt the existing in-domain parsers with high performance to a new domain scenario, cross-domain transfer learning methods are essential to solve the domain problem in parsing. This paper examines two scenarios for cross-domain transfer learning: semi-supervised and unsupervised cross-domain transfer learning. Specifically, we adopt a pre-trained language model BERT for training on the source domain (in-domain) data at the subword level and introduce self-training methods varied from tri-training for these two scenarios. The evaluation results on the NLPCC-2019 shared task and universal dependency parsing task indicate the effectiveness of the adopted approaches on cross-domain transfer learning and show the potential of self-learning to cross-lingual transfer learning.

PRHAN: Automated Pull Request Description Generation Based on Hybrid Attention Network

Descriptions of pull requests (PRs) are posted by developers for describing the modifications that they have made and the corresponding reasons in these PRs. Although PRs help developers improve the development efficiency, some developers usually ignore writing the descriptions for PRs. To alleviate the above problem, researchers generally utilize text summarization model to automatically generate descriptions for PRs. However, current RNN-based models still face the challenges such as low efficiency and out-of-vocabulary (OOV), which may influence the further performance improvement to their models. To break this bottleneck, we propose a novel model aiming at the above challenges, named PRHAN (Pull Requests Description Generation Based on Hybrid Attention Network). Specifically, the core of PRHAN is the hybrid attention network, which has faster execution efficiency than RNN-based model. Moreover, we address the OOV problem by the utilizing byte-pair encoding algorithm to build a vocabulary at the sub-word level. Such a vocabulary can represent the OOV words by combining sub-word units. To reduce the sensitivity of the model, we take a simple but effective method into the cross-entropy loss function, named label smoothing. We choose three baseline models, including LeadCM, Transformer and the state-of-the-art model built by Liu et al. and evaluate all the models on the open-source dataset through ROUGE, BLEU, and human evaluation. The experimental results demonstrate that PRHAN is more effective than baselines. Moreover, PRHAN can execute faster than the state-of-the-art model proposed by Liu et al.

An ensemble of neural models for nested adverse drug events and medication extraction with subwords.

ObjectiveThis article describes an ensembling system to automatically extract adverse drug events and drug related entities from clinical narratives, which was developed for the 2018 n2c2 Shared Task Track 2.Materials and MethodsWe designed a neural model to tackle both nested (entities embedded in other entities) and polysemous entities (entities annotated with multiple semantic types) based on MIMIC III discharge summaries. To better represent rare and unknown words in entities, we further tokenized the MIMIC III data set by splitting the words into finer-grained subwords. We finally combined all the models to boost the performance. Additionally, we implemented a featured-based conditional random field model and created an ensemble to combine its predictions with those of the neural model.ResultsOur method achieved 92.78% lenient micro F1-score, with 95.99% lenient precision, and 89.79% lenient recall, respectively. Experimental results showed that combining the predictions of either multiple models, or of a single model with different settings can improve performance.DiscussionAnalysis of the development set showed that our neural models can detect more informative text regions than feature-based conditional random field models. Furthermore, most entity types significantly benefit from subword representation, which also allows us to extract sparse entities, especially nested entities.ConclusionThe overall results have demonstrated that the ensemble method can accurately recognize entities, including nested and polysemous entities. Additionally, our method can recognize sparse entities by reconsidering the clinical narratives at a finer-grained subword level, rather than at the word level.

A Multilingual and Multidomain Study on Dialog Act Recognition Using Character-Level Tokenization

Automatic dialog act recognition is an important step for dialog systems since it reveals the intention behind the words uttered by its conversational partners. Although most approaches on the task use word-level tokenization, there is information at the sub-word level that is related to the function of the words and, consequently, their intention. Thus, in this study, we explored the use of character-level tokenization to capture that information. We explored the use of multiple character windows of different sizes to capture morphological aspects, such as affixes and lemmas, as well as inter-word information. Furthermore, we assessed the importance of punctuation and capitalization for the task. To broaden the conclusions of our study, we performed experiments on dialogs in three languages—English, Spanish, and German—which have different morphological characteristics. Furthermore, the dialogs cover multiple domains and are annotated with both domain-dependent and domain-independent dialog act labels. The achieved results not only show that the character-level approach leads to similar or better performance than the state-of-the-art word-level approaches on the task, but also that both approaches are able to capture complementary information. Thus, the best results are achieved by combining tokenization at both levels.

"To Name or Not to Name: That is the Question": The Role of Response Inhibition in Reading.

Reading is a complex process that includes the integration of information about letters (graphemes) and sounds (phonemes). In many circumstances, such as noisy environments, response inhibition is an additional factor that plays a marked role in successful oral reading. Response inhibition can take the form of task relevant inhibition (i.e., foils in a go/no-go task) and task irrelevant inhibition (i.e., distractor information). Here we investigated task relevant inhibition by having participants (N = 30) take part in two tasks: go/no-go naming with nonwords foils (GNG-NW) and go/no-go naming with pseudohomophones foils (GNG-PH). Also, we investigated the addition of task irrelevant inhibition by having participants (N = 28) take part in two tasks: GNG-NW + information masking and GNG-PH + information masking. We provide evidence that during a task relevant inhibition task, sub-word sound level information can be successfully inhibited, as evidenced by comparable response times for regular words and exception words, provided the foils do not contain familiar sound-based information (GNG-NW). In contrast, regular words were read aloud faster than exception words in a GNG-PH task, indicating that sub-word level interference occurs when the foils contain familiar sound-based information. The addition of task irrelevant inhibition (i.e., information masking at the phoneme level), served to increase response time overall, but did not impact the pattern of response times between regular words and exception words. Together these findings provide useful information regarding the role of response inhibition in word recognition and may be useful in computational models of word recognition and future work may benefit from accounting for the effects outlined in this paper.

Learning Simpler Language Models with the Differential State Framework.

Learning useful information across long time lags is a critical and difficult problem for temporal neural models in tasks such as language modeling. Existing architectures that address the issue are often complex and costly to train. The differential state framework (DSF) is a simple and high-performing design that unifies previously introduced gated neural models. DSF models maintain longer-term memory by learning to interpolate between a fast-changing data-driven representation and a slowly changing, implicitly stable state. Within the DSF framework, a new architecture is presented, the delta-RNN. This model requires hardly any more parameters than a classical, simple recurrent network. In language modeling at the word and character levels, the delta-RNN outperforms popular complex architectures, such as the long short-term memory (LSTM) and the gated recurrent unit (GRU), and, when regularized, performs comparably to several state-of-the-art baselines. At the subword level, the delta-RNN's performance is comparable to that of complex gated architectures.

A Novel Method to Detect Segmentation points of Arabic Words using Peaks and Neural Network

Many methods of segmentation using detection of segmentation points or where the location of segmentation points is expected before the segmentation process, the validity of segmentation points is verified by using ANNs. In this paper apply a novel method to detect correctly of location segmentation points by detect of peaks with neural networks for Arabic word. This method employs baseline and peaks identification; where using two steps to segmenting text. Where peaks identification function is applied which at the subword segment level to frame the minimum and maximum peaks, and baseline detection. Where these two steps have led to the best result through the model depends on minimum peaks attained by utilising a stroke operator with a view to extracting potential points of segmentation, and determining the baseline procedure was developed to approximate the parameters. Where this method has yielded highly accurate positive results for Arabic characters’ segmentation with four kinds of handwritten datasets as AHDB, IFN-ENIT, AHDB-FTR and ACDAR. Earlier results showed that the use of EDMS to MLP_ANN gives better results than GLCM and MOMENT in different groups and gives results of EDMS features on MNN with an accuracy level of 95.09% classifier for IFN-ENIT set of data.

Anti-Forensics of JPEG Images using Interpolation

Segmentation of Arabic text is a major challenge that shall be addressed by any recognition system. The cursive nature of Arabic writing makes it necessary to handle the segmentation issue at various levels. Arabic text line can be viewed as a sequence of words which in turn can be viewed as a sequence of sub-words. Sub-words have the frequently encountered intrinsic property of sharing the same vertical space which makes vertical projection based segmentation technique inefficient. In this paper, the task of segmenting handwritten Arabic text at sub-word level is taken up. The proposed algorithm is based on pulling away the connected components to overcome the impossibility of separating them by vertical projection based approach. Graph theoretic modeling is proposed to solve the problem of connected component extraction. In the sequel, these components are subjected to thorough analysis in order to obtain the constituent sub-words where a sub-word may consist of many components. The proposed algorithm was tested using variety of handwritten Arabic samples taken from different databases and the results obtained are encouraging.

面向双语教学的维吾尔语发音质量自动评估

Pursuing Chinese and Uygur bilingual teaching in XinJiang Province is an important government policy for improving communication between different language speakers. An automatic assessment system can yield many advantages in such bilingual teaching. However, as an agglutinative language, Uygurs particular way of word-building results in leaving many words out of the lexicon. Therefore, performance of the old pronunciation assessment system, which is based on a traditional ASR system, is poor. For building a high-performance system, we decide to use subwords as basic recognition units after analyzing the rules and habits of Uygur pronunciation. Experimental results indicate that the accuracy of the subword-based system is greatly improved by the implementation of machine segmentation based on double-level morphology analysis of Uygur subwords, calculating the posterior probabilitys denominator with a phoneme decoder, and calculating confidence at the subword level.

Discriminative Training for Phonetic Recognition of the Holy Quran

This paper presents the development of Holy Quran recitation recognizer. The decoder of recognizer performs sub-word level recognition at phoneme. The paper demonstrates high recognition accuracies achieved by applying incremental refinements to the HMM models of the phonemes during the training stage. The Maximum- likelihood (ML) criterion is first applied for HMMs parameter estimation, which produces average recognition accuracies of up to 83 %. This is followed by discriminative technique of minimum phone error (MPE), which is applied to minimize recognition error at phoneme level. Investigation shows that MPE based acoustic models improve generalization. The results show 3–4 % improvement in recognition accuracies, which are promising when compared with the case of ML approach applied alone.

Cultural literacy in Chinese and Malay

The discussion suggests that there are many meaningful ways of making important messages beyond lexical meaning. Speakers of Malay and Chinese have been applying meaningful linguistic systems consistently by accentuating meaning at subword levels to communicate, preach, scorn, and interact with members in their respective speech communities. Such cultural-specific verbal styling in speech, however, is not accounted for in language standardization. The word is commonly accepted as the basic meaning unit conducive to language planning and lexicography. Nonetheless, effort to understand meaningful linguistic units below the level of a word stem is relevant to both cultural literacy and group cohesion. The failure to understand and participate in the production of meaning in subword level sounds is a form of cultural autism. This results in a loss of interactive essence which is conventionally signaled subtly in native speech style. By highlighting versatile meaning-making in the selected language data, the discussion complements the focus on current lexical expansion in standard written literacy.

Identifying translationese at the word and sub-word level

We use text classification to distinguish automatically between original and translated texts in Hebrew, a morphologically complex language. To this end, we design several linguistically informed feature sets that capture word-level and sub-word-level (in particular, morphological) properties of Hebrew. Such features are abstract enough to allow for the development of accurate, robust classifiers, and they also lend themselves to linguistic interpretation. Careful evaluation shows that some of the classifiers we define are, indeed, highly accurate, and scale up nicely to domains that they were not trained on. In addition, analysis of the best features provides insight into the morphological properties of translated texts.

멀티미디어 전용 명령어를 내장한 멀티코어 프로세서 구현 및 검증

In this paper, we present a multi-core processor including multimedia specific instructions to process multimedia data efficiently in the mobile environment. Multimedia specific instructions exploit subword level parallelism (SLP), while the multi-core processor exploits data level parallelism (DLP). These combined parallelisms improve the performance of multimedia processing applications. The proposed multi-core processor including multimedia specific instructions is implemented and tested using a Xilinx ISE 10.1 tool and SoCMaster3 testbed system including Vertex 4 FPGA. Experimental results using a fire detection algorithm show that multimedia specific instructions outperform baseline instructions in the same multi-core architecture in terms of performance (1.2x better), energy efficiency (1.37x better), and area efficiency (1.23x better).