Character N-grams Research Articles

STYLOMETRIC ANALYSIS OF WORKS BYĞABDULLA TUQAYAND SÄĞİT SÜNÇÄLÄY: WHO IS THE AUTHOR OF THE POEM “SÖY ĞÖMERNE...”?

This article is devoted to determining the authorship of the Tatar quatrain “Söy ğömerne...” using stylometric methods. Ğabdulla Tuqay, to whom this text had been traditionally attributed, and Säğit Sünçäläy, evidence of whose authorship was discovered by literary scholars at the turn of the century, are considered possible authors. The paper tests several distance-based stylometric methods. The material of the study was a corpus of digitized original poems by Ğabdulla Tuqay and Säğit Sünçäläy in the modern literary variety of the Tatar language. The final sample for analysis included 10 texts by Tuqay and 8 texts by Sünçäläy. The distance between texts was measured using Burrows’ Delta, Cosine Delta and cosine distance. Using the package Stylo, calculations were performed and dendrograms were constructed from them. The Cosine Delta measure with extraction of character n-grams (with n = 4) proved to be the most effective. This is explained, firstly, by the agglutinative structure of the Tatar language, since among the most frequent n-grams extracted, elements corresponding to inflectional affixes were found. Secondly, the use of character n-grams allows to increase the size of the analyzed data. In some cases, good attribution results have also been achieved using Cosine Delta based on wordforms and cosine distance based on character n-grams. Despite the impossibility to draw a definite conclusion due to the extremely small size of the texts analyzed, the comparison of the results of applying different methods shows that the most likely author of the quatrain in question is Säğit Sünçäläy.

Corpus-based dialectometry with topic models

Abstract This paper presents a topic modeling approach to corpus-based dialectometry. Topic models are most often used in text mining to find latent structure in a collection of documents. They are based on the idea that frequently co-occurring words present the same underlying topic. In this study, topic models are used on interview transcriptions containing dialectal speech directly, without any annotations or preselected features. The transcriptions are modeled on complete words, on character n-grams, and after automatical segmentation. Data from three languages, Finnish, Norwegian, and Swiss German, are scrutinized. The proposed method is capable of discovering clear dialectal differences in all three datasets, while reflecting the differences between them. The method provides a significant simplification of the dialectometric workflow, simultaneously saving time and increasing objectivity. Using the method on non-normalized data could also benefit text mining, which is the traditional field of topic modeling.

Offensive language detection in low resource languages: A use case of Persian language.

THIS ARTICLE USES WORDS OR LANGUAGE THAT IS CONSIDERED PROFANE, VULGAR, OR OFFENSIVE BY SOME READERS. Different types of abusive content such as offensive language, hate speech, aggression, etc. have become prevalent in social media and many efforts have been dedicated to automatically detect this phenomenon in different resource-rich languages such as English. This is mainly due to the comparative lack of annotated data related to offensive language in low-resource languages, especially the ones spoken in Asian countries. To reduce the vulnerability among social media users from these regions, it is crucial to address the problem of offensive language in such low-resource languages. Hence, we present a new corpus of Persian offensive language consisting of 6,000 out of 520,000 randomly sampled micro-blog posts from X (Twitter) to deal with offensive language detection in Persian as a low-resource language in this area. We introduce a method for creating the corpus and annotating it according to the annotation practices of recent efforts for some benchmark datasets in other languages which results in categorizing offensive language and the target of offense as well. We perform extensive experiments with three classifiers in different levels of annotation with a number of classical Machine Learning (ML), Deep learning (DL), and transformer-based neural networks including monolingual and multilingual pre-trained language models. Furthermore, we propose an ensemble model integrating the aforementioned models to boost the performance of our offensive language detection task. Initial results on single models indicate that SVM trained on character or word n-grams are the best performing models accompanying monolingual transformer-based pre-trained language model ParsBERT in identifying offensive vs non-offensive content, targeted vs untargeted offense, and offensive towards individual or group. In addition, the stacking ensemble model outperforms the single models by a substantial margin, obtaining 5% respective macro F1-score improvement for three levels of annotation.

Normalizing Swiss German Dialects with the Power of Large Language Models

Swiss German dialects pose significant challenges for natural language processing (NLP) applications, due to their lack of standard orthography, linguistic diversity, and scarcity of annotated data. We introduce a novel method for normalizing Swiss German text to Standard German by employing the mT5 model, a cutting-edge large language model (LLM) that can perform various text-to-text transformations across multiple languages. Our approach not only aims to enhance the processing of Swiss German dialects but also seeks to broaden the understanding of the adaptability of pre-trained LLMs in the realm of dialect normalization. By fine-tuning the mT5 model across its small, base, and large variants with the SwissDial dataset under various hyperparameter settings, we evaluated the performance of these models using the character n-gram F-score (ChrF) and the COMET metrics. The results demonstrated that the mT5 model, particularly its smallest variant, can achieve high-quality normalization of Swiss German dialects with minimal performance differences between the model sizes. This indicates that the SwissDial dataset is sufficiently extensive for effective fine-tuning, suggesting that even less resource-intensive models are viable for this task. Our findings advocate for the potential of LLMs, like mT5, as powerful instruments for dialect normalization and other NLP challenges, offering a promising alternative to traditional methods.

A Synergistic Bidirectional LSTM and N-gram Multi-channel CNN Approach Based on BERT and FastText for Arabic Event Identification

Event extraction from texts continues to pose a challenge for many NLP systems. This article presents a novel neural network architecture that can extract and classify events from Arabic sentences. The model combines word representations and Part-Of-Speech (POS) tags and uses a bidirectional LSTM layer and a dual combined convolutional neural network. The first layer of the network focuses on sentence representations, while the second layer focuses on POS representations. The model takes advantage of both N-gram character features from FastText and contextual representations from bidirectional encoder representations from transformers. This combination proves to be successful, as evidenced by the good results obtained from evaluating the model on the Arabic TimeML corpus. Our results show that combining both contextual and N-gram representations outperforms the traditional skip-gram model.

Short text classification approach to identify child sexual exploitation material

Producing or sharing Child Sexual Exploitation Material (CSEM) is a severe crime that Law Enforcement Agencies (LEAs) fight daily. When the LEA seizes a computer from a potential producer or consumer of the CSEM, it analyzes the storage devices of the suspect looking for evidence. Manual inspection of CSEM is time-consuming given the limited time available for Spanish police to use a search warrant. Our approach to speeding up the identification of CSEM-related files is to analyze only the file names and their absolute paths rather than their content. The main challenge lies in handling short and sparse texts that are deliberately distorted by file owners using obfuscated words and user-defined naming patterns. We present two approaches to CSEM identification. The first employs two independent classifiers, one for the file name and the other for the file path, and their outputs are then combined. Conversely, the second approach uses only the file name classifier to iterate over an absolute path. Both operate at the character n-gram level, whereas novel binary and orthographic features are presented to enrich the text representation. We benchmarked six classification models based on machine learning and convolutional neural networks. The proposed classifier has an F1 score of 0.988, which can be a promising tool for LEAs.

String Kernel-Based Techniques for Native Language Identification

In recent years, Native Language Identification (NLI) has shown significant interest in computational linguistics. NLI uses an author’s speech or writing in a second language to figure out their native language. This may find applications in forensic linguistics, language teaching, second language acquisition, authorship attribution, identification of spam emails or phishing websites, etc. Conventional pairwise string comparison techniques are computationally expensive and time-consuming. This paper presents fast NLI techniques based on string kernels such as spectrum, presence bits, and intersection string kernels incorporating different learners such as a Support Vector Machine (SVM), Random Forest (RF), and Extreme Gradient Boosting-XGBoost (XGB). Feature sets for the proposed techniques are generated using different combinations of features such as n-word grams and noun phrases. Experimental analyses are carried out using 8235 English as a second language articles from 10 different linguistic backgrounds from a typical NLP benchmark dataset. The experimental results show that the proposed NLI technique incorporating a spectrum string kernel with an RF classifier outperformed existing character n-gram string kernels incorporating SVM, RF, and XGB classifiers. Also, comparable results were observed among different combinations of string kernels. Interestingly, the random forest classifier outperformed SVM and XGB classifiers with different feature sets. All the proposed NLI techniques demonstrated promising results with significant improvement in training time, with the best result attaining more than a 95 percent decrease in training time. The reduced training time of proposed techniques makes it well suited to scale NLI applications for production.

Automatic language identification: a case study of Pahari languages

In an attempt to expand the inclusiveness of Natural Language Processing, this paper focuses on developing resources and building machine learning models to identify four languages of the Northern Indo-Aryan family, also known as Pahari languages—Nepali, Garhwali, Kumaoni, and Dogri. This is the first attempt towards building identification models for Pahari languages and developing a plain text corpus for Garhwali and Kumaoni, both of which are lesser-known and under-resourced languages/mother tongues of India. The collected corpus, including data in Nepali and Dogri, is statistically analyzed at the word level. We also trained traditional machine learning models for Pahari language identification on this corpus and found that character n-grams based Linear Support Vector Machines performed best with 99.28% accuracy.

Research on the construction of event logic knowledge graph of supply chain management

Knowledge graph technology plays an important role in knowledge supporting for efficient supply chain management (SCM) of manufacturing enterprises, a SCM knowledge graph can be constructed based on the relevant case corpus. In order to solve the problems of coarse concept granularity of event ontology knowledge in SCM cases, potential words of characters in the existing entity recognition models may not be matched or matched incorrectly, key features of entities with different lengths may interfere with each other in the expression of attention mechanism, and lack of a large number of annotation training samples, a top-down construction method of SCM event logic knowledge graph (ELKG) is proposed. Firstly, SCM event argument classes and class relations are defined, an event logic ontology model is built, and the event argument entities according to event logic ontology are labeled. Then, an active learning event argument entity recognition (EAER) model based on two-stage generative adversarial network (GAN) is proposed. In GAN generator, an EAER model based on binocular attention-based stacked BiLSTM with CNN (BACSBN) is proposed, which combines word-level character feature attention mechanism and n-gram pooling feature attention mechanism to improve the attention to character features of constituent words and highlight entity key information with different lengths, respectively. Two-stage GAN adversarial training and label space attention mechanism are introduced to select the correct predicted label samples for active learning training. The experimental results show that BACSBN can improve the entity recognition accuracy, and the two-stage GAN’s active learning can further improve the recognition effect on the basis of full annotation sample training, and can still maintain a high accuracy in the absence of a large number of manual annotation data. Further, according to the sentence pattern and keyword matching, the matching relations of argument entities are completed, and the SCM ELKG is constructed to provide knowledge support for autonomous SCM.

Transfer and Triangulation Pivot Translation Approaches for Burmese Dialects

Parallel corpora for the languages of Myanmar (Beik, Burmese, Rakhine) are extremely scarce but a necessary requirement for machine translation R&D. Previous studies have proved that pivoting leads to better translation quality if the bridge language is closely related to the source and target language pair. The baseline study is conducted based on the three major approaches of machine translation; Weighted Finite State Transducer (WFST), Phrase-Based Statistical Machine Translation (PBSMT) and Deep Recurrent Neural Network (Deep-RNN). Based on the baseline results, this paper mainly investigated the pivot language technique for PBSMT with Burmese dialects. We employed two different pivot translation methods: transfer (sentence level) and triangulation (phrase level). We present the experimental results on Dawei-Beik, Beik-Dawei translations and Beik-Rakhine, Rakhine-Beik translation via Burmese. Both the transfer and triangulation approaches outperformed the baseline (direct translation), specifically for the Rakhine-Beik language pair. Moreover, the results of the average BiLingual Evaluation Understudy (BLEU), Character n-gram F-score (chrF), and Word Error Rate (WER) scores of the 10-fold cross-validation experiments proved that the triangulation pivot has significantly better acceleration than the transfer pivot. We plan to release the parallel corpora of Burmese dialects and present several avenues for further research.

Applying a Character-Level Model to a Short Arabic Dialect Sentence: A Saudi Dialect as a Case Study

Arabic dialect identification (ADI) has recently drawn considerable interest among researchers in language recognition and natural language processing fields. This study investigated the use of a character-level model that is effectively unrestricted in its vocabulary, to identify fine-grained Arabic language dialects in the form of short written text. The Saudi dialects, particularly the four main Saudi dialects across the country, were considered in this study. The proposed ADI approach consists of five main phases, namely dialect data collection, data preprocessing and labelling, character-based feature extraction, deep learning character-based model/classical machine learning character-based models, and model evaluation performance. Several classical machine learning methods, including logistic regression, stochastic gradient descent, variations of the naive Bayes models, and support vector classification, were applied to the dataset. For the deep learning, the character convolutional neural network (CNN) model was adapted with a bidirectional long short-term memory approach. The collected data were tested under various classification tasks, including two-, three- and four-way ADI tasks. The results revealed that classical machine learning algorithms outperformed the CNN approach. Moreover, the use of the term frequency–inverse document frequency, combined with a character n-grams model ranging from unigrams to four-grams achieved the best performance among the tested parameters.

A comparative study of syllables and character level N-grams for Dravidian multi-script and code-mixed offensive language identification

Curfews and lockdowns around the world in the Covid-19 era have increased the usage of the internet drastically and accordingly the amount of data shared on social media. In addition to using social media for sharing useful information, some miscreants are using the power of social media to spread hate speech and offensive content. Filtering the offensive language content manually is a laborious task due to the huge volume of data. Further, rapid developments in hardware and software technology have provided opportunities for users to post their comments not only in English but also in their native language scripts. However, based on the ease of Roman script usage, social media users specifically in multilingual countries like India, prefer to comment in code-mixed and multi-script texts. The typical systems that are employed to process and analyze monolingual texts are usually not appropriate for these kinds of texts. Further, as these texts do not adhere to the rules and regulations of any language to frame the words and sentences, the complexity of analyzing such texts increases. The novelty of the present study is to address the Offensive Language Identification (OLI) task in code-mixed and multi-script texts, this paper proposes to use relevant syllable and character n-grams features to train Machine Learning (ML) classifiers. The performance of the proposed models is evaluated on three Dravidian language pairs, namely: Malayalam-English, Tamil-English, and Kannada-English. The performances of ML classifiers prove the effectiveness of syllable and character n-grams features for code-mixed and multi-script texts analysis.

Stylistic Fingerprints, POS-tags, and Inflected Languages: A Case Study in Polish

ABSTRACT In stylometric investigations, frequencies of the most frequent words (MFWs) and character n-grams outperform other style-markers, even if their performance varies significantly across languages. In inflected languages, word endings play a prominent role, and hence different word forms cannot be recognized using generic text tokenization. Countless inflected word forms make frequencies sparse, making most statistical procedures complicated. Presumably, applying one of the NLP techniques, such as lemmatization and/or parsing, might increase the performance of classification. The aim of this paper is to examine the usefulness of grammatical features (as assessed via POS-tag n-grams) and lemmatized forms in recognizing authorial profiles, in order to address the underlying issue of the degree of freedom of choice within lexis and grammar. Using a corpus of Polish novels, we performed a series of supervised authorship attribution benchmarks, in order to compare the classification accuracy for different types of lexical and syntactic style-markers. Even if the performance of POS-tags as well as lemmatized forms was notoriously worse than that of lexical markers, the difference was not substantial and never exceeded ca. 15%.

Similarity Identification Based on Word Trigrams Using Exact String Matching Algorithms

Several studies regarding excellent exact string matching algorithms can be used to identify similarity, including the Rabin-Karp, Winnowing, and Horspool Boyer-Moore algorithms. In determining similarities, the Rabin-Karp and Winnowing algorithms use fingerprints, while the Horspool Boyer-Moore algorithm uses a bad-character table. However, previous research focused on identifying similarities using these algorithms based on character n-gram. In contrast, identification based on the word n-gram to determine the similarity based on its linguistic meaning, especially for longer strings, had not been covered yet. Therefore, a word-level trigram was proposed to identify similarities based on the word trigrams using the three algorithms and compare each performance. Based on precision, recall, and running time comparison, the Rabin-Karp algorithm results were 100%, 100%, and 0.19 ms, respectively; the Winnowing algorithm results with the smallest window were 100%, 56%, and 0.18 ms, respectively; and the Horspool algorithm results were 100%, 100%, and 0.06 ms. From these results, it can be concluded that the performance of the Horspool Boyer-Moore algorithm is better in terms of precision, recall, and running time.

Improving neural machine translation with POS-tag features for low-resource language pairs

Integrating linguistic features has been widely utilized in statistical machine translation (SMT) systems, resulting in improved translation quality. However, for low-resource languages such as Thai and Myanmar, the integration of linguistic features in neural machine translation (NMT) systems has yet to be implemented. In this study, we propose transformer-based NMT models (transformer, multi-source transformer, and shared-multi-source transformer models) using linguistic features for two-way translation of Thai-to-Myanmar, Myanmar-to-English, and Thai-to-English. Linguistic features such as part-of-speech (POS) tags or universal part-of-speech (UPOS) tags are added to each word on either the source or target side, or both the source and target sides, and the proposed models are conducted. The multi-source transformer and shared-multi-source transformer models take two inputs (i.e., string data and string data with POS tags) and produce string data or string data with POS tags. A transformer model that utilizes only word vectors was used as the first baseline model for comparison with the proposed models. The second baseline model, an Edit-Based Transformer with Repositioning (EDITOR) model, was also used to compare with our proposed models in addition to the baseline transformer model. The findings of the experiments show that adding linguistic features to the transformer-based models enhances the performance of a neural machine translation in low-resource language pairs. Moreover, the best translation results were yielded using shared-multi-source transformer models with linguistic features resulting in more significant Bilingual Evaluation Understudy (BLEU) scores and character n-gram F-score (chrF) scores than the baseline transformer and EDITOR models.

Post-OCR Document Correction with Large Ensembles of Character Sequence-to-Sequence Models

In this paper, we propose a novel method to extend sequence-to-sequence models to accurately process sequences much longer than the ones used during training while being sample- and resource-efficient, supported by thorough experimentation. To investigate the effectiveness of our method, we apply it to the task of correcting documents already processed with Optical Character Recognition (OCR) systems using sequence-to-sequence models based on characters. We test our method on nine languages of the ICDAR 2019 competition on post-OCR text correction and achieve a new state-of-the-art performance in five of them. The strategy with the best performance involves splitting the input document in character n-grams and combining their individual corrections into the final output using a voting scheme that is equivalent to an ensemble of a large number of sequence models. We further investigate how to weigh the contributions from each one of the members of this ensemble. Our code for post-OCR correction is shared at https://github.com/jarobyte91/post_ocr_correction.

Unsupervised authorship attribution using feature selection and weighted cosine similarity

This paper presents a computational model for the unsupervised authorship attribution task based on a traditional machine learning scheme. An improvement over the state of the art is achieved by comparing different feature selection methods on the PAN17 author clustering dataset. To achieve this improvement, specific pre-processing and features extraction methods were proposed, such as a method to separate tokens by type to assign them to only one category. Similarly, special characters are used as part of the punctuation marks to improve the result obtained when applying typed character n-grams. The Weighted cosine similarity measure is applied to improve the B3 F-score by reducing the vector values where attributes are exclusive. This measure is used to define distances between documents, which later are occupied by the clustering algorithm to perform authorship attribution.

Confusion2Vec 2.0: Enriching ambiguous spoken language representations with subwords.

Word vector representations enable machines to encode human language for spoken language understanding and processing. Confusion2vec, motivated from human speech production and perception, is a word vector representation which encodes ambiguities present in human spoken language in addition to semantics and syntactic information. Confusion2vec provides a robust spoken language representation by considering inherent human language ambiguities. In this paper, we propose a novel word vector space estimation by unsupervised learning on lattices output by an automatic speech recognition (ASR) system. We encode each word in Confusion2vec vector space by its constituent subword character n-grams. We show that the subword encoding helps better represent the acoustic perceptual ambiguities in human spoken language via information modeled on lattice-structured ASR output. The usefulness of the proposed Confusion2vec representation is evaluated using analogy and word similarity tasks designed for assessing semantic, syntactic and acoustic word relations. We also show the benefits of subword modeling for acoustic ambiguity representation on the task of spoken language intent detection. The results significantly outperform existing word vector representations when evaluated on erroneous ASR outputs, providing improvements up-to 13.12% relative to previous state-of-the-art in intent detection on ATIS benchmark dataset. We demonstrate that Confusion2vec subword modeling eliminates the need for retraining/adapting the natural language understanding models on ASR transcripts.

Integrating Character-level and Word-level Representation for Affect in Arabic Tweets

Affect tasks, which range from sentiment polarity classification to finer grained sentiment strength and emotional intensity detection, have become of increasing interest due to the vast amount of user-generated content and advanced learning models. Word representation models have been leveraged effectively within a variety of natural language processing tasks. However, these models are not always effective in the context of social media. When dealing with social media posts in Arabic, the use of Arabic dialects needs to be considered. Although using informal text to train word-level models can lead to the identification of words that convey the same meaning, these models are unable to capture the full extent of the words that are used in the real world due to out-of-vocabulary (OOV) words. The inability to identify such words is one of the main limitations of word-level models. One approach of overcoming OOV is through the use of character-level embeddings as they can effectively learn the vectors of word parts or character n-grams. This study uses a combination of character-level and word-level models to identify the most effective methods by which affective Arabic words in tweets can be represented semantically and morphologically. We evaluate our generated models and the proposed method by integrating them in a supervised learning framework that was used for a range of affect tasks and other related tasks. Our findings reveal that the developed models surpassed the performance of state-of-the-art Arabic pre-trained word embeddings over eight datasets. In addition, our models enhance previous state-of-the-art outcomes on tasks involving Arabic emotion intensity, outperforming the top-systems that used advanced ensemble learning models and several additional features.

Offensive-Language Detection on Multi-Semantic Fusion Based on Data Augmentation

The rapid identification of offensive language in social media is of great significance for preventing viral spread and reducing the spread of malicious information, such as cyberbullying and content related to self-harm. In existing research, the public datasets of offensive language are small; the label quality is uneven; and the performance of the pre-trained models is not satisfactory. To overcome these problems, we proposed a multi-semantic fusion model based on data augmentation (MSF). Data augmentation was carried out by back translation so that it reduced the impact of too-small datasets on performance. At the same time, we used a novel fusion mechanism that combines word-level semantic features and n-grams character features. The experimental results on the two datasets showed that the model proposed in this study can effectively extract the semantic information of offensive language and achieve state-of-the-art performance on both datasets.