Stylometric Features Research Articles

Enhancing phishing email detection with stylometric features and classifier stacking

Enhancing phishing email detection with stylometric features and classifier stacking

Investigating the Linguistic Fingerprint of GPT-4o in Arabic-to-English Translation Using Stylometry

This study explores the linguistic and stylistic characteristics of machine-generated texts, focusing on the output of GPT-4o. Using various natural language processing (NLP) techniques, including word frequency and stopword count analysis, readability and sentence structure metrics, lexical diversity measures, syntactic frequency analysis, and named entity recognition (NER), the research aims to uncover the stylometric fingerprints present in machine-generated content. The results reveal that GPT-4ogenerated texts exhibit moderate lexical diversity and syntactic complexity, with certain chapters reflecting higher readability and more varied sentence structures, while others lean toward simpler linguistic patterns. The findings also highlight thematic variation across chapters, as observed in the distribution of named entities, which contributes to understanding the model’s handling of different contextual content. The research suggests that while GPT-4o maintains a consistent style in its generated text, there are distinguishable characteristics that may serve as indicators of machine authorship. This provides valuable insights for stylometric analysis, authorship attribution, and the identification of machine-generated texts in various contexts. Future research could extend this work by exploring deeper stylometric features, conducting cross-model comparisons, and developing advanced authorship detection algorithms tailored for AI-generated content. Moreover, the ethical implications of stylometric analysis in the context of AI-generated texts warrant further investigation, particularly as machine-generated content becomes increasingly prevalent across different domains.

Genre Classification of Books in Russian with Stylometric Features: A Case Study

Within the literary domain, genres function as fundamental organizing concepts that provide readers, publishers, and academics with a unified framework. Genres are discrete categories that are distinguished by common stylistic, thematic, and structural components. They facilitate the categorization process and improve our understanding of a wide range of literary expressions. In this paper, we introduce a new dataset for genre classification of Russian books, covering 11 literary genres. We also perform dataset evaluation for the tasks of binary and multi-class genre identification. Through extensive experimentation and analysis, we explore the effectiveness of different text representations, including stylometric features, in genre classification. Our findings clarify the challenges present in classifying Russian literature by genre, revealing insights into the performance of different models across various genres. Furthermore, we address several research questions regarding the difficulty of multi-class classification compared to binary classification, and the impact of stylometric features on classification accuracy.

Unveiling ChatGPT text using writing style

Extensive use of AI-generated texts culminated recently after the advent of large language models. Although the use of AI text generators, such as ChatGPT, is beneficial, it also threatens the academic level as students may resort to it. In this work, we propose a technique leveraging the intrinsic stylometric features of documents to detect ChatGPT-based plagiarism. The stylometric features were normalized and fed to classical classifiers, such as k-Nearest Neighbors, Decision Tree, and Naïve Bayes, as well as ensemble classifiers, such as XGBoost and Stacking. A thorough examination of the classifier was conducted by using Cross-Fold validation, hyperparameters tuning, and multiple training iterations. The results show the efficacy of both classical and ensemble learning classifiers in distinguishing between human and ChatGPT writing styles with a noteworthy performance of XGBoost where 100 % was achieved for accuracy, recall, and precision metrics. Moreover, the proposed XGBoost classifier outperformed the state-of-the-art result on the same dataset and same classifier highlighting the superiority of the proposed feature style extraction method over TF-IDF techniques. The ensemble learning classifiers were also applied to the generated dataset with mixed texts, where paragraphs are written by ChatGPT and humans. The results show that 98 % of the documents were classified correctly as either mixed or human. The last contribution consists in the authorship attribution of the paragraphs of a single document where the accuracy reached 92.3 %.

Crossing Linguistic Barriers: Authorship Attribution in Sinhala Texts

Authorship attribution involves determining the original author of an anonymous text from a pool of potential authors. The author attribution task has applications in several domains, such as plagiarism detection, digital text forensics, and information retrieval. While these applications extend beyond any single language, existing research has predominantly centered on English, posing challenges for application in languages such as Sinhala due to linguistic disparities and a lack of language processing tools. We present the first comprehensive study on cross-topic authorship attribution for Sinhala texts and propose a solution that can effectively perform the authorship attribution task even if the topics within the test and training samples differ. Our solution consists of three main parts: (i) extraction of topic-independent stylometric features, (ii) generation of a small candidate author set with the help of similarity search, and (iii) identification of the true author. Several experimental studies were carried out to demonstrate that the proposed solution can effectively handle real-world scenarios involving a large number of candidate authors and a limited number of text samples for each candidate author.

ALISON: Fast and Effective Stylometric Authorship Obfuscation

Authorship Attribution (AA) and Authorship Obfuscation (AO) are two competing tasks of increasing importance in privacy research. Modern AA leverages an author's consistent writing style to match a text to its author using an AA classifier. AO is the corresponding adversarial task, aiming to modify a text in such a way that its semantics are preserved, yet an AA model cannot correctly infer its authorship. To address privacy concerns raised by state-of-the-art (SOTA) AA methods, new AO methods have been proposed but remain largely impractical to use due to their prohibitively slow training and obfuscation speed, often taking hours. To this challenge, we propose a practical AO method, ALISON, that (1) dramatically reduces training/obfuscation time, demonstrating more than 10x faster obfuscation than SOTA AO methods, (2) achieves better obfuscation success through attacking three transformer-based AA methods on two benchmark datasets, typically performing 15% better than competing methods, (3) does not require direct signals from a target AA classifier during obfuscation, and (4) utilizes unique stylometric features, allowing sound model interpretation for explainable obfuscation. We also demonstrate that ALISON can effectively prevent four SOTA AA methods from accurately determining the authorship of ChatGPT-generated texts, all while minimally changing the original text semantics. To ensure the reproducibility of our findings, our code and data are available at: https://github.com/EricX003/ALISON.

Can we spot fake public comments generated by ChatGPT(-3.5, -4)?: Japanese stylometric analysis expose emulation created by one-shot learning.

Public comments are an important opinion for civic when the government establishes rules. However, recent AI can easily generate large quantities of disinformation, including fake public comments. We attempted to distinguish between human public comments and ChatGPT-generated public comments (including ChatGPT emulated that of humans) using Japanese stylometric analysis. Study 1 conducted multidimensional scaling (MDS) to compare 500 texts of five classes: Human public comments, GPT-3.5 and GPT-4 generated public comments only by presenting the titles of human public comments (i.e., zero-shot learning, GPTzero), GPT-3.5 and GPT-4 emulated by presenting sentences of human public comments and instructing to emulate that (i.e., one-shot learning, GPTone). The MDS results showed that the Japanese stylometric features of the public comments were completely different from those of the GPTzero-generated texts. Moreover, GPTone-generated public comments were closer to those of humans than those generated by GPTzero. In Study 2, the performance levels of the random forest (RF) classifier for distinguishing three classes (human, GPTzero, and GPTone texts). RF classifiers showed the best precision for the human public comments of approximately 90%, and the best precision for the fake public comments generated by GPT (GPTzero and GPTone) was 99.5% by focusing on integrated next writing style features: phrase patterns, parts-of-speech (POS) bigram and trigram, and function words. Therefore, the current study concluded that we could discriminate between GPT-generated fake public comments and those written by humans at the present time.

Text classification by CEFR levels using machine learning methods and BERT language model

This paper presents a study of the problem of automatic classification of short coherent texts (essays) in English according to the levels of the international CEFR scale. Determining the level of text in natural language is an important component of assessing students knowledge, including checking open tasks in e-learning systems. To solve this problem, vector text models were considered based on stylometric numerical features of the character, word, sentence structure levels. The classification of the obtained vectors was carried out by standard machine learning classifiers. The article presents the results of the three most successful ones: Support Vector Classifier, Stochastic Gradient Descent Classifier, LogisticRegression. Precision, recall and F-score served as quality measures. Two open text corpora, CEFR Levelled English Texts and BEA-2019, were chosen for the experiments. The best classification results for six CEFR levels and sublevels from A1 to C2 were shown by the Support Vector Classifier with F-score 67 % for the CEFR Levelled English Texts. This approach was compared with the application of the BERT language model (six different variants). The best model, bert-base-cased, provided the F-score value of 69 %. The analysis of classification errors showed that most of them are between neighboring levels, which is quite understandable from the point of view of the domain. In addition, the quality of classification strongly depended on the text corpus, that demonstrated a significant difference in F-scores during application of the same text models for different corpora. In general, the obtained results showed the effectiveness of automatic text level detection and the possibility of its practical application.

Distinguishing ChatGPT(-3.5, -4)-generated and human-written papers through Japanese stylometric analysis.

In the first half of 2023, text-generative artificial intelligence (AI), including ChatGPT from OpenAI, has attracted considerable attention worldwide. In this study, first, we compared Japanese stylometric features of texts generated by ChatGPT, equipped with GPT-3.5 and GPT-4, and those written by humans. In this work, we performed multi-dimensional scaling (MDS) to confirm the distributions of 216 texts of three classes (72 academic papers written by 36 single authors, 72 texts generated by GPT-3.5, and 72 texts generated by GPT-4 on the basis of the titles of the aforementioned papers) focusing on the following stylometric features: (1) bigrams of parts-of-speech, (2) bigram of postpositional particle words, (3) positioning of commas, and (4) rate of function words. MDS revealed distinct distributions at each stylometric feature of GPT (3.5 and 4) and human. Although GPT-4 is more powerful than GPT-3.5 because it has more parameters, both GPT (3.5 and 4) distributions are overlapping. These results indicate that although the number of parameters may increase in the future, GPT-generated texts may not be close to that written by humans in terms of stylometric features. Second, we verified the classification performance of random forest (RF) classifier for two classes (GPT and human) focusing on Japanese stylometric features. This study revealed the high performance of RF in each stylometric feature: The RF classifier focusing on the rate of function words achieved 98.1% accuracy. Furthermore the RF classifier focusing on all stylometric features reached 100% in terms of all performance indexes (accuracy, recall, precision, and F1 score). This study concluded that at this stage we human discriminate ChatGPT from human limited to Japanese language.

TDRLM: Stylometric learning for authorship verification by Topic-Debiasing

Authorship verification is the task of determining if a candidate is the real author of the provided sequences of texts. Stylometric learning is the study of the writing style by analyzing the textual data through computational approaches for many linguistic tasks such as authorship verification, authorship identification, etc. In recent years, Deep learning has been used as an effective solution to stylometric learning for authorship verification. However, most of the existing studies do not consider the semantic biases captured by the model, and the learned stylometric features include a strong indicator of the topical biases. During the extraction of stylometric features, we found that specific words related to the latent topics impact the performance of machine learning algorithms and, therefore, are not generalizable toward out-of-sample authors. In this paper, we propose a latent topic score dictionary with a Topic-Debiasing Representation Learning Model (TDRLM) for stylometric representation learning on the problem of authorship verification. The model applies position-specific topic scores on a topic-debiasing attention mechanism in order to adjust the tokenized texts based on the topical bias. The experimental results show that our proposed approach achieves the best performance with the highest Area Under Curve (AUC) of 92.47% for the Twitter-Foursquare dataset and 93.11% for the (ICWSM) Twitter dataset, both better than the state-of-the-art stylometric learning models and the up-trained language models.

SSM: Stylometric and semantic similarity oriented multimodal fake news detection

Over the years, there has been a rise in the number of fabricated and fake news stories that utilize both textual and visual information formats. This coincides with the increased likelihood that users will acquire their news from websites and social media platforms. While there has been various research into the detection of fake news in text using machine learning techniques, less attention has been paid to the problem of multimedia data fabrication. In this paper, we propose a Stylometric, and Semantic similarity oriented for Multimodal Fake News Detection (SSM). There are five distinct modules that make up our methodology: Firstly, we used a Hyperbolic Hierarchical Attention Network (Hype-HAN) for extracting stylometric textual features. Secondly, we generated the news content summary and computed the similarity between Headline and summary. Thirdly, semantic similarity is computed between visual and textual features. Fourthly, images are analyzed for forgery. Lastly, the extracted features are fused for final classification. We have tested SSM framework on three standard fake news datasets. The results indicated that our suggested model has outperformed the base line and state-of-the-art methods and is more likely to detect fake news in complex environments.

Stylometric Appraisal of Coelho’s Selected Translated Novels: A Digital Humanities Study

To quantify the style markers of any writer is a laborious and erroneous task in close reading, while distant reading with the Voyant Summary tool presents stylometric features precisely. The current study employs the Voyant summary tool to analyse data through the lens of Stefan Sinclair and Geoffery Rockwell's Hermeneutica Theory (2016). The Voyant Summary tool performs text mining of Coelho's style markers; total words, unique words, vocabulary density, the average length of sentences and the most frequent words. Coelho's vocabulary density ranges from 0.075 to 0.263 words. Colhelo’s average length of sentences ranges from 13 to 26.2 words per sentence. The dialogic writing style is evident with the word 'said’. Moreover, themes of love and life are the most repeated ones. To enhance the readability of his works, he refrains from archaic vocabulary. The Zahir shows the lowest vocabulary density, while Adultery reveals the highest vocabulary density.

Weight of authorship evidence with multiple categories of stylometric features: A multinomial-based discrete model

This study empirically demonstrates the efficacy of a two-level Dirichlet-multinomial statistical model (the Multinomial system) for computing likelihood ratios (LR) for linguistic, textual evidence with multiple stylometric feature types with discrete values. The LRs are calculated separately for each feature type, namely, word, character and part of speech N-grams (N = 1,2,3), which are combined as overall LRs through logistic regression fusion. The Multinomial system’s performance is compared with that of a previously proposed system with the cosine distance (the Cosine system) using the same data (i.e., documents collated from 2160 authors). The experimental results show that: (1) the Multinomial system outperforms the Cosine system with the fused feature types by a log-LR cost of ca. 0.01 ∼ 0.05 bits; and (2) the Multinomial system is more advantageous in performance with longer documents than the Cosine system. Although the Cosine system is more robust overall against the sampling variability arising from the number of authors included in the reference and calibration databases, the Multinomial system can achieve reasonable stability in performance; for example, the standard deviation value of the log-LR cost becomes lower than 0.01 (10 random samplings of authors for the reference and calibration databases) with 60 or more authors in each database.

Fellow or foe? A quantitative thematic exploration into Putin’s and Trump’s stylometric features

Thematic concentration, a quantitative linguistic method, can reflect the speech style of a particular person. It may, to some degree, reflect the degree of a speaker’s intention to communicate certain themes. There has been limited empirical research on the similarity between Trump and Putin with respect to their linguistic features. Thus, the present study aims to compare Putin’s and Trump’s stylometric features and political themes based on thematic concentration with a corpus of Putin’s, Medvedev’s, Trump’s, and Obama’s speeches. Results show that 1) Both Putin’s and Trump’s speeches’ thematic concentration values are significantly or marginally significantly different from their precedents’. 2) Two leaders pay great attention to the concept of nationalism. 3) Thematic words of their speeches were slightly different across periods, reflecting the influence of external factors on the choice of language. The results of the present study may shed light on the stylometric studies of Putin and Trump.

A Survey on Authorship Analysis Tasks and Techniques

Abstract Authorship Analysis (AA) is a natural language processing field that examines the previous works of writers to identify the author of a text based on its features. Studies in authorship analysis include authorship identification, authorship profiling, and authorship verification. Due to its relevance, to many applications in this field attention has been paid. It is widely used in the attribution of historical literature. Other applications include legal linguistics, criminal law, forensic investigations, and computer forensics. This paper aims to provide an overview of the work done and the techniques applied in the authorship analysis domain. The examination of recent developments in this field is the principal focus. Many different criteria can be used to define a writer’s style. This paper investigates stylometric features in different author-related tasks, including lexical, syntactic, semantic, structural, and content-specific ones. A lot of classification methods have been applied to authorship analysis tasks. We examine many research studies that use different machine learning and deep learning techniques. As a means of pointing the direction for future studies, we present the most relevant methods recently proposed. The reviewed studies include documents of different types and different languages. In summary, due to the fact that each natural language has its own set of features, there is no standard technique generically applicable for solving the AA problem.

An Optimal Feature Set for Stylometry-based Style Change detection at Document and Sentence Level

Writing style change detection models focus on determining the number of authors of documents with or without known authors. Determining the exact number of authors contributing in writing a document particularly when the authors contribute short texts in form of a sentence is still challenging because of the lack of standardized feature sets able to discriminate between the works of authors. Therefore, the task of identifying the best feature set for all the tasks of the writing style change detection is still considered important. This paper sought to determine the best feature set for the writing style change detection tasks; separating documents with several style changes (multi-authorship) from documents without any style changes (single-authorship), and determining the number and location of style changes in the case of multi-authorship. We performed exploratory research on existing stylometric features to determine the best document level and sentence level features. Document level features were extracted and used to separate single authored from multi-authored documents, while sentence level features were used to answer the question of determining the number of style changes To answer this question, we trained a random forest classifier to rank document level features and sentence level features separately, and applied an ablation test on the top 15 sentence level features using k-means clustering algorithm to confirm the effect of these features on model performance. The study found out that the best document level feature set for separating documents with and without style change was provided by an ensemble of features including number of sentence repetitions (num_sentence_repetitions) as the most determinant feature, 5-grams, 4-grams, Special_character, sentence_begin_lower, sentence_begin_upper, diversity, automated_readability_index, parenthesis_count, first_word_uppercase, lensear_write_formula, dale_chall_readability, difficult_words, type_token_ratio. These were the top ranked features in experiment one. On the other hand, the top fifteen sentence level features based on feature ranks using random forest classifier were diversity, dale_chall_readability grade, check_available_vowel, flesch_kincaid grade, parenthesis_count, colon_count, verbs, bigrams, alphabets, personal pronouns, coordinating conjunctions, interjections, modals, type_token ratio and punctuations_count. Consequently, the optimal feature set for determining the number of style changes in documents was considered based on the results of the ablation study on the top fifteen sentence level features, and was provided by an ensemble of features including personal pronouns, check_available_vowels, punctuations_counts, parenthesis count, coordinating conjunctions and colon count.

Translator attribution for Arabic using machine learning

AbstractGiven a set of target language documents and their translators, the translator attribution task aims at identifying which translator translated which documents. The attribution and the identification of the translator’s style could contribute to fields including translation studies, digital humanities, and forensic linguistics. To conduct this investigation, firstly, we develop a new corpus containing the translations of world-famous books into Arabic. We then pre-process the books in our corpus which mainly involves cleaning irrelevant material, morphological segmentation analysis of words, and devocalization. After pre-processing the books, we propose to use 100 most frequent words and/or morphologically segmented function words as writing style markers of the translators (i.e. stylometric features) to differentiate between translations of different translators. After the completion of features extraction process, we applied several supervised and unsupervised machine-learning algorithms along with our novel cluster-to-author index to perform this task. We found that the translators are not invisible, and morphological analysis may not be more useful than just using the 100 most frequent words as features. The support vector machine linear kernel algorithm reported 99% classification accuracy. Similar findings were reported by the unsupervised machine-learning methods, namely, K-mean clustering and hierarchical clustering.

A Comprehensive Study of Learning Approaches for Author Gender Identification

In recent years, author gender identification is an important yet challenging task in the fields of information retrieval and computational linguistics. In this paper, different learning approaches are presented to address the problem of author gender identification for Turkish articles. First, several classification algorithms are applied to the list of representations based on different paradigms: fixed-length vector representations such as Stylometric Features (SF), Bag-of-Words (BoW) and distributed word/document embeddings such as Word2vec, fastText and Doc2vec. Secondly, deep learning architectures, Convolution Neural Network (CNN), Recurrent Neural Network (RNN), special kinds of RNN such as Long-Short Term Memory (LSTM) and Gated Recurrent Unit (GRU), C-RNN, Bidirectional LSTM (bi-LSTM), Bidirectional GRU (bi-GRU), Hierarchical Attention Networks and Multi-head Attention (MHA) are designated and their comparable performances are evaluated. We conducted a variety of experiments and achieved outstanding empirical results. To conclude, ML algorithms with BoW have promising results. fast-Text is also probably suitable between embedding models. This comprehensive study contributes to literature utilizing different learning approaches based on several ways of representations. It is also first important attempt to identify author gender applying SF, fastText and DNN architectures to the Turkish language.

Stylometric features in librarian’s responses to user queries: implications for user interaction in digital information services

PurposeThe purpose of this study is to explore linguistic stylometric patterns encompassing lexical, syntactic, structural, sentiment and politeness features that are found in librarians’ responses to user queries.Design/methodology/approachA total of 462 online texts/transcripts comprising answers of librarians to users’ questions drawn from the Internet Public Library were examined. A Principal Component Analysis, which is a data reduction technique, was conducted on the texts and transcripts. Data analysis illustrates the three principal components that predominantly occur in librarians’ answers: stylometric richness, stylometric brevity and interpersonal support.FindingsThe results of the study have important implications in digital information services because stylometric features such as lexical richness, structural clarity and interpersonal support may interplay with the degree of complexity of user queries, the (a)synchronous communication mode, application of information service guideline and manuals and overall characteristics and quality of a given digital information service. Such interplay may bring forth a direct impact on user perceptions and satisfaction regarding interaction with librarians and the information service received through the computer-mediated communication channel.Originality/valueTo the best of the authors’ knowledge, the stylometric features encompassing lexical, syntactic, structural, sentiment and politeness using Principal Component Analysis have not been explored in digital information/reference services. Thus, there is an emergent need to explore more fully how linguistic stylometric features interplay with the types of user queries, the asynchronous online communication mode, application of information service guidelines and the quality of a particular digital information service.

Post-Authorship Attribution Using Regularized Deep Neural Network

Post-authorship attribution is a scientific process of using stylometric features to identify the genuine writer of an online text snippet such as an email, blog, forum post, or chat log. It has useful applications in manifold domains, for instance, in a verification process to proactively detect misogynistic, misandrist, xenophobic, and abusive posts on the internet or social networks. The process assumes that texts can be characterized by sequences of words that agglutinate the functional and content lyrics of a writer. However, defining an appropriate characterization of text to capture the unique writing style of an author is a complex endeavor in the discipline of computational linguistics. Moreover, posts are typically short texts with obfuscating vocabularies that might impact the accuracy of authorship attribution. The vocabularies include idioms, onomatopoeias, homophones, phonemes, synonyms, acronyms, anaphora, and polysemy. The method of the regularized deep neural network (RDNN) is introduced in this paper to circumvent the intrinsic challenges of post-authorship attribution. It is based on a convolutional neural network, bidirectional long short-term memory encoder, and distributed highway network. The neural network was used to extract lexical stylometric features that are fed into the bidirectional encoder to extract a syntactic feature-vector representation. The feature vector was then supplied as input to the distributed high networks for regularization to minimize the network-generalization error. The regularized feature vector was ultimately passed to the bidirectional decoder to learn the writing style of an author. The feature-classification layer consists of a fully connected network and a SoftMax function to make the prediction. The RDNN method was tested against thirteen state-of-the-art methods using four benchmark experimental datasets to validate its performance. Experimental results have demonstrated the effectiveness of the method when compared to the existing state-of-the-art methods on three datasets while producing comparable results on one dataset.