Word N-grams Research Articles

Detection of Cyberbullying Patterns in Low Resource Colloquial Roman Urdu Microtext using Natural Language Processing, Machine Learning, and Ensemble Techniques

Social media platforms have become a substratum for people to enunciate their opinions and ideas across the globe. Due to anonymity preservation and freedom of expression, it is possible to humiliate individuals and groups, disregarding social etiquette online, inevitably proliferating and diversifying the incidents of cyberbullying and cyber hate speech. This intimidating problem has recently sought the attention of researchers and scholars worldwide. Still, the current practices to sift the online content and offset the hatred spread do not go far enough. One factor contributing to this is the recent prevalence of regional languages in social media, the dearth of language resources, and flexible detection approaches, specifically for low-resource languages. In this context, most existing studies are oriented towards traditional resource-rich languages and highlight a huge gap in recently embraced resource-poor languages. One such language currently adopted worldwide and more typically by South Asian users for textual communication on social networks is Roman Urdu. It is derived from Urdu and written using a Left-to-Right pattern and Roman scripting. This language elicits numerous computational challenges while performing natural language preprocessing tasks due to its inflections, derivations, lexical variations, and morphological richness. To alleviate this problem, this research proposes a cyberbullying detection approach for analyzing textual data in the Roman Urdu language based on advanced preprocessing methods, voting-based ensemble techniques, and machine learning algorithms. The study has extracted a vast number of features, including statistical features, word N-Grams, combined n-grams, and BOW model with TFIDF weighting in different experimental settings using GridSearchCV and cross-validation techniques. The detection approach has been designed to tackle users’ textual input by considering user-specific writing styles on social media in a colloquial and non-standard form. The experimental results show that SVM with embedded hybrid N-gram features produced the highest average accuracy of around 83%. Among the ensemble voting-based techniques, XGboost achieved the optimal accuracy of 79%. Both implicit and explicit Roman Urdu instances were evaluated, and the categorization of severity based on prediction probabilities was performed. Time complexity is also analyzed in terms of execution time, indicating that LR, using different parameters and feature combinations, is the fastest algorithm. The results are promising with respect to standard assessment metrics and indicate the feasibility of the proposed approach in cyberbullying detection for the Roman Urdu language.

FreCDo: A Large Corpus for French Cross-Domain Dialect Identification

We present a novel corpus for French dialect identification comprising 413,522 French text samples collected from public news websites in Belgium, Canada, France and Switzerland. To ensure an accurate estimation of the dialect identification performance of models, we designed the corpus to eliminate potential biases related to topic, writing style, and publication source. More precisely, the training, validation and test splits are collected from different news websites, while searching for different keywords (topics). This leads to a French cross-domain (FreCDo) dialect identification task. We conduct experiments with four competitive baselines, a fine-tuned CamemBERT model, an XGBoost based on fine-tuned CamemBERT features, a Support Vector Machines (SVM) classifier based on fine-tuned CamemBERT features, and an SVM based on word n-grams. Aside from presenting quantitative results, we also make an analysis of the most discriminative features learned by CamemBERT. Our corpus is available in open-source format at https://github.com/MihaelaGaman/FreCDo.

Identification of offensive language in Urdu using semantic and embedding models.

Automatic identification of offensive/abusive language is very necessary to get rid of unwanted behavior. However, it is more challenging to generalize the solution due to the different grammatical structures and vocabulary of each language. Most of the prior work targeted western languages, however, one study targeted a low-resource language (Urdu). The prior study used basic linguistic features and a small dataset. This study designed a new dataset (collected from popular Pakistani Facebook pages) containing 7,500 posts for offensive language detection in Urdu. The proposed methodology used four types of feature engineering models: three are frequency-based and the fourth one is the embedding model. Frequency-based are either determined by the term frequency-inverse document frequency (TF-IDF) or bag-of-words or word n-gram feature vectors. The fourth is generated by the word2vec model, trained on the Urdu embeddings using a corpus of 196,226 Facebook posts. The experiments demonstrate that the stacking-based ensemble model with word2vec shows the best performance as a standalone model by achieving 88.27% accuracy. In addition, the wrapper-based feature selection method further improves performance. The hybrid combination of TF-IDF, bag-of-words, and word2vec feature models achieved 90% accuracy and 97% AUC. In addition, it outperformed the baseline with an improvement of 3.55% in accuracy, 3.68% in the recall, 3.60% in f1-measure, 3.67% in precision, and 2.71% in AUC. The findings of this research provide practical implications for commercial applications and future research.

Big-Data-Based Text Mining and Social Network Analysis of Landscape Response to Future Environmental Change

Climate change impacts the urban environment and landscape changes worldwide. To understand how South Korean citizens perceive these changes and what they expect for the future landscape, this study analyzed urban residents’ perceptions through text mining. Data related to the keywords “future landscape”, “future environment”, “well-being”, and “climate change” were collected from July 2020 to July 2021 from the Korean search engines Naver, Daum, and Google using the tool TEXTOM. Keywords, importance, and related words were derived through word frequency, TF-IDF, and N-gram analysis. CONCOR analysis was used to derive the meaning and relevance of the words. In “future landscape”, results showed a high frequency of the words “complex”, “apartment”, “future value”, and “sale”, and the connection strength was higher between “complex”, “landscape”, and “future value”. In “future environment”, “eco-friendly” showed the highest word frequency, and the words “New Deal”, “hydrogen” and “mobility” showed a high frequency and correlation. For “well-being”, “Green Cross” (a well-being-related company) showed the highest frequency, and the connection strength between satisfaction indexes was high. For “climate change”, “response” showed the highest frequency, and the connection strength between “carbon-neutral”, “UN-convention”, and “plan-establishment” was high. These results showed that South Koreans associate landscaping with the value of apartment complexes, that they expect solutions to mitigate climate change impacts with green and eco-friendly strategies, and lastly that well-being-related companies are receiving a great deal of public attention. Thus, it is expected that the results will help plan effective landscaping approaches to respond to environmental changes.

Optimal alphabet for single text compression

A text written using symbols from a given alphabet can be compressed using the Huffman code, which minimizes the length of the encoded text. It is necessary, however, to employ a text-specific codebook, i.e. the symbol-codeword dictionary, to decode the original text. Thus, the compression performance should be evaluated by the full code length, i.e. the length of the encoded text plus the length of the codebook. We studied several alphabets for compressing texts – letters, n-grams of letters, syllables, words, and phrases. If only sufficiently short texts are retained, an alphabet of letters or two-grams of letters is optimal. For the majority of Project Gutenberg texts, the best alphabet (the one that minimizes the full code length) is given by syllables or words, depending on the representation of the codebook. Letter 3 and 4-grams, having on average comparable length to syllables/words, perform noticeably worse than syllables or words. Word 2-grams also are never the best alphabet, on the account of having a very large codebook. We also show that the codebook representation is important – switching from a naive representation to a compact one significantly improves the matters for alphabets with large number of symbols, most notably the words.Thus, meaning-expressing elements of the language (syllables or words) provide the best compression alphabet.

직업성 피부질환의 텍스트마이닝, CONCOR 분석을 통한 연구동향 분석

Purpose: The purpose of this study is to analyze occupational skin diseases and construction skin diseases in Korea using word frequency analysis, N-gram analysis, and CONCOR analysis to examine the research trends of occupational skin diseases. Method: The research method was conducted in a total of five stages, followed by data collection, data preprocessing, word frequency analysis, N-gram analysis, and CONCOR analysis. On June 1, 2022, 66 search results were analyzed under the theme of occupational and constructive skin diseases, and duplicate papers, contents unrelated to research purposes, and data with low relevance were excluded. Result: As a result of analyzing the word frequency, disease ranked the highest, and as a result of analyzing the connection strength between words, disease showed the highest connection strength in the skin. CONCOR analysis showed the occurrence status of skin diseases in Group 1, allergic contact dermatitis in Group 2, chemical exposure reaction in Group 3, disease prevention and monitoring system in Group 4 workplace, and occupational disease prevention in Group 5 were classified as health diseases and two were identified as chemicals. Conclusion: In summary, immediate countermeasures are needed to prevent occupational skin diseases, and programs that provide appropriate protective equipment and compensation along with health education, disease treatment, and counseling should be developed.

『주택연구』의 30년간 연구경향: 텍스트 마이닝 접근법

We analyze the research trend of the Housing Research Review over the period of 1993 to 2021. In particular, adopting text-mining methodologies: word frequency analysis, wordcloud technique, N-gram, and Topic Modeling, we analyze titles and keywords of all the papers on the journal. Based on those analyses, we find that words that the papers on the journal most frequently used over the past 30 years are ‘housing’ and ‘research’, which are followed by ‘price’, ‘househhold’ and ‘income’. Our word proximity analsyis shows the most-related keywords; housing prices, housing markets, fluctuations, and trading volume in terms of market and price structure; housing occupancy and household changes in terms of housing demand.
 Our analysis of Latent Dirichlet Allocation reveals six categories; ‘the ownership and apartments’, ‘price and space’, ‘real estate policy and regulation’, ‘household’s decision in housing market’, ‘regional characteristics evaluation’, and ‘residency and rental housing’. The most interested topics are related to ‘price’, and we confirm that the topics of interest has been changed over the decades; from ‘development and management’ and ‘rental housing and real estate’ to ‘public housing’. In addition, our analysis confirms that housing research on Housing Studies Review is comprehensive and includes a variety of housing topics.

기업의 해외진출에 따른 지역별 무역기술장벽 특성 비교: 수산업을 중심으로

The purpose of this study was to investigate the characteristics of each country’s Technical Barriers to Trade(TBT) notification according to each region as the TBTrelated regulations in the fisheries became more sensitive. The research data used the overseas overseas technical regulation information system of the Korean Agency for Technology and Standards and includes 201 notices affecting Korea. The study areas were Africa, Asia, Europe, North America, Latin America. As the analysis method, the text mining technique was applied, and there are word frequency and N-gram. As a result, the commonly extracted words include aquatic products production, processing, and packaging, as well as compliance with standards and technical regulations. By region, developing countries in Africa, Asia, Latin America showed formal regulations such as additional certification and an increase in required documents. Conversely, developed countries such as Europe and North America showed word connectivity related to practical regulations such as environmental protection and hygiene. Korea is in a position to respond to both practical and formal regulations at the same time. Therefore, it should be possible to evade regulations on technical barriers to trade through technical support such as Korea's fisheries standards to host countries.

ExpFinder: A hybrid model for expert finding from text-based expertise data

Finding an expert plays a crucial role in driving successful collaborations and speeding up high-quality research development and innovations. However, the rapid growth of scientific publications and digital data makes identifying the right experts a challenging problem. Existing approaches for finding experts given a topic can be categorised into information retrieval techniques such as vector space models, document language models, and graph-based models. In this paper, we propose ExpFinder , a new hybrid model for expert finding, that integrates a novel N-gram vector space model, denoted as nVSM, and a graph-based model, denoted as μCO-HITS, that is a proposed variation of the CO-HITS algorithm. The key of nVSM is to exploit recent inverse document frequency weighting method for N-gram words, and ExpFinder incorporates nVSM into μCO-HITS to achieve expert finding. We comprehensively evaluate ExpFinder on four different datasets from the academic domains in comparison with six different expert finding models. The evaluation results show that ExpFinder is an highly effective model for expert finding, substantially outperforming all the compared models in 19% to 160.2%.

Similarity Identification Based on Word Trigrams Using Exact String Matching Algorithms

Several studies regarding excellent exact string matching algorithms can be used to identify similarity, including the Rabin-Karp, Winnowing, and Horspool Boyer-Moore algorithms. In determining similarities, the Rabin-Karp and Winnowing algorithms use fingerprints, while the Horspool Boyer-Moore algorithm uses a bad-character table. However, previous research focused on identifying similarities using these algorithms based on character n-gram. In contrast, identification based on the word n-gram to determine the similarity based on its linguistic meaning, especially for longer strings, had not been covered yet. Therefore, a word-level trigram was proposed to identify similarities based on the word trigrams using the three algorithms and compare each performance. Based on precision, recall, and running time comparison, the Rabin-Karp algorithm results were 100%, 100%, and 0.19 ms, respectively; the Winnowing algorithm results with the smallest window were 100%, 56%, and 0.18 ms, respectively; and the Horspool algorithm results were 100%, 100%, and 0.06 ms. From these results, it can be concluded that the performance of the Horspool Boyer-Moore algorithm is better in terms of precision, recall, and running time.

Keyphrase Extraction Using Enhanced Word and Document Embedding

ABSTRACT We live in a world where information is available, in all areas of human activity, increasingly in digital text documents. It is necessary to explore the knowledge implied in these documents considering its fast-growing availability. The use of keywords provides for a more effective search for a document of interest, as keywords highlight a document's primary concept and, therefore, allow the researcher's interest to be readily aligned with that text. In this article, an unsupervised keyword extraction approach is proposed. The proposed approach retrofitted the concept of n-grams with state-of-the-art words and document embeddings. The approach simultaneously proposed a new method to compose document vectors using important word vectors and their idf-scores. Here we use higher-order word n-grams to improve various unigram embeddings and introduce a novel task to produce document embedding for document representation. The performance of the proposed embeddings is evaluated using four different datasets. The combination of higher-order word n-grams retrofitted Glove, and document embedding is the best embedding to be used for extracting key phrases. The bi-gram retrofitted embedding improves the results significantly over the baseline approaches.

Machine Learning Techniques, Features, Datasets, and Algorithm Performance Parameters for Sentiment Analysis: A Systematic Review

The purpose of this paper is to review various studies on current machine learning techniques used in sentiment analysis with the primary focus on finding the most suitable combinations of the techniques, datasets, data features, and algorithm performance parameters used in most applications. To accomplish this, we performed a systematic review of 24 articles published between 2013 and 2020 covering machine learning techniques for sentiment analysis. The review shows that Support Vector Machine as well as Naïve Bayes techniques are the most popular machine learning techniques; word stem and n-grams are the most extensively applied features, and the Twitter dataset is the most predominant. This review further revealed that machine learning algorithms' performance depends on many factors, including the dataset, extracted features, and size of data used. Accuracy is the most commonly used algorithm performance metric. These findings offer important information for researchers and businesses to use when selecting suitable techniques, features, and datasets for sentiment analysis for various business applications such as brand reputation monitoring.

Multi-class sentiment analysis of urdu text using multilingual BERT

Sentiment analysis (SA) is an important task because of its vital role in analyzing people’s opinions. However, existing research is solely based on the English language with limited work on low-resource languages. This study introduced a new multi-class Urdu dataset based on user reviews for sentiment analysis. This dataset is gathered from various domains such as food and beverages, movies and plays, software and apps, politics, and sports. Our proposed dataset contains 9312 reviews manually annotated by human experts into three classes: positive, negative and neutral. The main goal of this research study is to create a manually annotated dataset for Urdu sentiment analysis and to set baseline results using rule-based, machine learning (SVM, NB, Adabbost, MLP, LR and RF) and deep learning (CNN-1D, LSTM, Bi-LSTM, GRU and Bi-GRU) techniques. Additionally, we fine-tuned Multilingual BERT(mBERT) for Urdu sentiment analysis. We used four text representations: word n-grams, char n-grams,pre-trained fastText and BERT word embeddings to train our classifiers. We trained these models on two different datasets for evaluation purposes. Finding shows that the proposed mBERT model with BERT pre-trained word embeddings outperformed deep learning, machine learning and rule-based classifiers and achieved an F1 score of 81.49%.

Fake news spreaders profiling using N-grams of various types and SHAP-based feature selection

Complex learning approaches along with complicated and expensive features are not always the best or the only solution for Natural Language Processing (NLP) tasks. Despite huge progress and advancements in learning approaches such as Deep Learning (DL) and Transfer Learning (TL), there are many NLP tasks such as Text Classification (TC), for which basic Machine Learning (ML) classifiers perform superior to DL or TL approaches. Added to this, an efficient feature engineering step can significantly improve the performance of ML based systems. To check the efficacy of ML based systems and feature engineering on TC, this paper explores char, character sequences, syllables, word n-grams as well as syntactic n-grams as features and SHapley Additive exPlanations (SHAP) values to select the important features from the collection of extracted features. Voting Classifiers (VC) with soft and hard voting of four ML classifiers, namely: Support Vector Machine (SVM) with Linear and Radial Basis Function (RBF) kernel, Logistic Regression (LR), and Random Forest (RF) was trained and evaluated on Fake News Spreaders Profiling (FNSP) shared task dataset in PAN 2020. This shared task consists of profiling fake news spreaders in English and Spanish languages. The proposed models exhibited an average accuracy of 0.785 for both languages in this shared task and outperformed the best models submitted to this task.

코로나19 팬데믹 상황의 건강 및 운동 인식에 대한 빅데이터 분석

In this paper uses big data to find keywords related to the keywords “health” and “exercise” in 2020 that were dominated by coronavirus 19, the purpose is to analyze how people perceive it and to identify changed perceptions and trends. It conducted word frequency analysis, TF-IDF analysis, N-gram analysis, network analysis, and emotional analysis using Textom, a solution for big data analysis. Analysis results, First, Health and exercise had the highest frequency of simultaneous appearance with “health insurance, the National Health Insurance Corporation, and health checkups.” Second, health is often seen alongside “diet, help, specialized products,” and exercise is often seen alongside “acute, diet, and participation.” Third, as a result of the network analysis, four clusters were formed: the Exercise Effects Group, the Health Care Group, the Health Insurance Group, and the Health Care Group. Fourth, people’s perceptions of health and exercise were more positive than negative ones, and most neutral documents such as advertising, introduction and methods were found. These findings are meaningful in suggesting the direction of physical education during the post-corona era and using them as basic data.

Deep Learning Model for Sentiment Analysis on Short Informal Texts

This paper proposes a classification model to classify short informal texts. Those short informal texts were texts that were noisy, typos, irregular, and could consist of a very small number of words or even only a single word. The proposed model was trained using a dataset collected from student comments from an application called Evaluasi Dosen Oleh Mahasiswa (EDOM). This application assesses the lecturers using questionnaires filled out by students. It also records the student's comments but is not part of the evaluation calculation, therefore this work makes the data possible to be part of the assessment through sentiment analysis. This work focuses on building suitable preprocessing algorithm and building a simple deep learning network. The preprocessing algorithm was based on multiple word n-gram and Term Frequency-Inverse Document Frequency (TF-IDF) vectorization, and the network was built with a relatively shallow network. To evaluate the model in real usage, an application was built. The results were very convincing, reaching 0.979 in accuracy and 0.63 in F1-Score. Nonetheless, the imbalanced dataset was the only factor that needed to be investigated further for better overall performance.

Improving Mandarin End-to-End Speech Recognition With Word N-Gram Language Model

Despite the rapid progress of end-to-end (E2E) automatic speech recognition (ASR), it has been shown that incorporating external language models (LMs) into the decoding can further improve the recognition performance of E2E ASR systems. To align with the modeling units adopted in E2E ASR systems, subword-level (e.g., characters, BPE) LMs are usually used to cooperate with current E2E ASR systems. However, the use of subword-level LMs will ignore the word-level information, which may limit the strength of the external LMs in E2E ASR. Although several methods have been proposed to incorporate word-level external LMs in E2E ASR, these methods are mainly designed for languages with clear word boundaries such as English and cannot be directly applied to languages like Mandarin, in which each character sequence can have multiple corresponding word sequences. To this end, we propose a novel decoding algorithm where a word-level lattice is constructed on-the-fly to consider all possible word sequences for each partial hypothesis. Then, the LM score of the hypothesis is obtained by intersecting the generated lattice with an external word N-gram LM. The proposed method is examined on both Attention-based Encoder-Decoder (AED) and Neural Transducer (NT) frameworks. Experiments suggest that our method consistently outperforms subword-level LMs, including N-gram LM and neural network LM. We achieve state-of-the-art results on both Aishell-1 (CER 4.18%) and Aishell-2 (CER 5.06%) datasets and reduce CER by 14.8% relatively on a 21K-hour Mandarin dataset.

Zeta revisited

Abstract It has been claimed that traditional interpretations of Zeta based on graphs with bisector lines (Craig and Kinney, 2009, Shakespeare, Computers, and the Mystery of Authorship. Cambridge: Cambridge University Press.) are unsound, that validation with a held-out segments is inadequate, that counting types rather than tokens is suspect, and that the relationship between stronger Zeta scores and shorter word n-grams is an artifact of the method rather than a research finding (Rizvi, 2019a, The interpretation of zeta test results. Digital Scholarship in the Humanities, 34(2): 401–18). All of these claims are unsound. The separation of base and counter segments in a Zeta analysis is in fact an important research finding, the traditional interpretation of Zeta results based on a bisector line remain sound, and validation with a held-out segments is an appropriate method. Zeta’s reliance on consistency rather than frequency is not a bug but valuable feature that provides an important complementary method to those based on frequency. The results of extensive testing on corpora of prose, poetry, and drama show that increasing word n-gram length (but not character n-gram length) is often negatively correlated with classification accuracy. Higher Zeta scores are often correlated with better results, though the results vary a great deal depending on the corpus, the classification method, and the type and length of n-gram. Most importantly, these results show that Zeta gives strong results that are competitive with methods like Cosine Delta and Support Vector Machine on these classification tasks.

Development of Dialogue Management System for Banking Services

Rapid increase in conversational AI and user chat data lead to intensive development of dialogue management systems (DMS) for various industries. Yet, for low-resource languages, such as Azerbaijani, very little research has been conducted. The main purpose of this work is to experiment with various DMS pipeline set-ups to decide on the most appropriate natural language understanding and dialogue manager settings. In our project, we designed and evaluated different DMS pipelines with respect to the conversational text data obtained from one of the leading retail banks in Azerbaijan. In the work, the main two components of DMS—Natural language Understanding (NLU) and Dialogue Manager—have been investigated. In the first step of NLU, we utilized a language identification (LI) component for language detection. We investigated both built-in LI methods such as fastText and custom machine learning (ML) models trained on the domain-based dataset. The second step of the work was a comparison of the classic ML classifiers (logistic regression, neural networks, and SVM) and Dual Intent and Entity Transformer (DIET) architecture for user intention detection. In these experiments we used different combinations of feature extractors such as CountVectorizer, Term Frequency-Inverse Document Frequency (TF-IDF) Vectorizer, and word embeddings for both word and character n-gram based tokens. To extract important information from the text messages, Named Entity Extraction (NER) component was added to the pipeline. The best NER model was chosen among conditional random fields (CRF) tagger, deep neural networks (DNN), models and build in entity extraction component inside DIET architecture. Obtained entity tags fed to the Dialogue Management module as features. All NLU set-ups were followed by the Dialogue Management module that contains a Rule-based Policy to handle FAQs and chitchats as well as a Transformer Embedding Dialogue (TED) Policy to handle more complex and unexpected dialogue inputs. As a result, we suggest a DMS pipeline for a financial assistant, which is capable of identifying intentions, named entities, and a language of text followed by policies that allow generating a proper response (based on the designed dialogues) and suggesting the best next action.

Linguistic features evaluation for hadith authenticity through automatic machine learning

Abstract There has not been any research that provides an evaluation of the linguistic features extracted from the matn (text) of a Hadith. Moreover, none of the fairly large corpora are publicly available as a benchmark corpus for Hadith authenticity, and there is a need to build a ‘gold standard’ corpus for good practices in Hadith authentication. We write a scraper in Python programming language and collect a corpus of 3,651 authentic prophetic traditions and 3,593 fake ones. We process the corpora with morphological segmentation and perform extensive experimental studies using a variety of machine learning algorithms, mainly through automatic machine learning, to distinguish between these two categories. With a feature set including words, morphological segments, characters, top N words, top N segments, function words, and several vocabulary richness features, we analyze the results in terms of both prediction and interpretability to explain which features are more characteristic of each class. Many experiments have produced good results and the highest accuracy (i.e. 78.28%) is achieved using word n-grams as features using the Multinomial Naive Bayes classifier. Our extensive experimental studies conclude that, at least for Digital Humanities, feature engineering may still be desirable due to the high interpretability of the features. The corpus and software (scripts) will be made publicly available to other researchers in an effort to promote progress and replicability.