Sentence Classification Research Articles

TOWARDS CURBING CYBER-BULLYING IN MALAYSIA BY AUTHOR IDENTIFICATION OF IBAN AND KADAZANDUSUN OSN TEXT USING DEEP LEARNING

Online Social Network (OSN) is frequently used to carry out cyber-criminal actions such as cyberbullying. As a developing country in Asia that keeps abreast of ICT advancement, Malaysia is no exception when it comes to cyberbullying. Author Identification (AI) task plays a vital role in social media forensic investigation (SMF) to unveil the genuine identity of the offender by analysing the text written in OSN by the candidate culprits. Several challenges in AI dealing with OSN text, including limited text length and informal language full of internet jargon and grammatical errors that further impact AI's performance in SMF. The traditional AI system that analyses long text documents seems inadequate to analyse short OSN text's writing style. N-gram features are proven to efficiently represent the authors' writing style for shot text. However, representing N-grams in traditional representation like Tf-IDF resulted in sparse and difficult in grasping the semantic information from text. Besides, most AI works have been done in English but receive less attention in indigenous languages. In West Malaysia, the supreme languages that transcend ethnic boundaries are Iban of Sarawak and KadazanDusun of Sabah, which both are inherently under-resourced. This paper presented a proposed workflow of AI for short OSN text using two Under-Resourced Language (U-RL), Iban and KadazanDusun tweets, to curb the cyberbullying issue in Malaysia. This paper compares Tf-Idf (sparse) and SoA embedding-based (dense) feature representations to observe which representations best represent the stylistic features of the authors’ writing. N-grams of word, character, and POS were extracted as the features. The representation models were learned by different classifiers using machine learning (Naïve Bayes, Random Forest, and SVM). The convolutional neural network (CNN), a SoA deep learning model in sentence classification, was tested against the traditional classifiers. The result was observed by combining different representation models and classifiers on three datasets (English, Iban, and KadazanDusun). The best result was achieved when CNN learned embedding-based models with a combination of all features. KadazanDusun achieved the highest accuracy with 95.76%, English with 95.02%, and Iban with 94%..

Biomedical Abstract Sentence Classification by BERT-Based Reading Comprehension

Biomedical Abstract Sentence Classification by BERT-Based Reading Comprehension

News text classification using Long-Term Short Memory (LSTM) algorithm

Over the past few years, the classification of texts has become increasingly important. Because knowledge is now available to users through various sources namely electronic media, digital media, print media, and many more. One of them is the development of so much news every day. LSTM is one of the algorithms of deep learning methods that can classify a text. This research proves for the LSTM algorithm on the classification of news text sentences. The data used is the news text from the Kaggle data center set i.e. aggregator news data. The results of the LSTM experiment from 10 epochs obtained with an accuracy value of 93,15% on the classification of texts into four categories, namely entertainment, bussines, science, and health.

An unsupervised linguistic-based model for automatic glossary term extraction from a single PDF textbook

Term extraction from textbooks is the cornerstone of many different intelligent natural language processing systems, especially those that support learners and educators in the education system. This paper proposes a novel unsupervised domain-independent model that automatically extracts relevant and domain-related key terms from a single PDF textbook, without relying on a statistical technique or external knowledge base. It only relies on the basic linguistic techniques of the natural language processing: pattern recognition, sentence tokenization, part-of-speech tagging, and chunking. The model takes a PDF textbook as an input and produces a list of key terms as an output. Furthermore, the model proposes a novel classification of sentences from which the concept of defining sentences is proposed. The defining sentences are the main textual units that the model revolves around to identify the key terms. The architecture of the proposed work consists of 21 processes distributed across three phases. The first phase consists of five processes for extracting text from a PDF textbook and cleaning it for the next phases. The second phase consists of eight processes for identifying the defining sentences and extracting them from all the textbook’s sentences. The last phase consists of eight processes for identifying and extracting the key terms from every defining sentence. The proposed work was evaluated by two experiments in which two PDF textbooks from different fields are used. The experimental evaluation showed that the results were promising.

Few-Shot Learning for Clinical Natural Language Processing Using Siamese Neural Networks: Algorithm Development and Validation Study

Background Natural language processing (NLP) has become an emerging technology in health care that leverages a large amount of free-text data in electronic health records to improve patient care, support clinical decisions, and facilitate clinical and translational science research. Recently, deep learning has achieved state-of-the-art performance in many clinical NLP tasks. However, training deep learning models often requires large, annotated data sets, which are normally not publicly available and can be time-consuming to build in clinical domains. Working with smaller annotated data sets is typical in clinical NLP; therefore, ensuring that deep learning models perform well is crucial for real-world clinical NLP applications. A widely adopted approach is fine-tuning existing pretrained language models, but these attempts fall short when the training data set contains only a few annotated samples. Few-shot learning (FSL) has recently been investigated to tackle this problem. Siamese neural network (SNN) has been widely used as an FSL approach in computer vision but has not been studied well in NLP. Furthermore, the literature on its applications in clinical domains is scarce. Objective The aim of our study is to propose and evaluate SNN-based approaches for few-shot clinical NLP tasks. Methods We propose 2 SNN-based FSL approaches, including pretrained SNN and SNN with second-order embeddings. We evaluate the proposed approaches on the clinical sentence classification task. We experiment with 3 few-shot settings, including 4-shot, 8-shot, and 16-shot learning. The clinical NLP task is benchmarked using the following 4 pretrained language models: bidirectional encoder representations from transformers (BERT), BERT for biomedical text mining (BioBERT), BioBERT trained on clinical notes (BioClinicalBERT), and generative pretrained transformer 2 (GPT-2). We also present a performance comparison between SNN-based approaches and the prompt-based GPT-2 approach. Results In 4-shot sentence classification tasks, GPT-2 had the highest precision (0.63), but its recall (0.38) and F score (0.42) were lower than those of BioBERT-based pretrained SNN (0.45 and 0.46, respectively). In both 8-shot and 16-shot settings, SNN-based approaches outperformed GPT-2 in all 3 metrics of precision, recall, and F score. Conclusions The experimental results verified the effectiveness of the proposed SNN approaches for few-shot clinical NLP tasks.

QNLP in Practice: Running Compositional Models of Meaning on a Quantum Computer

Quantum Natural Language Processing (QNLP) deals with the design and implementation of NLP models intended to be run on quantum hardware. In this paper, we present results on the first NLP experiments conducted on Noisy Intermediate-Scale Quantum (NISQ) computers for datasets of size greater than 100 sentences. Exploiting the formal similarity of the compositional model of meaning by Coecke, Sadrzadeh, and Clark (2010) with quantum theory, we create representations for sentences that have a natural mapping to quantum circuits. We use these representations to implement and successfully train NLP models that solve simple sentence classification tasks on quantum hardware. We conduct quantum simulations that compare the syntax-sensitive model of Coecke et al. with two baselines that use less or no syntax; specifically, we implement the quantum analogues of a “bag-of-words” model, where syntax is not taken into account at all, and of a word-sequence model, where only word order is respected. We demonstrate that all models converge smoothly both in simulations and when run on quantum hardware, and that the results are the expected ones based on the nature of the tasks and the datasets used. Another important goal of this paper is to describe in a way accessible to AI and NLP researchers the main principles, process and challenges of experiments on quantum hardware. Our aim in doing this is to take the first small steps in this unexplored research territory and pave the way for practical Quantum Natural Language Processing.

SIMPLE SENTENCE IN A·WE

A·we is the standard variety of the A·chik language also known as A·chikku which is also commonly known as the Garo language. The Garo language belongs to the Tibeto-Burman branch of the Sino-Tibetan language family. Since Syntax involves arranging words to create logical phrases, clauses, and sentences, simple sentence is an important part of syntax and thus knowing about the simple sentence in A·we forms the basis of writing correct sentences in this variety of the Garo language. Noam Chomsky, the famous Linguist used the phrase, “ colorless green ideas sleep furiously’ in his book, Syntactic Structures (1957) as an example of a sentence which is syntactically and grammatically correct because it has the correct word order and the verb is consistent with the subject but is semantically incorrect. Chomsky (1957) thus illustrates that the rules governing syntax are different from the meaning conveyed by words. We can observe that there are subject and predicate in a simple sentence in language, which is the same for A·we. It is not essential that a simple sentence must be a short sentence and it is also possible to write a simple sentence if there is only one predicate used with a number of subjects to make a long sentence. Such sentences are still called a simple sentence. In this paper, we will discuss some classification of simple sentence in A·we which will further contribute to the study of syntax in A·we as well as aid in constructing proper sentences in the language.

Synwmd: Syntax-aware word Mover’s distance for sentence similarity evaluation

Word Mover’s Distance (WMD) computes the distance between words and models text similarity with the moving cost between words in two text sequences. Yet, it does not offer good performance in sentence similarity evaluation since it does not incorporate word importance and fails to take inherent contextual and structural information in a sentence into account. An improved WMD method using the syntactic parse tree, called Syntax-aware Word Mover’s Distance (SynWMD), is proposed to address these two shortcomings in this work. First, a weighted graph is built upon the word co-occurrence statistics extracted from the syntactic parse trees of sentences. The importance of each word is inferred from graph connectivities. Second, the local syntactic parsing structure of words is considered in computing the distance between words. To demonstrate the effectiveness of the proposed SynWMD, we conduct experiments on 6 textual semantic similarity (STS) datasets and 4 sentence classification datasets. Experimental results show that SynWMD achieves state-of-the-art performance on STS tasks. It also outperforms other WMD-based methods on sentence classification tasks.

A Neuro Symbolic Approach for Contradiction Detection in Persian Text

Detection of semantic contradictory sentences is a challenging and fundamental issue for some NLP applications, such as textual entailments recognition. In this study, contradiction means different types of semantic confrontation, such as negation, antonymy, and numerical. Due to the lack of sufficient data to apply precise machine learning and, specifically, deep learning methods to Persian and other low-resource languages, rule-based approaches are of great interest. Also, recently, the emergence of new methods such as transfer learning has opened up the possibility of deep learning for low-resource languages. This paper introduces a hybrid contradiction detection approach for detecting seven categories of contradictions in Persian texts: Antonymy, negation, numerical, factive, structural, lexical and world knowledge. The proposed method consists of 1) a novel data mining method and 2) a transformer-based deep neural method for contradiction detection . Also, a simple baseline is presented for comparison. The data mining method uses frequent rule mining to extract appropriate contradiction detection rules employing a development set. Extracted rules are tested for different categories of contradictory sentences. In the first step, a classifier checks whether the rules work for an input sentence pair. Then, according to the result, rules are used for three categories of negation, numerical, and antonym. In this part, the highest F-measure is obtained for detecting the negation category (90%), the average F-measure for these three categories is 86%, and for the other four categories, in which the rules have a lower F-measure of 62%, the transformer-based method achieved 76%. The proposed hybrid approach has an overall f-measure of higher than 80%. 

Joint Syntax-Enhanced and Topic-Driven Graph Networks for Emotion Recognition in Multi-Speaker Conversations

Daily conversations contain rich emotional information, and identifying this emotional information has become a hot task in the field of natural language processing. The traditional dialogue sentiment analysis method studies one-to-one dialogues and cannot be effectively applied to multi-speaker dialogues. This paper focuses on the relationship between participants in a multi-speaker conversation and analyzes the influence of each speaker on the emotion of the whole conversation. We summarize the challenges of emotion recognition work in multi-speaker dialogue, focusing on the context-topic switching problem caused by multi-speaker dialogue due to its free flow of topics. For this challenge, this paper proposes a graph network that combines syntactic structure and topic information. A syntax module is designed to convert sentences into graphs, using edges to represent dependencies between words, solving the colloquial problem of daily conversations. We use graph convolutional networks to extract the implicit meaning of discourse. In addition, we focus on the impact of topic information on sentiment, so we design a topic module to optimize the topic extraction and classification of sentences by VAE. Then, we use the combination of attention mechanism and syntactic structure to strengthen the model’s ability to analyze sentences. In addition, the topic segmentation technology is adopted to solve the long-term dependencies problem, and a heterogeneous graph is used to model the dialogue. The nodes of the graph combine speaker information and utterance information. Aiming at the interaction relationship between the subject and the object of the dialogue, different edge types are used to represent different interaction relationships, and different weights are assigned to them. The experimental results of our work on multiple public datasets show that the new model outperforms several other alternative methods in sentiment label classification results. In the multi-person dialogue dataset, the classification accuracy is increased by more than 4%, which verifies the effectiveness of constructing heterogeneous dialogue graphs.

Art of Use of Exclusive Language in the Epic of Khing Ju of the Ede People in the Central Highlands, Vietnam

The Ede community in the Central Highlands, Vietnam has created famous epics, such as Dam san, Khing Ju, Xinh Nha, etc. These are oral works of art, the result of the past collective composition process, which is attached to and directly serves different activities in community life. The Ede epic is a literary genre with high integrity, in addition to its literary and linguistic value, it also contains precious documents about culture, history, ideology, and indigenous people. Based on the theory of the art of exaggeration, combining survey, statistics, and classification of sentences using the art of exaggeration in the epic Khing Ju of the Ede people in the Central Highlands, Vietnam, the article provides an overview of the model. exaggerated structural model in the Khing Ju epic, explaining the elements in the model, such as The element is exaggerated; Exaggerated aspect factor; Exaggerated relational factors; Magnifying factor; Magnified enlargement factor. Simultaneous analysis of objects participating in the exaggerated structure, such as people; animals, and plants; some objects in daily life. Thereby clarifying the value of the exaggeration method for the content of the work and expressing the cultural characteristics of the community.
 
 Received: 10 January 2023 / Accepted: 25 February 2023 / Published: 5 March 2023

A deep penetration network for sentence classification

Sentence classification is an important task in natural language processing. The task makes use of deep networks to enclose a mass of features with different granularities in a sentence. However, the classification usually suffer from severe performance degradation when stacking a large number of networks. The main reason is that, in a deep architecture, the silent feature representations are easily weakened and mixed with noisy information, which is not effective in learning contextual features and constructing semantic dependencies in a sentence. In this paper, a deep penetration network (DPN) is designed to improve deep architectures’ ability to preserve the favourable semantic features. The DPN enables salient features to penetrate through a deeper architecture and to construct long semantic dependencies between them. This approach is evaluated on seven public datasets. Our experiments show that the DPN exhibits a stable performance with deeper architectures. It improves the performance on three types of sentence classification tasks, outperforming the existing state-of-the-art models.

Grammar-aware sentence classification on quantum computers

Natural language processing (NLP) is at the forefront of great advances in contemporary AI, and it is arguably one of the most challenging areas of the field. At the same time, in the area of quantum computing (QC), with the steady growth of quantum hardware and notable improvements towards implementations of quantum algorithms, we are approaching an era when quantum computers perform tasks that cannot be done on classical computers with a reasonable amount of resources. This provides an new range of opportunities for AI, and for NLP specifically. In this work, we work with the Categorical Distributional Compositional (DisCoCat) model of natural language meaning, whose underlying mathematical underpinnings make it amenable to quantum instantiations. Earlier work on fault-tolerant quantum algorithms has already demonstrated potential quantum advantage for NLP, notably employing DisCoCat. In this work, we focus on the capabilities of noisy intermediate-scale quantum (NISQ) hardware and perform the first implementation of an NLP task on a NISQ processor, using the DisCoCat framework. Sentences are instantiated as parameterised quantum circuits; word-meanings are embedded in quantum states using parameterised quantum-circuits and the sentence’s grammatical structure faithfully manifests as a pattern of entangling operations which compose the word-circuits into a sentence-circuit. The circuits’ parameters are trained using a classical optimiser in a supervised NLP task of binary classification. Our novel QNLP model shows concrete promise for scalability as the quality of the quantum hardware improves in the near future and solidifies a novel branch of experimental research at the intersection of QC and AI.

An Approach to Summarizing Product Reviews

Product reviews are an important feature of e-commerce because they influence the communicative behavior of a potential addressee (customers). Users often read online product reviews to get either general or specific information about a product or service. Besides, companies analyze customer reviews to improve their product quality or to adjust their marketing strategy. However, many reviews are unstructured and long. With the number of product reviews growing rapidly, reading a large number of reviews becomes a time-consuming process for both users and companies. Therefore, review summarization becomes a serious issue. In this paper, we propose a new approach to summarizing reviews of electronics and household appliances. The proposed approach makes it possible to structure information for every aspect category; it provides calculation sentiment score for each aspect category, and shows the most relevant sentences for each aspect. We used product reviews from Yandex.Market as target data. Our task was performed in five main phases: 1) expert identification of thematic aspect categories; 2) classifying sentences into the predefined aspect categories; 3) sentence classification into two classes—positive and negative—with calculating the number of both type sentences within each aspect category; 4) sentence ranking; 5) visualization of the results obtained in the previous phases. The quality of the algorithm for creating a resume from a large collection of reviews has been tested on five models of products from the following categories: coffee machines, robot vacuum cleaners, e-books, TV-sets, and washing machines.

Automatic recognition and classification of future work sentences from academic articles in a specific domain

Future work sentences (FWS) are the particular sentences in academic papers that contain the author's description of their proposed follow-up research direction. This paper presents methods to automatically extract FWS from academic papers and classify them according to the different future directions embodied in the paper's content. FWS recognition methods will enable subsequent researchers to locate future work sentences more accurately and quickly and reduce the time and cost of acquiring the corpus. At the same time, changes in the content of future work will be illuminated, and a foundation will be laid for a more in-depth semantic analysis of future work sentences. The current work on automatic identification of future work sentences is relatively small, and the existing research cannot accurately identify FWS from academic papers, and thus cannot conduct data mining on a large scale. Furthermore, there are many aspects to the content of future work, and the subdivision of the content is conducive to the analysis of specific development directions. In this paper, Nature Language Processing (NLP) is used as a case study, and FWS are extracted from academic papers and classified into different types. We manually build an annotated corpus with six different types of FWS. Then, automatic recognition and classification of FWS are implemented using machine learning models, and the performance of these models is compared based on the evaluation metrics. The results show that the Bernoulli Bayesian model has the best performance in the automatic recognition task, with the Macro F1 reaching 90.73%, and the SCIBERT model has the best performance in the automatic classification task, with the weighted average F1 reaching 72.63%. Finally, we extract keywords from FWS and gain a deep understanding of the key content described in FWS, and we also demonstrate that content determination in FWS will be reflected in the subsequent research work by measuring the similarity between future work sentences and the abstracts.

Monitoring Indonesian online news for COVID-19 event detection using deep learning

Even though coronavirus disease 2019 (COVID-19) vaccination has been done, preparedness for the possibility of the next outbreak wave is still needed with new mutations and virus variants. A near real-time surveillance system is required to provide the stakeholders, especially the public, to act in a timely response. Due to the hierarchical structure, epidemic reporting is usually slow particularly when passing jurisdictional borders. This condition could lead to time gaps for public awareness of new and emerging events of infectious diseases. Online news is a potential source for COVID-19 monitoring because it reports almost every infectious disease incident globally. However, the news does not report only about COVID-19 events, but also various information related to COVID-19 topics such as the economic impact, health tips, and others. We developed a framework for online news monitoring and applied sentence classification for news titles using deep learning to distinguish between COVID-19 events and non-event news. The classification results showed that the fine-tuned bidirectional encoder representations from transformers (BERT) trained with Bahasa Indonesia achieved the highest performance (accuracy: 95.16%, precision: 94.71%, recall: 94.32%, F1-score: 94.51%). Interestingly, our framework was able to identify news that reports the new COVID strain from the United Kingdom (UK) as an event news, 13 days before the Indonesian officials closed the border for foreigners.

Label informed hierarchical transformers for sequential sentence classification in scientific abstracts

AbstractSegmenting scientific abstracts into discourse categories like background, objective, method, result, and conclusion is useful in many downstream tasks like search, recommendation and summarization. This task of classifying each sentence in the abstract into one of a given set of discourse categories is called sequential sentence classification. Existing machine learning‐based approaches to this problem consider the content of only the abstract to obtain the neural representation of each sentence, which is then labelled with a discourse category. But this ignores the semantic information offered by the discourse labels themselves. In this paper, we propose LIHT, Label Informed Hierarchical Transformers – a method for sequential sentence classification that explicitly and hierarchically exploits the semantic information in the labels to learn label‐aware neural sentence representations. The hierarchical model helps to capture not only the fine‐grained interactions between the discourse labels and the words in the abstract at the sentence level but also the potential dependencies that may exist in the label sequence. Thus, LIHT generates label‐aware contextual sentence representations that are then labelled with a conditional random field. We evaluate LIHT on three publicly available datasets, namely, PUBMED‐RCT, NICTA‐PIBOSO and CSAbstract. The incremental gain in F1‐score in all the three cases over the respective state‐of‐the‐art approaches is around . Though the gains are modest, LIHT establishes a new performance benchmark for this task and is a novel technique of independent interest. We also perform an ablation study to identify the contribution of each component of LIHT in the observed performance, and a case study to visualize the roles of the different components of our model.

Exploiting All Samples in Low-Resource Sentence Classification: Early Stopping and Initialization Parameters

To improve deep-learning performance in low-resource settings, many researchers have redesigned model architectures or applied additional data (e.g., external resources, unlabeled samples). However, there have been relatively few discussions on how to make good use of small amounts of labeled samples, although it is potentially beneficial and should be done before applying additional data or redesigning models. In this study, we assume a low-resource setting in which only a few labeled samples (i.e., 30–100 per class) are available, and we discuss how to exploit them without additional data or model redesigns. We explore possible approaches in the following three aspects: training validation splitting, early stopping, and weight initialization. Extensive experiments are conducted on six public sentence classification datasets. Performance on various evaluation metrics (e.g., accuracy, loss, and calibration error) significantly varied depending on the approaches that were combined in the three aspects. Based on the results, we propose an integrated method, which is to initialize the model with a weight averaging method and use a non-validation stop method to train all samples. This simple integrated method consistently outperforms the competitive methods; e.g., the average accuracy of six datasets of this method was 1.8% higher than those of conventional validation-based methods. In addition, the integrated method further improves the performance when adapted to several state-of-the-art models that use additional data or redesign the network architecture (e.g., self-training and enhanced structural models). Our results highlight the importance of the training strategy and suggest that the integrated method can be the first step in the low-resource setting. This study provides empirical knowledge that will be helpful when dealing with low-resource data in future efforts. Our code is publicly available at https://github.com/DMCB-GIST/exploit_all_samples.

Classification of the elliptical sentences in the question-answer dialogue unity in modern English language communication

The article provides an arrangement of different points of view of Russian and foreign linguists on the versatile issue of ellipsis and elliptical sentences. It shows how these constructions are de-scribed within the formal-grammatical, textual, stylistic, logical, comparative and communicative approaches; and the correlation between ellipsis and syntactic anaphoric zero qualification is given. The second part of the article is devoted to the statistics of different types of elliptical sentences based on the established basic models of elliptical sentences in questionand-answer dialogue unities of the modern English language communication. The author identifies five structural models of elliptical sentences in the question-answer dialogical unity of the modern English language. They are: one-member elliptical interrogative and answer sentences with noun, pronoun, adjective and adverb as a key component; one-member elliptical sentences with functional parts of speech as a key component: an elliptical question presented by question words, conjunctions; and elliptical answers in the form of conjunctions, interjections, prepositions. Then come one-member elliptical answer yesand no-sentences; two-member elliptical general and special questions with direct word order; two-member elliptical answer sentences with the incomplete structure of the predicate and also two-member elliptical interrogative and answer sentences with represented members.

Verbless predicative structures in Persian paroemias

Many books and articles are devoted to Persian phraseology, but insuffi cient attention has been paid to the description and classification of verbless sentences. Among all sentential idioms, proverbs, sayings and other figurative statements are of great interest. The purpose of this work is to describe the most common types of verbless sentential idioms, their syntactic and semantic structures and patterns of functioning. The structures of verbless sentences (VS) that form Persian proverbs are based on ellipsis. The elimination of the verb enhances the style of statements. The language of paroemias becomes more capacious. Combined with other means, such as use of archaic lexemes and their obsolete forms, this achieves an antique stylization. The deviation of the style of proverbs from the style of everyday speech attracts the addressee’s attention. Some nominal parts of speech possess inherent predicativity and therefore do not combine with other predicative units, such as personal forms of the verb. The majority of bipartite phraseological units oppose the first judgment to the second. Within complex judgments, the components are most often connected asyndetically or with the enclitical conjunction -o ‘and’. In many polypredicative paroemias, the first clause contains a finite verb, and the subsequent ones are elliptical sentences with an omitted verb. The copula is usually omitted in all clauses. A number of verbless sentences in paroemias are distinguished by the emphatic word order, with the attribute preceding its head. Most of the sayings, from the point of view of information structure, refer to the situation as a whole and therefore are thetic sentences with a non-inherent topic and the given which is expressed in a preliminary context.