Supervised Text Classification Research Articles

Ideological self-selection in online news exposure: Evidence from Europe and the US.

Today's high-choice digital media environments allow citizens to completely refrain from online news exposure and, if they do use news, to select sources that align with their ideological preferences. Yet due to measurement problems and cross-country differences, recent research has been inconclusive regarding the prevalence of ideological self-selection into like-minded online news. We introduce a multi-method design combining the web-browsing histories and survey responses of more than 7000 participants from six major democracies with supervised text classification to separate political from nonpolitical news exposure. We find that political online news exposure is both substantially less prevalent and subject to stronger ideological self-selection than nonpolitical online news exposure, especially in the United States. By highlighting the peculiar role of political news content, the results improve the understanding of online news exposure and the role of digital media in democracy.

Living review framework for better policy design and management of hazardous waste in Australia

The significant increase in hazardous waste generation in Australia has led to the discussion over the incorporation of artificial intelligence into the hazardous waste management system. Recent studies explored the potential applications of artificial intelligence in various processes of managing waste. However, no study has examined the use of text mining in the hazardous waste management sector for the purpose of informing policymakers. This study developed a living review framework which applied supervised text classification and text mining techniques to extract knowledge using the domain literature data between 2022 and 2023. The framework employed statistical classification models trained using iterative training and the best model XGBoost achieved an F1 score of 0.87. Using a small set of 126 manually labelled global articles, XGBoost automatically predicted the labels of 678 Australian articles with high confidence. Then, keyword extraction and unsupervised topic modelling with Latent Dirichlet Allocation (LDA) were performed. Results indicated that there were 2 main research themes in Australian literature: (1) the key waste streams and (2) the resource recovery and recycling of waste. The implication of this framework would benefit the policymakers, researchers, and hazardous waste management organisations by serving as a real time guideline of the current key waste streams and research themes in the literature which allow robust knowledge to be applied to waste management and highlight where the gap in research remains.

Neurosurgical literature classification – Evaluation of three automated methods and time trend analysis of the literature

BackgroundAutomated supervised text classification methods require preclassified training data. Their application in scenarios that a large amount of preclassified data is not accessible is challenging. Neurosurgical literature classification into subspecialties is an example of this situation. We have introduced an automated similarity-based text classification method, evaluated it along with two other automated methods and applied the introduced method in neurosurgical literature classification. MethodsPerformance of an introduced similarity-based text classification method along with two other automated methods (Lbl2Vec and keyword counting-based methods) was compared with performance of two senior neurosurgery registrars in classification of neurosurgical literature to 5 subspecialties. The Kappa-statistic measure of interrater agreement, overall marginal homogeneity using the Stuart-Maxwell test, marginal homogeneity relative to individual categories using McNemar tests and the sensitivity and specificity of each of the three methods were calculated.The introduced method was used to classify 211617 neurosurgical publications indexed in Pubmed to different subspecialties based on keywords extracted from subspecialty sections of a neurosurgery textbook. ResultsThe introduced similarity-based method showed the highest agreement with the registrars (raw agreement and Kappa value) followed by the Lbl2Vec and the counting-based method. Classifications of the English neurosurgical publications indexed in Pubmed into categories of Oncology, Vascular, Spine and functional using the introduced similarity-based method were more reliable (closer to the registrars’ classifications) than Cranial trauma. The classifications and future forecast showed highest publications in Oncology, followed by Cranial trauma, Vascular, spine and functional neurosurgery. ConclusionThe classification of the English neurosurgical publications indexed in Pubmed to different subspecialties, using the introduced method, shows that Oncology and tumour has been the main battleground for the neurosurgeons over years and probably in the near future. The performance of the introduced classification method in comparison with the human performance shows its potential application in the situations that enough preclassified data are not accessible for automated text classification.

A deep multiple-instance text binary classification for topic relevant content extraction on social media

Social media platforms have rich text data, which can be used in data mining and analysis. However, given the fact, the evolution speed of natural languages is rapid in social media, and data on social media is very noisy. This is a great challenge to the accuracy of data analysis. To overcome this problem, we propose a topic-relevant content extraction (TRCE) based on deep multiple instance classification, leveraging existing information and hierarchical relationships among texts under a thread on social media as weak supervision to extract topic-strong-relevant data and filter out noise accurately without manually labeling data. The proposed method introduces latent variables, Bernoulli distribution, and variational inference into multiple-instance learning (MIL) to generate pseudo labels. Then we employ a dual-stream neural network with a 3-stage training process to achieve training MIL end-to-end. Experimental results show TRCE has a significant improvement compared with other MIL methods. Meanwhile, it only has a little decrease compared with supervised text classification on accuracy and F1 score. Given the fact TRCE does not need manually labeled data at all, while supervised classification relies heavily on labeled data, TRCE is a competitive method to extracting topic-relevant data and filtering out noise on social media.

AI Algorithm to Predict Acute Coronary Syndrome in Prehospital Cardiac Care: Retrospective Cohort Study

Background Overcrowding of hospitals and emergency departments (EDs) is a growing problem. However, not all ED consultations are necessary. For example, 80% of patients in the ED with chest pain do not have an acute coronary syndrome (ACS). Artificial intelligence (AI) is useful in analyzing (medical) data, and might aid health care workers in prehospital clinical decision-making before patients are presented to the hospital. Objective The aim of this study was to develop an AI model which would be able to predict ACS before patients visit the ED. The model retrospectively analyzed prehospital data acquired by emergency medical services' nurse paramedics. Methods Patients presenting to the emergency medical services with symptoms suggestive of ACS between September 2018 and September 2020 were included. An AI model using a supervised text classification algorithm was developed to analyze data. Data were analyzed for all 7458 patients (mean 68, SD 15 years, 54% men). Specificity, sensitivity, positive predictive value (PPV), and negative predictive value (NPV) were calculated for control and intervention groups. At first, a machine learning (ML) algorithm (or model) was chosen; afterward, the features needed were selected and then the model was tested and improved using iterative evaluation and in a further step through hyperparameter tuning. Finally, a method was selected to explain the final AI model. Results The AI model had a specificity of 11% and a sensitivity of 99.5% whereas usual care had a specificity of 1% and a sensitivity of 99.5%. The PPV of the AI model was 15% and the NPV was 99%. The PPV of usual care was 13% and the NPV was 94%. Conclusions The AI model was able to predict ACS based on retrospective data from the prehospital setting. It led to an increase in specificity (from 1% to 11%) and NPV (from 94% to 99%) when compared to usual care, with a similar sensitivity. Due to the retrospective nature of this study and the singular focus on ACS it should be seen as a proof-of-concept. Other (possibly life-threatening) diagnoses were not analyzed. Future prospective validation is necessary before implementation.

Low-Resource Personal Attribute Prediction from Conversations

Personal knowledge bases (PKBs) are crucial for a broad range of applications such as personalized recommendation and Web-based chatbots. A critical challenge to build PKBs is extracting personal attribute knowledge from users' conversation data. Given some users of a conversational system, a personal attribute and these users' utterances, our goal is to predict the ranking of the given personal attribute values for each user. Previous studies often rely on a relative number of resources such as labeled utterances and external data, yet the attribute knowledge embedded in unlabeled utterances is underutilized and their performance of predicting some difficult personal attributes is still unsatisfactory. In addition, it is found that some text classification methods could be employed to resolve this task directly. However, they also perform not well over those difficult personal attributes. In this paper, we propose a novel framework PEARL to predict personal attributes from conversations by leveraging the abundant personal attribute knowledge from utterances under a low-resource setting in which no labeled utterances or external data are utilized. PEARL combines the biterm semantic information with the word co-occurrence information seamlessly via employing the updated prior attribute knowledge to refine the biterm topic model's Gibbs sampling process in an iterative manner. The extensive experimental results show that PEARL outperforms all the baseline methods not only on the task of personal attribute prediction from conversations over two data sets, but also on the more general weakly supervised text classification task over one data set.

Comparing BERT Against Traditional Machine Learning Models in Text Classification

The BERT model has arisen as a popular state-of-the-art model in recent years. It is able to cope with NLP tasks such as supervised text classification without human supervision. Its flexibility to cope with any corpus delivering great results has make this approach very popular in academia and industry. Although, other approaches have been used before successfully. We first present BERT and a review on classical NLP approaches. Then, we empirically test with a suite of different scenarios the behaviour of BERT against traditional TF-IDF vocabulary fed to machine learning models. The purpose of this work is adding empirical evidence to support the use of BERT as a default on NLP tasks. Experiments show the superiority of BERT and its independence of features of the NLP problem such as the language of the text adding empirical evidence to use BERT as a default technique in NLP problems.

Cross-Lingual Classification of Political Texts Using Multilingual Sentence Embeddings

AbstractEstablished approaches to analyze multilingual text corpora require either a duplication of analysts’ efforts or high-quality machine translation (MT). In this paper, I argue that multilingual sentence embedding (MSE) is an attractive alternative approach to language-independent text representation. To support this argument, I evaluate MSE for cross-lingual supervised text classification. Specifically, I assess how reliably MSE-based classifiers detect manifesto sentences’ topics and positions compared to classifiers trained using bag-of-words representations of machine-translated texts, and how this depends on the amount of training data. These analyses show that when training data are relatively scarce (e.g., 20K or less-labeled sentences), MSE-based classifiers can be more reliable and are at least no less reliable than their MT-based counterparts. Furthermore, I examine how reliable MSE-based classifiers label sentences written in languages not in the training data, focusing on the task of discriminating sentences that discuss the issue of immigration from those that do not. This analysis shows that compared to the within-language classification benchmark, such “cross-lingual transfer” tends to result in fewer reliability losses when relying on the MSE instead of the MT approach. This study thus presents an important addition to the cross-lingual text analysis toolkit.

#ChronicPain: Automated Building of a Chronic Pain Cohort from Twitter Using Machine Learning

Background Due to the high burden of chronic pain, and the detrimental public health consequences of its treatment with opioids, there is a high-priority need to identify effective alternative therapies. Social media is a potentially valuable resource for knowledge about self-reported therapies by chronic pain sufferers. Methods We attempted to (a) verify the presence of large-scale chronic pain-related chatter on Twitter, (b) develop natural language processing and machine learning methods for automatically detecting self-disclosures, (c) collect longitudinal data posted by them, and (d) semiautomatically analyze the types of chronic pain-related information reported by them. We collected data using chronic pain-related hashtags and keywords and manually annotated 4,998 posts to indicate if they were self-reports of chronic pain experiences. We trained and evaluated several state-of-the-art supervised text classification models and deployed the best-performing classifier. We collected all publicly available posts from detected cohort members and conducted manual and natural language processing-driven descriptive analyses. Results Interannotator agreement for the binary annotation was 0.82 (Cohen’s kappa). The RoBERTa model performed best (F 1 score: 0.84; 95% confidence interval: 0.80 to 0.89), and we used this model to classify all collected unlabeled posts. We discovered 22,795 self-reported chronic pain sufferers and collected over 3 million of their past posts. Further analyses revealed information about, but not limited to, alternative treatments, patient sentiments about treatments, side effects, and self-management strategies. Conclusion Our social media based approach will result in an automatically growing large cohort over time, and the data can be leveraged to identify effective opioid-alternative therapies for diverse chronic pain types.

Transparency and strategic promotion: How court press releases facilitate judicial agenda-building in Germany

Press releases disseminate selected information and promote particular issues. Although several courts regularly publish press releases on decisions, it is unclear which policy issues are more likely to be promoted by courts. This study argues that courts publish press releases to increase transparency and promote politically contested issues strategically. Using original data on the German Federal Constitutional Court, it assesses the role of press releases for judicial agenda-setting. Policy issues in 7114 court decisions between 1998 and 2019 were coded through supervised text classification. The results show that the press releases cover a greater range of policy issues than the Court’s decisions. They are more likely to be published for decisions that deal with politically contested issues. This study contributes to the policy agenda literature by adding a new case, the German Federal Constitutional Court, and court press releases as a new perspective.

Dissecting Non-Use of Online News – Systematic Evidence from Combining Tracking and Automated Text Classification

A high proportion of non-users of news is considered a concern for a functioning democracy. However, existing empirical assessments on the share of news avoiders rely exclusively on survey data and the results vary drastically between studies, making it difficult to evaluate the severity of the issue. This study relies on tracking data of Swiss Internet users and applies and discusses two computational methods, identifying news at the domain and article level, to realistically assess the extent of non-users of online news. Results indicate that at least 14.2% of Internet users do not use news online. Furthermore, this study suggests that identifying news use solely based on tracking data at the domain level is distorted by a faux news effect, i.e., non-news use on news domains, and an invisible news effect, i.e., news use on small and unknown news domains. The parallel use of tracking data and supervised text classification allows to dissect and discuss these effects systematically. Similarly, it is found that not accounting for news use via apps overestimates the extent of non-use of online news. The findings provide valuable insights for future applications of these methods in similar contexts.

The Executive Branch decisions in Brazil: A study of administrative decrees through machine learning and network analysis.

This paper dissects the potential of state-of-the-art computational analysis to promote the investigation of government’s administrative decisions and politics. The Executive Branch generates massive amounts of textual data comprising daily decisions in several levels and stages of the law and decree-making processes. The use of automated text analysis to explore this data based on the substantive interests of scholars runs into computational challenges. Computational methods have been applied to texts from the Legislative and Judicial Branches; however, there barely are suitable taxonomies to automate the classification and analysis of the Executive’s administrative decrees. To solve this problem, we put forward a computational framework to analyze the Brazilian administrative decrees from 2000 to 2019. Our strategy to uncover the contents and patterns of the presidential decree-making is developed in three main steps. First, we conduct an unsupervised text analysis through the LDA algorithm for topic modeling. Second, building upon the LDA results, we propose two taxonomies for the classification of decrees: (a) the ministerial coauthorship of the decrees to map policy areas and (b) the decrees’ fields of law based on a tagging system provided by the Brazilian Senate. Using these taxonomies, we compare the performance of three supervised text classification algorithms: SVM, Convolutional Neural Network, and Hierarchical Attention Network, achieving F1-scores of up to 80% when automatically classifying decrees. Third, we analyze the network generated by links between decrees through centrality and clustering approaches, distinguishing a set of administrative decisions related to the president’s priorities in the economic policy area. Our findings confirm the potential of our computational framework to explore N-large datasets, advance exploratory studies, and generate testable propositions in different research areas. They advance the monitoring of Brazil’s administrative decree-making process that is shaped by the president’s priorities and by the interplay among cabinet members.

Rule-Enhanced Active Learning for Semi-Automated Weak Supervision

A major bottleneck preventing the extension of deep learning systems to new domains is the prohibitive cost of acquiring sufficient training labels. Alternatives such as weak supervision, active learning, and fine-tuning of pretrained models reduce this burden but require substantial human input to select a highly informative subset of instances or to curate labeling functions. REGAL (Rule-Enhanced Generative Active Learning) is an improved framework for weakly supervised text classification that performs active learning over labeling functions rather than individual instances. REGAL interactively creates high-quality labeling patterns from raw text, enabling a single annotator to accurately label an entire dataset after initialization with three keywords for each class. Experiments demonstrate that REGAL extracts up to 3 times as many high-accuracy labeling functions from text as current state-of-the-art methods for interactive weak supervision, enabling REGAL to dramatically reduce the annotation burden of writing labeling functions for weak supervision. Statistical analysis reveals REGAL performs equal or significantly better than interactive weak supervision for five of six commonly used natural language processing (NLP) baseline datasets.

Spreading the word? European Union agencies and social media attention

Public agencies need to distribute information to their manifold audience quickly and directly. The emergence of social media platforms has sparked positive projections about future government-public interactions via the internet and almost every EU agency has created social media presences on the leading social media platforms. However, social media accounts of agencies receive strongly varying amounts of public attention and therefore display varying degrees of usefulness to connect with the public. This research examines which factors influence how much long-standing and temporal attention social media accounts of EU agencies receive. Using an extensive Twitter dataset of EU agencies and a new methodology that employs supervised text classification through the novel BERT language model to classify agency tweets, possible explanations of social media attention are tested. Results show that long-standing social media attention (i.e., size of the followership) is mostly explained by salience in traditional news, account age, and tweeting frequency, whilst a more interactive communication style tends to yield more temporal attention (i.e., number of retweets). The findings underline previous assumptions that employing a more interactive communication style maximizes public organizations' potential to connect with their audiences on social media.

A new document representation based on global policy for supervised term weighting schemes in text categorization.

There are two main factors involved in documents classification, document representation method and classification algorithm. In this study, we focus on document representation method and demonstrate that the choice of representation methods has impacts on quality of classification results. We propose a document representation strategy for supervised text classification named document representation based on global policy (DRGP), which can obtain an appropriate document representation according to the distribution of terms. The main idea of DRGP is to construct the optimization function through the importance of terms to different categories. In the experiments, we investigate the effects of DRGP on the 20 Newsgroups, Reuters21578 datasets, and using the SVM as classifier. The results show that the DRGP outperforms other text representation strategy schemes, such as Document Max, Document Two Max and global policy.

Greasing the wheels for comparative communication research: Supervised text classification for multilingual corpora

Employing supervised machine learning for text classification is already a resource-intensive endeavor in a monolingual setting. However, facing the challenge to classify a multilingual corpus, the cost of producing the required annotated documents quickly exceeds even generous time and financial constraints. We show how tools like automated annotation and machine translation can not only efficiently but also effectively be employed for the classification of a multilingual corpus with supervised machine learning. Our findings demonstrate that good results can already be achieved with the machine translation of about 250 to 350 documents per category class and language and a dictionary in just one language, which we perceive as a realistic scenario for many projects. The methodological strategy is applied to study migration frames in seven languages (news discourse in seven European countries) and discussed and evaluated for its usability in comparative communication research.

Hoax analyzer for Indonesian news using RNNs with fasttext and glove embeddings

Misinformation has become an innocuous yet potentially harmful problem ever since the development of internet. Numbers of efforts are done to prevent the consumption of misinformation, including the use of artificial intelligence (AI), mainly natural language processing (NLP). Unfortunately, most of natural language processing use English as its linguistic approach since English is a high resource language. On the contrary, Indonesia language is considered a low resource language thus the amount of effort to diminish consumption of misinformation is low compared to English-based natural language processing. This experiment is intended to compare fastText and GloVe embeddings for four deep neural networks (DNN) models: long short-term memory (LSTM), bidirectional long short-term memory (BI-LSTM), gated recurrent unit (GRU) and bidirectional gated recurrent unit (BI-GRU) in terms of metrics score when classifying news between three classes: fake, valid, and satire. The latter results show that fastText embedding is better than GloVe embedding in supervised text classification, along with BI-GRU + fastText yielding the best result.

Enhancing text classification performance by preprocessing misspelled words in Indonesian language

Supervised learning using shallow machine learning methods is still a popular method in processing text, despite the rapidly advancing sector of unsupervised methodologies using deep learning. Supervised text classification for application user feedback sentiments in Indonesian Language is one of the applications which is quite popular in both the research community and industry. However, due to the nature of shallow machine learning approaches, various text preprocessing techniques are required to clean the input data. This research aims to implement and evaluate the role of Levenshtein distance algorithm in detecting and preprocessing misspelled words in Indonesian language, before the text data is then used to train a user feedback sentiment classification model using multinomial Naïve Bayes. This research experimented with various evaluation scenarios, and found that preprocessing misspelled words in Indonesian language using the Levenshtein distance algorithm could be useful and showed a promising 8.2% increase on the accuracy of the model’s ability to classify user feedback text according to their sentiments.

Unsupervised Text Classification for Natural Language Interactive Narratives

Natural language interactive narratives are a variant of traditional branching storylines where player actions are expressed in natural language rather than by selecting among choices. Previous efforts have handled the richness of natural language input using machine learning technologies for text classification, bootstrapping supervised machine learning approaches with human-in-the-loop data acquisition or by using expected player input as fake training data. This paper explores a third alternative, where unsupervised text classifiers are used to automatically route player input to the most appropriate storyline branch. We describe the Data-driven Interactive Narrative Engine (DINE), a web-based tool for authoring and deploying natural language interactive narratives. To compare the performance of different algorithms for unsupervised text classification, we collected thousands of user inputs from hundreds of crowdsourced participants playing 25 different scenarios, and hand-annotated them to create a gold-standard test set. Through comparative evaluations, we identified an unsupervised algorithm for narrative text classification that approaches the performance of supervised text classification algorithms. We discuss how this technology supports authors in the rapid creation and deployment of interactive narrative experiences, with authorial burdens similar to that of traditional branching storylines.

Hoax Analyzer for Indonesian News Using Deep Learning Models

Abstract Fake news has always been in a problem in many parts of the world. Since English is the most dominant language in the world, hoax analyzers are mostly made to cater to news done in English. This study presents various Deep Neural Network (DNN) models: Long Short-Term Memory (LSTM), Bidirectional LSTM (BI-LSTM), Gated Recurrent Unit (GRU), Bidirectional GRU (BI-GRU), and 1-Dimensional Convolutional Neural Network (1D-CNN) as well as two classifiers: Support Vector Machine (SVM) and Naive Bayes used to predict the validity of news done in Bahasa Indonesia. The results show that DNN models are superior to classifiers in supervised text classification tasks, with 1D-CNN achieving the best result.