Human Annotators Research Articles

Learning to Identify Illegal Landfills through Scene Classification in Aerial Images

Illegal landfills are uncontrolled disposals of waste that cause severe environmental and health risk. Discovering them as early as possible is of prominent importance for preventing hazards, such as fire pollution and leakage. Before the digital era, the only means to detect illegal waste dumps was the on site inspection of potentially suspicious sites, a procedure extremely costly and impossible to scale to a vast territory. With the advent of Earth observation technology, scanning the territory via aerial images has become possible. However, manual image interpretation remains a complex and time-consuming task that requires expert skill. Photo interpretation can be partially automated by embedding the expert knowledge within a data driven classifier trained with samples provided by human annotators. In this paper, the detection of illegal landfills is formulated as a multi-scale scene classification problem. Scene elements positioning and spatial relations constitute hints of the presence of illegal waste dumps. A dataset of ≈3000 images (20 cm resolution per pixel) was created with the help of expert photo interpreters. A combination of ResNet50 and Feature Pyramid Network (FPN) elements accounting for different object scales achieves 88% precision with an 87% of recall in a test area. The results proved the feasibility of applying convolutional neural networks for scene classification in this scenario to optimize the process of waste dumps detection.

Argumentation schemes: From genetics to international relations to environmental science policy to AI ethics

Argumentation schemes have played a key role in our research projects on computational models of natural argument over the last decade. The catalogue of schemes in Walton, Reed and Macagno’s 2008 book, Argumentation Schemes, served as our starting point for analysis of the naturally occurring arguments in written text, i.e., text in different genres having different types of author, audience, and subject domain (genetics, international relations, environmental science policy, AI ethics), for different argument goals, and for different possible future applications. We would often first attempt to analyze the arguments in our corpora in terms of those schemes, then adapt schemes as needed for the goals of the project, and in some cases implement them for use in computational models. Among computational researchers, the main interest in argumentation schemes has been for use in argument mining by applying machine learning methods to existing argument corpora. In contrast, a primary goal of our research has been to learn more about written arguments themselves in various contemporary fields. Our approach has been to manually analyze semantics, discourse structure, argumentation, and rhetoric in texts. Another goal has been to create sharable digital corpora containing the results of our studies. Our approach has been to define argument schemes for use by human corpus annotators or for use in logic programs for argument mining. The third goal is to design useful computer applications based upon our studies, such as argument diagramming systems that provide argument schemes as building blocks. This paper describes each of the various projects: the methods, the argument schemes that were identified, and how they were used. Then a synthesis of the results is given with a discussion of open issues.

Performance and Efficiency Evaluation of ASR Inference on the Edge

Automatic speech recognition, a process of converting speech signals to text, has improved a great deal in the past decade thanks to the deep learning based systems. With the latest transformer based models, the recognition accuracy measured as word-error-rate (WER), is even below the human annotator error (4%). However, most of these advanced models run on big servers with large amounts of memory, CPU/GPU resources and have huge carbon footprint. This server based architecture of ASR is not viable in the long run given the inherent lack of privacy for user data, reliability and latency issues of the network connection. On the other hand, on-device ASR (meaning, speech to text conversion on the edge device itself) solutions will fix deep-rooted privacy issues while at same time being more reliable and performant by avoiding network connectivity to the back-end server. On-device ASR can also lead to a more sustainable solution by considering the energy vs. accuracy trade-off and choosing right model for specific use cases/applications of the product. Hence, in this paper we evaluate energy-accuracy trade-off of ASR with a typical transformer based speech recognition model on an edge device. We have run evaluations on Raspberry Pi with an off-the-shelf USB meter for measuring energy consumption. We conclude that, in the case of CPU based ASR inference, the energy consumption grows exponentially as the word error rate improves linearly. Additionally, based on our experiment we deduce that, with PyTorch mobile optimization and quantization, the typical transformer based ASR on edge performs reasonably well in terms of accuracy and latency and comes close to the accuracy of server based inference.

Microstructure classification in the unsupervised context

Traditional microstructure classification requires human annotations provided by a subject matter expert. The requirement of human input is both costly and subjective and cannot keep up with the current volume of experimentally and computationally generated microstructure images. In this work, we develop a framework that is capable of reducing the cost of human annotation in this process by leveraging novel machine learning procedures for class discovery and label assignment. To reduce the penalty of a poor label assignment made by this automated process, labels are only assigned to high-confidence observations while ambiguous data are left unlabeled. Semi-supervised classification is then employed to leverage the high- and low-confidence label assignments, and a novel generalization of an established semi-supervised error estimation technique to the multi-class context is introduced to assess the resulting classifiers. Finally, it is shown that this framework can be used to produce highly accurate classifiers over microstructure image class taxonomies which are discovered solely through data-driven methods and which display consistent structural trends within and distinct morphological differences between classes.

Refining epileptogenic high-frequency oscillations using deep learning: a reverse engineering approach.

Intracranially recorded interictal high-frequency oscillations have been proposed as a promising spatial biomarker of the epileptogenic zone. However, its visual verification is time-consuming and exhibits poor inter-rater reliability. Furthermore, no method is currently available to distinguish high-frequency oscillations generated from the epileptogenic zone (epileptogenic high-frequency oscillations) from those generated from other areas (non-epileptogenic high-frequency oscillations). To address these issues, we constructed a deep learning-based algorithm using chronic intracranial EEG data via subdural grids from 19 children with medication-resistant neocortical epilepsy to: (i) replicate human expert annotation of artefacts and high-frequency oscillations with or without spikes, and (ii) discover epileptogenic high-frequency oscillations by designing a novel weakly supervised model. The ‘purification power’ of deep learning is then used to automatically relabel the high-frequency oscillations to distill epileptogenic high-frequency oscillations. Using 12 958 annotated high-frequency oscillation events from 19 patients, the model achieved 96.3% accuracy on artefact detection (F1 score = 96.8%) and 86.5% accuracy on classifying high-frequency oscillations with or without spikes (F1 score = 80.8%) using patient-wise cross-validation. Based on the algorithm trained from 84 602 high-frequency oscillation events from nine patients who achieved seizure-freedom after resection, the majority of such discovered epileptogenic high-frequency oscillations were found to be ones with spikes (78.6%, P < 0.001). While the resection ratio of detected high-frequency oscillations (number of resected events/number of detected events) did not correlate significantly with post-operative seizure freedom (the area under the curve = 0.76, P = 0.06), the resection ratio of epileptogenic high-frequency oscillations positively correlated with post-operative seizure freedom (the area under the curve = 0.87, P = 0.01). We discovered that epileptogenic high-frequency oscillations had a higher signal intensity associated with ripple (80–250 Hz) and fast ripple (250–500 Hz) bands at the high-frequency oscillation onset and with a lower frequency band throughout the event time window (the inverted T-shaped), compared to non-epileptogenic high-frequency oscillations. We then designed perturbations on the input of the trained model for non-epileptogenic high-frequency oscillations to determine the model’s decision-making logic. The model confidence significantly increased towards epileptogenic high-frequency oscillations by the artificial introduction of the inverted T-shaped signal template (mean probability increase: 0.285, P < 0.001), and by the artificial insertion of spike-like signals into the time domain (mean probability increase: 0.452, P < 0.001). With this deep learning-based framework, we reliably replicated high-frequency oscillation classification tasks by human experts. Using a reverse engineering technique, we distinguished epileptogenic high-frequency oscillations from others and identified its salient features that aligned with current knowledge.

Optimized Detection of Central Apneas Preceding Late-Onset Sepsis in Premature Infants.

In neonatal intensive care units, respiratory traces of premature infants developing late onset sepsis (LOS) may also show episodes of apneas. However, since clinical patient monitors often underdetect apneas, clinical experts are required to investigate patients' traces looking for these events. In this work we present a method to optimize an existing algorithm for central apnea (CA) detection and how we used it together with human annotations to investigate the occurrence of CAs preceding LOS.The algorithm was optimized by using a previously-annotated dataset consisting of 90 hours, extracted from 10 premature infants. This allowed to double precision (19.7% vs 9.3%, median values per patient) without affecting recall (90.5% vs 94.5%) compared to the original algorithm. This choice caused the missed identification of just 1 additional CA (4 vs 3) in the whole dataset. The optimized algorithm was then used to annotate a second dataset consisting of 480 hours, extracted from 10 premature infants diagnosed with LOS. Annotations were corrected by two clinical experts.A significantly higher number of CA annotations was found in the 6 hours prior to sepsis onset (p-value < 0.05). The use of the optimized algorithm followed by human annotations proved to be a suitable, time-efficient method to annotate CAs before sepsis in premature infants, enabling future use in large datasets.

3DSNP 2.0: update and expansion of the noncoding genomic variant annotation database.

The rapid development of single-molecule long-read sequencing (LRS) and single-cell assay for transposase accessible chromatin sequencing (scATAC-seq) technologies presents both challenges and opportunities for the annotation of noncoding variants. Here, we updated 3DSNP, a comprehensive database for human noncoding variant annotation, to expand its applications to structural variation (SV) and to implement variant annotation down to single-cell resolution. The updates of 3DSNP include (i) annotation of 108 317 SVs from a full spectrum of functions, especially their potential effects on three-dimensional chromatin structures, (ii) evaluation of the accessible chromatin peaks flanking the variants across 126 cell types/subtypes in 15 human fetal tissues and 54 cell types/subtypes in 25 human adult tissues by integrating scATAC-seq data and (iii) expansion of Hi-C data to 49 human cell types. In summary, this version is a significant and comprehensive improvement over the previous version. The 3DSNP v2.0 database is freely available at https://omic.tech/3dsnpv2/.

Using weak supervision to generate training datasets from social media data: a proof of concept to identify drug mentions.

Twitter has been a remarkable resource for research in pharmacovigilance in the last decade. Traditionally, rule- or lexicon-based methods have been utilized for automatically extracting drug tweets for human annotation. The process of human annotation to create labeled sets for machine learning models is laborious, time consuming and not scalable. In this work, we demonstrate the feasibility of applying weak supervision (noisy labeling) to select drug data, and build machine learning models using large amounts of noisy labeled data instead of limited gold standard labelled sets. Our results demonstrate the models built with large amounts of noisy data achieve similar performance than models trained on limited gold standard datasets, hence demonstrating that weak supervision helps reduce the need to rely on manual annotation, allowing more data to be easily labeled and useful for downstream machine learning applications, in this case drug mention identification.

Convolutional Autoencoding and Gaussian Mixture Clustering for Unsupervised Beat-to-Beat Heart Rate Estimation of Electrocardiograms from Wearable Sensors.

Heart rate is one of the most important diagnostic bases for cardiovascular disease. This paper introduces a deep autoencoding strategy into feature extraction of electrocardiogram (ECG) signals, and proposes a beat-to-beat heart rate estimation method based on convolution autoencoding and Gaussian mixture clustering. The high-level heartbeat features were first extracted in an unsupervised manner by training the convolutional autoencoder network, and then the adaptive Gaussian mixture clustering was applied to detect the heartbeat locations from the extracted features, and calculated the beat-to-beat heart rate. Compared with the existing heartbeat classification/detection methods, the proposed unsupervised feature learning and heartbeat clustering method does not rely on accurate labeling of each heartbeat location, which could save a lot of time and effort in human annotations. Experimental results demonstrate that the proposed method maintains better accuracy and generalization ability compared with the existing ECG heart rate estimation methods and could be a robust long-time heart rate monitoring solution for wearable ECG devices.

Segmentation of vestibular schwannoma from MRI, an open annotated dataset and baseline algorithm

Automatic segmentation of vestibular schwannomas (VS) from magnetic resonance imaging (MRI) could significantly improve clinical workflow and assist patient management. We have previously developed a novel artificial intelligence framework based on a 2.5D convolutional neural network achieving excellent results equivalent to those achieved by an independent human annotator. Here, we provide the first publicly-available annotated imaging dataset of VS by releasing the data and annotations used in our prior work. This collection contains a labelled dataset of 484 MR images collected on 242 consecutive patients with a VS undergoing Gamma Knife Stereotactic Radiosurgery at a single institution. Data includes all segmentations and contours used in treatment planning and details of the administered dose. Implementation of our automated segmentation algorithm uses MONAI, a freely-available open-source framework for deep learning in healthcare imaging. These data will facilitate the development and validation of automated segmentation frameworks for VS and may also be used to develop other multi-modal algorithmic models.

A Statistical Analysis of Summarization Evaluation Metrics Using Resampling Methods

Abstract The quality of a summarization evaluation metric is quantified by calculating the correlation between its scores and human annotations across a large number of summaries. Currently, it is unclear how precise these correlation estimates are, nor whether differences between two metrics’ correlations reflect a true difference or if it is due to mere chance. In this work, we address these two problems by proposing methods for calculating confidence intervals and running hypothesis tests for correlations using two resampling methods, bootstrapping and permutation. After evaluating which of the proposed methods is most appropriate for summarization through two simulation experiments, we analyze the results of applying these methods to several different automatic evaluation metrics across three sets of human annotations. We find that the confidence intervals are rather wide, demonstrating high uncertainty in the reliability of automatic metrics. Further, although many metrics fail to show statistical improvements over ROUGE, two recent works, QAEval and BERTScore, do so in some evaluation settings.1

Explore the Lipid-Lowering and Weight-Reducing Mechanism of Lotus Leaf Based on Network Pharmacology and Molecular Docking.

Objective To predict the target of the active ingredient of lotus leaf for lowering fat and losing weight. Explore its multicomponent, multitarget, multipath mechanism. Methods Screen the main active ingredients of lotus leaves through the TCMSP database, and use the TCMSP database to predict the potential targets of the active ingredients. Obtain obesity-related targets from the human genome annotation (GeneCards) database. Use Venn software to take the intersection of the two to obtain the effect target of the lotus leaf lipid-lowering and weight-reducing effects. Use Cytoscape 3.6.0 software to construct an effective ingredient-target network. Use the STRING database to construct an intersection target protein interaction (PPI) network, visualize it with Cytoscape 3.6.0 software, and perform network topology analysis to obtain the core target. Use the DAVID database to perform gene ontology (GO) and metabolic pathway (KEGG) enrichment analysis for the above targets. Use AutoDockTools software for molecular docking to verify the binding strength. Results A total of 15 main active ingredients such as quercetin, isorhamnetin, sitosterol, and kaempferol were obtained, which can act on 135 targets related to obesity. These targets are significantly enriched in multiple GO and KEGG entries such as hypoxia response, positive regulation of gene expression, response to toxic substances, aging, and positive regulation of RNA polymerase II promoter transcription. Molecular docking shows that flavonoids such as quercetin have better binding to the target protein Akt1. Conclusion The lipid-lowering and weight-reducing effects of lotus leaf embody the characteristics of multicomponent, multitarget, and multipathway of traditional Chinese medicine, which provides a certain scientific basis for the screening and in-depth study of the effective ingredients of lotus leaf.

Non-Visual Accessibility Assessment of Videos.

Video accessibility is crucial for blind screen-reader users as online videos are increasingly playing an essential role in education, employment, and entertainment. While there exist quite a few techniques and guidelines that focus on creating accessible videos, there is a dearth of research that attempts to characterize the accessibility of existing videos. Therefore in this paper, we define and investigate a diverse set of video and audio-based accessibility features in an effort to characterize accessible and inaccessible videos. As a ground truth for our investigation, we built a custom dataset of 600 videos, in which each video was assigned an accessibility score based on the number of its wins in a Swiss-system tournament, where human annotators performed pairwise accessibility comparisons of videos. In contrast to existing accessibility research where the assessments are typically done by blind users, we recruited sighted users for our effort, since videos comprise a special case where sight could be required to better judge if any particular scene in a video is presently accessible or not. Subsequently, by examining the extent of association between the accessibility features and the accessibility scores, we could determine the features that signifcantly (positively or negatively) impact video accessibility and therefore serve as good indicators for assessing the accessibility of videos. Using the custom dataset, we also trained machine learning models that leveraged our handcrafted features to either classify an arbitrary video as accessible/inaccessible or predict an accessibility score for the video. Evaluation of our models yielded an F 1 score of 0.675 for binary classification and a mean absolute error of 0.53 for score prediction, thereby demonstrating their potential in video accessibility assessment while also illuminating their current limitations and the need for further research in this area.

Creating Computational Models for Planning TTFields Treatment for Tumors in the Infratentorial Brain

Tumor Treating Fields (TTFields) is an FDA-approved treatment for glioblastoma multiforme (GBM) and malignant pleural mesothelioma (MPM), and is currently being investigated in a phase III trial for the treatment of brain metastases from non-small cell lung cancer (NSCLC). Increased TTFields dose density at the tumor is associated with longer patient survival. Therefore, estimating dose at the tumor utilizing computational simulations with patient-specific computational models are integral for treatment planning. NSCLC brain metastases are often observed in infratentorial brain regions. In these cases, TTFields arrays may be placed on the head, neck, and upper back. Manual segmentation of this large area is impractical and time consuming. Therefore, we developed a computational method for semi-automatic segmentation of infratentorial tissues relevant to TTFields treatment planning.We incorporated a dataset of 20 head T1w Gad MRIs of patients treated with TTFields with segmentation of the infratentorial area, and 20 CT images that incorporate head, neck and upper torso from a public database for segmentation of extracranial structures. For the segmentation of the infratentorial zone we developed an atlas-based method that deforms a predefined statistical atlas to best fit the patient's MRI. Our method then revises the initial fitting to ensure the segmented structures adhere to known anatomical relations. For supratentorial brain areas, we incorporate a custom atlas-based method. Extra-cranial zones (head, neck and upper torso) were segmented with threshold-based methods for the skin, muscle, fat, bones, and bronchi. User-defined landmarks were used to segment the esophagus and the artery. The spine and CSF were segmented automatically by identifying the relevant area and defining a constant ratio between their diameters. To evaluate the segmentation of the infratentorial regions, a trained annotator manually segmented the infratentorial area of the 20 head MRIs. We compared manual and automatic segmentations and measured the Dice overlap score.The average Dice coefficient between the human annotator and the algorithms' segmentation was 0.84 (SD = 0.05) for cerebellum and 0.67 (SD = 0.05) for brainstem. These results are comparable to those observed in supratentorial regions with a method that was verified previously. The radiation oncologist confirmed that the semi-automatic segmentation of the infratentorial, and extracranial head, neck and upper torso provides a quality result for TTFields treatment planning in a reasonable amount of time for all cases. The infratentorial segmentation is fully automatic and typically lasts a few minutes. Segmentation of extracranial structures is semi-automatic and typically lasts less than an hour.We have developed a fast method for the segmentation of infratentorial structures for TTFields planning.

Human-Guided Modality Informativeness for Affective States.

This paper studies the hypothesis that not all modalities are always needed to predict affective states. We explore this hypothesis in the context of recognizing three affective states that have shown a relation to a future onset of depression: positive, aggressive, and dysphoric. In particular, we investigate three important modalities for face-to-face conversations: vision, language, and acoustic modality. We first perform a human study to better understand which subset of modalities people find informative, when recognizing three affective states. As a second contribution, we explore how these human annotations can guide automatic affect recognition systems to be more interpretable while not degrading their predictive performance. Our studies show that humans can reliably annotate modality informativeness. Further, we observe that guided models significantly improve interpretability, i.e., they attend to modalities similarly to how humans rate the modality informativeness, while at the same time showing a slight increase in predictive performance.

Understanding Human-side Impact of Sampling Image Batches in Subjective Attribute Labeling

Capturing human annotators' subjective responses in image annotation has become crucial as vision-based classifiers expand the range of application areas. While there has been significant progress in image annotation interface design in general, relatively little research has been conducted to understand how to elicit reliable and cost-efficient human annotation when the nature of the task includes a certain level of subjectivity. To bridge this gap, we aim to understand how different sampling methods in image batch labeling, a design that allows human annotators to label a batch of images simultaneously, can impact human annotation performances. In particular, we developed three different strategies in forming image batches: (1) uncertainty-based labeling (UL) that prioritizes images that a classifier predicts with the highest uncertainty, (2) certainty-based labeling (CL), a reverse strategy of UL, and (3) random, a baseline approach that randomly selects images. Although UL and CL solely select images to be labeled from a classifier's point of view, we hypothesized that human-side perception and labeling performance may also vary depending on the different sampling strategies. In our study, we observed that participants were able to recognize a different level of perceived cognitive load across three conditions (CL the easiest while UL the most difficult). We also observed a trade-off between annotation task effectiveness (CL and UL more reliable than random) and task efficiency (UL the most efficient while CL the least efficient). Based on the results, we discuss the implications of design and possible future research directions of image batch labeling.

The Challenge of Variable Effort Crowdsourcing and How Visible Gold Can Help

We consider a class of variable effort human annotation tasks in which the number of labels required per item can greatly vary (e.g., finding all faces in an image, named entities in a text, bird calls in an audio recording, etc.). In such tasks, some items require far more effort than others to annotate. Furthermore, the per-item annotation effort is not known until after each item is annotated since determining the number of labels required is an implicit part of the annotation task itself. On an image bounding-box task with crowdsourced annotators, we show that annotator accuracy and recall consistently drop as effort increases. We hypothesize reasons for this drop and investigate a set of approaches to counteract it. Firstly, we benchmark on this task a set of general best-practice methods for quality crowdsourcing. Notably, only one of these methods actually improves quality: the use of visible gold questions that provide periodic feedback to workers on their accuracy as they work. Given these promising results, we then investigate and evaluate variants of the visible gold approach, yielding further improvement. Final results show a 7% improvement in bounding-box accuracy over the baseline. We discuss the generality of the visible gold approach and promising directions for future research.

Impact of human gene annotations on RNA-seq differential expression analysis

BackgroundDifferential expression (DE) analysis of RNA-seq data typically depends on gene annotations. Different sets of gene annotations are available for the human genome and are continually updated–a process complicated with the development and application of high-throughput sequencing technologies. However, the impact of the complexity of gene annotations on DE analysis remains unclear.ResultsUsing “mappability”, a metric of the complexity of gene annotation, we compared three distinct human gene annotations, GENCODE, RefSeq, and NONCODE, and evaluated how mappability affected DE analysis. We found that mappability was significantly different among the human gene annotations. We also found that increasing mappability improved the performance of DE analysis, and the impact of mappability mainly evident in the quantification step and propagated downstream of DE analysis systematically.ConclusionsWe assessed how the complexity of gene annotations affects DE analysis using mappability. Our findings indicate that the growth and complexity of gene annotations negatively impact the performance of DE analysis, suggesting that an approach that excludes unnecessary gene models from gene annotations improves the performance of DE analysis.

Segmentation-Less, Automated, Vascular Vectorization.

Recent advances in two-photon fluorescence microscopy (2PM) have allowed large scale imaging and analysis of blood vessel networks in living mice. However, extracting network graphs and vector representations for the dense capillary bed remains a bottleneck in many applications. Vascular vectorization is algorithmically difficult because blood vessels have many shapes and sizes, the samples are often unevenly illuminated, and large image volumes are required to achieve good statistical power. State-of-the-art, three-dimensional, vascular vectorization approaches often require a segmented (binary) image, relying on manual or supervised-machine annotation. Therefore, voxel-by-voxel image segmentation is biased by the human annotator or trainer. Furthermore, segmented images oftentimes require remedial morphological filtering before skeletonization or vectorization. To address these limitations, we present a vectorization method to extract vascular objects directly from unsegmented images without the need for machine learning or training. The Segmentation-Less, Automated, Vascular Vectorization (SLAVV) source code in MATLAB is openly available on GitHub. This novel method uses simple models of vascular anatomy, efficient linear filtering, and vector extraction algorithms to remove the image segmentation requirement, replacing it with manual or automated vector classification. Semi-automated SLAVV is demonstrated on three in vivo 2PM image volumes of microvascular networks (capillaries, arterioles and venules) in the mouse cortex. Vectorization performance is proven robust to the choice of plasma- or endothelial-labeled contrast, and processing costs are shown to scale with input image volume. Fully-automated SLAVV performance is evaluated on simulated 2PM images of varying quality all based on the large (1.4×0.9×0.6 mm3 and 1.6×108 voxel) input image. Vascular statistics of interest (e.g. volume fraction, surface area density) calculated from automatically vectorized images show greater robustness to image quality than those calculated from intensity-thresholded images.

Identification of user’s credibility on twitter social networks

&lt;p&gt;Twitter is one of the most influential social media platforms, facilitates the spreading of information in the form of text, images, and videos. However, the credibility of posted content is still trailed by an interrogation mark. Introduction: In this paper, a model has been developed for finding the user’s credibility based on the tweets which they had posted on Twitter social networks. The model consists of machine learning algorithms that assist not only in categorizing the tweets into credibility classes but also helps in finding user’s credibility ratings on the social media platform. Methods and results: The dataset and associated features of 100,000 tweets were extracted and pre-processed. Furthermore, the credibility class labelling of tweets was performed using four different human annotators. The meaning cloud and natural language understanding platforms were used for calculating the polarity, sentiment, and emotions score. The K-Means algorithm was applied for finding the clusters of tweets based on features set, whereas, random forest, support vector machine, naïve Bayes, K-nearest-neighbours (KNN), J48 decision tree, and multilayer perceptron were used for classifying the tweets into credibility classes. A significant level of accuracy, precision, and recall was provided by all the classifiers for all the given credibility classes.&lt;/p&gt;