Annotated Instances Research Articles

Active Learning with Particle Swarm Optimization for Enhanced Skin Cancer Classification Utilizing Deep CNN Models.

Skin cancer is a critical global health issue, with millions of non-melanoma and melanoma cases diagnosed annually. Early detection is essential to improving patient outcomes, yet traditional deep learning models for skin cancer classification are often limited by the need for large, annotated datasets and extensive computational resources. The aim of this study is to address these limitations by proposing an efficient skin cancer classification framework that integrates active learning (AL) with particle swarm optimization (PSO). The AL framework selectively identifies the most informative unlabeled instances for expert annotation, minimizing labeling costs while optimizing classifier performance. PSO, a nature-inspired metaheuristic algorithm, enhances the selection process within AL, ensuring the most relevant data points are chosen. This method was applied to train multiple Convolutional Neural Network (CNN) models on the HAM10000 skin lesion dataset. Experimental results demonstrate that the proposed AL-PSO approach significantly improves classification accuracy, with the Least Confidence strategy achieving approximately 89.4% accuracy while using only 40% of the labeled training data. This represents a substantial improvement over traditional approaches in terms of both accuracy and efficiency. The findings indicate that the integration of AL and PSO can accelerate the adoption of AI in clinical settings for skin cancer detection. The code for this study is publicly available at ( https://github.com/Sayantani-31/AL-PSO ).

Motorcycle Recognition System Using Convolutional Neural Network

Vehicle detection is becoming increasingly important for highway management, particularly in Aklan, where motorcycles and tricycles are the predominant modes of transportation. Due to their diverse designs, accurate detection of these vehicles remains challenging. This study addresses this issue by developing a vision-based motorcycle recognition system using a Convolutional Neural Network (CNN) implemented in MATLAB. A new high-definition dataset, comprising 34,002 annotated instances from 17,785 motorcycle images and 16,217 tricycle images, was created. The dataset was collected from various locations around Ibajay, Aklan, using regular cameras and smartphones. The images were classified into motorcycles and tricycles and then used to train the CNN model, which was compared with the You Only Look Once (YOLO) version 2 network. Experimental results indicate that the proposed CNN-based motorcycle recognition system achieves high detection accuracy, comparable to the YOLOv2 model. This system contributes to Sustainable Development Goal (SDG) 11 by improving sustainable urban transportation systems and traffic management. Additionally, it holds significant commercial value for motorcycle brands through enhanced vehicle tracking and market analysis.

Overall positive prototype for few-shot open-set recognition

Few-shot open-set recognition (FSOR) is the task of recognizing samples in known classes with a limited number of annotated instances while also detecting samples that do not belong to any known class. This is a challenging problem because the models must learn to generalize from a small number of labeled samples and distinguish them from an unlimited number of potential negative examples. In this paper, we propose a novel approach called overall positive prototype to effectively improve performance. Conceptually, negative samples would distribute throughout the feature space and are hard to be described. From the opposite viewpoint, we propose to construct an overall positive prototype that acts as a cohesive representation for positive samples that distribute in a relatively smaller neighborhood. By measuring the distance between a query sample and the overall positive prototype, we can effectively classify it as either positive or negative. We show that this simple yet innovative approach provides the state-of-the-art FSOR performance in terms of accuracy and AUROC.

Answering List-Type Questions in Health Domain with Pretrained Large Language Model: A Case for COVID-19 Symptoms.

List-type questions, which can have a varying number of answers, are more common in the health domain where people seek for health-related information from a passage or passages. An example of this type of question answering task is to find COVID-19 symptoms from a Twitter post. However, due to the lack of annotated instances for supervised learning, automatic identification of COVID-19 symptoms from Twitter posts is challenging. We investigated detection of symptom mentions in Twitter posts using GPT-3, a pre-trained large language model, along with few-shot learning. Our results of 5-shot and 10-shot learning on a corpus of 655 annotated tweets demonstrate that few-shot learning with pre-trained large language model is a promising approach to answering list-type questions with a minimal amount of effort of annotation.

PDSE-Lite: lightweight framework for plant disease severity estimation based on Convolutional Autoencoder and Few-Shot Learning.

Plant disease diagnosis with estimation of disease severity at early stages still remains a significant research challenge in agriculture. It is helpful in diagnosing plant diseases at the earliest so that timely action can be taken for curing the disease. Existing studies often rely on labor-intensive manually annotated large datasets for disease severity estimation. In order to conquer this problem, a lightweight framework named "PDSE-Lite" based on Convolutional Autoencoder (CAE) and Few-Shot Learning (FSL) is proposed in this manuscript for plant disease severity estimation with few training instances. The PDSE-Lite framework is designed and developed in two stages. In first stage, a lightweight CAE model is built and trained to reconstruct leaf images from original leaf images with minimal reconstruction loss. In subsequent stage, pretrained layers of the CAE model built in the first stage are utilized to develop the image classification and segmentation models, which are then trained using FSL. By leveraging FSL, the proposed framework requires only a few annotated instances for training, which significantly reduces the human efforts required for data annotation. Disease severity is then calculated by determining the percentage of diseased leaf pixels obtained through segmentation out of the total leaf pixels. The PDSE-Lite framework's performance is evaluated on Apple-Tree-Leaf-Disease-Segmentation (ATLDS) dataset. However, the proposed framework can identify any plant disease and quantify the severity of identified diseases. Experimental results reveal that the PDSE-Lite framework can accurately detect healthy and four types of apple tree diseases as well as precisely segment the diseased area from leaf images by using only two training samples from each class of the ATLDS dataset. Furthermore, the PDSE-Lite framework's performance is compared with existing state-of-the-art techniques, and it is found that this framework outperformed these approaches. The proposed framework's applicability is further verified by statistical hypothesis testing using Student t-test. The results obtained from this test confirm that the proposed framework can precisely estimate the plant disease severity with a confidence interval of 99%. Hence, by reducing the reliance on large-scale manual data annotation, the proposed framework offers a promising solution for early-stage plant disease diagnosis and severity estimation.

Enhancing random forest predictive performance for foot and mouth disease outbreaks in Uganda: a calibrated uncertainty prediction approach for varying distributions.

Foot-and-mouth disease poses a significant threat to both domestic and wild cloven-hoofed animals, leading to severe economic losses and jeopardizing food security. While machine learning models have become essential for predicting foot-and-mouth disease outbreaks, their effectiveness is often compromised by distribution shifts between training and target datasets, especially in non-stationary environments. Despite the critical impact of these shifts, their implications in foot-and-mouth disease outbreak prediction have been largely overlooked. This study introduces the Calibrated Uncertainty Prediction approach, designed to enhance the performance of Random Forest models in predicting foot-and-mouth disease outbreaks across varying distributions. The Calibrated Uncertainty Prediction approach effectively addresses distribution shifts by calibrating uncertain instances for pseudo-label annotation, allowing the active learner to generalize more effectively to the target domain. By utilizing a probabilistic calibration model, Calibrated Uncertainty Prediction pseudo-annotates the most informative instances, refining the active learner iteratively and minimizing the need for human annotation and outperforming existing methods known to mitigate distribution shifts. This reduces costs, saves time, and lessens the dependence on domain experts while achieving outstanding predictive performance. The results demonstrate that Calibrated Uncertainty Prediction significantly enhances predictive performance in non-stationary environments, achieving an accuracy of 98.5%, Area Under the Curve of 0.842, recall of 0.743, precision of 0.855, and an F1 score of 0.791. These findings underscore Calibrated Uncertainty Prediction's ability to overcome the vulnerabilities of existing ML models, offering a robust solution for foot-and-mouth disease outbreak prediction and contributing to the broader field of predictive modeling in infectious disease management.

Dimensions of constructional meanings in the German Constructicon: Why collo-profiles matter

Abstract A constructicon, i.e., a structured inventory of constructions, essentially aims at documenting functions of lexical and grammatical constructions. Among other parameters, so-called constructional collo-profiles, as introduced by Herbst (2018, 2020), are conclusive for determining constructional meanings. They provide information on how relevant individual words are for construction slots, they hint at usage preferences of constructions and serve as a helpful indicator for semantic peculiarities of constructions. However, even though collo-profiles constitute an indispensable component of constructicon entries, they pose major challengers for constructicographers: For a constructicographic enterprise it is not feasible to conduct collostructional analyses for hundreds or even thousands of constructions. In this article, we introduce a procedure based on the large language model BERT that allows to predict collo-profiles without having to extensively annotate instances of constructions in a given corpus. Specifically, by discussing the constructions X macht Y ADJP (‘x makes Y ADJ’, e.g. he drives him crazy) and N1 PREP N1 (e.g., bumper to bumper, constructions over constructions), we show how the developed automated system generates collo-profiles based on a limited number of annotated instances. Finally, we place collo-profiles alongside other dimensions of constructional meanings included in the German Constructicon.

Groups of experts often differ in their decisions: What are the implications for AI and machine learning? A commentary onNoise: A Flaw in Human Judgment, byKahneman, Sibony, and Sunstein (2021)

AbstractMachine Learning systems rely heavily on annotated instances. Such annotations are frequently done by human experts, or by tools developed by experts, and so the central message of this book,Noise: A Flaw in Human Judgment(Kahneman, Sibony, and Sunstein 2021) is of considerable importance to AI/Machine Learning community. The core message is that if a number of experts are asked to annotate tasks that involve judgments, these responses will frequently differ. This observation poses a problem for how analysts choose a particular annotated dataset (from the group), or process the set of responses to give a “balanced” response, or whether to reject all the annotated datasets. A further important aspect of this book is the case studies which demonstrate that differences in judgments between fellow experts have been reported in a significant number of disciplines including, business, the law, government, and medicine. Kahneman, Sibony and Sunstein (2021), referred to as KSS subsequently, discuss how Expert Biases can be reduced, but the main focus of this book is a discussion of Noise, that is, differences that often occur between fellow experts, and how Noise can often be reduced. To address the last point KSS have formulated a set of six decision hygiene principles which include the recommendation that complex tasks should be subdivided, and then each subtask should be solved separately. A further principle is that each task should be solved by individual experts before the various judgments are discussed with fellow experts. Effectively, the book being reviewed covers three main topics: First, it reports several motivating studies that show how judgments of fellow experts varied significantly in the pricing of insurance premiums, and in setting the lengths of custodial sentences. These motivating studies very effectively illustrate the central concepts of Judgment, Noise, and Bias; that section also provides definitions of these core concepts and discusses how Noise is often amplified in group meetings. Secondly, the authors provide detailed discussion of further studies, in a variety of domains, which report the levels of disagreement between experts. Thirdly, KSS discusses how to reduce the levels of Noise between experts, as noted above, the authors refer to these as Principles of Noise Hygiene. These three parts are interwoven in a complex way throughout the book; in our view, the best overview of the book is given in the section Review and Conclusions: Taking Noise Seriously (KSS, p. 361).

Adaptive class augmented prototype network for few-shot relation extraction

Relation extraction is one of the most essential tasks of knowledge construction, but it depends on a large amount of annotated data corpus. Few-shot relation extraction is proposed as a new paradigm, which is designed to learn new relationships between entities with merely a small number of annotated instances, effectively mitigating the cost of large-scale annotation and long-tail problems. To generalize to novel classes not included in the training set, existing approaches mainly focus on tuning pre-trained language models with relation instructions and developing class prototypes based on metric learning to extract relations. However, the learned representations are extremely sensitive to discrepancies in intra-class and inter-class relationships and hard to adaptively classify the relations due to biased class features and spurious correlations, such as similar relation classes having closer inter-class prototype representation. In this paper, we introduce an adaptive class augmented prototype network with instance-level and representation-level augmented mechanisms to strengthen the representation space. Specifically, we design the adaptive class augmentation mechanism to expand the representation of classes in instance-level augmentation, and class augmented representation learning with Bernoulli perturbation context attention to enhance the representation of class features in representation-level augmentation and explore adaptive debiased contrastive learning to train the model. Experimental results have been demonstrated on FewRel and NYT-25 under various few-shot settings, and the proposed model has improved accuracy and generalization, especially for cross-domain and different hard tasks.

Meta In-Context Learning: Harnessing Large Language Models for Electrical Data Classification

The evolution of communication technology has driven the demand for intelligent power grids and data analysis in power systems. However, obtaining and annotating electrical data from intelligent terminals is time-consuming and challenging. We propose Meta In-Context Learning (M-ICL), a new approach that harnesses large language models to classify time series electrical data, which largely alleviates the need for annotated data when adapting to new tasks. The proposed M-ICL consists of two stages: meta-training and meta-testing. In meta-training, the model is trained on various tasks that have an adequate amount of training data. The meta-training stage aims to learn the mapping between electrical data and the embedding space of large language models. In the meta-testing stage, the trained model makes predictions on new tasks. By utilizing the in-context learning ability of large language models, M-ICL adapts models to new tasks effectively with only a few annotated instances (e.g., 1–5 training instances per class). Our contributions lie in the new application of large language models to electrical data classification and the introduction of M-ICL to improve the classification performance with the strong in-context learning ability of large language models. Furthermore, we conduct extensive experiments on 13 real-world datasets, and the experimental results show that the proposed M-ICL improves the average accuracy over all datasets by 19.06%, 12.06%, and 6.63% when only one, two, and five training instances for each class are available, respectively. In summary, M-ICL offers a promising solution to the challenges of electrical data classification.

Towards Hard Few-Shot Relation Classification

Few-shot relation classification (FSRC) focuses on recognizing novel relations by learning with merely a handful of annotated instances. Meta-learning has been widely adopted for such a task, which trains on randomly generated few-shot tasks to learn generic data representations. Despite impressive results achieved, existing models still perform suboptimally when handling hard FSRC tasks with similar categories that confuse the model to distinguish correctly. We argue this is largely due to two reasons, 1) ignoring pivotal and discriminate information that is crucial to distinguish confusing classes, and 2) training indiscriminately via randomly sampled tasks of varying difficulty. In this paper, we introduce a novel prototypical network approach with contrastive learning that learns more informative and discriminative representations by exploiting relation label information. We further design two strategies that increase the difficulty of training tasks and allow the model to adaptively learn to focus on hard tasks. By doing so, our model can better represent subtle inter-relation variance and grow up through task difficulty. Extensive experiments on three standard benchmarks demonstrate the effectiveness of our method.

Interaction Information Guided Prototype Representation Rectification for Few-Shot Relation Extraction

Few-shot relation extraction aims to identify and extract semantic relations between entity pairs using only a small number of annotated instances. Many recently proposed prototype-based methods have shown excellent performance. However, existing prototype-based methods ignore the hidden inter-instance interaction information between the support and query sets, leading to unreliable prototypes. In addition, the current optimization of the prototypical network only relies on cross-entropy loss. It is only concerned with the accuracy of the predicted probability for the correct label, ignoring the differences of other non-correct labels, which cannot account for relation discretization in semantic space. This paper proposes an attentional network of interaction information to obtain a more reliable relation prototype. Firstly, an inter-instance interaction information attention module is designed to mitigate prototype unreliability through interaction information between the support set and query set instances, utilizing category information hidden in the query set. Secondly, the similarity scalar, which is defined by the mixed features of the prototype and the relation and is added to the focal loss to improve the attention of hard examples. We conducted extensive experiments on two standard datasets and demonstrated the validity of our proposed model.

"Dummy Grandpa, Do You Know Anything?": Identifying and Characterizing Ad Hominem Fallacy Usage in the Wild

Today, participating in discussions on online forums is extremely commonplace and these discussions have started rendering a strong influence on the overall opinion of online users. Naturally, twisting the flow of the argument can have a strong impact on the minds of naive users, which in the long run might have socio-political ramifications, for example, winning an election or spreading targeted misinformation. Thus, these platforms are potentially highly vulnerable to malicious players who might act individually or as a cohort to breed fallacious arguments with a motive to sway public opinion. Ad hominem arguments are one of the most effective forms of such fallacies. Although a simple fallacy, it is effective enough to sway public debates in offline world and can be used as a precursor to shutting down the voice of opposition by slander. In this work, we take a first step in shedding light on the usage of ad hominem fallacies in the wild. First, we build a powerful ad hominem detector based on transformer architecture with high accuracy (F1 more than 83%, showing a significant improvement over prior work), even for datasets for which annotated instances constitute a very small fraction. We then used our detector on 265k arguments collected from the online debate forum – CreateDebate. Our crowdsourced surveys validate our in-the-wild predictions on CreateDebate data (94% match with manual annotation). Our analysis revealed that a surprising 31.23% of CreateDebate content contains ad hominem fallacy, and a cohort of highly active users post significantly more ad hominem to suppress opposing views. Then, our temporal analysis revealed that ad hominem argument usage increased significantly since the 2016 US Presidential election, not only for topics like Politics, but also for Science and Law. We conclude by discussing important implications of our work to detect and defend against ad hominem fallacies.

Online Active Learning for Drifting Data Streams.

Classification methods for streaming data are not new, but very few current frameworks address all three of the most common problems with these tasks: concept drift, noise, and the exorbitant costs associated with labeling the unlabeled instances in data streams. Motivated by this gap in the field, we developed an active learning framework based on a dual-query strategy and Ebbinghaus's law of human memory cognition. Called CogDQS, the query strategy samples only the most representative instances for manual annotation based on local density and uncertainty, thus significantly reducing the cost of labeling. The policy for discerning drift from noise and replacing outdated instances with new concepts is based on the three criteria of the Ebbinghaus forgetting curve: recall, the fading period, and the memory strength. Simulations comparing CogDQS with baselines on six different data streams containing gradual drift or abrupt drift with and without noise show that our approach produces accurate, stable models with good generalization ability at minimal labeling, storage, and computation costs.

Beyond Learning Features: Training a Fully-Functional Classifier with ZERO Instance-Level Labels

We attempt to train deep neural networks for classification without using any labeled data. Existing unsupervised methods, though mine useful clusters or features, require some annotated samples to facilitate the final task-specific predictions. This defeats the true purpose of unsupervised learning and hence we envisage a paradigm of `true' self-supervision, where absolutely no annotated instances are used for training a classifier. The proposed method first pretrains a deep network through self-supervision and performs clustering on the learned features. A classifier layer is then appended to the self-supervised network and is trained by matching the distribution of the predictions to that of a predefined prior. This approach leverages the distribution of labels for supervisory signals and consequently, no image-label pair is needed. Experiments reveal that the method works on major nominal as well as ordinal classification datasets and delivers significant performance.

Instance exploitation for learning temporary concepts from sparsely labeled drifting data streams

Continual learning from streaming data sources becomes more and more popular due to the increasing number of online tools and systems. Dealing with dynamic and everlasting problems poses new challenges for which traditional batch-based offline algorithms turn out to be insufficient in terms of computational time and predictive performance. One of the most crucial limitations is that we cannot assume having an access to a finite and complete data set – we always have to be ready for new data that may complement our model. This poses a critical problem of providing labels for potentially unbounded streams. In real world, we are forced to deal with very strict budget limitations, therefore, we will most likely face the scarcity of annotated instances, which are essential in supervised learning. In our work, we emphasize this problem and propose a novel instance exploitation technique. We show that when: (i) data is characterized by temporary non-stationary concepts, and (ii) there are very few labels spanned across a long time horizon, it is actually better to risk overfitting and adapt models more aggressively by exploiting the only labeled instances we have, instead of sticking to a standard learning mode and suffering from severe underfitting. We present different strategies and configurations for our methods, as well as an ensemble algorithm that attempts to maintain a sweet spot between risky and normal adaptation. Finally, we conduct a complex in-depth comparative analysis of our methods, using state-of-the-art streaming algorithms relevant for the given problem.

Identifying Adverse Drug Reaction-Related Text from Social Media: A Multi-View Active Learning Approach with Various Document Representations

Adverse drug reactions (ADRs) are a huge public health issue. Identifying text that mentions ADRs from a large volume of social media data is important. However, we need to address two challenges for high-performing ADR-related text detection: the data imbalance problem and the requirement of simultaneously using data-driven information and handcrafted information. Therefore, we propose an approach named multi-view active learning using domain-specific and data-driven document representations (MVAL4D), endeavoring to enhance the predictive capability and alleviate the requirement of labeled data. Specifically, a new view-generation mechanism is proposed to generate multiple views by simultaneously exploiting various document representations obtained using handcrafted feature engineering and by performing deep learning methods. Moreover, different from previous active learning studies in which all instances are chosen using the same selection criterion, MVAL4D adopts different criteria (i.e., confidence and informativeness) to select potentially positive instances and potentially negative instances for manual annotation. The experimental results verify the effectiveness of MVAL4D. The proposed approach can be generalized to many other text classification tasks. Moreover, it can offer a solid foundation for the ADR mention extraction task, and improve the feasibility of monitoring drug safety using social media data.

MiDTD: A Simple and Effective Distillation Framework for Distantly Supervised Relation Extraction

Relation extraction (RE), an important information extraction task, faced the great challenge brought by limited annotation data. To this end, distant supervision was proposed to automatically label RE data, and thus largely increased the number of annotated instances. Unfortunately, lots of noise relation annotations brought by automatic labeling become a new obstacle. Some recent studies have shown that the teacher-student framework of knowledge distillation can alleviate the interference of noise relation annotations via label softening. Nevertheless, we find that they still suffer from two problems: propagation of inaccurate dark knowledge and constraint of a unified distillation temperature . In this article, we propose a simple and effective Multi-instance Dynamic Temperature Distillation (MiDTD) framework, which is model-agnostic and mainly involves two modules: multi-instance target fusion (MiTF) and dynamic temperature regulation (DTR). MiTF combines the teacher’s predictions for multiple sentences with the same entity pair to amend the inaccurate dark knowledge in each student’s target. DTR allocates alterable distillation temperatures to different training instances to enable the softness of most student’s targets to be regulated to a moderate range. In experiments, we construct three concrete MiDTD instantiations with BERT, PCNN, and BiLSTM-based RE models, and the distilled students significantly outperform their teachers and the state-of-the-art (SOTA) methods.

Using Pre-trained Language Model to Enhance Active Learning for Sentence Matching

Active learning is an effective method to substantially alleviate the problem of expensive annotation cost for data-driven models. Recently, pre-trained language models have been demonstrated to be powerful for learning language representations. In this article, we demonstrate that the pre-trained language model can also utilize its learned textual characteristics to enrich criteria of active learning. Specifically, we provide extra textual criteria with the pre-trained language model to measure instances, including noise, coverage, and diversity. With these extra textual criteria, we can select more efficient instances for annotation and obtain better results. We conduct experiments on both English and Chinese sentence matching datasets. The experimental results show that the proposed active learning approach can be enhanced by the pre-trained language model and obtain better performance.

Automated Travel History Extraction From Clinical Notes for Informing the Detection of Emergent Infectious Disease Events: Algorithm Development and Validation.

BackgroundPatient travel history can be crucial in evaluating evolving infectious disease events. Such information can be challenging to acquire in electronic health records, as it is often available only in unstructured text.ObjectiveThis study aims to assess the feasibility of annotating and automatically extracting travel history mentions from unstructured clinical documents in the Department of Veterans Affairs across disparate health care facilities and among millions of patients. Information about travel exposure augments existing surveillance applications for increased preparedness in responding quickly to public health threats.MethodsClinical documents related to arboviral disease were annotated following selection using a semiautomated bootstrapping process. Using annotated instances as training data, models were developed to extract from unstructured clinical text any mention of affirmed travel locations outside of the continental United States. Automated text processing models were evaluated, involving machine learning and neural language models for extraction accuracy.ResultsAmong 4584 annotated instances, 2659 (58%) contained an affirmed mention of travel history, while 347 (7.6%) were negated. Interannotator agreement resulted in a document-level Cohen kappa of 0.776. Automated text processing accuracy (F1 85.6, 95% CI 82.5-87.9) and computational burden were acceptable such that the system can provide a rapid screen for public health events.ConclusionsAutomated extraction of patient travel history from clinical documents is feasible for enhanced passive surveillance public health systems. Without such a system, it would usually be necessary to manually review charts to identify recent travel or lack of travel, use an electronic health record that enforces travel history documentation, or ignore this potential source of information altogether. The development of this tool was initially motivated by emergent arboviral diseases. More recently, this system was used in the early phases of response to COVID-19 in the United States, although its utility was limited to a relatively brief window due to the rapid domestic spread of the virus. Such systems may aid future efforts to prevent and contain the spread of infectious diseases.