Annotation Scheme Research Articles

COLD: Annotation scheme and evaluation data set for complex offensive language in English

COLD: Annotation scheme and evaluation data set for complex offensive language in English

Comparing Posture Classifications From Wearable Camera Still-images To Accelerometry

PURPOSE: To assess the convergent validity of a novel image annotation schema for determining posture from wearable camera still-images (IMGs) and activPAL (AP) posture classifications. METHODS: Participants (n=5, mean age 45y, range 21-81y, 3F) wore an AP monitor at the right anterior-mid-thigh and an Autographer wearable camera (WC) above the xiphoid process during two or three, 2hr visits. WC was set to capture IMGs every 5-60 sec. IMGs were annotated with the Oxford Image Browser software for 3 postures that matched AP classifications: sitting, standing and movement. A sequence of 3 IMGs denoted a postural “event”. Sequences of fewer than 3 IMGs and when IMGs could not be accurately classified were annotated as “transitions”. For analyses, IMGs and AP output were converted to one-sec epochs and matched sec-by-sec. Total visit time and event time is reported in min. Overall and event percent agreement between AP and IMGs were calculated. Within events, statistical bias and CIs for posture times from IMGs to AP posture times were calculated to determine accuracy and precision with mean posture times from AP stated as reference. Confusion matrices were computed to determine misclassification. RESULTS: 13 visits were analyzed with a total visit time of 1546 min and total event time of 1375 min. Mean overall percent agreement including events and transitions was 72%, while mean event percent agreement was 80%. Fig.1 shows bias and mean event posture min from AP, where bias was low for sitting but IMGs tended to underestimate standing and overestimate movement. From confusion matrices, IMGs misclassify standing as movement 46% of the time. CONCLUSION: Within events, IMGs annotated with the novel schema can differentiate sitting and upright postures. Future work is warranted to both improve posture classification and examine IMG accuracy and precision in assessing activity behavior. Supported by NIH 1R01CA215318

Attraction, selection, and attrition in online health communities: Initial conversations and their association with subsequent activity levels

BackgroundThe effectiveness of online health communities (OHCs) for improving outcomes for health care consumers, health professionals, and health services has already been well investigated. However, research on determinants of OHC users’ activity levels, what is associated with attrition or attraction to these communities, and the impacts of initial posts is limited. ObjectivesWe sought to explore topic exchanges in OHCs and determine how users’ initial posts and community reactions to them are associated with their subsequent activity levels. We also aimed to extend the theory of Attraction-Selection-Attrition for Online Communities (OCASA) to this area. MethodsWe examined exchanges in a major Australian OHC for cancer patients, analyzing about 2500 messages posted over 2009–18. We developed a novel annotation scheme to examine new members’ initial posts and the community’s reactions to them. ResultsThe annotation scheme includes five themes: informational support provision, emotional support provision, requests for help, self-reflection & disclosures, and conversational cues. Initial conversations were associated with future activity levels in terms of active posting versus non-active engagement in the community. We found that most OHC members disclosed personal reflections to bond with the community, and many actively posted to the community solely to provide informational and emotional support to others. ConclusionOur work extends OCASA theory to bond-based contexts, presents a new annotation scheme for OHC support topics, and makes an important contribution to knowledge about the relationship between users’ activity levels and their initial posts. The findings help managers and owners understand how members use OHCs and how to encourage active participation. They also suggest how to attract new members and minimize attrition among existing members.

Narrative production in autistic adults: A systematic analysis of the microstructure, macrostructure and internal state language

While narrative competence has been well documented in autistic children and young adolescents, fairly little is known about narrative performance of autistic adults. However, narrative abilities continue to develop well into adulthood. Hence, the main objective of the present study is to provide a clearer linguistic and communicative profile of ASD in adulthood by performing a systematic description of narrative performance in autistic adults. A specific annotation scheme was developed to code narrative production, in order to be able to compare the production of autistic participants to pairwise matched neurotypical adults relative to microstructure (syntactic complexity), macrostructure (overall story structure and cohesive ties) and internal state language of the corpus' narratives. The results suggest that autistic adults performed worse than their neurotypical peers on all three dimensions of narrative production, resulting in less coherent narratives overall for autistic adults.

“Talk to you later”

Abstract This article presents an applied discussion of the possibility of integrating conversation analysis (CA) methodology into that of machine learning. The aim is to improve the detection of that which resembles disengagement in the interaction between a robot and a human. We offer a novel analytical assemblage at the heart of the two disciplines, and namely on the level of the annotation schemes provided by conversation analysis transcription methods. First, we demonstrate that the need for a stable structure in establishing an interaction scenario and in designing robot behaviours does not prevent the emergence of ordinariness or creativity among the participants engaged in this interaction. Secondly, based on an actual case, we emphasize the possibility of systematicness in CA transcription to support the choice (a) of the categories targeted by prediction methods and defined by the annotation scheme, and (b) of the verbal and non-verbal features used to create prediction models.

Heuristische Textpraktiken in den Wissenschaften

Abstract This article contains a comparative study of heuristic textual practices in various scientific disciplines. By this we mean formulation practices with which new knowledge is generated in institutionally influenced routines and connected to existing knowledge, e. g. ‚highlighting the relevance of a research topic‘, ‚defining a concept‘ or ‚supporting a statement argumentatively‘. The aim is to find out to what extent such textual practices occur in different scientific disciplines, how they are distributed and combined. Furthermore, we study the effects domain-specific contexts have on heuristic textual practices. The data basis of our study is a corpus of 65 dissertations from the 13 different faculties of the TU Darmstadt. In the pilot study we report here, we examined the introductory chapters of the dissertations. Methodologically, it is an annotation study: Based on the current state of research on the subject, we have derived a basic annotation scheme, which we have developed and refined in a collaborative process of guideline creation. Our study affiliates on socio-pragmatic research on text production and formulation routines in the sciences. It is theoretically informed by the philosophy of science research on heuristics, methodically we make a contribution to the scientific debate on collaborative annotation procedures.

TeamTat: a collaborative text annotation tool.

Manually annotated data is key to developing text-mining and information-extraction algorithms. However, human annotation requires considerable time, effort and expertise. Given the rapid growth of biomedical literature, it is paramount to build tools that facilitate speed and maintain expert quality. While existing text annotation tools may provide user-friendly interfaces to domain experts, limited support is available for figure display, project management, and multi-user team annotation. In response, we developed TeamTat (https://www.teamtat.org), a web-based annotation tool (local setup available), equipped to manage team annotation projects engagingly and efficiently. TeamTat is a novel tool for managing multi-user, multi-label document annotation, reflecting the entire production life cycle. Project managers can specify annotation schema for entities and relations and select annotator(s) and distribute documents anonymously to prevent bias. Document input format can be plain text, PDF or BioC(uploaded locally or automatically retrieved from PubMed/PMC), and output format is BioC with inline annotations. TeamTat displays figures from the full text for the annotator's convenience. Multiple users can work on the same document independently in their workspaces, and the team manager can track task completion. TeamTat provides corpus quality assessment via inter-annotator agreement statistics, and a user-friendly interface convenient for annotation review and inter-annotator disagreement resolution to improve corpus quality.

The brain in a box: A toolbox for creating Cartesian geometric representations with isometric dimensions (Cgrids)

BackgroundThe folding of the human cortex complicates extraction of position information and recognition of patterns across the cortical surface. New MethodAs straight lines correspond better to our intuitions in spatial orientation, we developed an approach for imposing Cartesian grids on portions of the cortical surface, which can then be represented in a rectangular matrix. These functions have been implemented in the Cgrid (Cartesian Geometric Representation with Isometric Dimensions) toolbox. Cgrids can be generated based on regions of interest, or combinations thereof, according to any one of the Freesurfer’s annotation schemes. ResultsThe toolbox was evaluated using the surface reconstructions of T1-weighted images of 30 subjects, and 17 different Cgrids that in combination covered nearly the entire surface area of the brain. The vast majority of Cgrids (90.4 %) could be generated without issues. Comparison with Existing method(s)The toolbox facilitates spatial orientation and pattern recognition, in addition to allowing detailed comparison between the left and right hemisphere, and bringing existing volumetric tools to bear on surface-based data. The output of the toolbox is fully compatible with most existing fMRI/MRI analyses packages, and is immediately suitable as input for second level analysis. ConclusionsThe toolbox has the potential for broad applicability, especially when ease of data handling and representation are critical factors. The toolbox can be downloaded from: https://github.com/mathijsraemaekers/Cgrid-toolbox.

Network aggregation improves gene function prediction of grapevine gene co-expression networks.

Aggregation across multiple networks highlights robust co-expression interactions and improves the functional connectivity of grapevine gene co-expression networks. In recent years, the rapid accumulation of transcriptome datasets from diverse experimental conditions has enabled the widespread use of gene co-expression network (GCN) analysis in plants. In grapevine, GCN analysis has shown great promise for gene function prediction, however, measurable progress is currently lacking. Using accumulated microarray datasets from the grapevine whole-genome array (33 experiments, 1359 samples), we explored how meta-analysis through aggregation influences the functional connectivity (performance) of derived networks using guilt-by-association neighbor voting. Two annotation schemes, i.e. MapMan BIN and Pfam, at two sparsity thresholds, i.e. top 100 (stringent) and 300 (relaxed) ranked genes were evaluated. We observed that aggregating across multiple networks improves performance dramatically, with the aggregate outperforming the majority of functional terms across individual networks. Network sparsity and size (i.e. the number of samples and aggregates) were key factors influencing performance while the choice of annotation scheme had little. Systematic comparison with various state-of-the-art microarray andRNA-seq networks was also performed, however, none outperformed the aggregate microarray network despite having good predictive performance. Repeating these series of tests using a functional enrichment-based performance metric also showed remarkably consistent findings with guilt-by-association neighbor voting. To demonstrate its functionality, we explore the function and transcriptional regulation of grapevine EXPANSIN genes. We envisage that network aggregation will offer new and unique opportunities for gene function prediction in future grapevine functional genomics studies.To this end, we make the aggregate networks and associated metadata publicly available at VTC-Agg (https://sites.google.com/view/vtc-agg).

An Annotated Corpus of Reference Resolution for Interpreting Common Grounding

Common grounding is the process of creating, repairing and updating mutual understandings, which is a fundamental aspect of natural language conversation. However, interpreting the process of common grounding is a challenging task, especially under continuous and partially-observable context where complex ambiguity, uncertainty, partial understandings and misunderstandings are introduced. Interpretation becomes even more challenging when we deal with dialogue systems which still have limited capability of natural language understanding and generation. To address this problem, we consider reference resolution as the central subtask of common grounding and propose a new resource to study its intermediate process. Based on a simple and general annotation schema, we collected a total of 40,172 referring expressions in 5,191 dialogues curated from an existing corpus, along with multiple judgements of referent interpretations. We show that our annotation is highly reliable, captures the complexity of common grounding through a natural degree of reasonable disagreements, and allows for more detailed and quantitative analyses of common grounding strategies. Finally, we demonstrate the advantages of our annotation for interpreting, analyzing and improving common grounding in baseline dialogue systems.

End-to-End Trainable Non-Collaborative Dialog System

End-to-end task-oriented dialog models have achieved promising performance on collaborative tasks where users willingly coordinate with the system to complete a given task. While in non-collaborative settings, for example, negotiation and persuasion, users and systems do not share a common goal. As a result, compared to collaborate tasks, people use social content to build rapport and trust in these non-collaborative settings in order to advance their goals. To handle social content, we introduce a hierarchical intent annotation scheme, which can be generalized to different non-collaborative dialog tasks. Building upon TransferTransfo (Wolf et al. 2019), we propose an end-to-end neural network model to generate diverse coherent responses. Our model utilizes intent and semantic slots as the intermediate sentence representation to guide the generation process. In addition, we design a filter to select appropriate responses based on whether these intermediate representations fit the designed task and conversation constraints. Our non-collaborative dialog model guides users to complete the task while simultaneously keeps them engaged. We test our approach on our newly proposed AntiScam dataset and an existing PersuasionForGood dataset. Both automatic and human evaluations suggest that our model outperforms multiple baselines in these two non-collaborative tasks.

Corpus Wide Argument Mining—A Working Solution

One of the main tasks in argument mining is the retrieval of argumentative content pertaining to a given topic. Most previous work addressed this task by retrieving a relatively small number of relevant documents as the initial source for such content. This line of research yielded moderate success, which is of limited use in a real-world system. Furthermore, for such a system to yield a comprehensive set of relevant arguments, over a wide range of topics, it requires leveraging a large and diverse corpus in an appropriate manner. Here we present a first end-to-end high-precision, corpus-wide argument mining system. This is made possible by combining sentence-level queries over an appropriate indexing of a very large corpus of newspaper articles, with an iterative annotation scheme. This scheme addresses the inherent label bias in the data and pinpoints the regions of the sample space whose manual labeling is required to obtain high-precision among top-ranked candidates.

Corpora Annotated with Negation: An Overview

Negation is a universal linguistic phenomenon with a great qualitative impact on natural language processing applications. The availability of corpora annotated with negation is essential to training negation processing systems. Currently, most corpora have been annotated for English, but the presence of languages other than English on the Internet, such as Chinese or Spanish, is greater every day. In this study, we present a review of the corpora annotated with negation information in several languages with the goal of evaluating what aspects of negation have been annotated and how compatible the corpora are. We conclude that it is very difficult to merge the existing corpora because we found differences in the annotation schemes used, and most importantly, in the annotation guidelines: the way in which each corpus was tokenized and the negation elements that have been annotated. Differently than for other well established tasks like semantic role labeling or parsing, for negation there is no standard annotation scheme nor guidelines, which hampers progress in its treatment.

Towards Automatic and Fast Annotation of Seismocardiogram Signals Using Machine Learning

The automatic annotation of Seismocardiogram (SCG) potentially aid to estimate various cardiac health parameters continuously. However, the inter-subject variability of SCG poses great difficulties to automate its accurate annotation. The objective of the research is to design SCG peak retrieval methods on the top of the ensemble features extracted from the SCG morphology for the automatic annotation of SCG signals. The annotation scheme is formulated as a binary classification problem. Three binary classifiers such as Naive Bayes (NB), Support Vector Machine (SVM), and Logistic Regression (LR) are employed for the annotation and the results are compared with the recent state-of-the-art schemes. The performance evaluation is carried out using 9000 SCG signals of 20 presumably healthy volunteers with no known serious cardiac abnormalities. The SCG signals are acquired from the Physionet public repository “cebsdb”. The models are rigorously validated using metrics “Precision”, “Recall”, and “F-measure” followed by 5-fold cross-validation. The experimental validation with recent state-of-the-art solutions establishes the robustness of the proposed NB, SVM and LR with average annotation accuracy of 0.86, 0.925 and 0.935, respectively. The mean response time of proposed models is in the fraction of $\frac {{1}}{{10}}$ sec, which establishes its application for the real-time annotation.

Automatic Posture and Movement Tracking of Infants with Wearable Movement Sensors

Infants’ spontaneous and voluntary movements mirror developmental integrity of brain networks since they require coordinated activation of multiple sites in the central nervous system. Accordingly, early detection of infants with atypical motor development holds promise for recognizing those infants who are at risk for a wide range of neurodevelopmental disorders (e.g., cerebral palsy, autism spectrum disorders). Previously, novel wearable technology has shown promise for offering efficient, scalable and automated methods for movement assessment in adults. Here, we describe the development of an infant wearable, a multi-sensor smart jumpsuit that allows mobile accelerometer and gyroscope data collection during movements. Using this suit, we first recorded play sessions of 22 typically developing infants of approximately 7 months of age. These data were manually annotated for infant posture and movement based on video recordings of the sessions, and using a novel annotation scheme specifically designed to assess the overall movement pattern of infants in the given age group. A machine learning algorithm, based on deep convolutional neural networks (CNNs) was then trained for automatic detection of posture and movement classes using the data and annotations. Our experiments show that the setup can be used for quantitative tracking of infant movement activities with a human equivalent accuracy, i.e., it meets the human inter-rater agreement levels in infant posture and movement classification. We also quantify the ambiguity of human observers in analyzing infant movements, and propose a method for utilizing this uncertainty for performance improvements in training of the automated classifier. Comparison of different sensor configurations also shows that four-limb recording leads to the best performance in posture and movement classification.

Unsupervised Knowledge Transfer for Object Detection in Marine Environmental Monitoring and Exploration

The volume of digital image data collected in the field of marine environmental monitoring and exploration has been growing in rapidly increasing rates in recent years. Computational support is essential for the timely evaluation of the high volume of marine imaging data, but often modern techniques such as deep learning cannot be applied due to the lack of training data. In this article, we present Unsupervised Knowledge Transfer (UnKnoT), a new method to use the limited amount of training data more efficiently. In order to avoid time-consuming annotation, it employs a technique we call “scale transfer” and enhanced data augmentation to reuse existing training data for object detection of the same object classes in new image datasets. We introduce four fully annotated marine image datasets acquired in the same geographical area but with different gear and distance to the sea floor. We evaluate the new method on the four datasets and show that it can greatly improve the object detection performance in the relevant cases compared to object detection without knowledge transfer. We conclude with a recommendation for an image acquisition and annotation scheme that ensures a good applicability of modern machine learning methods in the field of marine environmental monitoring and exploration.

#Brexit: Leave or remain? the role of user’s community and diachronic evolution on stance detection

Interest has grown around the classification of stance that users assume within online debates in recent years. Stance has been usually addressed by considering users posts in isolation, while social studies highlight that social communities may contribute to influence users’ opinion. Furthermore, stance should be studied in a diachronic perspective, since it could help to shed light on users’ opinion shift dynamics that can be recorded during the debate. We analyzed the political discussion in UK about the BREXIT referendum on Twitter, proposing a novel approach and annotation schema for stance detection, with the main aim of investigating the role of features related to social network community and diachronic stance evolution. Classification experiments show that such features provide very useful clues for detecting stance.

EusTimeML: A mark-up language for temporal information in Basque

We present EusTimeML, a mark-up language for temporal information in texts written in Basque. It is compliant with the TimeML specifications, while offering some adapted attributes and attribute values in order to represent the language-specific features of Basque. In particular, alterations have been carried out for verb tense, aspect and modality coding, as well as for time expression and signal annotation. EusTimeML also provides a major extension to the existing TimeML schemes, since the attributes and values for factuality annotation have been added to the existing temporal information annotation scheme. EusTimeML has been used to annotate the EusTimeBank Corpus, the news and history narratives corpus that has been used as the gold standard in temporal information processing in Basque.

OpenSARUrban: A Sentinel-1 SAR Image Dataset for Urban Interpretation

The Sentinel-1 mission provides a freely accessible opportunity for urban image interpretation based on synthetic aperture radar (SAR) data with a specific resolution, which is of paramount importance for Earth observation. In parallel, with the rapid development of advanced technologies, especially deep learning, we urgently need a large-scale SAR dataset supporting urban image interpretation. This article presents OpenSARUrban : a Sentinel-1 dataset dedicated to the content-related interpretation of urban SAR images, including a well-defined hierarchical annotation scheme, data collection, well-established procedures for dataset compilation and organization as well as properties, visualizations, and applications of this dataset. Particularly, our OpenSARUrban collection provides 33 358 image patches of urban SAR scenes, covering 21 major cities of China, including 10 different target area categories, 4 kinds of data formats, 2 kinds of polarization modes, and owning 5 essential properties: large-scale coverage, diversity, specificity, reliability, and sustainability. These properties guarantee the achievement of several goals for OpenSARUrban . The first one is to support urban target characterization. The second one is to help develop well-applicable and advanced algorithms for Sentinel-1 urban target classification. The third one is to explore content-based image retrieval for these kinds of data. In addition, dataset visualization is implemented from the perspective of manifolds to give an intuitive understanding. Besides a detailed description and visualization of the dataset, we present results of some benchmarking algorithms, demonstrating that this dataset is practical and challenging. Notably, developing algorithms to enhance the classification performance on the whole dataset and considering the data imbalance are especially demanding.

Self-learning AI Framework for Skin Lesion Image Segmentation and Classification

Image segmentation and classification are the two main fundamental steps in pattern recognition. To perform medical image segmentation or classification with deep learning models, it requires training on large image dataset with annotation. The dermoscopy images (ISIC archive) considered for this work does not have ground truth information for lesion segmentation. Performing manual labelling on this dataset is time-consuming. To overcome this issue, self-learning annotation scheme was proposed in the two-stage deep learning algorithm. The two-stage deep learning algorithm consists of U-Net segmentation model with the annotation scheme and CNN classifier model. The annotation scheme uses a K-means clustering algorithm along with merging conditions to achieve initial labelling information for training the U-Net model. The classifier models namely ResNet-50 and LeNet-5 were trained and tested on the image dataset without segmentation for comparison and with the U-Net segmentation for implementing the proposed self-learning Artificial Intelligence (AI) framework. The classification results of the proposed AI framework achieved training accuracy of 93.8% and testing accuracy of 82.42% when compared with the two classifier models directly trained on the input images.