Annotation Process Research Articles

Development of a Corpus Annotated With Mentions of Pain in Mental Health Records: Natural Language Processing Approach.

Pain is a widespread issue, with 20% of adults (1 in 5) experiencing it globally. A strong association has been demonstrated between pain and mental health conditions, and this association is known to exacerbate disability and impairment. Pain is also known to be strongly related to emotions, which can lead to damaging consequences. As pain is a common reason for people to access health care facilities, electronic health records (EHRs) are a potential source of information on this pain. Mental health EHRs could be particularly beneficial since they can show the overlap of pain with mental health. Most mental health EHRs contain the majority of their information within the free-text sections of the records. However, it is challenging to extract information from free text. Natural language processing (NLP) methods are therefore required to extract this information from the text. This research describes the development of a corpus of manually labeled mentions of pain and pain-related entities from the documents of a mental health EHR database, for use in the development and evaluation of future NLP methods. The EHR database used, Clinical Record Interactive Search, consists of anonymized patient records from The South London and Maudsley National Health Service Foundation Trust in the United Kingdom. The corpus was developed through a process of manual annotation where pain mentions were marked as relevant (ie, referring to physical pain afflicting the patient), negated (ie, indicating absence of pain), or not relevant (ie, referring to pain affecting someone other than the patient, or metaphorical and hypothetical mentions). Relevant mentions were also annotated with additional attributes such as anatomical location affected by pain, pain character, and pain management measures, if mentioned. A total of 5644 annotations were collected from 1985 documents (723 patients). Over 70% (n=4028) of the mentions found within the documents were annotated as relevant, and about half of these mentions also included the anatomical location affected by the pain. The most common pain character was chronic pain, and the most commonly mentioned anatomical location was the chest. Most annotations (n=1857, 33%) were from patients who had a primary diagnosis of mood disorders (International Classification of Diseases-10th edition, chapter F30-39). This research has helped better understand how pain is mentioned within the context of mental health EHRs and provided insight into the kind of information that is typically mentioned around pain in such a data source. In future work, the extracted information will be used to develop and evaluate a machine learning-based NLP application to automatically extract relevant pain information from EHR databases.

Everyone’s Voice Matters: Quantifying Annotation Disagreement Using Demographic Information

In NLP annotation, it is common to have multiple annotators label the text and then obtain the ground truth labels based on major annotators’ agreement. However, annotators are individuals with different backgrounds and various voices. When annotation tasks become subjective, such as detecting politeness, offense, and social norms, annotators’ voices differ and vary. Their diverse voices may represent the true distribution of people’s opinions on subjective matters. Therefore, it is crucial to study the disagreement from annotation to understand which content is controversial from the annotators. In our research, we extract disagreement labels from five subjective datasets, then fine-tune language models to predict annotators’ disagreement. Our results show that knowing annotators’ demographic information (e.g., gender, ethnicity, education level), in addition to the task text, helps predict the disagreement. To investigate the effect of annotators’ demographics on their disagreement level, we simulate different combinations of their artificial demographics and explore the variance of the prediction to distinguish the disagreement from the inherent controversy from text content and the disagreement in the annotators’ perspective. Overall, we propose an innovative disagreement prediction mechanism for better design of the annotation process that will achieve more accurate and inclusive results for NLP systems. Our code and dataset are publicly available.

Identification of annotation artifacts concerning the chalcone synthase (CHS)

ObjectiveChalcone synthase (CHS) catalyzes the initial step of the flavonoid biosynthesis. The CHS encoding gene is well studied in numerous plant species. Rapidly growing sequence databases contain hundreds of CHS entries that are the result of automatic annotation. In this study, we evaluated apparent multiplication of CHS domains in CHS gene models of four plant species.Main findingsCHS genes with an apparent triplication of the CHS domain encoding part were discovered through database searches. Such genes were found in Macadamia integrifolia, Musa balbisiana, Musa troglodytarum, and Nymphaea colorata. A manual inspection of the CHS gene models in these four species with massive RNA-seq data suggests that these gene models are the result of artificial fusions in the annotation process. While there are hundreds of seemingly correct CHS records in the databases, it is not clear why these annotation artifacts appeared.

A semi-automatic annotation methodology that combines Summarization and Human-In-The-Loop to create disinformation detection resources

Early detection of disinformation is one of the most challenging big-scale problems facing present day society. This is why the application of technologies such as Artificial Intelligence and Natural Language Processing is necessary. The vast majority of Artificial Intelligence approaches require annotated data, and generating these resources is very expensive. This proposal aims to improve the efficiency of the annotation process with a two-level semi-automatic annotation methodology. The first level extracts relevant information through summarization techniques. The second applies a Human-in-the-Loop strategy whereby the labels are pre-annotated by the machine, corrected by the human and reused by the machine to retrain the automatic annotator. After evaluating the system, the average annotation time per news item is reduced by 50%. In addition, a set of experiments on the semi-automatically annotated dataset that is generated are performed so as to demonstrate the effectiveness of the proposal. Although the dataset is annotated in terms of unreliable content, it is applied to the veracity detection task with very promising results (0.95 accuracy in reliability detection and 0.78 in veracity detection).

OrbiFragsNets. A tool for automatic annotation of orbitrap MS2 spectra using networks grade as selection criteria

We introduce OrbiFragsNets, a tool for automatic annotation of MS2 spectra generated by Orbitrap instruments, as well as the concepts of chemical consistency and fragments networks. OrbiFragsNets takes advantage of the specific confidence interval for each peak in every MS2 spectrum, which is an unclear idea across the high-resolution mass spectrometry literature. The spectrum annotations are expressed as fragments networks, a set of networks with the possible combinations of annotations for the fragments. The model behind OrbiFragsNets is briefly described here and explained in detail in the constantly updated manual available in the GitHub repository. This new approach in MS2 spectrum de novo automatic annotation proved to perform as good as well established tools such as RMassBank and SIRIUS.•A new approach on automatic annotation of Orbitrap MS2 spectra is introduced.•Possible spectrum annotation are described as independent consistent networks, with annotations for each fragment as nodes, and annotations for the mass difference between fragments as edges.•Annotation process is described as the selection of the most connected fragments network.

Inter-rater agreement for the annotation of neurologic signs and symptoms in electronic health records.

The extraction of patient signs and symptoms recorded as free text in electronic health records is critical for precision medicine. Once extracted, signs and symptoms can be made computable by mapping to signs and symptoms in an ontology. Extracting signs and symptoms from free text is tedious and time-consuming. Prior studies have suggested that inter-rater agreement for clinical concept extraction is low. We have examined inter-rater agreement for annotating neurologic concepts in clinical notes from electronic health records. After training on the annotation process, the annotation tool, and the supporting neuro-ontology, three raters annotated 15 clinical notes in three rounds. Inter-rater agreement between the three annotators was high for text span and category label. A machine annotator based on a convolutional neural network had a high level of agreement with the human annotators but one that was lower than human inter-rater agreement. We conclude that high levels of agreement between human annotators are possible with appropriate training and annotation tools. Furthermore, more training examples combined with improvements in neural networks and natural language processing should make machine annotators capable of high throughput automated clinical concept extraction with high levels of agreement with human annotators.

Evaluation of cerebrovascular disease risk with carotid ultrasonography imaging in artificial intelligence framework

Carotid plaque is a biomarker of generalized atherosclerosis, and may predict ischemic stroke. Carotid intima-media thickness (C-IMT) measurement with ultrasonography imaging could capture the condition of carotid plaque. However, manual measurement of C-IMT is observer- dependent, resulting in observer bias and low reproducibility. In this study, we develop artificial intelligence (AI) framework that could automatically measure the C-IMT, and compared it with C-IMT measured by board of expert. This is a retrospective study done in Dr. Moewardi General Hospital, Surakarta, Indonesia. Carotid B-mode ultrasonography images were measured by panel of expert and by AI. After annotation process on Neurabot platform, AI could detect region of interest (ROI), and would do segmentation on the area to measure C-IMT autonomously. Dependent T-test was used to evaluate validity, and Cronbach’s alpha was used to find the reliability of C-IMT measured by panel of expert and AI. There was strong correlation (r=0.874; p=0.014) on dependent t-test for C-IMT measured by AI with C-IMT measured by board of expert. The internal consistency reliability coefficients (Cronbach’s alpha) were 0.938 and 0.909, for pretest and posttest, respectively. We also analyzed the test-retest reliability by comparing pretest and posttest score with dependent t-test, and we observed strong correlation with r=0.871 (p=0.000). AI developed on Neurabot platform are valid and reliable to measure C-IMT.

Automatic knowledge extraction from Chinese electronic medical records and rheumatoid arthritis knowledge graph construction.

Knowledge graphs are a powerful tool for organizing knowledge, processing information and integrating scattered information, effectively visualizing the relationships among entities and supporting further intelligent applications. One of the critical tasks in building knowledge graphs is knowledge extraction. The existing knowledge extraction models in the Chinese medical domain usually require high-quality and large-scale manually labeled corpora for model training. In this study, we investigate rheumatoid arthritis (RA)-related Chinese electronic medical records (CEMRs) and address the automatic knowledge extraction task with a small number of annotated samples from CEMRs, from which an authoritative RA knowledge graph is constructed. After constructing the domain ontology of RA and completing manual labeling, we propose the MC-bidirectional encoder representation from transformers-bidirectional long short-term memory-conditional random field (BERT-BiLSTM-CRF) model for the named entity recognition (NER) task and the MC-BERT + feedforward neural network (FFNN) model for the entity extraction task. The pretrained language model (MC-BERT) is trained with many unlabeled medical data and fine-tuned using other medical domain datasets. We apply the established model to automatically label the remaining CEMRs, and then an RA knowledge graph is constructed based on the entities and entity relations, a preliminary assessment is conducted, and an intelligent application is presented. The proposed model achieved better performance than that of other widely used models in knowledge extraction tasks, with mean F1 scores of 92.96% in entity recognition and 95.29% in relation extraction. This study preliminarily confirmed that using a pretrained medical language model could solve the problem that knowledge extraction from CEMRs requires a large number of manual annotations. An RA knowledge graph based on the above identified entities and extracted relations from 1,986 CEMRs was constructed. Experts verified the effectiveness of the constructed RA knowledge graph. In this paper, an RA knowledge graph based on CEMRs was established, the processes of data annotation, automatic knowledge extraction, and knowledge graph construction were described, and a preliminary assessment and an application were presented. The study demonstrated the viability of a pretrained language model combined with a deep neural network for knowledge extraction tasks from CEMRs based on a small number of manually annotated samples.

A novel low-cost multi-sensor solution for pavement distress segmentation and characterization at night

The pavement distress measurement is a crucial aspect in guaranteeing the safety of transportation infrastructure. In this regard, we introduce a novel and cost-effective multi-sensor approach for pavement segmentation during low-light night conditions. By utilizing the low-cost Azure Kinect multi-sensor system, we generated a multi-sensor dataset that encompasses aligned IR, RGB, and depth images. Then we carried out the data annotation process on the RGB images. A total of 11,343 manual annotations were meticulously made on 791 images, which were randomly selected from a collection of 96,891 frames. Subsequently, four different deep learning-based image segmentation models were analyzed both quantitatively and qualitatively. The results indicated that the segmentation performance on the IR dataset outperformed that of the RGB dataset. The model with the highest mIoU (mean Intersection over Union) of 0.7169 was ConvNext when trained on the IR dataset. Furthermore, we proposed the use of relative height for evaluating the severity of pavement distress. On the aligned depth map, the relative height was calculated using the depth data from the corresponding pavement distress area. Additionally, a quantitative comparison between manual annotations and the results obtained through deep learning revealed that the latter was more effective in identifying more severe forms of pavement distress. Through this study, we established the feasibility of collecting pavement distress data during nighttime using a low-cost multi-sensor system.

Consensus label propagation with graph convolutional networks for single-cell RNA sequencing cell type annotation.

Single-cell RNA sequencing (scRNA-seq) data, annotated by cell type, is useful in a variety of downstream biological applications, such as profiling gene expression at the single-cell level. However, manually assigning these annotations with known marker genes is both time-consuming and subjective. We present a Graph Convolutional Network (GCN)-based approach to automate the annotation process. Our process builds upon existing labeling approaches, using state-of-the-art tools to find cells with highly confident label assignments through consensus and spreading these confident labels with a semi-supervised GCN. Using simulated data and two scRNA-seq datasets from different tissues, we show that our method improves accuracy over a simple consensus algorithm and the average of the underlying tools. We also compare our method to a nonparametric neighbor majority approach, showing comparable results. We then demonstrate that our GCN method allows for feature interpretation, identifying important genes for cell type classification. We present our completed pipeline, written in PyTorch, as an end-to-end tool for automating and interpreting the classification of scRNA-seq data. Our code for conducting the experiments in this paper and using our model is available at https://github.com/lewinsohndp/scSHARP.

Cardiac Adipose Tissue Segmentation via Image-Level Annotations.

Automatically identifying the structural substrates underlying cardiac abnormalities can potentially provide real-time guidance for interventional procedures. With the knowledge of cardiac tissue substrates, the treatment of complex arrhythmias such as atrial fibrillation and ventricular tachycardia can be further optimized by detecting arrhythmia substrates to target for treatment (i.e., adipose) and identifying critical structures to avoid. Optical coherence tomography (OCT) is a real-time imaging modality that aids in addressing this need. Existing approaches for cardiac image analysis mainly rely on fully supervised learning techniques, which suffer from the drawback of workload on labor-intensive annotation process of pixel-wise labeling. To lessen the need for pixel-wise labeling, we develop a two-stage deep learning framework for cardiac adipose tissue segmentation using image-level annotations on OCT images of human cardiac substrates. In particular, we integrate class activation maps with superpixel segmentation to solve the sparse tissue seed challenge raised in cardiac tissue segmentation. Our study bridges the gap between the demand for automatic tissue analysis and the lack of high-quality pixel-wise annotations. To the best of our knowledge, this is the first study that attempts to address cardiac tissue segmentation on OCT images via weakly supervised learning techniques. Within an in-vitro human cardiac OCT dataset, we demonstrate that our weakly supervised approach on image-level annotations achieves comparable performance as fully supervised methods trained on pixel-wise annotations.

OCTAve: 2D En Face Optical Coherence Tomography Angiography Vessel Segmentation in Weakly-Supervised Learning With Locality Augmentation.

While the microvasculature annotation within Optical Coherence Tomography Angiography (OCTA) can be leveraged using deep-learning techniques, expensive annotation processes are required to create sufficient training data. One way to avoid the expensive annotation is to use a type of weak annotation in which only the center of the vessel is annotated. However, retaining the final segmentation quality with roughly annotated data remains a challenge. Our proposed methods, called OCTAve, provide a new way of using weak-annotation for microvasculature segmentation. Since the centerline labels are similar to scribble annotations, we attempted to solve this problem by using the scribble-based weakly-supervised learning method. Even though the initial results look promising, we found that the method could be significantly improved by adding our novel self-supervised deep supervision method based on Kullback-Liebler divergence. The study on large public datasets with different annotation styles (i.e., ROSE, OCTA-500) demonstrates that our proposed method gives better quantitative and qualitative results than the baseline methods and a naive approach, with a p-value less than 0.001 on dice's coefficient and a lot fewer artifacts. The segmentation results are both qualitatively and quantitatively superior to baseline weakly-supervised methods when using scribble-based weakly-supervised learning augmented with self-supervised deep supervision, with an average drop in segmentation performance of less than 10%. This work gives a new perspective on how weakly-supervised learning can be used to reduce the cost of annotating microvasculature, which can make the annotating process easier and reduce the amount of work for domain experts.

Progress, challenge and prospect of plant plastome annotation.

The plastome (plastid genome) represents an indispensable molecular data source for studying phylogeny and evolution in plants. Although the plastome size is much smaller than that of nuclear genome, and multiple plastome annotation tools have been specifically developed, accurate annotation of plastomes is still a challenging task. Different plastome annotation tools apply different principles and workflows, and annotation errors frequently occur in published plastomes and those issued in GenBank. It is therefore timely to compare available annotation tools and establish standards for plastome annotation. In this review, we review the basic characteristics of plastomes, trends in the publication of new plastomes, the annotation principles and application of major plastome annotation tools, and common errors in plastome annotation. We propose possible methods to judge pseudogenes and RNA-editing genes, jointly consider sequence similarity, customed algorithms, conserved domain or protein structure. We also propose the necessity of establishing a database of reference plastomes with standardized annotations, and put forward a set of quantitative standards for evaluating plastome annotation quality for the scientific community. In addition, we discuss how to generate standardized GenBank annotation flatfiles for submission and downstream analysis. Finally, we prospect future technologies for plastome annotation integrating plastome annotation approaches with diverse evidences and algorithms of nuclear genome annotation tools. This review will help researchers more efficiently use available tools to achieve high-quality plastome annotation, and promote the process of standardized annotation of the plastome.

PhD-SNPg: updating a webserver and lightweight tool for scoring nucleotide variants.

One of the primary challenges in human genetics is determining the functional impact of single nucleotide variants (SNVs) and insertion and deletions (InDels), whether coding or noncoding. In the past, methods have been created to detect disease-related single amino acid changes, but only some can assess the influence of noncoding variations. CADD is the most commonly used and advanced algorithm for predicting the diverse effects of genome variations. It employs a combination of sequence conservation and functional features derived from the ENCODE project data. To use CADD, a large set of pre-calculated information must be downloaded during the installation process. To streamline the variant annotation process, we developed PhD-SNPg, a machine-learning tool that is easy to install and lightweight, relying solely on sequence-based features. Here we present an updated version, trained on a larger dataset, that can also predict the impact of the InDel variations. Despite its simplicity, PhD-SNPg performs similarly to CADD, making it ideal for rapid genome interpretation and as a benchmark for tool development.

Counting Activities Using Weakly Labeled Raw Acceleration Data: A Variable-Length Sequence Approach with Deep Learning to Maintain Event Duration Flexibility.

This paper presents a novel approach for counting hand-performed activities using deep learning and inertial measurement units (IMUs). The particular challenge in this task is finding the correct window size for capturing activities with different durations. Traditionally, fixed window sizes have been used, which occasionally result in incorrectly represented activities. To address this limitation, we propose segmenting the time series data into variable-length sequences using ragged tensors to store and process the data. Additionally, our approach utilizes weakly labeled data to simplify the annotation process and reduce the time to prepare annotated data for machine learning algorithms. Thus, the model receives only partial information about the performed activity. Therefore, we propose an LSTM-based architecture, which takes into account both the ragged tensors and the weak labels. To the best of our knowledge, no prior studies attempted counting utilizing variable-size IMU acceleration data with relatively low computational requirements using the number of completed repetitions of hand-performed activities as a label. Hence, we present the data segmentation method we employed and the model architecture that we implemented to show the effectiveness of our approach. Our results are evaluated using the Skoda public dataset for Human activity recognition (HAR) and demonstrate a repetition error of ±1 even in the most challenging cases. The findings of this study have applications and can be beneficial for various fields, including healthcare, sports and fitness, human-computer interaction, robotics, and the manufacturing industry.

AssistMED project - a tool for rapid data extraction from textual electronic health records for cardiovascular research- system design and preliminary results

Abstract Funding Acknowledgements Type of funding sources: Public grant(s) – National budget only. Main funding source(s): Ministry of Science and Higher Education Background Retrospective and observational analyses are an important part of cardiovascular research. The adoption of Electronic Health Records (EHR) improved the availability of medical documentation for research purposes, but rapid data collection is hampered by the predominantly unstructured nature of EHR. A significant proportion of data is textual information that cannot be utilized for research purposes until is manually coded into a database by a healthcare professional through manual chart review- this conventional method is both cost and time- consuming. Text mining could be utilized to accelerate the process, but little is known about attainable data accuracy and estimation of time gain. Purpose We developed a comprehensive text-processing tool for the cardiology field. The algorithm employs advanced text processing based on a specifically designed, vast database of medical terminology, drug lists and echocardiography parameters with a data structure tailored to the needs of cardiovascular research conduction. The algorithm can automatically analyze three types of textual data which are universal parts of discharge summary in Poland: (1) descriptive list of medical diagnoses; (2) discharge recommendations; (3) echocardiography report (if performed). A set of discharge summaries was analyzed with both the conventional method and the algorithm to demonstrate the process of acquisition of basic characteristics of the cohort of patients with atrial fibrillation/flutter (AF/AFl). Methods Discharge summaries of 394 patients hospitalized at one cardiology department were analyzed (1) automatically and (2) manually coded into the database by a healthcare professional utilizing a proprietary developed annotation tool to accelerate the annotation process, minimize errors and calculate total effective data acquisition time. Results The time of manual and automatic data analysis was 19:11 and 0:15 hours respectively. Of the total 394 patients, 319 (81%) had AF/AFl according to both manual identification and the algorithm (full agreement). The characteristics of the study group obtained with manual and automatic method is presented in Table. There was high agreement in the detection of diseases, presence of drug groups and echocardiographic parameters; some differences between the two classifications were noted but did not reach statistical significance. The median CHA2DS2VASc score calculated by the algorithm was 4 (IQR 3-5) versus 4 (IQR 3-5) for the manual method (p=0.92) and HAS-BLED score by the algorithm was 1 (IQR 1-2) versus 1 (IQR 1-2) for the manual method (p=0.98). Conclusions The utilization of the algorithm greatly reduced the time required for basic characteristics of the study group acquisition without significantly compromising the quality of the data. Automatic detection of retrospective study cohorts through the application of text processing techniques from electronic health records in Polish is promising and feasible.

Active learning with deep autoencoders for seismic facies interpretation

Machine-learning (ML)-assisted seismic interpretation tasks require large quantities of labeled data annotated by expert interpreters, which is a costly and time-consuming process. Whereas existing works to minimize dependence on labeled data assume the data annotation process to already be completed, active learning — a field of ML — works by selecting the most important training samples for the interpreter to annotate in real time simultaneously with the training of the interpretation model itself, resulting in high levels of performance with fewer labeled data samples than otherwise possible. Whereas there exists significant literature on active learning for classification tasks with respect to natural images, there exists very little to no work for dense prediction tasks in geophysics such as interpretation. We have developed a unique and first-of-its-kind active learning framework for seismic facies interpretation using the manifold learning properties of deep autoencoders. By jointly learning representations for supervised and unsupervised tasks and then ranking unlabeled samples by their nearness to the data manifold, we can identify the most relevant training samples to be labeled by the interpreter in each training round. On the popular F3 data set, we obtain close to a 10% point difference in terms of the interpretation accuracy between the proposed method and the baseline with only three fully annotated seismic sections.

Coaches and teachers shift noticing across coaching cycles: analysis of video-based annotations

ABSTRACT We share results of a study on coaches’ and teachers’ noticing as they annotated key moments from classroom video of the teachers’ lessons over multiple online coaching cycles. Coaches and teachers engaged in mathematics coaching cycles that included a pre-lesson meeting, a video-recorded lesson, an annotation process, and a post-lesson meeting. We analysed 554 annotations from coaches and teachers across 22 coaching cycles, with an emphasis on the subject, analytic stance, and content of the coaches’ and teachers’ annotations. Findings indicate that as teachers participated in multiple coaching cycles, their annotations progressively focused on students and they increased the frequency of connections between student thinking and aspects of instructional practice. Across the coaching cycles, coaches provided fewer interpretations and included more questions with a shift toward focusing on instructional practice goals. We emphasise the possible benefits of coaching with respect to noticing and provide actionable recommendations for coaching and research.

Mammography Datasets for Neural Networks-Survey.

Deep neural networks have gained popularity in the field of mammography. Data play an integral role in training these models, as training algorithms requires a large amount of data to capture the general relationship between the model's input and output. Open-access databases are the most accessible source of mammography data for training neural networks. Our work focuses on conducting a comprehensive survey of mammography databases that contain images with defined abnormal areas of interest. The survey includes databases such as INbreast, the Curated Breast Imaging Subset of Digital Database for Screening Mammography (CBIS-DDSM), the OPTIMAM Medical Image Database (OMI-DB), and The Mammographic Image Analysis Society Digital Mammogram Database (MIAS). Additionally, we surveyed recent studies that have utilized these databases in conjunction with neural networks and the results they have achieved. From these databases, it is possible to obtain at least 3801 unique images with 4125 described findings from approximately 1842 patients. The number of patients with important findings can be increased to approximately 14,474, depending on the type of agreement with the OPTIMAM team. Furthermore, we provide a description of the annotation process for mammography images to enhance the understanding of the information gained from these datasets.

A study on methods for revising dependency treebanks: in search of gold

Reliably annotated corpora with reliable annotation are a valuable resource for Natural Language Processing, which justifies the search for methods capable of assisting linguistic revision. In this context, we present a study on methods for revising dependency treebanks, investigating the contribution of three different strategies to the corpus review: (i) linguistic rules; (ii) an adaptation of the n-grams method proposed by Boyd et al. (2008) applied to Portuguese; and (iii) Inter-Annotator Disagreement, a linguistically motivated approach that draws inspiration from the human annotation process. The results are promising, and taken together the three methods can lead to the revision of up to 58% of the errors in a specific corpus at the cost of revising only 20% of the corpus. We also present a tool that integrates treebank editing, evaluation and search capabilities with the review methods, as well as a gold-standard Portuguese corpus from the oil and gas domain.