Annotation Errors Research Articles

Progression of Geographic Atrophy: Epistemic Uncertainties Affecting Mathematical Models and Machine Learning.

PurposeThe purpose of this study was to identify a taxonomy of epistemic uncertainties that affect results for geographic atrophy (GA) assessment and progression.MethodsAn important source of variability is called “epistemic uncertainty,” which is due to incomplete system knowledge (i.e. limitations in measurement devices, artifacts, and human subjective evaluation, including annotation errors). In this study, different epistemic uncertainties affecting the analysis of GA were identified and organized into a taxonomy. The uncertainties were discussed and analyzed, and an example was provided in the case of model structure uncertainty by characterizing progression of GA by mathematical modelling and machine learning. It was hypothesized that GA growth follows a logistic (sigmoidal) function. Using case studies, the GA growth data were used to test the sigmoidal hypothesis.ResultsEpistemic uncertainties were identified, including measurement error (imperfect outcomes from measuring tools), subjective judgment (grading affected by grader's vision and experience), model input uncertainties (data corruption or entry errors), and model structure uncertainties (elucidating the right progression pattern). Using GA growth data from case studies, it was demonstrated that GA growth can be represented by a sigmoidal function, where growth eventually approaches an upper limit.ConclusionEpistemic uncertainties contribute to errors in study results and are reducible if identified and addressed. By prior identification of epistemic uncertainties, it is possible to (a) quantify uncertainty not accounted for by natural statistical variability, and (b) reduce the presence of these uncertainties in future studies.Translational RelevanceLowering epistemic uncertainty will reduce experimental error, improve consistency and reproducibility, and increase confidence in diagnostics.

Fungal genomes: suffering with functional annotation errors

BackgroundThe genome sequence data of more than 65985 species are publicly available as of October 2021 within the National Center for Biotechnology Information (NCBI) database alone and additional genome sequences are available in other databases and also continue to accumulate at a rapid pace. However, an error-free functional annotation of these genome is essential for the research communities to fully utilize these data in an optimum and efficient manner.ResultsAn analysis of proteome sequence data of 689 fungal species (7.15 million protein sequences) was conducted to identify the presence of functional annotation errors. Proteins associated with calcium signaling events, including calcium dependent protein kinases (CDPKs), calmodulins (CaM), calmodulin-like (CML) proteins, WRKY transcription factors, selenoproteins, and proteins associated with the terpene biosynthesis pathway, were targeted in the analysis. Gene associated with CDPKs and selenoproteins are known to be absent in fungal genomes. Our analysis, however, revealed the presence of proteins that were functionally annotated as CDPK proteins. However, InterproScan analysis indicated that none of the protein sequences annotated as “calcium dependent protein kinase” were found to encode calcium binding EF-hands at the regulatory domain. Similarly, none of a protein sequences annotated as a “selenocysteine” were found to contain a Sec (U) amino acid. Proteins annotated as CaM and CMLs also had significant discrepancies. CaM proteins should contain four calcium binding EF-hands, however, a range of 2–4 calcium binding EF-hands were present in the fungal proteins that were annotated as CaM proteins. Similarly, CMLs should possess four calcium binding EF-hands, but some of the CML annotated fungal proteins possessed either three or four calcium binding EF-hands. WRKY transcription factors are characterized by the presence of a WRKY domain and are confined to the plant kingdom. Several fungal proteins, however, were annotated as WRKY transcription factors, even though they did not contain a WRKY domain.ConclusionThe presence of functional annotation errors in fungal genome and proteome databases is of considerable concern and needs to be addressed in a timely manner.

Automated detection of former field systems from airborne laser scanning data: a new approach for Historical Ecology

Former field systems (FFS) are the most widespread traces of ancient activities in present European landscapes and represent significant perturbations to ecosystems. Through its ability to penetrate forest canopies and detect microlandforms, Airborne Laser Scanning data reveal archaeological relics over large areas, from periods older than the first available Historical Topographic Maps. Mapping these traces from ALS-derived data (e.g. Digital Elevation Model (DEM)) thus allows for a determination of a new temporal baseline in order to evaluate the effects of a longer history on current patterns of biodiversity. Here, we evaluate the ability of traditional machine learning (Random Forest-RF) and deep learning (Fully Connected Networks-FCN) models to detect Medieval Terraced slopes and Ridges and Furrows (RaF) from an ALS-derived DEM in the southern Vosges (1462 km2). We used a combination of Local Binary Patterns and topographical metrics to measure properties of FFS and to train detection models. We then assessed the relative performance of each model semantically and spatially. Our results demonstrated the high suitability of our approach for reproducing major trends in the landscape with a high level of similarity between the predicted and reference spatial patterns (Structural Similarity Index - SSIM > 0.75). RF outperformed FCN for Terraced Slopes, whilst minimizing the false positive rate. FCN slightly outperformed RF for the RaF dataset but showed promising abilities to survey unseen data with a low sensitivity to annotation errors. We suggest that this approach has the potential to offer new spatio-temporal possibilities in Historical Ecology studies as a means of automatically detecting past archaeological ecosystems from a landscape to a regional scale.

Manual Annotation Studio (MAS): a collaborative platform for manual functional annotation of viral and microbial genomes

BackgroundFunctional genome annotation is the process of labelling functional genomic regions with descriptive information. Manual curation can produce higher quality genome annotations than fully automated methods. Manual annotation efforts are time-consuming and complex; however, software can help reduce these drawbacks.ResultsWe created Manual Annotation Studio (MAS) to improve the efficiency of the process of manual functional annotation prokaryotic and viral genomes. MAS allows users to upload unannotated genomes, provides an interface to edit and upload annotations, tracks annotation history and progress, and saves data to a relational database. MAS provides users with pertinent information through a simple point and click interface to execute and visualize results for multiple homology search tools (blastp, rpsblast, and HHsearch) against multiple databases (Swiss-Prot, nr, CDD, PDB, and an internally generated database). MAS was designed to accept connections over the local area network (LAN) of a lab or organization so multiple users can access it simultaneously. MAS can take advantage of high-performance computing (HPC) clusters by interfacing with SGE or SLURM and data can be exported from MAS in a variety of formats (FASTA, GenBank, GFF, and excel).ConclusionsMAS streamlines and provides structure to manual functional annotation projects. MAS enhances the ability of users to generate, interpret, and compare results from multiple tools. The structure that MAS provides can improve project organization and reduce annotation errors. MAS is ideal for team-based annotation projects because it facilitates collaboration.

PASS: Protein Annotation Surveillance Site for Protein Annotation Using Homologous Clusters, NLP, and Sequence Similarity Networks.

Advances in genome sequencing have accelerated the growth of sequenced genomes but at a cost in the quality of genome annotation. At the same time, computational analysis is widely used for protein annotation, but a dearth of experimental verification has contributed to inaccurate annotation as well as to annotation error propagation. Thus, a tool to help life scientists with accurate protein annotation would be useful. In this work we describe a website we have developed, the Protein Annotation Surveillance Site (PASS), which provides such a tool. This website consists of three major components: a database of homologous clusters of more than eight million protein sequences deduced from the representative genomes of bacteria, archaea, eukarya, and viruses, together with sequence information; a machine-learning software tool which periodically queries the UniprotKB database to determine whether protein function has been experimentally verified; and a query-able webpage where the FASTA headers of sequences from the cluster best matching an input sequence are returned. The user can choose from these sequences to create a sequence similarity network to assist in annotation or else use their expert knowledge to choose an annotation from the cluster sequences. Illustrations demonstrating use of this website are presented.

Next-Generation Sequencing Somatic and Germline Assay Troubleshooting Guide Derived From Proficiency Testing Data.

Next-generation sequencing-based assays are increasingly used in clinical molecular laboratories to detect somatic variants in solid tumors and hematologic malignancies and to detect constitutional variants. Proficiency testing data are potential sources of information about challenges in performing these assays. To examine the most common sources of unacceptable results from the College of American Pathologists Next-Generation Sequencing Bioinformatics, Hematological Malignancies, Solid Tumor, and Germline surveys and provide recommendations on how to avoid these pitfalls and improve performance. The College of American Pathologists next-generation sequencing somatic and germline proficiency testing survey results from 2016 to 2019 were analyzed to identify the most common causes of unacceptable results. On somatic and germline proficiency testing surveys, 95.9% (18 815/19 623) and 97.8% (33 890/34 641) of all variants were correctly identified, respectively. The most common causes of unacceptable results related to sequencing were false-negative errors in genomic regions that were difficult to sequence because of high GC content. False-positive errors occurred in the context of homopolymers and pseudogenes. Recurrent errors in variant annotation were seen for dinucleotide and duplication variants and included unacceptable transcript selection and outdated variant nomenclature. A small percentage of preanalytic or postanalytic errors were attributed to specimen swaps and transcription errors. Laboratories demonstrate overall excellent performance for detecting variants in both somatic and germline proficiency testing surveys. Proficiency testing survey results highlight infrequent, but recurrent, analytic and nonanalytic challenges in performing next- generation sequencing-based assays and point to remedies to help laboratories improve performance.

Open Issues for Protein Function Assignment in Haloferax volcanii and Other Halophilic Archaea.

Background: Annotation ambiguities and annotation errors are a general challenge in genomics. While a reliable protein function assignment can be obtained by experimental characterization, this is expensive and time-consuming, and the number of such Gold Standard Proteins (GSP) with experimental support remains very low compared to proteins annotated by sequence homology, usually through automated pipelines. Even a GSP may give a misleading assignment when used as a reference: the homolog may be close enough to support isofunctionality, but the substrate of the GSP is absent from the species being annotated. In such cases, the enzymes cannot be isofunctional. Here, we examined a variety of such issues in halophilic archaea (class Halobacteria), with a strong focus on the model haloarchaeon Haloferax volcanii. Results: Annotated proteins of Hfx. volcanii were identified for which public databases tend to assign a function that is probably incorrect. In some cases, an alternative, probably correct, function can be predicted or inferred from the available evidence, but this has not been adopted by public databases because experimental validation is lacking. In other cases, a probably invalid specific function is predicted by homology, and while there is evidence that this assigned function is unlikely, the true function remains elusive. We listed 50 of those cases, each with detailed background information, so that a conclusion about the most likely biological function can be drawn. For reasons of brevity and comprehension, only the key aspects are listed in the main text, with detailed information being provided in a corresponding section of the Supplementary Materials. Conclusions: Compiling, describing and summarizing these open annotation issues and functional predictions will benefit the scientific community in the general effort to improve the evaluation of protein function assignments and more thoroughly detail them. By highlighting the gaps and likely annotation errors currently in the databases, we hope this study will provide a framework for experimentalists to systematically confirm (or disprove) our function predictions or to uncover yet more unexpected functions.

Cross-Context News Corpus for Protest Event-Related Knowledge Base Construction

Abstract We describe a gold standard corpus of protest events that comprise various local and international English language sources from various countries. The corpus contains document-, sentence-, and token-level annotations. This corpus facilitates creating machine learning models that automatically classify news articles and extract protest event-related information, constructing knowledge bases that enable comparative social and political science studies. For each news source, the annotation starts with random samples of news articles and continues with samples drawn using active learning. Each batch of samples is annotated by two social and political scientists, adjudicated by an annotation supervisor, and improved by identifying annotation errors semi-automatically. We found that the corpus possesses the variety and quality that are necessary to develop and benchmark text classification and event extraction systems in a cross-context setting, contributing to the generalizability and robustness of automated text processing systems. This corpus and the reported results will establish a common foundation in automated protest event collection studies, which is currently lacking in the literature.

An in-depth analysis of the individual impact of controlled language rules on machine translation output: a mixed-methods approach

Examining the general impact of Controlled Language (CL) rules in the context of Machine Translation (MT) has been an area of research for many years. The present study focuses on the following question: how do CL rules impact MT output individually? By analysing a German corpus-based test suite of technical texts that have been translated into English by different MT systems, this study endeavours to answer this question at different levels: the general impact of CL rules (rule- and system-independent), their impact at rule level (system-independent) as well as at rule and system level. The results of five MT systems are analysed and contrasted: a rule-based system, a statistical system, two differently constructed hybrid systems, and a neural system. For this, a mixed-methods triangulation approach that includes error annotation, human evaluation, and automatic evaluation was applied. The data was analysed both qualitatively and quantitatively in terms of CL influence on the following parameters: number and type of MT errors, style and content quality, and scores of two automatic evaluation metrics. In line with many studies, the results show a general positive impact of the applied CL rules on the MT output. However, at rule level, only four rules proved to have positive effects on the aforementioned parameters; three rules had negative effects on the parameters; and two rules did not show any significant impact. At rule and system level, the rules affected the MT systems differently, as expected. Rules that had a positive impact on earlier MT approaches did not show the same impact on the neural MT approach. Furthermore, neural MT delivered distinctly better results than earlier MT approaches, namely the highest error-free, style and content quality rates both before and after applying the rules, which indicates that neural MT offers a promising solution that no longer requires CL rules for improving the MT output.

Non-essential ribosomal proteins in bacteria and archaea identified using COGs.

Ribosomal proteins (RPs) are highly conserved across the bacterial and archaeal domains. Although many RPs are essential for survival, genome analysis demonstrates the absence of some RP genes in many bacterial and archaeal genomes. Furthermore, global transposon mutagenesis and/or targeted deletion showed that elimination of some RP genes had only a moderate effect on the bacterial growth rate. Here, we systematically analyze the evolutionary conservation of RPs in prokaryotes by compiling the list of the ribosomal genes that are missing from one or more genomes in the recently updated version of the Clusters of Orthologous Genes (COG) database. Some of these absences occurred because the respective genes carried frameshifts, presumably, resulting from sequencing errors, while others were overlooked and not translated during genome annotation. Apart from these annotation errors, we identified multiple genuine losses of RP genes in a variety of bacteria and archaea. Some of these losses are clade-specific, whereas others occur in symbionts and parasites with dramatically reduced genomes. The lists of computationally and experimentally defined non-essential ribosomal genes show a substantial overlap, revealing a common trend in prokaryote ribosome evolution that could be linked to the architecture and assembly of the ribosomes. Thus, RPs that are located at the surface of the ribosome and/or are incorporated at a late stage of ribosome assembly are more likely to be non-essential and to be lost during microbial evolution, particularly, in the course of genome compaction.IMPORTANCEIn many prokaryote genomes, one or more ribosomal protein (RP) genes are missing. Analysis of 1,309 prokaryote genomes included in the COG database shows that only about half of the RPs are universally conserved in bacteria and archaea. In contrast, up to 16 other RPs are missing in some genomes, primarily, tiny (<1 Mb) genomes of host-associated bacteria and archaea. Ten universal and nine archaea-specific ribosomal proteins show clear patterns of lineage-specific gene loss. Most of the RPs that are frequently lost from bacterial genomes are located on the ribosome periphery and are non-essential in Escherichia coli and Bacillus subtilis These results reveal general trends and common constraints in the architecture and evolution of ribosomes in prokaryotes.

Mitochondrial analysis of oribatid mites provides insights into their atypical tRNA annotation, genome rearrangement and evolution

BackgroundThe mitochondrial (mt) genomes of Sarcoptiformes mites typically contain 37 genes. Although the loss of genes is rare in Sarcoptiformes mite mitogenomes, two of the six previously reported oribatid mites (Acariforms: Sarcoptiformes) are reported to have lost parts of their tRNA genes. To confirm whether the tRNA genes were indeed lost and whether the loss is universal, we re-annotated the available oribatid mite sequences and sequenced the mitogenome of Oribatula sakamorii.MethodsThe mitogenome of O. sakamorii was sequenced using an Illumina HiSeq sequencer. The mt tRNA gene was annotated using multi-software combined with a manual annotation approach. Phylogenetic analyses were performed using the maximum likelihood and Bayesian inference methods with concatenated nucleotide and amino acid sequences.ResultsThe mitogenomes of O. sakamorii contained 37 genes, including 22 tRNA genes. We identified all mt tRNA genes that were reported as “lost” in Steganacarus magnus and Paraleius leontonychus and revealed certain atypical tRNA annotation errors in oribatid mite sequences. Oribatid mite mitogenomes are characterized by low rates of genetic rearrangement, with six or seven gene blocks conserved between the mitogenome of all species and that of ancestral arthropods. Considering the relative order of the major genes (protein-coding genes and rRNAs), only one or two genes were rearranged with respect to their positions in the ancestral genome. We explored the phylogenetic relationships among the available oribatid mites, and the results confirmed the systematic position of Hermannia in the Crotonioidea superfamily. This was also supported by the synapomorphic gene-derived boundaries.ConclusionsThe tRNA “lost” phenomenon is not universal in oribatid mites. Rather, highly atypical secondary structure of the inferred mt tRNA genes made them unidentifiable using a single type of tRNA search program. The use of multi-software combined with a manual annotation approach can improve the accuracy of tRNA gene annotation. In addition, we identified the precise systematic position of Hermannia and validated that Astigmata is nested in Oribatida.Graphical

Polytonia

This paper first proposes a labeling scheme for tonal aspects of speech and then describes an automatic annotation system using this transcription. This fine-grained transcription provides labels indicating pitch level and pitch movement of individual syllables. Of the five pitch levels, three (low, mid, high) are defined on the basis of pitch changes in the local context and two (bottom, top) are defined relative to the boundaries of the speaker’s global pitch range. For pitch movements, both simple and compound, the transcription indicates direction (rise, fall, level) and size, using size categories (pitch intervals) adjusted relative to the speaker’s pitch range. The automatic tonal annotation system combines several processing steps: segmentation into syllable peaks, pause detection, pitch stylization, pitch range estimation, classification of the intra-syllabic pitch contour, and pitch level assignment. It uses a dedicated and rule-based procedure, which unlike commonly used supervised learning techniques does not require a labeled corpus for training the model. The paper also includes a preliminary evaluation of the annotation system, for a reference corpus of nearly 14 minutes of spontaneous speech in French and Dutch, in order to quantify the annotation errors. The results, expressed in terms of standard measures of precision, recall, accuracy and Fmeasure are encouraging. For pitch levels low, mid and high an F-measure between 0.946 and 0.815 is obtained and for pitch movements a value between 0.708 and 1. Provided additional modules for the detection of prominence and prosodic boundaries, the resulting annotation may serve as an input for a phonological annotation.

Evolution of fibrinogen domain related proteins in Aedes aegypti: Their expression during Arbovirus infections

Aedes aegypti (Ae. aegypti) mosquito species is a major vector for the transmission of Arboviruses to humans, causing viral diseases of immense importance for public health. Its genome sequence provides an opportunity for comparative genomics and helps to understand evolution of important gene/protein families. Among these, fibrinogen domain related proteins (FREPs) are an interesting family found in vertebrates and invertebrates. These proteins are characterized by conserved C-terminal fibrinogen and diverse N-terminal domains. These features make FREPs interesting targets for their evolution studies. Vertebrate FREPs are one of the well-studied in terms of phylogenetic origin, structure, and function. Although, invertebrate FREPs from few insects and snails are shown to be involved in pathogen recognition and pathogenesis, their role in Ae. aegypti is unknown. From structural classification and evolutionary point of view, invertebrate FREPs are poorly understood in comparison to their vertebrate homologs. Analysis of 33 FREP genes from Ae. aegypti suggests, diverse set of proteins clubbed together based on conserved C-terminal identifier with diverse N-terminal sequence. Gene clustering revealed that few FREPs have evolved through gene duplication and diversification, whereas others are different genes. Based on annotations, FREPs were classified into six major classes. Few errors in annotations were identified and the corrected sequences are presented. Gene expression analysis shows interesting changes in expression pattern of few genes, whereas that transcripts for nine FREPs were not detected in these data set. Wide range of phylogenetic lineage and sequence diversity suggests they might be involved in different physiological functions, which are yet to be explored.

Post-Editing Oriented Human Quality Evaluation of Neural Machine Translation in Translator Training: A Study on Perceived Difficulties and Benefits

The aim of this article is to investigate translation trainees’ perceived difficulties and benefits of a post-editing oriented neural machine translation (NMT) error annotation and quality evaluation task which was carried out for the language pair English-Turkish and for two separate text types and domains, i.e., environmental blogs and movie/TV reviews, within the scope of an MA course on translation quality standards. The data to be analyzed were collected through a semi-structured questionnaire which was given to the trainees attending the course after the completion of the task. The questionnaire was prepared with the aim of understanding perceived difficulties and benefits of the task. Analysis of the answers revealed that most of the trainees were in the opinion that the task was difficult. Majority of the trainees also believed that the task was beneficial and enabled them to feel empowered to make decisions on human translation quality evaluation of machine translation (MT) and to carry out error annotation and post-editing activities in the future. According to a significant number of trainees, error annotation facilitated their post-editing process and reduced the effort in post-editing. Enhanced understanding of MT error annotation and enhanced ability to perform post-editing were the significant benefits stated by the trainees. The difficulties were associated with being introduced and assigned to perform tasks they were not familiar with. Yet, as displayed in the answers to the questionnaire, a considerable majority of the trainees were positive about the learning experience. The results have also shown that integration of MT-related activities into translator training with a focus on the empowerment of the human translator has its difficulties and benefits also for translation trainers. While the difficulties for the trainers concern the decisions on the design, implementation, and planning of the task and the responsibility to carry out the task in constant interaction and collaboration with the trainees, the benefits are the sense of fulfillment and enrichment brought by positive feedback from the trainees and the discovery of the fact that the so-called ‘teaching experience’ becomes a ‘learning experience’ for trainers as well as trainees.

ФРАГМЕНТ РАЗМЕТКИ В ЛИНГВИСТИЧЕСКОМ КОРПУСЕ RuTuBiC. ПЕРЕКЛЮЧЕНИЕ КОДА ИЛИ ЛЕКСИЧЕСКОЕ ЗАИМСТВОВАНИЕ?

The article presents a solution to one of the problems of special linguistic markup in the RuTuBiC corpus – the Russian Speech Corpus of Russian-Turkic Bilinguals, asso-ciated with error annotation at the lexical level. The corpus includes three subcorpuses representing materials of the Russian speech of Shor-Russian, Tatar-Russian and Khakass-Russian bilinguals. The article presents solutions developed on the basis of all subcorpuses; the illustrative contexts are drawn from the Shor-Russian subcorpus, recordings of interviews with 14 respondents, about 20 hours of sound. The recordings were made during expeditions to Shoria in 2017–2019. Bilingualism of the respondents is defined as early natural bilingualism with the dominance of the second Russian lan-guage, mother tongues are languages of the family heritage. The theoretical basis of the research was works on linguistic contact at the lexical level. Solutions based on the differentiation of lexemes fully mastered by the system of standard Russian and units with the status of borrowings from other subsystems of the national language and other languages are proposed. In the latter case, linguistic and contextual features are distin-guished that oppose lexical borrowing and code-switching. The typical errors singled out at the lexical level are: [LexId] – idiomatic expressions that are not fixed in the standard language (dialectal and vernacular, slang, etc.), they can also be Turkic calques; [LexSem] – general Russian words used in meanings different from those fixed in the normative sources; [LexSemAgr] – violations of the lexical and semantic agreement norms. The units borrowed from the mother tongue of the respondents are located on the scale of transitions from nuclear to borderline. The nuclear units marked with the [Lex] tag are dialectal units, common words, other word usage cases that are outside the standard, as well as borrowings from the Turkic languages that are not included in the dictionaries of standard Russian. On the border “to the left” are borrowings assimilated to different degrees. On the border “to the right” are non-assimilated borrowings and code-switches. The [CodeSw] marks code-switching, insertion of mother tongue elements into Russian speech. The author considers the inclusion of statements as nuclear cases of code-switching, and single lexical inclusions as transitional cases. Code-switching is evidenced by metatext and linguistic proper, primarily phonetic, indicators. There is an insignificant number of both lexical borrowings and cases of code-switching in the speech of the respondents of the RuTuBiC corpus, which depends on the type of bilingualism. The typicality of metatext marking of borrowings and code-switches is determined by the discursive, genre and thematic limitations of the corpus.

Automated Metric Analysis of Spanish Poetry: Two Complementary Approaches

The automatic metric analysis (commonly referred to as scansion ) of Spanish poetry is not a trivial problem since it combines the nuances of the language, the different poetic traditions related to melodic patterns, and the personal stylistic preferences and intentions of the author. In this paper, we explore two alternative algorithmic approaches tailored to different applications scenarios. The first approach, Rantanplan, is a rule-based method that consists of four Natural Language Processing modules that work together to perform scansion and other related analysis: Part of Speech tagging, syllabification, stress assignment, and metrical adjustment. The second approach, Jumper, explores the possibility of performing scansion without syllabification, with a twofold purpose: to minimize the errors propagated in different parts of the linguistic processing pipeline (including the syllabification step), and to improve the efficiency of the process. Both systems outperform the state of the art and provide either a more informative solution (suitable, for instance, for teaching purposes) or a more efficient processing (when a correct scansion is all the linguistic knowledge required, as in scholar philological studies). The combined use of both systems turns out to provide a practical tool to clean-up manual annotation errors in corpora.

DeepHistoClass: A Novel Strategy for Confident Classification of Immunohistochemistry Images Using Deep Learning

A multitude of efforts worldwide aim to create a single-cell reference map of the human body, for fundamental understanding of human health, molecular medicine, and targeted treatment. Antibody-based proteomics using immunohistochemistry (IHC) has proven to be an excellent technology for integration with large-scale single-cell transcriptomics datasets. The golden standard for evaluation of IHC staining patterns is manual annotation, which is expensive and may lead to subjective errors. Artificial intelligence holds much promise for efficient and accurate pattern recognition, but confidence in prediction needs to be addressed. Here, the aim was to present a reliable and comprehensive framework for automated annotation of IHC images. We developed a multilabel classification of 7848 complex IHC images of human testis corresponding to 2794 unique proteins, generated as part of the Human Protein Atlas (HPA) project. Manual annotation data for eight different cell types was generated as a basis for training and testing a proposed Hybrid Bayesian Neural Network. By combining the deep learning model with a novel uncertainty metric, DeepHistoClass (DHC) Confidence Score, the average diagnostic performance improved from 86.9% to 96.3%. This metric not only reveals which images are reliably classified by the model, but can also be utilized for identification of manual annotation errors. The proposed streamlined workflow can be developed further for other tissue types in health and disease and has important implications for digital pathology initiatives or large-scale protein mapping efforts such as the HPA project.

Automated Confirmation of Protein Annotation Using NLP and the UniProtKB Database

Advances in genome sequencing technology and computing power have brought about the explosive growth of sequenced genomes in public repositories with a concomitant increase in annotation errors. Many protein sequences are annotated using computational analysis rather than experimental verification, leading to inaccuracies in annotation. Confirmation of existing protein annotations is urgently needed before misannotation becomes even more prevalent due to error propagation. In this work we present a novel approach for automatically confirming the existence of manually curated information with experimental evidence of protein annotation. Our ensemble learning method uses a combination of recurrent convolutional neural network, logistic regression, and support vector machine models. Natural language processing in the form of word embeddings is used with journal publication titles retrieved from the UniProtKB database. Importantly, we use recall as our most significant metric to ensure the maximum number of verifications possible; results are reported to a human curator for confirmation. Our ensemble model achieves 91.25% recall, 71.26% accuracy, 65.19% precision, and an F1 score of 76.05% and outperforms the Bidirectional Encoder Representations from Transformers for Biomedical Text Mining (BioBERT) model with fine-tuning using the same data.

Robust Normalized Softmax Loss for Deep Metric Learning-Based Characterization of Remote Sensing Images With Label Noise

Most deep metric learning-based image characterization methods exploit supervised information to model the semantic relations among the remote sensing (RS) scenes. Nonetheless, the unprecedented availability of large-scale RS data makes the annotation of such images very challenging, requiring automated supportive processes. Whether the annotation is assisted by aggregation or crowd-sourcing, the RS large-variance problem, together with other important factors [e.g., geo-location/registration errors, land-cover changes, even low-quality Volunteered Geographic Information (VGI), etc.] often introduce the so-called label noise, i.e., semantic annotation errors. In this article, we first investigate the deep metric learning-based characterization of RS images with label noise and propose a novel loss formulation, named robust normalized softmax loss (RNSL), for robustly learning the metrics among RS scenes. Specifically, our RNSL improves the robustness of the normalized softmax loss (NSL), commonly utilized for deep metric learning, by replacing its logarithmic function with the negative Box–Cox transformation in order to down-weight the contributions from noisy images on the learning of the corresponding class prototypes. Moreover, by truncating the loss with a certain threshold, we also propose a truncated robust normalized softmax loss (t-RNSL) which can further enforce the learning of class prototypes based on the image features with high similarities between them, so that the intraclass features can be well grouped and interclass features can be well separated. Our experiments, conducted on two benchmark RS data sets, validate the effectiveness of the proposed approach with respect to different state-of-the-art methods in three different downstream applications (classification, clustering, and retrieval). The codes of this article will be publicly available from <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/jiankang1991</uri> .

Automated Removal of Non-homologous Sequence Stretches with PREQUAL.

Large-scale multigene datasets used in phylogenomics and comparative genomics often contain sequence errors inherited from source genomes and transcriptomes. These errors typically manifest as stretches of non-homologous characters and derive from sequencing, assembly, and/or annotation errors. The lack of automatic tools to detect and remove sequence errors leads to the propagation of these errors in large-scale datasets. PREQUAL is a command line tool that identifies and masks regions with non-homologous adjacent characters in sets of unaligned homologous sequences. PREQUAL uses a full probabilistic approach based on pair hidden Markov models. On the front end, PREQUAL is user-friendly and simple to use while also allowing full customization to adjust filtering sensitivity. It is primarily aimed at amino acid sequences but can handle protein-coding nucleotide sequences. PREQUAL is computationally efficient and shows high sensitivity and accuracy. In this chapter, we briefly introduce the motivation for PREQUAL and its underlying methodology, followed by a description of basic and advanced usage, and conclude with some notes and recommendations. PREQUAL fills an important gap in the current bioinformatics tool kit for phylogenomics, contributing toward increased accuracy and reproducibility in future studies.