Ambiguous Annotations Research Articles

Orthogonalized layer features with wavelet for multi-object tracking

Abstract Multi-object tracking (MOT) in crowded scenes presents challenges such as target occlusion and interference from similar objects. The detection models that rely on the target center as positive samples often struggle with noise introduced by ambiguous data annotations. To address these issues, we propose to segregate the target features of distinct frequency tiers by leveraging wavelet decomposition. Features obtained from two-dimensional wavelet decomposition can exhibit orthogonality and complementarity along the horizontal and vertical directions. Low-frequency components are usually related to visible targets, while high-frequency energy often emanates from targets undergoing occlusion, enabling discrimination between multi-layered objectives. Experimental evaluations are conducted on the MOT17 and MOT20 benchmarks (motchallenge.net), which demonstrate that our approach has better competitive performance compared to the current state-of-the-art methods.

Collect, Measure, Repeat: Reliability Factors for Responsible AI Data Collection

The rapid entry of machine learning approaches in our daily activities and high-stakes domains demands transparency and scrutiny of their fairness and reliability. To help gauge machine learning models' robustness, research typically focuses on the massive datasets used for their deployment, e.g., creating and maintaining documentation for understanding their origin, process of development, and ethical considerations. However, data collection for AI is still typically a one-off practice, and oftentimes datasets collected for a certain purpose or application are reused for a different problem. Additionally, dataset annotations may not be representative over time, contain ambiguous or erroneous annotations, or be unable to generalize across issues or domains. Recent research has shown these practices might lead to unfair, biased, or inaccurate outcomes. We argue that data collection for AI should be performed in a responsible manner where the quality of the data is thoroughly scrutinized and measured through a systematic set of appropriate metrics. In this paper, we propose a Responsible AI (RAI) methodology designed to guide the data collection with a set of metrics for an iterative in-depth analysis of the factors influencing the quality and reliability of the generated data. We propose a granular set of measurements to inform on the internal reliability of a dataset and its external stability over time. We validate our approach across nine existing datasets and annotation tasks and four content modalities. This approach impacts the assessment of data robustness used for AI applied in the real world, where diversity of users and content is eminent. Furthermore, it deals with fairness and accountability aspects in data collection by providing systematic and transparent quality analysis for data collections.

MAN: Mining Ambiguity and Noise for Facial Expression Recognition in the Wild

Due to the ambiguous expressions and the subjectiveness of annotators, annotation ambiguity is a serious obstacle for facial expression recognition (FER). Ambiguous annotation exists in similar and dissimilar classes, which we call ambiguity and noise. The previous state-of-the-art approaches use uncertainty to generalize the two categories, and adopt uncertainty learning to suppress uncertainty samples. However, ambiguous expressions are confused with noisy label expressions may bias the model toward easy samples and hurt the generalization capability. To solve this problem, we propose a novel approach to mine ambiguity and noise (MAN) in FER datasets. Specifically, we design a co-division module, which divides the datasets into clean, ambiguous and noisy label expressions based on the consistency and inconsistency between the predictions of two networks and the given labels. To effectively learn the clean expressions, improve discriminative ability and avoid memorizing noisy labels, the tri-regularization module employs supervised learning, mutuality learning and unsupervised learning for the three subsets, respectively. Extensive experiments have shown that MAN can effectively mine the real ambiguity and noise, and achieve state-of-the-art performance in both synthetic noisy datasets and popular benchmarks.

The protein common assembly database (ProtCAD)-a comprehensive structural resource of protein complexes.

Proteins often act through oligomeric interactions with other proteins. X-ray crystallography and cryo-electron microscopy provide detailed information on the structures of biological assemblies, defined as the most likely biologically relevant structures derived from experimental data. In crystal structures, the most relevant assembly may be ambiguously determined, since multiple assemblies observed in the crystal lattice may be plausible. It is estimated that 10-15% of PDB entries may have incorrect or ambiguous assembly annotations. Accurate assemblies are required for understanding functional data and training of deep learning methods for predicting assembly structures. As with any other kind of biological data, replication via multiple independent experiments provides important validation for the determination of biological assembly structures. Here we present the Protein Common Assembly Database (ProtCAD), which presents clusters of protein assembly structures observed in independent structure determinations of homologous proteins in the Protein Data Bank (PDB). ProtCAD is searchable by PDB entry, UniProt identifiers, or Pfam domain designations and provides downloads of coordinate files, PyMol scripts, and publicly available assembly annotations for each cluster of assemblies. About 60% of PDB entries contain assemblies in clusters of at least 2 independent experiments. All clusters and coordinates are available on ProtCAD web site (http://dunbrack2.fccc.edu/protcad).

Ensemble learning via supervision augmentation for white matter hyperintensity segmentation.

Since the ambiguous boundary of the lesion and inter-observer variability, white matter hyperintensity segmentation annotations are inherently noisy and uncertain. On the other hand, the high capacity of deep neural networks (DNN) enables them to overfit labels with noise and uncertainty, which may lead to biased models with weak generalization ability. This challenge has been addressed by leveraging multiple annotations per image. However, multiple annotations are often not available in a real-world scenario. To mitigate the issue, this paper proposes a supervision augmentation method (SA) and combines it with ensemble learning (SA-EN) to improve the generalization ability of the model. SA can obtain diverse supervision information by estimating the uncertainty of annotation in a real-world scenario that per image have only one ambiguous annotation. Then different base learners in EN are trained with diverse supervision information. The experimental results on two white matter hyperintensity segmentation datasets demonstrate that SA-EN gets the optimal accuracy compared with other state-of-the-art ensemble methods. SA-EN is more effective on small datasets, which is more suitable for medical image segmentation with few annotations. A quantitative study is presented to show the effect of ensemble size and the effectiveness of the ensemble model. Furthermore, SA-EN can capture two types of uncertainty, aleatoric uncertainty modeled in SA and epistemic uncertainty modeled in EN.

Learning From Ambiguous Labels for Lung Nodule Malignancy Prediction.

Lung nodule malignancy prediction is an essential step in the early diagnosis of lung cancer. Besides the difficulties commonly discussed, the challenges of this task also come from the ambiguous labels provided by annotators, since deep learning models have in some cases been found to reproduce or amplify human biases. In this paper, we propose a multi-view 'divide-and-rule' (MV-DAR) model to learn from both reliable and ambiguous annotations for lung nodule malignancy prediction on chest CT scans. According to the consistency and reliability of their annotations, we divide nodules into three sets: a consistent and reliable set (CR-Set), an inconsistent set (IC-Set), and a low reliable set (LR-Set). The nodule in IC-Set is annotated by multiple radiologists inconsistently, and the nodule in LR-Set is annotated by only one radiologist. Although ambiguous, inconsistent labels tell which label(s) is consistently excluded by all annotators, and the unreliable labels of a cohort of nodules are largely correct from the statistical point of view. Hence, both IC-Set and LR-Set can be used to facilitate the training of MV-DAR. Our MV-DAR contains three DAR models to characterize a lung nodule from three orthographic views and is trained following a two-stage procedure. Each DAR consists of three networks with the same architecture, including a prediction network (Prd-Net), a counterfactual network (CF-Net), and a low reliable network (LR-Net), which are trained on CR-Set, IC-Set, and LR-Set respectively in the pretraining phase. In the fine-tuning phase, the image representation ability learned by CF-Net and LR-Net is transferred to Prd-Net by negative-attention module (NA-Module) and consistent-attention module (CA-Module), aiming to boost the prediction ability of Prd-Net. The MV-DAR model has been evaluated on the LIDC-IDRI dataset and LUNGx dataset. Our results indicate not only the effectiveness of the MV-DAR in learning from ambiguous labels but also its superiority over present noisy label-learning models in lung nodule malignancy prediction.

FineAction: A Fine-Grained Video Dataset for Temporal Action Localization

Temporal action localization (TAL) is an important and challenging problem in video understanding. However, most existing TAL benchmarks are built upon the coarse granularity of action classes, which exhibits two major limitations in this task. First, coarse-level actions can make the localization models overfit in high-level context information, and ignore the atomic action details in the video. Second, the coarse action classes often lead to the ambiguous annotations of temporal boundaries, which are inappropriate for temporal action localization. To tackle these problems, we develop a novel large-scale and fine-grained video dataset, coined as FineAction, for temporal action localization. In total, FineAction contains 103K temporal instances of 106 action categories, annotated in 17K untrimmed videos. Compared to the existing TAL datasets, our FineAction takes distinct characteristics of fine action classes with rich diversity, dense annotations of multiple instances, and co-occurring actions of different classes, which introduces new opportunities and challenges for temporal action localization. To benchmark FineAction, we systematically investigate the performance of several popular temporal localization methods on it, and deeply analyze the influence of fine-grained instances in temporal action localization. As a minor contribution, we present a simple baseline approach for handling the fine-grained action detection, which achieves an mAP of 13.17% on our FineAction. We believe that FineAction can advance research of temporal action localization and beyond. The dataset is available at https://deeperaction.github.io/datasets/fineaction.

Detecting sample swaps in diverse NGS data types using linkage disequilibrium

As the number of genomics datasets grows rapidly, sample mislabeling has become a high stakes issue. We present CrosscheckFingerprints (Crosscheck), a tool for quantifying sample-relatedness and detecting incorrectly paired sequencing datasets from different donors. Crosscheck outperforms similar methods and is effective even when data are sparse or from different assays. Application of Crosscheck to 8851 ENCODE ChIP-, RNA-, and DNase-seq datasets enabled us to identify and correct dozens of mislabeled samples and ambiguous metadata annotations, representing ~1% of ENCODE datasets.

Ontology-Based Image Classification and Annotation

With the rapid growth of image collections, image classification and annotation has been active areas of research with notable recent progress. Bag-of-Visual-Words (BoVW) model, which relies on building visual vocabulary, has been widely used in this area. Recently, attention has been shifted to the use of advanced architectures which are characterized by multi-level processing. Hierarchical Max-Pooling (HMAX) model has attracted a great deal of attention in image classification. To improve image classification and annotation, several approaches based on ontologies have been proposed. However, image classification and annotation remain a challenging problem due to many related issues like the problem of ambiguity between classes. This problem can affect the quality of both classification and annotation results. In this paper, we propose an ontology-based image classification and annotation approach. Our contributions consist of the following: (1) exploiting ontological relationships between classes during both image classification and annotation processes; (2) combining the outputs of hypernym–hyponym classifiers to lead to a better discrimination between classes; and (3) annotating images by combining hypernym and hyponym classification results in order to improve image annotation and to reduce the ambiguous and inconsistent annotations. The aim is to improve image classification and annotation by using ontologies. Several strategies have been experimented, and the obtained results have shown that our proposal improves image classification and annotation.

Presence\u2013absence polymorphisms of single-copy genes in the stony coral Acropora digitifera

BackgroundDespite the importance of characterizing genetic variation among coral individuals for understanding phenotypic variation, the correlation between coral genomic diversity and phenotypic expression is still poorly understood.ResultsIn this study, we detected a high frequency of genes showing presence–absence polymorphisms (PAPs) for single-copy genes in Acropora digitifera. Among 10,455 single-copy genes, 516 (5%) exhibited PAPs, including 32 transposable element (TE)-related genes. Five hundred sixteen genes exhibited a homozygous absence in one (102) or more than one (414) individuals (n = 33), indicating that most of the absent alleles were not rare variants. Among genes showing PAPs (PAP genes), roughly half were expressed in adults and/or larvae, and the PAP status was associated with differential expression among individuals. Although 85% of PAP genes were uncharacterized or had ambiguous annotations, 70% of these genes were specifically distributed in cnidarian lineages in eumetazoa, suggesting that these genes have functional roles related to traits related to cnidarians or the family Acroporidae or the genus Acropora. Indeed, four of these genes encoded toxins that are usually components of venom in cnidarian-specific cnidocytes. At least 17% of A. digitifera PAP genes were also PAPs in A. tenuis, the basal lineage in the genus Acropora, indicating that PAPs were shared among species in Acropora.ConclusionsExpression differences caused by a high frequency of PAP genes may be a novel genomic feature in the genus Acropora; these findings will contribute to improve our understanding of correlation between genetic and phenotypic variation in corals.

Visual Social Relationship Recognition

Social relationships form the basis of social structure of humans. Developing computational models to understand social relationships from visual data is essential for building intelligent machines that can better interact with humans in a social environment. In this work, we study the problem of visual social relationship recognition in images. We propose a dual-glance model for social relationship recognition, where the first glance fixates at the person of interest and the second glance deploys attention mechanism to exploit contextual cues. To enable this study, we curated a large scale People in Social Context dataset, which comprises of 23,311 images and 79,244 person pairs with annotated social relationships. Since visually identifying social relationship bears certain degree of uncertainty, we further propose an adaptive focal loss to leverage the ambiguous annotations for more effective learning. We conduct extensive experiments to quantitatively and qualitatively demonstrate the efficacy of our proposed method, which yields state-of-the-art performance on social relationship recognition.

Bisupervised network with pyramid pooling module for land cover classification of satellite remote sensing imagery

Land cover classification is a significant task in remote sensing that aims at land cover monitoring and adjustment. Instance segmentation in deep learning has been widely used in land cover classification. However, this method requires high quality, labor-intensive pixel-level annotations. A bisupervised pipeline is proposed especially for datasets with ambiguous samples, which reduces the requirement of labeling accuracy. In order to supervise learning with ambiguous annotation samples, the pipeline has two losses to feedback, namely main loss and auxiliary loss. The main loss is still responsible for the expected output. According to the difference degree between ambiguous samples, categories are constructed artificially. The auxiliary loss generates feedback through categories, sharing decoder layers of main output during the training process. In the prediction process, auxiliary loss is used to update the main loss results. The neural network adopts the structure of U-Net with the pyramid pooling module, in which the multiscale feature is used in the feature extraction process. We also compare five different backbones and choose Inception-V3 as the backbone to improve the feature extraction capabilities of network encoders. The use of transposed convolution instead of traditional upsampling in decoders can improve the segmentation details. Our bisupervise network obtains Jaccard index of 83.936% and F 1-score of 91.17% on Gaofen-2 imagery dataset. The results demonstrate that the proposed method can achieve better classification performance in datasets with ambiguous annotations.

PhylotaR: An Automated Pipeline for Retrieving Orthologous DNA Sequences from GenBank in R.

The exceptional increase in molecular DNA sequence data in open repositories is mirrored by an ever-growing interest among evolutionary biologists to harvest and use those data for phylogenetic inference. Many quality issues, however, are known and the sheer amount and complexity of data available can pose considerable barriers to their usefulness. A key issue in this domain is the high frequency of sequence mislabeling encountered when searching for suitable sequences for phylogenetic analysis. These issues include, among others, the incorrect identification of sequenced species, non-standardized and ambiguous sequence annotation, and the inadvertent addition of paralogous sequences by users. Taken together, these issues likely add considerable noise, error or bias to phylogenetic inference, a risk that is likely to increase with the size of phylogenies or the molecular datasets used to generate them. Here we present a software package, phylotaR that bypasses the above issues by using instead an alignment search tool to identify orthologous sequences. Our package builds on the framework of its predecessor, PhyLoTa, by providing a modular pipeline for identifying overlapping sequence clusters using up-to-date GenBank data and providing new features, improvements and tools. We demonstrate and test our pipeline’s effectiveness by presenting trees generated from phylotaR clusters for two large taxonomic clades: Palms and primates. Given the versatility of this package, we hope that it will become a standard tool for any research aiming to use GenBank data for phylogenetic analysis.

MutHTP: mutations in human transmembrane proteins.

Existing sources of experimental mutation data do not consider the structural environment of amino acid substitutions and distinguish between soluble and membrane proteins. They also suffer from a number of further limitations, including data redundancy, lack of disease classification, incompatible information content, and ambiguous annotations (e.g. the same mutation being annotated as disease and benign). We have developed a novel database, MutHTP, which contains information on 183 395 disease-associated and 17 827 neutral mutations in human transmembrane proteins. For each mutation site MutHTP provides a description of its location with respect to the membrane protein topology, structural environment (if available) and functional features. Comprehensive visualization, search, display and download options are available. The database is publicly available at http://www.iitm.ac.in/bioinfo/MutHTP/. The website is implemented using HTML, PHP and javascript and supports recent versions of all major browsers, such as Firefox, Chrome and Opera. Supplementary data are available at Bioinformatics online.

Identification of the ATPase Subunit of the Primary Maltose Transporter in the Hyperthermophilic Anaerobe Thermotoga maritima.

Thermotoga maritima is a hyperthermophilic anaerobic bacterium that produces molecular hydrogen (H2) by fermentation. It catabolizes a broad range of carbohydrates through the action of diverse ABC transporters. However, in T. maritima and related species, highly similar genes with ambiguous annotation obscure a precise understanding of genome function. In T. maritima, three putative malK genes, all annotated as ATPase subunits, exhibited high identity to each other. To distinguish between these genes, malK disruption mutants were constructed by gene replacement, and the resulting mutant cell lines were characterized. Only a disruption of malK3 produced a defect in maltose catabolism. To verify that the mutant phenotype arose specifically from malK3 inactivation, the malK3 mutation was repaired by recombination, and maltose catabolism was restored. This study demonstrates the importance of a maltose ABC-type transporter and its relationship to sugar metabolism in T. maritimaIMPORTANCE The application and further development of a genetic system was used here to investigate gene paralogs in the hyperthermophile Thermotoga maritima The occurrence of three ABC transporter ATPase subunits all annotated as malK was evaluated using a combination of genetic and bioinformatic approaches. The results clarify the role of only one malK gene in maltose catabolism in a nonmodel organism noted for fermentative hydrogen production.

DisProt 7.0: a major update of the database of disordered proteins.

The Database of Protein Disorder (DisProt, URL: www.disprot.org) has been significantly updated and upgraded since its last major renewal in 2007. The current release holds information on more than 800 entries of IDPs/IDRs, i.e. intrinsically disordered proteins or regions that exist and function without a well-defined three-dimensional structure. We have re-curated previous entries to purge DisProt from conflicting cases, and also upgraded the functional classification scheme to reflect continuous advance in the field in the past 10 years or so. We define IDPs as proteins that are disordered along their entire sequence, i.e. entirely lack structural elements, and IDRs as regions that are at least five consecutive residues without well-defined structure. We base our assessment of disorder strictly on experimental evidence, such as X-ray crystallography and nuclear magnetic resonance (primary techniques) and a broad range of other experimental approaches (secondary techniques). Confident and ambiguous annotations are highlighted separately. DisProt 7.0 presents classified knowledge regarding the experimental characterization and functional annotations of IDPs/IDRs, and is intended to provide an invaluable resource for the research community for a better understanding structural disorder and for developing better computational tools for studying disordered proteins.

Regular expressions of MS/MS spectra for partial annotation of metabolite features

Partial annotation and characterization of metabolite structures on the basis of data from tandem mass spectrometry (MS/MS) spectra are technical bottlenecks in metabolomics. Novel approaches should be explored for evaluation of spectral similarities among structurally related compounds as well as for description of fragmentation motifs commonly observed in MS/MS spectra. A regular expression of MS/MS data was developed to search for structurally similar metabolites and to describe spectral motifs for partial annotation and characterization of metabolite structures. After definition of an MS/MS string as a text representation of an MS/MS spectrum, a regular expression of MS/MS strings involving meta characters, anchors, and quantifiers was introduced. Here it was also demonstrated that spectral motifs can be described by a regular expression to define a common fragmentation pattern observed among structurally related metabolites. The regular expression was applied to a search for similar MS/MS spectra. Analysis of MassBank data with fragment assignment information (fragment ion and neutral loss matrix, http://metabolomics.jp/wiki/Index:MassBank ) suggested that the regular expression of MS/MS spectra can detect spectral similarities among structurally related metabolites. Analysis of MS/MS spectral libraries of Arabidopsis and rice revealed that the metabolite features can be partially annotated or characterized by the spectral motifs and can be assigned the corresponding ontology codes produced by Chemical Entities of Biological Interest (ChEBI). The MS/MS spectral motifs represent a method for partial annotation or characterization of metabolite features. A regular expression of MS/MS data holds promise for further enrichment of metabolite annotations and for easy sharing of ambiguous annotation data among metabolomic studies.

MEGAnnotator: a user-friendly pipeline for microbial genomes assembly and annotation.

Genome annotation is one of the key actions that must be undertaken in order to decipher the genetic blueprint of organisms. Thus, a correct and reliable annotation is essential in rendering genomic data valuable. Here, we describe a bioinformatics pipeline based on freely available software programs coordinated by a multithreaded script named MEGAnnotator (Multithreaded Enhanced prokaryotic Genome Annotator). This pipeline allows the generation of multiple annotated formats fulfilling the NCBI guidelines for assembled microbial genome submission, based on DNA shotgun sequencing reads, and minimizes manual intervention, while also reducing waiting times between software program executions and improving final quality of both assembly and annotation outputs. MEGAnnotator provides an efficient way to pre-arrange the assembly and annotation work required to process NGS genome sequence data. The script improves the final quality of microbial genome annotation by reducing ambiguous annotations. Moreover, the MEGAnnotator platform allows the user to perform a partial annotation of pre-assembled genomes and includes an option to accomplish metagenomic data set assemblies. MEGAnnotator platform will be useful for microbiologists interested in genome analyses of bacteria as well as those investigating the complexity of microbial communities that do not possess the necessary skills to prepare their own bioinformatics pipeline.

MALT1 is not alone after all: identification of novel paracaspases.

Paracaspases and metacaspases are two families of caspase-like proteins identified in 2000. Up until now paracaspases were considered a single gene family with one known non-metazoan paracaspase in the slime mold Dictyostelium and a single animal paracaspase called MALT1. Human MALT1 is a critical signaling component in many innate and adaptive immunity pathways that drive inflammation, and when it is overly active, it can also cause certain forms of cancer. Here, we report the identification and functional analysis of two new vertebrate paracaspases, PCASP2 and PCASP3. Functional characterization indicates that both scaffold and protease functions are conserved across the three vertebrate paralogs. This redundancy might explain the loss of two of the paralogs in mammals and one in Xenopus. Several of the vertebrate paracaspases currently have incorrect or ambiguous annotations. We propose to annotate them accordingly as PCASP1, PCASP2, and PCASP3 similar to the caspase gene nomenclature. A comprehensive search in other metazoans and in non-metazoan species identified additional new paracaspases. We also discovered the first animal metacaspase in the sponge Amphimedon. Comparative analysis of the active site suggests that paracaspases constitute one of the several subclasses of metacaspases that have evolved several times independently.

Multiple evidence strands suggest that there may be as few as 19,000 human protein-coding genes.

Determining the full complement of protein-coding genes is a key goal of genome annotation. The most powerful approach for confirming protein-coding potential is the detection of cellular protein expression through peptide mass spectrometry (MS) experiments. Here, we mapped peptides detected in seven large-scale proteomics studies to almost 60% of the protein-coding genes in the GENCODE annotation of the human genome. We found a strong relationship between detection in proteomics experiments and both gene family age and cross-species conservation. Most of the genes for which we detected peptides were highly conserved. We found peptides for >96% of genes that evolved before bilateria. At the opposite end of the scale, we identified almost no peptides for genes that have appeared since primates, for genes that did not have any protein-like features or for genes with poor cross-species conservation. These results motivated us to describe a set of 2001 potential non-coding genes based on features such as weak conservation, a lack of protein features, or ambiguous annotations from major databases, all of which correlated with low peptide detection across the seven experiments. We identified peptides for just 3% of these genes. We show that many of these genes behave more like non-coding genes than protein-coding genes and suggest that most are unlikely to code for proteins under normal circumstances. We believe that their inclusion in the human protein-coding gene catalogue should be revised as part of the ongoing human genome annotation effort.