Annotation Styles Research Articles

Meta-learners for few-shot weakly-supervised medical image segmentation

Most uses of Meta-Learning in visual recognition are very often applied to image classification, with a relative lack of work in other tasks such as segmentation and detection. We propose a new generic Meta-Learning framework for few-shot weakly supervised segmentation in medical imaging domains. The proposed approach includes a meta-training phase that uses a meta-dataset. It is deployed on an out-of-distribution few-shot target task, where a single highly generalizable model, trained via a selective supervised loss function, is used as a predictor. The model can be trained in several distinct ways, such as second-order optimization, metric learning, and late fusion. Some relevant improvements of existing methods that are part of the proposed approach are presented. We conduct a comparative analysis of meta-learners from distinct paradigms adapted to few-shot image segmentation in different sparsely annotated radiological tasks. The imaging modalities include 2D chest, mammographic, and dental X-rays, as well as 2D slices of volumetric tomography and resonance images. Our experiments consider in total 9 meta-learners, 4 backbones, and multiple target organ segmentation tasks. We explore small-data scenarios in radiology with varying weak annotation styles and densities. Our analysis shows that metric-based meta-learning approaches achieve better segmentation results in tasks with smaller domain shifts compared to the meta-training datasets, while some gradient- and fusion-based meta-learners are more generalizable to larger domain shifts. Guidelines learned from the comparative performance assessment of the analyzed methods are summarized to support those readers interested in the field.

A robust deep learning detector for sleep spindles and K-complexes: towards population norms

Sleep spindles (SSs) and K-complexes (KCs) are brain patterns involved in cognitive functions that appear during sleep. Large-scale sleep studies would benefit from precise and robust automatic sleep event detectors, capable of adapting the variability in both electroencephalography (EEG) signals and expert annotation rules. We introduce the Sleep EEG Event Detector (SEED), a deep learning system that outperforms existing approaches in SS and KC detection, reaching an F1-score of 80.5% and 83.7%, respectively, on the MASS2 dataset. SEED transfers well and requires minimal fine-tuning for new datasets and annotation styles. Remarkably, SEED substantially reduces the required amount of annotated data by using a novel pretraining approach that leverages the rule-based detector A7. An analysis of 11,224 subjects revealed that SEED's detections provide better estimates of SS population statistics than existing approaches. SEED is a powerful resource for obtaining sleep-event statistics that could be useful for establishing population norms.

Statistical and topological summaries aid disease detection for segmented retinal vascular images.

Disease complications can alter vascular network morphology and disrupt tissue functioning. Microvascular diseases of the retina are assessed by visual inspection of retinal images, but this can be challenging when diseases exhibit silent symptoms or patients cannot attend in-person meetings. We examine the performance of machine learning algorithms in detecting microvascular disease when trained on statistical and topological summaries of segmented retinal vascular images. We compute 13 separate descriptor vectors (5 statistical, 8 topological) to summarize the morphology of retinal vessel segmentation images and train support vector machines to predict each image's disease classification from the summary vectors. We assess the performance of each descriptor vector, using five-fold cross validation to estimate their accuracy. We apply these methods to four datasets that were assembled from four existing data repositories; three datasets contain segmented retinal vascular images from one of the repositories, whereas the fourth "All" dataset combines images from four repositories. Among the 13 total descriptor vectors considered, either a statistical Box-counting descriptor vector or a topological Flooding descriptor vector achieves the highest accuracy levels. On the combined "All" dataset, the Box-counting vector outperforms all other descriptors, including the topological Flooding vector which is sensitive to differences in the annotation styles between the different datasets. Our work represents a first step to establishing which computational methods are most suitable for identifying microvascular disease and assessing their current limitations. These methods could be incorporated into automated disease assessment tools.

Rethinking Generalization: The Impact of Annotation Style on Medical Image Segmentation

Generalization is an important attribute of machine learning models, particularly for those that are to be deployed in a medical context, where unreliable predictions can have real world consequences. While the failure of models to generalize across datasets is typically attributed to a mismatch in the data distributions, performance gaps are often a consequence of biases in the "ground-truth" label annotations. This is particularly important in the context of medical image segmentation of pathological structures (e.g. lesions), where the annotation process is much more subjective, and affected by a number underlying factors, including the annotation protocol, rater education/experience, and clinical aims, among others. In this paper, we show that modeling annotation biases, rather than ignoring them, poses a promising way of accounting for differences in annotation style across datasets. To this end, we propose a generalized conditioning framework to (1) learn and account for different annotation styles across multiple datasets using a single model, (2) identify similar annotation styles across different datasets in order to permit their effective aggregation, and (3) fine-tune a fully trained model to a new annotation style with just a few samples. Next, we present an image-conditioning approach to model annotation styles that correlate with specific image features, potentially enabling detection biases to be more easily identified.

Regressive Pseudo Label for Weakly Supervised Facial Landmark Detection

The progress of the deep neural network and visual sensors promote the facial landmark detection. However, faces are easily collected from Internet of Things (IoT) devices worldwide but are hard-labeled facial landmarks in consistent style. Though many weak supervision (WS) algorithms and theories have been proposed to handle the labeling problem, most of them are tailored for classification tasks and fail in regression tasks, especially in facial landmark detection. To tackle this WS regression task, first, we propose a regressive pseudo-labeling method by analyzing the cluster assumption of facial landmark detection, where overlaps are reduced and interval areas are increased among clusters of facial parts for unlabeled faces. Moreover, auxiliary information and domain loss are utilized to adapt model to samples with different styles. Second, we design a generator–regressor network to first estimate facial boundary attentions and then to locate facial landmarks. However, when generating pseudo labels based on predictive models automatically, there are two major issues. One is that the results of the multistage network highly depend on the former-stage accuracy, and another is that jointing different annotation styles always produces ambiguity feature representation. Thus, we propose two ideas. During training, the generator and regressor are decoupled to alleviate the inner dependence of the multistage network, and the heatmap discriminator is introduced to improve the quality of the predicted facial boundary. We design a transformer structure to fuse face image and boundary attentions, so that it can further complement useful features. Based on these ideas, our methods can automatically annotate accurate pseudo facial landmarks for unlabeled faces. Extensive experiments show that our model achieves good performance on different benchmark data sets, both in accuracy and efficiency.

Assessing the impact of OCR noise on multilingual event detection over digitised documents

Event detection is a crucial task for natural language processing and it involves the identification of instances of specified types of events in text and their classification into event types. The detection of events from digitised documents could enable historians to gather and combine a large amount of information into an integrated whole, a panoramic interpretation of the past. However, the level of degradation of digitised documents and the quality of the optical character recognition (OCR) tools might hinder the performance of an event detection system. While several studies have been performed in detecting events from historical documents, the transcribed documents needed to be hand-validated which implied a great effort of human expertise and manual labour-intensive work. Thus, in this study, we explore the robustness of two different event detection language-independent models to OCR noise, over two datasets that cover different event types and multiple languages. We aim at analysing their ability to mitigate problems caused by the low quality of the digitised documents and we simulate the existence of transcribed data, synthesised from clean annotated text, by injecting synthetic noise. For creating the noisy synthetic data, we chose to utilise four main types of noise that commonly occur after the digitisation process: Character Degradation, Bleed Through, Blur, and Phantom Character. Finally, we conclude that the imbalance of the datasets, the richness of the different annotation styles, and the language characteristics are the most important factors that can influence event detection in digitised documents.

Neural Character-Level Syntactic Parsing for Chinese

In this work, we explore character-level neural syntactic parsing for Chinese with two typical syntactic formalisms: the constituent formalism and a dependency formalism based on a newly released character-level dependency treebank. Prior works in Chinese parsing have struggled with whether to de ne words when modeling character interactions. We choose to integrate full character-level syntactic dependency relationships using neural representations from character embeddings and richer linguistic syntactic information from human-annotated character-level Parts-Of-Speech and dependency labels. This has the potential to better understand the deeper structure of Chinese sentences and provides a better structural formalism for avoiding unnecessary structural ambiguities. Specifically, we first compare two different character-level syntax annotation styles: constituency and dependency. Then, we discuss two key problems for character-level parsing: (1) how to combine constituent and dependency syntactic structure in full character-level trees and (2) how to convert from character-level to word-level for both constituent and dependency trees. In addition, we also explore several other key parsing aspects, including di erent character-level dependency annotations and joint learning of Parts-Of-Speech and syntactic parsing. Finally, we evaluate our models on the Chinese Penn Treebank (CTB) and our published Shanghai Jiao Tong University Chinese Character Dependency Treebank (SCDT). The results show the e effectiveness of our model on both constituent and dependency parsing. We further provide empirical analysis and suggest several directions for future study.

Automated Annotator Variability Inspection for Biomedical Image Segmentation

Supervised deep learning approaches for automated diagnosis support require datasets annotated by experts. Intra-annotator variability of a single annotator and inter-annotator variability between annotators can affect the quality of the diagnosis support. As medical experts will always differ in annotation details, quantitative studies concerning the annotation quality are of particular interest. A consistent and noise-free annotation of large-scale datasets by, for example, dermatologists or pathologists is a current challenge. Hence, methods are needed to automatically inspect annotations in datasets. In this paper, we categorize annotation noise in image segmentation tasks, present methods to simulate annotation noise, and examine the impact on the segmentation quality. Two novel automated methods to identify intra-annotator and inter-annotator inconsistencies based on uncertainty-aware deep neural networks are proposed. We demonstrate the benefits of our automated inspection methods such as focused re-inspection of noisy annotations or the detection of generally different annotation styles using the biomedical ISIC 2017 Melanoma image segmentation dataset.

Large-scale atlas of microarray data reveals the distinct expression landscape of different tissues in Arabidopsis.

Transcriptome data sets from thousands of samples of the model plant Arabidopsis thaliana have been collectively generated by multiple individual labs. Although integration and meta-analysis of these samples has become routine in the plant research community, it is often hampered by a lack of metadata or differences in annotation styles of different labs. In this study, we carefully selected and integrated 6057 Arabidopsis microarray expression samples from 304 experiments deposited to the Gene Expression Omnibus (GEO) at the National Center for Biotechnology Information (NCBI). Metadata such as tissue type, growth conditions and developmental stage were manually curated for each sample. We then studied the global expression landscape of the integrated data set and found that samples of the same tissue tend to be more similar to each other than to samples of other tissues, even in different growth conditions or developmental stages. Root has the most distinct transcriptome, compared with aerial tissues, but the transcriptome of cultured root is more similar to the transcriptome of aerial tissues, as the cultured root samples lost their cellular identity. Using a simple computational classification method, we showed that the tissue type of a sample can be successfully predicted based on its expression profile, opening the door for automatic metadata extraction and facilitating the re-use of plant transcriptome data. As a proof of principle, we applied our automated annotation pipeline to 708 RNA-seq samples from public repositories and verified the accuracy of our predictions with sample metadata provided by the authors.

Annotation As an Index to Critical Writing

The differences in the ability to write critical and analytical essays among students with individual annotation styles were investigated. Critical and analytical writing was determined by the writer's ability to respond to a text with logical and critical analysis and attention to its thematic argument. Annotation styles were determined by ways of annotating a text: critical inquiry as skillful and simple highlighting as verbatim. The results indicated that skillful annotators produced more critical and analytical writing samples than did verbatim annotators. Verbatim annotators recycled information rather than analyzing it. The findings are congruent with theories that promote explicit metacognitive skills and support the position that teaching tactics consonant with students' cultural backgrounds are more likely to succeed in fostering critical thinking reflected in writing.