Ground Truth Annotations Research Articles

LAKE ICE MONITORING WITH WEBCAMS AND CROWD-SOURCED IMAGES

Abstract. Lake ice is a strong climate indicator and has been recognised as part of the Essential Climate Variables (ECV) by the Global Climate Observing System (GCOS). The dynamics of freezing and thawing, and possible shifts of freezing patterns over time, can help in understanding the local and global climate systems. One way to acquire the spatio-temporal information about lake ice formation, independent of clouds, is to analyse webcam images. This paper intends to move towards a universal model for monitoring lake ice with freely available webcam data. We demonstrate good performance, including the ability to generalise across different winters and lakes, with a state-of-the-art Convolutional Neural Network (CNN) model for semantic image segmentation, Deeplab v3+. Moreover, we design a variant of that model, termed Deep-U-Lab, which predicts sharper, more correct segmentation boundaries. We have tested the model’s ability to generalise with data from multiple camera views and two different winters. On average, it achieves Intersection-over-Union (IoU) values of ≈71% across different cameras and ≈69% across different winters, greatly outperforming prior work. Going even further, we show that the model even achieves 60% IoU on arbitrary images scraped from photo-sharing websites. As part of the work, we introduce a new benchmark dataset of webcam images, Photi-LakeIce, from multiple cameras and two different winters, along with pixel-wise ground truth annotations.

A bluetooth low energy dataset for the analysis of social interactions with commercial devices.

This paper describes a data collection campaign and a dataset of BLE beacons for detecting and analysing human social interactions. The dataset has been collected by involving 15 volunteers that interacted in indoor environments for a total of 11 hours of activity. The dataset is released as a collection of CSV files with a timestamp, RSSI (Received Signal Strength Indicator) and a unique identifier of the emitting and of the receiving devices. Volunteers wear a wristband equipped with BLE tags emitting beacons at a fixed rate, and a mobile application able to collect and to store beacons. We organized 6 interaction sessions, designed to reproduce the three common stages of an interaction (Non Interaction, Approaching and Interaction). Moreover, we reproduced interactions by varying the volunteer’s posture as well as the position of the receiving device. The dataset is released with a ground truth annotation reporting the exact time intervals during which volunteers actually interacted. The combination of such factors, provides a rich dataset useful to experiment algorithms for detecting interactions and for analyzing dynamics of interactions in a real-world setting. We present in detail the dataset and its evaluation in “Sensing Social Interactions through BLE Beacons and Commercial Mobile Devices”, in which we focus on two orthogonal analysis: quality of the dataset and RSSI symmetry of the channel during the interaction stage between pairs of users.

Neural side effect discovery from user credibility and experience-assessed online health discussions

BackgroundHealth 2.0 allows patients and caregivers to conveniently seek medical information and advice via e-portals and online discussion forums, especially regarding potential drug side effects. Although online health communities are helpful platforms for obtaining non-professional opinions, they pose risks in communicating unreliable and insufficient information in terms of quality and quantity. Existing methods in extracting user-reported adverse drug reactions (ADRs) in online health forums are not only insufficiently accurate as they disregard user credibility and drug experience, but are also expensive as they rely on supervised ground truth annotation of individual statement. We propose a NEural ArchiTecture for Drug side effect prediction (NEAT), which is optimized on the task of drug side effect discovery based on a complete discussion while being attentive to user credibility and experience, thus, addressing the mentioned shortcomings. We train our neural model in a self-supervised fashion using ground truth drug side effects from mayoclinic.org. NEAT learns to assign each user a score that is descriptive of their credibility and highlights the critical textual segments of their post.ResultsExperiments show that NEAT improves drug side effect discovery from online health discussion by 3.04% from user-credibility agnostic baselines, and by 9.94% from non-neural baselines in term of F1. Additionally, the latent credibility scores learned by the model correlate well with trustworthiness signals, such as the number of “thanks” received by other forum members, and improve credibility heuristics such as number of posts by 0.113 in term of Spearman’s rank correlation coefficient. Experience-based self-supervised attention highlights critical phrases such as mentioned side effects, and enhances fully supervised ADR extraction models based on sequence labelling by 5.502% in terms of precision.ConclusionsNEAT considers both user credibility and experience in online health forums, making feasible a self-supervised approach to side effect prediction for mentioned drugs. The derived user credibility and attention mechanism are transferable and improve downstream ADR extraction models. Our approach enhances automatic drug side effect discovery and fosters research in several domains including pharmacovigilance and clinical studies.

Temporal social network reconstruction using wireless proximity sensors: model selection and consequences

The emerging technologies of wearable wireless devices open entirely new ways to record various aspects of human social interactions in a broad range of settings. Such technologies allow to log the temporal dynamics of face-to-face interactions by detecting the physical proximity of participants. However, despite the wide usage of this technology and the collected datasets, precise reconstruction methods transforming the raw recorded communication data packets to social interactions are still missing.In this study we analyse a proximity dataset collected during a longitudinal social experiment aiming to understand the co-evolution of children’s language development and social network. Physical proximity and verbal communication of hundreds of pre-school children and their teachers are recorded over three years using autonomous wearable low power wireless devices. The dataset is accompanied with three annotated ground truth datasets, which record the time, distance, relative orientation, and interaction state of interacting children for validation purposes.We use this dataset to explore several pipelines of dynamical event reconstruction including earlier applied naïve approaches, methods based on Hidden Markov Model, or on Long Short-Term Memory models, some of them combined with supervised pre-classification of interaction packets. We find that while naïve models propose the worst reconstruction, Long Short-Term Memory models provide the most precise way to reconstruct real interactions up to {sim} 90% accuracy. Finally, we simulate information spreading on the reconstructed networks obtained by the different methods. Results indicate that small improvement of network reconstruction accuracy may lead to significantly different spreading dynamics, while sometimes large differences in accuracy have no obvious effects on the dynamics. This not only demonstrates the importance of precise network reconstruction but also the careful choice of the reconstruction method in relation with the data collected. Missing this initial step in any study may seriously mislead conclusions made about the emerging properties of the observed network or any dynamical process simulated on it.

Deep Learning based Quantification of Ovary and Follicles using 3D Transvaginal Ultrasound in Assisted Reproduction.

Quantification of ovarian and follicular volume and follicle count are performed in clinical practice for diagnosis and management in assisted reproduction. Ovarian volume and Antral Follicle Count (AFC) are typically tracked over the ovulation cycle. Volumetric analysis of ovary and follicle is manual and largely operator dependent. In this manuscript, we have proposed a deep-learning method for automatic simultaneous segmentation of ovary and follicles in 3D Transvaginal Ultrasound (TVUS), namely S-Net. The proposed loss function restricts false detection of follicles outside the ovary. Additionally, we have used multi-layered loss to provide deep supervision for training the network. S-Net is optimized for inference time and memory while utilizing 3D context in the 2D deep-learning network. 66 3D TVUS volumes (13,200 2D image slices) were acquired from 66 subjects in this Institutional Review Board (IRB) approved study. The segmentation framework provides approximately 92% and 87% average DICE overlap with the ground truth annotations for ovary and follicles, respectively. We have obtained state-of-the-art results with a detection rate of 88%, 91% and 98% for follicles of size 2-4mm, 4-12mm and >12mm.

Investigation of tumor and vessel motion correlation in the liver.

Intrafraction imaging‐based motion management systems for external beam radiotherapy can rely on internal surrogate structures when the target is not easily visualized. This work evaluated the validity of using liver vessels as internal surrogates for the estimation of liver tumor motion. Vessel and tumor motion were assessed using ten two‐dimensional sagittal MR cine datasets collected on the ViewRay MRIdian. For each case, a liver tumor and at least one vessel were tracked for 175 s. A tracking approach utilizing block matching and multiple simultaneous templates was applied. Accuracy of the tracked motion was calculated from the error between the tracked centroid position and manually defined ground truth annotations. The patient’s abdomen surface and diaphragm were manually annotated in all frames. The Pearson correlation coefficient (CC) was used to compare the motion of the features and tumor in the anterior–posterior (AP) and superior–inferior (SI) directions. The distance between the centroids of the features and the tumors was calculated to assess if feature proximity affects relative correlation, and the tumor range of motion was determined. Intra‐ and interfraction motion amplitude variabilities were evaluated to further assess the relationship between tumor and feature motion. The mean CC between the motion of the vessel and the tumor were 0.85 ± 0.11 (AP) and 0.92 ± 0.04 (SI), 0.83 ± 0.11 (AP) and −0.89 ± 0.06 (SI) for the surface and tumor, and 0.80 ± 0.17 (AP) and 0.94 ± 0.03 (SI) for the diaphragm and tumor. For intrafraction analysis, the average amplitude variability was 2.47 ± 0.77 mm (AP) and 3.14 ± 1.49 mm (SI) for the vessels, 2.70 ± 1.08 mm (AP) and 3.43 ± 1.73 mm (SI) for the surface, and 2.76 ± 1.41 mm (AP) and 2.91 ± 1.38 mm (SI) for the diaphragm. No relationship between distance and motion correlation was observed. The motion of liver tumors and liver vessels was well correlated, making vessels a suitable surrogate for tumor motion in the liver.

Design of a Category Independent, Aspect Based Automated Opinion Analysis Technique for Online Product Reviews

ABSTRACT Customer feedback in the form of online reviews is an important source of information to manufacturers or service providers for evaluation of their products or services. Online reviews also help potential buyers in making their decisions. Manual checking of these huge amount of unstructured texts is time consuming. Several attempts have been made for opinion aggregation of online reviews but a generalized automated technique has yet to be developed. In this work, an efficient rule based technique for aspect wise summarization of online product reviews irrespective of their categories has been designed. The proposed technique develops the rules for extracting aspects and associating the opinion words to the respective aspects followed by effective grouping and summarization of aspect-opinion pairs into human interpretable form. The algorithm has been implemented on Amazon Product Reviews and evaluated against manually annotated ground truth. The result shows promising similarity with human judgement.

PB-Net: Automatic peak integration by sequential deep learning for multiple reaction monitoring

Mass spectrometry (MS) based proteomics has become an indispensable component of modern molecular and cellular biochemistry analysis. Multiple reaction monitoring (MRM) is one of the most well-established MS techniques for molecule detection and quantification. Despite its wide usage, there lacks an accurate computational framework to analyze MRM data, and expert annotation is often required, especially to perform peak integration. Here we propose a deep learning method PB-Net (Peak Boundary Neural Network), built upon recent advances in sequential neural networks, for fully automatic chromatographic peak integration. To train PB-Net, we generated a large dataset of over 170,000 expert annotated peaks from MS transitions spanning a wide dynamic range, including both peptides and intact glycopeptides. Our model demonstrated outstanding performances on unseen test samples, reaching near-perfect agreement (Pearson's r 0.997) with human annotated ground truth. Systematic evaluations also show that PB-Net is substantially more robust and accurate compared to previous state-of-the-art peak integration software. PB-Net can benefit the wide community of mass spectrometry data analysis, especially in applications involving high-throughput MS experiments. Codes and test data used in this work are available at https://github.com/miaecle/PB-net. SignificanceHuman annotations serve an important role in accurate quantification of multiple reaction monitoring (MRM) experiments, though they are costly to collect and limit analysis throughput. In this work we proposed and developed a novel technique for the peak-integration step in MRM, based on recent innovations in sequential deep learning models. We collected in total 170,000 expert-annotated MRM peaks and trained a set of accurate and robust neural networks for the task. Results demonstrated a substantial improvement over the current state-of-the-art software for mass spectrometry analysis and comparable level of accuracy and precision as human annotators.

Learned optical flow for intra-operative tracking of the retinal fundus

PurposeSustained delivery of regenerative retinal therapies by robotic systems requires intra-operative tracking of the retinal fundus. We propose a supervised deep convolutional neural network to densely predict semantic segmentation and optical flow of the retina as mutually supportive tasks, implicitly inpainting retinal flow information missing due to occlusion by surgical tools.MethodsAs manual annotation of optical flow is infeasible, we propose a flexible algorithm for generation of large synthetic training datasets on the basis of given intra-operative retinal images. We evaluate optical flow estimation by tracking a grid and sparsely annotated ground truth points on a benchmark of challenging real intra-operative clips obtained from an extensive internally acquired dataset encompassing representative vitreoretinal surgical cases.ResultsThe U-Net-based network trained on the synthetic dataset is shown to generalise well to the benchmark of real surgical videos. When used to track retinal points of interest, our flow estimation outperforms variational baseline methods on clips containing tool motions which occlude the points of interest, as is routinely observed in intra-operatively recorded surgery videos.ConclusionsThe results indicate that complex synthetic training datasets can be used to specifically guide optical flow estimation. Our proposed algorithm therefore lays the foundation for a robust system which can assist with intra-operative tracking of moving surgical targets even when occluded.

Deep learning detection and quantification of pneumothorax in heterogeneous routine chest computed tomography

BackgroundAutomatically detecting and quantifying pneumothorax on chest computed tomography (CT) may impact clinical decision-making. Machine learning methods published so far struggle with the heterogeneity of technical parameters and the presence of additional pathologies, highlighting the importance of stable algorithms.MethodsA deep residual UNet was developed and evaluated for automated, volume-level pneumothorax grading (i.e., labelling a volume whether a pneumothorax was present or not), and pixel-level classification (i.e., segmentation and quantification of pneumothorax), on a retrospective series of routine chest CT data. Ground truth annotations were provided by radiologists. The fully automated pixel-level pneumothorax segmentation method was trained using 43 chest CT scans and evaluated on 9 chest CT scans with pixel-level annotation basis and 567 chest CT scans on a volume-level basis.ResultsThis method achieved a receiver operating characteristic area under the curve (AUC) of 0.98, an average precision of 0.97, and a Dice similarity coefficient (DSC) of 0.94. This segmentation performance resulted to be similar to the inter-rater segmentation accuracy of two radiologists, who achieved a DSC of 0.92. The comparison of manual and automated pneumothorax quantification yielded a Pearson correlation coefficient of 0.996. The volume-level pneumothorax grading accuracy was evaluated on 567 chest CT scans and yielded an AUC of 0.98 and an average precision of 0.95.ConclusionsWe proposed a deep learning method for the detection and quantification of pneumothorax in heterogeneous routine clinical data that may facilitate the automated triage of urgent examinations and enable treatment decision support.

Evaluation of Automatic Facial Wrinkle Detection Algorithms.

Facial wrinkles (considered to be natural features) appear as people get older. Wrinkle detection is an important aspect of applications that depend on facial skin changes, such as face age estimation and soft biometrics. While existing wrinkle detection algorithms focus on forehead horizontal lines, it is necessary to develop new methods to detect all wrinkles (vertical and horizontal) on whole face. Therefore, we evaluated the performance of wrinkle detection algorithms on the whole face and proposed an enhancement technique to improve the performance. More specifically, we used 45 images of the Face Recognition Technology dataset (FERET) and 25 images of the Sudanese dataset. For ground truth annotations, the selected images were manually annotated by the researcher. The experiments showed that the method with enhancement performed better at detecting facial wrinkles when compared to the state-of-the-art methods. When evaluated on FERET, the average Jaccard similarity indices were 56.17%, 31.69% and 15.87% for the enhancement method, Hybrid Hessian Filter and Gabor Filter, respectively.

Self-Supervised Representation Learning for Ultrasound Video.

Recent advances in deep learning have achieved promising performance for medical image analysis, while in most cases ground-truth annotations from human experts are necessary to train the deep model. In practice, such annotations are expensive to collect and can be scarce for medical imaging applications. Therefore, there is significant interest in learning representations from unlabelled raw data. In this paper, we propose a self-supervised learning approach to learn meaningful and transferable representations from medical imaging video without any type of human annotation. We assume that in order to learn such a representation, the model should identify anatomical structures from the unlabelled data. Therefore we force the model to address anatomy-aware tasks with free supervision from the data itself. Specifically, the model is designed to correct the order of a reshuffled video clip and at the same time predict the geometric transformation applied to the video clip. Experiments on fetal ultrasound video show that the proposed approach can effectively learn meaningful and strong representations, which transfer well to downstream tasks like standard plane detection and saliency prediction.

SeaGrassDetect: A Novel Method for the Detection of Seagrass from Unlabelled Underwater Videos

Benthic vegetation is arguably one of the most important indicators of the state of marine environment. Assessment of the status of eelgrass (Zostera marina) is commonly done using various remote sensing methods such as aerial photography or satellite images. These methods often fail to capture the true scenario beneath the surface of the water if the water is turbid or the satellite image is masked by cloud cover which makes it impossible to see beneath them. As a second line of defense, researchers have used under water videos (obtained either with scuba,snorkel or other visual observations) to assess the ground truth. Lots of man-hours are spent browsing through many hours of video data manually by an expert and assessing the status (presence/absence) which is a very common practice. Here we propose two methods for detection of eelgrass (presence/absence) from under water videos obtained from Roskilde Fjord in Denmark. We extend these methods to show that it can be used as a proxy to estimate coverage of eelgrass in a given area which match well with an expert's estimation. The benefit of using this method is that it is objective, less biased, cost efficient, robust to noisy environment, does not require pixel-level annotated ground truth images and can be used on existing video transects. This method can also detect rare errors from domain expert's visual estimation.

Automatic “Ground Truth” Annotation and Industrial Workpiece Dataset Generation for Deep Learning

In industry, it is becoming common to detect and recognize industrial workpieces using deep learning methods. In this field, the lack of datasets is a big problem, and collecting and annotating datasets in this field is very labor intensive. The researchers need to perform dataset annotation if a dataset is generated by themselves. It is also one of the restrictive factors that the current method based on deep learning cannot expand well. At present, there are very few workpiece datasets for industrial fields, and the existing datasets are generated from ideal workpiece computer aided design (CAD) models, for which few actual workpiece images were collected and utilized. We propose an automatic industrial workpiece dataset generation method and an automatic ground truth annotation method. Included in our methods are three algorithms that we proposed: a point cloud based spatial plane segmentation algorithm to segment the workpieces in the real scene and to obtain the annotation information of the workpieces in the images captured in the real scene; a random multiple workpiece generation algorithm to generate abundant composition datasets with random rotation workpiece angles and positions; and a tangent vector based contour tracking and completion algorithm to get improved contour images. With our procedures, annotation information can be obtained using the algorithms proposed in this paper. Upon completion of the annotation process, a json format file is generated. Faster R-CNN (Faster R-convolutional neural network), SSD (single shot multibox detector) and YOLO (you only look once: unified, real-time object detection) are trained using the datasets proposed in this paper. The experimental results show the effectiveness and integrity of this dataset generation and annotation method.

Semantic Relational Object Tracking

This paper addresses the topic of semantic world modeling by conjoining probabilistic reasoning and object anchoring. The proposed approach uses a so-called bottom-up object anchoring method that relies on the rich continuous data from perceptual sensor data. A novel anchoring matching function method learns to maintain object entities in space and time and is validated using a large set of trained humanly annotated ground truth data of real-world objects. For more complex scenarios, a high-level probabilistic object tracker has been integrated with the anchoring framework and handles the tracking of occluded objects via reasoning about the state of unobserved objects. We demonstrate the performance of our integrated approach through scenarios such as the shell game scenario, where we illustrate how anchored objects are retained by preserving relations through probabilistic reasoning.

Estimating the imageability of words by mining visual characteristics from crawled image data

Natural Language Processing and multi-modal analyses are key elements in many applications. However, the semantic gap is an everlasting problem, leading to unnatural results disconnected from the user’s perception. To understand semantics in multimedia applications, human perception needs to be taken into consideration. Imageability is an approach originating from Pyscholinguistics to quantize the human perception of words. Research shows a relationship between language usage and the imageability of words, making it useful for multimodal applications. However, the creation of imageability datasets is often manual and labor-intensive. In this paper, we propose a method using image data mining of a variety of visual features to estimate the imageability of words. The main assumption is a relationship between the imageability of concepts, human perception, and the contents of Web-crawled images. Using a set of low- and high-level visual features from Web-crawled images, a model is trained to predict imageability. The evaluations show that the imageability can be predicted with both a sufficiently low error, and a high correlation to the ground-truth annotations. The proposed method can be used to increase the corpus of imageability dictionaries.

Text alignment in early printed books combining deep learning and dynamic programming

We describe a technique for transcript alignment in early printed books by using deep models in combination with dynamic programming algorithms. Two object detection models, based on Faster R-CNN, are trained to locate words. We first train an initial model to recognize generic words and hyphens by using information about the number of words in text lines. Using the model prediction on pages with a line-by-line ground-truth annotation is available, we train a second model able to detect landmark words. The alignment is then based on the identification of landmark words in pages where we only know the text corresponding to zones in the page. The proposed technique is evaluated on a publicly available digitization of the Gutenberg Bible while the transcription is based on the Vulgata, a late 4th century Latin translation of the Bible.

I Feel I Feel You: A Theory of Mind Experiment in Games

In this study into the player’s emotional theory of mind (ToM) of gameplaying agents, we investigate how an agent’s behaviour and the player’s own performance and emotions shape the recognition of a frustrated behaviour. We focus on the perception of frustration as it is a prevalent affective experience in human-computer interaction. We present a testbed game tailored towards this end, in which a player competes against an agent with a frustration model based on theory. We collect gameplay data, an annotated ground truth about the player’s appraisal of the agent’s frustration, and apply face recognition to estimate the player’s emotional state. We examine the collected data through correlation analysis and predictive machine learning models, and find that the player’s observable emotions are not correlated highly with the perceived frustration of the agent. This suggests that our subject’s ToM is a cognitive process based on the gameplay context. Our predictive models—using ranking support vector machines—corroborate these results, yielding moderately accurate predictors of players’ ToM.

VOSTR: Video Object Segmentation via Transferable Representations

In order to learn video object segmentation models, conventional methods require a large amount of pixel-wise ground truth annotations. However, collecting such supervised data is time-consuming and labor-intensive. In this paper, we exploit existing annotations in source images and transfer such visual information to segment videos with unseen object categories. Without using any annotations in the target video, we propose a method to jointly mine useful segments and learn feature representations that better adapt to the target frames. The entire process is decomposed into three tasks: (1) refining the responses with fully-connected CRFs, (2) solving a submodular function for selecting object-like segments, and (3) learning a CNN model with a transferable module for adapting seen categories in the source domain to the unseen target video. We present an iterative update scheme between three tasks to self-learn the final solution for object segmentation. Experimental results on numerous benchmark datasets demonstrate that the proposed method performs favorably against the state-of-the-art algorithms.

Deep learning-based segmentation of breast masses in dedicated breast CT imaging: Radiomic feature stability between radiologists and artificial intelligence

A deep learning (DL) network for 2D-based breast mass segmentation in unenhanced dedicated breast CT images was developed and validated, and its robustness in radiomic feature stability and diagnostic performance compared to manual annotations of multiple radiologists was investigated. 93 mass-like lesions were extensively augmented and used to train the network (n = 58 masses), which was then tested (n = 35 masses) against manual ground truth of a qualified breast radiologist with experience in breast CT imaging using the Conformity coefficient (with a value equal to 1 indicating a perfect performance). Stability and diagnostic power of 672 radiomic descriptors were investigated between the computerized segmentation, and 4 radiologists' annotations for the 35 test set cases. Feature stability and diagnostic performance in the discrimination between benign and malignant cases were quantified using intraclass correlation (ICC) and multivariate analysis of variance (MANOVA), performed for each segmentation case (4 radiologists and DL algorithm). DL-based segmentation resulted in a Conformity of 0.85 ± 0.06 against the annotated ground truth. For the stability analysis, although modest agreement was found among the four annotations performed by radiologists (Conformity 0.78 ± 0.03), over 90% of all radiomic features were found to be stable (ICC>0.75) across multiple segmentations. All MANOVA analyses were statistically significant (p ≤ 0.05), with all dimensions equal to 1, and Wilks’ lambda ≤0.35. In conclusion, DL-based mass segmentation in dedicated breast CT images can achieve high segmentation performance, and demonstrated to provide stable radiomic descriptors with comparable discriminative power in the classification of benign and malignant tumors to expert radiologist annotation.