Annotated Images Research Articles

Imaging Evaluation of the Pediatric Craniovertebral Junction

The craniovertebral junction is imaged in both head and cervical spine imaging. An understanding of the normal morphometric appearance and common anatomic variants of this region is important for making correct diagnoses and avoiding misinterpretation, e.g., differentiating trauma from congenital variants. We describe a practical approach to evaluating the pediatric craniovertebral junction and recognizing common variants using annotated images. Pathology of the craniovertebral junction, including platybasia, basilar invagination, and traumatic injury, is illustrated, and syndromic associations of these findings are reviewed.Learning Objective: Review normal and variant anatomy of the pediatric craniovertebral junction and describe useful measurements for evaluating for traumatic and nontraumatic conditions in children.

Extraction of snow cover from high-resolution remote sensing imagery using deep learning on a small dataset

ABSTRACTSnow cover is of great significance for many applications. However, automatic extraction of snow cover from high spatial resolution remote sensing (HSRRS) imagery remains challenging, owing to its multiscale characteristics, similarities to clouds, and occlusion by the shadows of mountains and clouds. Deep convolutional neural networks for semantic segmentation are the most popular approach to automatic map generation, but they require huge computing time and resources, as well as a large dataset of pixel-wise annotated HSRRS images, which precludes the application of many superior models. In this study, these limitations are overcome by using a sequence of transfer learning steps. The method starts with a modified aligned ‘Xception’ model pre-trained for object classification on ImageNet. Subsequently, a ‘DeepLab version three plus’ (DeepLabv3+) model is trained using a large dataset of Landsat images and corresponding snow cover products. Finally, a second transfer learning step is employed to fine-tune the model on the small dataset of GaoFen-2, the highest resolution HSRRS satellite in China. Experiments demonstrate the feasibility and effectiveness of this framework for automatic snow cover extraction.

Strategies to Reduce the Expert Supervision Required for Deep Learning-Based Segmentation of Histopathological Images.

The emergence of computational pathology comes with a demand to extract more and more information from each tissue sample. Such information extraction often requires the segmentation of numerous histological objects (e.g., cell nuclei, glands, etc.) in histological slide images, a task for which deep learning algorithms have demonstrated their effectiveness. However, these algorithms require many training examples to be efficient and robust. For this purpose, pathologists must manually segment hundreds or even thousands of objects in histological images, i.e., a long, tedious and potentially biased task. The present paper aims to review strategies that could help provide the very large number of annotated images needed to automate the segmentation of histological images using deep learning. This review identifies and describes four different approaches: the use of immunohistochemical markers as labels, realistic data augmentation, Generative Adversarial Networks (GAN), and transfer learning. In addition, we describe alternative learning strategies that can use imperfect annotations. Adding real data with high-quality annotations to the training set is a safe way to improve the performance of a well configured deep neural network. However, the present review provides new perspectives through the use of artificially generated data and/or imperfect annotations, in addition to transfer learning opportunities.

Synthetic dataset generation for object-to-model deep learning in industrial applications.

The availability of large image data sets has been a crucial factor in the success of deep learning-based classification and detection methods. Yet, while data sets for everyday objects are widely available, data for specific industrial use-cases (e.g., identifying packaged products in a warehouse) remains scarce. In such cases, the data sets have to be created from scratch, placing a crucial bottleneck on the deployment of deep learning techniques in industrial applications. We present work carried out in collaboration with a leading UK online supermarket, with the aim of creating a computer vision system capable of detecting and identifying unique supermarket products in a warehouse setting. To this end, we demonstrate a framework for using data synthesis to create an end-to-end deep learning pipeline, beginning with real-world objects and culminating in a trained model. Our method is based on the generation of a synthetic dataset from 3D models obtained by applying photogrammetry techniques to real-world objects. Using 100K synthetic images for 10 classes, an InceptionV3 convolutional neural network was trained, which achieved accuracy of 96% on a separately acquired test set of real supermarket product images. The image generation process supports automatic pixel annotation. This eliminates the prohibitively expensive manual annotation typically required for detection tasks. Based on this readily available data, a one-stage RetinaNet detector was trained on the synthetic, annotated images to produce a detector that can accurately localize and classify the specimen products in real-time.

Unsupervised natural image patch learning

A metric for natural image patches is an important tool for analyzing images. An efficient means of learning one is to train a deep network to map an image patch to a vector space, in which the Euclidean distance reflects patch similarity. Previous attempts learned such an embedding in a supervised manner, requiring the availability of many annotated images. In this paper, we present an unsupervised embedding of natural image patches, avoiding the need for annotated images. The key idea is that the similarity of two patches can be learned from the prevalence of their spatial proximity in natural images. Clearly, relying on this simple principle, many spatially nearby pairs are outliers. However, as we show, these outliers do not harm the convergence of the metric learning. We show that our unsupervised embedding approach is more effective than a supervised one or one that uses deep patch representations. Moreover, we show that it naturally lends itself to an efficient self-supervised domain adaptation technique onto a target domain that contains a common foreground object.

Construction of Virtual Datasets for Bin Picking

Bin Picking is a typical problem for identifying and sorting complex stacked objects, the difficulty of which lies in the identification of objects. This paper uses deep learning method to identify objects preliminarily. Because the deep learning method requires high-quality datasets, the traditional manual annotation method is too difficult to complete under the industrial production environment. Therefore, this paper proposes a method to build virtual datasets based on virtual simulation technology. It uses OpenGL technology and Bullet physics engine to build virtual scenes. Then through camera loading and scene drawing, the datasets including RGB images, segmentation annotation images and so on are generated. Experimental results show that the model trained on virtual datasets by deep learning can achieve good accuracy, so the feasibility and practical application value of the method are verified.

The role of public challenges and data sets towards algorithm development, trust, and use in clinical practice

In the past decade, machine learning and artificial intelligence have made significant advancements in pattern analysis, including speech and natural language processing, image recognition, object detection, facial recognition, and action categorization. Indeed, in many of these applications, accuracy has reached or exceeded human levels of performance. Subsequently, a multitude of studies have begun to examine the application of these technologies to health care, and in particular, medical image analysis. Perhaps the most difficult subdomain involves skin imaging because of the lack of standards around imaging hardware, technique, color, and lighting conditions. In addition, unlike radiological images, skin image appearance can be significantly affected by skin tone as well as the broad range of diseases. Furthermore, automated algorithm development relies on large high-quality annotated image data sets that incorporate the breadth of this circumstantial and diagnostic variety. These issues, in combination with unique complexities regarding integrating artificial intelligence systems into a clinical workflow, have led to difficulty in using these systems to improve sensitivity and specificity of skin diagnostics in health care networks around the world. In this article, we summarize recent advancements in machine learning, with a focused perspective on the role of public challenges and data sets on the progression of these technologies in skin imaging. In addition, we highlight the remaining hurdles toward effective implementation of technologies to the clinical workflow and discuss how public challenges and data sets can catalyze the development of solutions.

Multi-Pig Part Detection and Association with a Fully-Convolutional Network.

Computer vision systems have the potential to provide automated, non-invasive monitoring of livestock animals, however, the lack of public datasets with well-defined targets and evaluation metrics presents a significant challenge for researchers. Consequently, existing solutions often focus on achieving task-specific objectives using relatively small, private datasets. This work introduces a new dataset and method for instance-level detection of multiple pigs in group-housed environments. The method uses a single fully-convolutional neural network to detect the location and orientation of each animal, where both body part locations and pairwise associations are represented in the image space. Accompanying this method is a new dataset containing 2000 annotated images with 24,842 individually annotated pigs from 17 different locations. The proposed method achieves over 99% precision and over 96% recall when detecting pigs in environments previously seen by the network during training. To evaluate the robustness of the trained network, it is also tested on environments and lighting conditions unseen in the training set, where it achieves 91% precision and 67% recall. The dataset is publicly available for download.

CDBV: A Driving Dataset With Chinese Characteristics From a Bike View

Public datasets are valuable for the development of the learning-based algorithms. In the field of autonomous driving, almost all the existing public datasets are collected by manually driving cars, and most of them are contributed by American and European researchers and represent the traffic scenarios typical for these countries. To diversify the public driving datasets and to provide data support for researches on the self-driving vehicles moving on bike lanes, such as wheelchairs, bikes, and some robots, we present the Chinese Driving from a Bike View (CDBV), which is a driving dataset collected by a bike in Beijing, China. The CDBV dataset contains annotated images, raw videos, and code tools. The annotations are for the object detection tasks and especially focus on the categories with Chinese characteristics, including the express tricycles, the motorbikes specializing in takeouts, the bicycle landmarks, the zebra crossings, and the traffic lights. We totally annotated 40 064 traffic elements in 13 427 images. The CDBV dataset is publicly available at http://www.sujingwang.name/CDBV.html.

Weakly-Supervised Learning of a Deep Convolutional Neural Networks for Semantic Segmentation

Deep convolutional neural networks (DCNNs) trained on the pixel-wise annotated images have dramatically improved the state-of-the-art in semantic segmentation. However, due to the high cost of labeling training data, its application has great limitation. In this paper, we propose a DCNNs model for generating the pixel-level labels using the image-level annotation. The model consists of an encoder-decoder, a feature decomposer, and a multi-label classifier. The encoder extracts the deep convolutional feature maps of the input image. The feature decomposer can decompose the convolutional feature extracted by the encoder into feature components of different semantics. The decomposer is based on the orthogonal non-negative matrix factorization (NMF) technology. The function of the decoder is to map the feature components of different semantics to the input resolution images. The decoder uses the position index of maximum pooling provided by the corresponding encoder to perform non-linear up-sampling, which eliminates the learning requirement of up-sampling. Since the image reconstruction is conducted according to the semantic categories, image regions of different semantics are restored to different images. Then, the regions of different semantics can be segmented through the posted-processing algorithms. The experimental results on open data set show that the proposed model outperforms some recently developed methods.

Understanding Medical Findings in Child Sexual Abuse: An Update For 2018.

When child sexual abuse is suspected, it is important for medical professionals to understand normal as well as abnormal features of the external genital and anal tissues in children. Physicians and other healthcare professionals who rarely perform a detailed examination of these areas may mistakenly interpret normal findings, or findings caused by conditions unrelated to abuse, as signs of injury due to sexual abuse. For this reason, it is important for all medical professionals to have a basic understanding of the appearance of the genital and anal tissues in children. Examples of normal and abnormal genital and anal findings are provided via annotated images. This review presents a research-based summary of how medical findings in children should be interpreted with respect to possible sexual abuse. Resources for obtaining expert review of medical findings are described.

A Noise-Resilient Online Learning Algorithm for Scene Classification

The proliferation of remote sensing imagery motivates a surge of research interest in image processing such as feature extraction and scene recognition, etc. Among them, scene recognition (classification) is a typical learning task that focuses on exploiting annotated images to infer the category of an unlabeled image. Existing scene classification algorithms predominantly focus on static data and are designed to learn discriminant information from clean data. They, however, suffer from two major shortcomings, i.e., the noisy label may negatively affect the learning procedure and learning from scratch may lead to a huge computational burden. Thus, they are not able to handle large-scale remote sensing images, in terms of both recognition accuracy and computational cost. To address this problem, in the paper, we propose a noise-resilient online classification algorithm, which is scalable and robust to noisy labels. Specifically, ramp loss is employed as loss function to alleviate the negative affect of noisy labels, and we iteratively optimize the decision function in Reproducing Kernel Hilbert Space under the framework of Online Gradient Descent (OGD). Experiments on both synthetic and real-world data sets demonstrate that the proposed noise-resilient online classification algorithm is more robust and sparser than state-of-the-art online classification algorithms.

Adversarial learning for viewpoints invariant 3D human pose estimation

2D pose estimation have achieve remarkable performance with deep convolutional neural networks. However 3D pose estimation is current constrained by the limited datasets of 3D annotations. Meanwhile most annotated images are captured using Motion Capture system in lab or certain studio, which has large variations with large-scale monocular 2D pose datasets. We propose an adversarial learning framework, which can learn invariant human pose latent from 3D annotated datasets to optimize the estimation of monocular images with only 2D annotations. However there is large difference in observation coordinates between 2D datasets and 3D datasets, and this viewpoints issue should be separated from invariant pose latent. We add a viewpoints invariant module to automatically regulate observation viewpoints for generated 3D pose, which transforming the generated pose to more suitable observation in the 3D datasets. Our method achieve competitive results on both 2D and 3D benchmarks.

Weakly supervised semantic segmentation using distinct class specific saliency maps

Weakly supervised segmentation has drawn considerable attention, because of the high costs associated with the creation of pixel-wise annotated image datasets that are used for training fully supervised segmentation models. We propose a weakly supervised semantic segmentation method using CNN-based class-specific saliency maps and fully connected CRF. To obtain distinct class-specific saliency maps (DCSM) that can be used as unary potentials of CRF, we propose a novel method of estimating class saliency maps, which significantly improves the method proposed by Simonyan et al. (2014) through the following improvements: (1) using CNN derivatives with respect to feature maps of the intermediate convolutional layers with up-sampling instead of an input image; (2) subtracting the saliency maps of other classes from the saliency maps of the target class to differentiate target objects among other objects; (3) aggregating multiple-scale class saliency maps to compensate for the low resolution in feature maps. In addition, we propose the use of a novel algorithm for estimating segmentation “Easiness” combined with the proposed saliency-based method. Wei et al. (2016) recently demonstrated that a fully supervised segmentation model enhanced the performance of weakly supervised segmentation by training the model using the estimated initial masks in a weakly supervised setting. However, the initial estimated masks tend to include some noise, which sometimes produces erroneous results. Therefore, we focus on improving the quality of the initial estimated masks for training a fully supervised segmentation model. We propose a method for retrieving “good seeds” by predicting the segmentation “Easiness” of images based on the consistency of the outputs under different conditions. We illustrate that our proposed method can retrieve “good seeds”. Despite of the trade-off between training data quality and the number of training images, retrieved images can improve the accuracy of weakly supervised segmentation by combining data augmentation.

Imaging biomarkers and staging in IPF.

The application of computer-based imaging analysis in patients with idiopathic pulmonary fibrosis is a rapidly developing field. The purpose of this review is to provide insights into the problems associated with visual interpretation of HRCT patterns and describe some of the current technologies used to provide objective quantification of disease on HRCT. Future directions are also discussed. Although there is strong evidence that visual quantification of disease on HRCT in idiopathic pulmonary fibrosis provides prognostic information, this approach is hampered by its subjective nature and interobserver variability. In contrast, computer-based quantification of disease on HRCT provides objective and reproducible data, which may help to predict mortality and time to decline in patients with idiopathic pulmonary fibrosis. The use of these technologies may also help to stratify clinical risk in patients enrolled in drug trials. The future of imaging-based biomarker research in idiopathic pulmonary fibrosis is undoubtedly computer-based HRCT evaluation. However, if this field is to continue to innovate, large, well annotated imaging datasets for developing and testing. new computer-based tools are needed as well as prospective trials for biomarker validation.

Iterative multi-path tracking for video and volume segmentation with sparse point supervision.

Recent machine learning strategies for segmentation tasks have shown great ability when trained on large pixel-wise annotated image datasets. It remains a major challenge however to aggregate such datasets, as the time and monetary cost associated with collecting extensive annotations is extremely high. This is particularly the case for generating precise pixel-wise annotations in video and volumetric image data. To this end, this work presents a novel framework to produce pixel-wise segmentations using minimal supervision. Our method relies on 2D point supervision, whereby a single 2D location within an object of interest is provided on each image of the data. Our method then estimates the object appearance in a semi-supervised fashion by learning object-image-specific features and by using these in a semi-supervised learning framework. Our object model is then used in a graph-based optimization problem that takes into account all provided locations and the image data in order to infer the complete pixel-wise segmentation. In practice, we solve this optimally as a tracking problem using a K-shortest path approach. Both the object model and segmentation are then refined iteratively to further improve the final segmentation. We show that by collecting 2D locations using a gaze tracker, our approach can provide state-of-the-art segmentations on a range of objects and image modalities (video and 3D volumes), and that these can then be used to train supervised machine learning classifiers.

Fast and robust ellipse detection algorithm for head-mounted eye tracking systems

In head-mounted eye tracking systems, the correct detection of pupil position is a key factor in estimating gaze direction. However, this is a challenging issue when the videos are recorded in real-world conditions, due to the many sources of noise and artifacts that exist in these scenarios, such as rapid changes in illumination, reflections, occlusions and an elliptical appearance of the pupil. Thus, it is an indispensable prerequisite that a pupil detection algorithm is robust in these challenging conditions. In this work, we present one pupil center detection method based on searching the maximum contribution point to the radial symmetry of the image. Additionally, two different center refinement steps were incorporated with the aim of adapting the algorithm to images with highly elliptical pupil appearances. The performance of the proposed algorithm is evaluated using a dataset consisting of 225,569 head-mounted annotated eye images from publicly available sources. The results are compared with the better algorithm found in the bibliography, with our algorithm being shown as superior.

Understanding subjects contained in Dunhuang mural images for deep semantic annotation

PurposeThe purpose of this paper is to understand the subjects contained in the Dunhuang mural images as well as their relation structures.Design/methodology/approachThis paper performed content analysis based on Panofsky’s theory and 237 research papers related to the Dunhuang mural images. UNICET software was also used to study the correlation structures of subject network.FindingsThe results show that the three levels of subject have all captured the attention of Dunhuang mural researchers, the iconology occupy the critical position in the whole image study, and the correlation between iconography and iconology was strong. Further analysis reveals that cultural development, production, and power and domination have high centralities in the subject network.Research limitations/implicationsThe research samples come from three major Chinese journal databases. However, there are still many authoritative monographs and foreign publications about the Dunhuang murals which are not included in this study.Originality/valueThe results uncover the subject hierarchies and structures contained in the Dunhuang murals from the angle of image scholarship which express scholars’ intention and contribute to the deep semantic annotation on digital Dunhuang mural images.

Multimedia retrieval based on non-linear graph-based fusion and partial least squares regression

Heterogeneous sources of information, such as images, videos, text and metadata are often used to describe different or complementary views of the same multimedia object, especially in the online news domain and in large annotated image collections. The retrieval of multimedia objects, given a multimodal query, requires the combination of several sources of information in an efficient and scalable way. Towards this direction, we provide a novel unsupervised framework for multimodal fusion of visual and textual similarities, which are based on visual features, visual concepts and textual metadata, integrating non-linear graph-based fusion and Partial Least Squares Regression. The fusion strategy is based on the construction of a multimodal contextual similarity matrix and the non-linear combination of relevance scores from query-based similarity vectors. Our framework can employ more than two modalities and high-level information, without increase in memory complexity, when compared to state-of-the-art baseline methods. The experimental comparison is done in three public multimedia collections in the multimedia retrieval task. The results have shown that the proposed method outperforms the baseline methods, in terms of Mean Average Precision and Precision@20.

Multi-Class Latent Concept Pooling for Computer-Aided Endoscopy Diagnosis

Successful computer-aided diagnosis systems typically rely on training datasets containing sufficient and richly annotated images. However, detailed image annotation is often time consuming and subjective, especially for medical images, which becomes the bottleneck for the collection of large datasets and then building computer-aided diagnosis systems. In this article, we design a novel computer-aided endoscopy diagnosis system to deal with the multi-classification problem of electronic endoscopy medical records (EEMRs) containing sets of frames, while labels of EEMRs can be mined from the corresponding text records using an automatic text-matching strategy without human special labeling. With unambiguous EEMR labels and ambiguous frame labels, we propose a simple but effective pooling scheme called Multi-class Latent Concept Pooling, which learns a codebook from EEMRs with different classes step by step and encodes EEMRs based on a soft weighting strategy. In our method, a computer-aided diagnosis system can be extended to new unseen classes with ease and applied to the standard single-instance classification problem even though detailed annotated images are unavailable. In order to validate our system, we collect 1,889 EEMRs with more than 59K frames and successfully mine labels for 348 of them. The experimental results show that our proposed system significantly outperforms the state-of-the-art methods. Moreover, we apply the learned latent concept codebook to detect the abnormalities in endoscopy images and compare it with a supervised learning classifier, and the evaluation shows that our codebook learning method can effectively extract the true prototypes related to different classes from the ambiguous data.