Large Image Collections Research Articles

Fine-Grained Land Use Classification at the City Scale Using Ground-Level Images

Multimedia researchers have exploited large collections of community-contributed geo-referenced images to better understand a particular image, such as its subject matter or where it was taken, as well as to better understand a geographic location, such as the most visited tourist spots in a city or what the local cuisine is like. The goal of this paper is to better understand location. In particular, we use geo-referenced image collections to better understand what occurs in different parts of a city at fine spatial and activity class scales. This problem is known as land use mapping in the geographical sciences. We propose a novel framework to perform fine-grained land use mapping at the city scale using ground-level images. Mapping land use is considerably more difficult than mapping land cover and is generally not possible using overhead imagery as it requires close-up views and seeing inside buildings. We postulate that the growing collections of geo-referenced, ground-level images suggest an alternate approach to this geographic knowledge discovery problem. We develop a general framework that uses Flickr images to map 45 different land-use classes for the city of San Francisco, CA, USA. Individual images are classified using a novel convolutional neural network containing two streams: one for recognizing objects and another for recognizing scenes. This network is trained in an end-to-end manner directly on the labeled training images. We propose several novel strategies to overcome the noisiness of our user-generated data including search-based training set augmentation and online adaptive training. We derive a ground truth map of San Francisco in order to evaluate our method. We demonstrate the effectiveness of our approach through geovisualization and quantitative analysis. Our framework achieves over 29% recall at the individual land parcel level that represents a strong baseline for the challenging 45-way land use classification problem, especially given the noisiness of the image data.

On-Demand Processing of Data Cubes from Satellite Image Collections with the gdalcubes Library

Earth observation data cubes are increasingly used as a data structure to make large collections of satellite images easily accessible to scientists. They hide complexities in the data such that data users can concentrate on the analysis rather than on data management. However, the construction of data cubes is not trivial and involves decisions that must be taken with regard to any particular analyses. This paper proposes on-demand data cubes, which are constructed on the fly when data users process the data. We introduce the open-source C++ library and R package gdalcubes for the construction and processing of on-demand data cubes from satellite image collections, and show how it supports interactive method development workflows where data users can initially try methods on small subsamples before running analyses on high resolution and/or large areas. Two study cases, one on processing Sentinel-2 time series and the other on combining vegetation, land surface temperature, and precipitation data, demonstrate and evaluate this implementation. While results suggest that on-demand data cubes implemented in gdalcubes support interactivity and allow for combining multiple data products, the speed-up effect also strongly depends on how original data products are organized. The potential for cloud deployment is discussed.

Acquiring periodic tilings of regular polygons from images

We describe how we have acquired geometrical models of many periodic tilings of regular polygons from two large collections of images. These models are based on a simplification of the representation recently proposed by us that uses complex numbers. We also describe an algorithm for deciding when two representations give the same tiling, which was used to identify coincidences in these collections.

Benchmarking unsupervised near-duplicate image detection

Unsupervised near-duplicate detection has many practical applications ranging from social media analysis and web-scale retrieval, to digital image forensics. It entails running a threshold-limited query on a set of descriptors extracted from the images, with the goal of identifying all possible near-duplicates, while limiting the false positives due to visually similar images. Since the rate of false alarms grows with the dataset size, a very high specificity is thus required, up to 1–10−9 for realistic use cases; this important requirement, however, is often overlooked in literature. In recent years, descriptors based on deep convolutional neural networks have matched or surpassed traditional feature extraction methods in content-based image retrieval tasks. To the best of our knowledge, ours is the first attempt to establish the performance range of deep learning-based descriptors for unsupervised near-duplicate detection on a range of datasets, encompassing a broad spectrum of near-duplicate definitions. We leverage both established and new benchmarks, such as the Mir-Flick Near-Duplicate (MFND) dataset, in which a known ground truth is provided for all possible pairs over a general, large scale image collection. To compare the specificity of different descriptors, we reduce the problem of unsupervised detection to that of binary classification of near-duplicate vs. not-near-duplicate images. The latter can be conveniently characterized using Receiver Operating Curve (ROC). Our findings in general favor the choice of fine-tuning deep convolutional networks, as opposed to using off-the-shelf features, but differences at high specificity settings depend on the dataset and are often small. The best performance was observed on the MFND benchmark, achieving 96% sensitivity at a false positive rate of 1.43×10−6.

Deep learning enables robust assessment and selection of human blastocysts after in vitro fertilization

Visual morphology assessment is routinely used for evaluating of embryo quality and selecting human blastocysts for transfer after in vitro fertilization (IVF). However, the assessment produces different results between embryologists and as a result, the success rate of IVF remains low. To overcome uncertainties in embryo quality, multiple embryos are often implanted resulting in undesired multiple pregnancies and complications. Unlike in other imaging fields, human embryology and IVF have not yet leveraged artificial intelligence (AI) for unbiased, automated embryo assessment. We postulated that an AI approach trained on thousands of embryos can reliably predict embryo quality without human intervention. We implemented an AI approach based on deep neural networks (DNNs) to select highest quality embryos using a large collection of human embryo time-lapse images (about 50,000 images) from a high-volume fertility center in the United States. We developed a framework (STORK) based on Google’s Inception model. STORK predicts blastocyst quality with an AUC of >0.98 and generalizes well to images from other clinics outside the US and outperforms individual embryologists. Using clinical data for 2182 embryos, we created a decision tree to integrate embryo quality and patient age to identify scenarios associated with pregnancy likelihood. Our analysis shows that the chance of pregnancy based on individual embryos varies from 13.8% (age ≥41 and poor-quality) to 66.3% (age <37 and good-quality) depending on automated blastocyst quality assessment and patient age. In conclusion, our AI-driven approach provides a reproducible way to assess embryo quality and uncovers new, potentially personalized strategies to select embryos.

Animal Scanner: Software for classifying humans, animals, and empty frames in camera trap images.

Camera traps are a popular tool to sample animal populations because they are noninvasive, detect a variety of species, and can record many thousands of animal detections per deployment. Cameras are typically set to take bursts of multiple photographs for each detection and are deployed in arrays of dozens or hundreds of sites, often resulting in millions of photographs per study. The task of converting photographs to animal detection records from such large image collections is daunting, and made worse by situations that generate copious empty pictures from false triggers (e.g., camera malfunction or moving vegetation) or pictures of humans. We developed computer vision algorithms to detect and classify moving objects to aid the first step of camera trap image filtering—separating the animal detections from the empty frames and pictures of humans. Our new work couples foreground object segmentation through background subtraction with deep learning classification to provide a fast and accurate scheme for human–animal detection. We provide these programs as both Matlab GUI and command prompt developed with C++. The software reads folders of camera trap images and outputs images annotated with bounding boxes around moving objects and a text file summary of results. This software maintains high accuracy while reducing the execution time by 14 times. It takes about 6 seconds to process a sequence of ten frames (on a 2.6 GHZ CPU computer). For those cameras with excessive empty frames due to camera malfunction or blowing vegetation automatically removes 54% of the false‐triggers sequences without influencing the human/animal sequences. We achieve 99.58% on image‐level empty versus object classification of Serengeti dataset. We offer the first computer vision tool for processing camera trap images providing substantial time savings for processing large image datasets, thus improving our ability to monitor wildlife across large scales with camera traps.

Tattoo Image Search at Scale: Joint Detection and Compact Representation Learning.

The explosive growth of digital images in video surveillance and social media has led to the significant need for efficient search of persons of interest in law enforcement and forensic applications. Despite tremendous progress in primary biometric traits (e.g., face and fingerprint) based person identification, a single biometric trait alone can not meet the desired recognition accuracy in forensic scenarios. Tattoos, as one of the important soft biometric traits, have been found to be valuable for assisting in person identification. However, tattoo search in a large collection of unconstrained images remains a difficult problem, and existing tattoo search methods mainly focus on matching cropped tattoos, which is different from real application scenarios. To close the gap, we propose an efficient tattoo search approach that is able to learn tattoo detection and compact representation jointly in a single convolutional neural network (CNN) via multi-task learning. While the features in the backbone network are shared by both tattoo detection and compact representation learning, individual latent layers of each sub-network optimize the shared features toward the detection and feature learning tasks, respectively. We resolve the small batch size issue inside the joint tattoo detection and compact representation learning network via random image stitch and preceding feature buffering. We evaluate the proposed tattoo search system using multiple public-domain tattoo benchmarks, and a gallery set with about 300K distracter tattoo images compiled from these datasets and images from the Internet. In addition, we also introduce a tattoo sketch dataset containing 300 tattoos for sketch-based tattoo search. Experimental results show that the proposed approach has superior performance in tattoo detection and tattoo search at scale compared to several state-of-the-art tattoo retrieval algorithms.

Annotations, Ontologies, and Whole Slide Images – Development of an Annotated Ontology-Driven Whole Slide Image Library of Normal and Abnormal Human Tissue

Objective: Digital pathology is today a widely used technology, and the digitalization of microscopic slides into whole slide images (WSIs) allows the use of machine learning algorithms as a tool in the diagnostic process. In recent years, “deep learning” algorithms for image analysis have been applied to digital pathology with great success. The training of these algorithms requires a large volume of high-quality images and image annotations. These large image collections are a potent source of information, and to use and share the information, standardization of the content through a consistent terminology is essential. The aim of this project was to develop a pilot dataset of exhaustive annotated WSI of normal and abnormal human tissue and link the annotations to appropriate ontological information. Materials and Methods: Several biomedical ontologies and controlled vocabularies were investigated with the aim of selecting the most suitable ontology for this project. The selection criteria required an ontology that covered anatomical locations, histological subcompartments, histopathologic diagnoses, histopathologic terms, and generic terms such as normal, abnormal, and artifact. WSIs of normal and abnormal tissue from 50 colon resections and 69 skin excisions, diagnosed 2015-2016 at the Department of Clinical Pathology in Linköping, were randomly collected. These images were manually and exhaustively annotated at the level of major subcompartments, including normal or abnormal findings and artifacts. Results: Systemized nomenclature of medicine clinical terms (SNOMED CT) was chosen, and the annotations were linked to its codes and terms. Two hundred WSI were collected and annotated, resulting in 17,497 annotations, covering a total area of 302.19 cm2, equivalent to 107,7 gigapixels. Ninety-five unique SNOMED CT codes were used. The time taken to annotate a WSI varied from 45 s to over 360 min, a total time of approximately 360 h. Conclusion: This work resulted in a dataset of 200 exhaustive annotated WSIs of normal and abnormal tissue from the colon and skin, and it has informed plans to build a comprehensive library of annotated WSIs. SNOMED CT was found to be the best ontology for annotation labeling. This project also demonstrates the need for future development of annotation tools in order to make the annotation process more efficient.

Integration of SAR Data Into Monitoring of the 2014–2015 Holuhraun Eruption, Iceland: Contribution of the Icelandic Volcanoes Supersite and the FutureVolc Projects

We report how data from satellite and aerial synthetic aperture radar (SAR) observations were integrated into monitoring of the 2014-2015 Holuhraun eruption in the Barðarbunga volcanic system, the largest effusive eruption in Iceland since the 1783-84 Laki eruption. A lava field formed in one of the most remote areas in Iceland, after the propagation of a ~50 km-long dyke beneath the Vatnajokull ice cap, where the Barðarbunga caldera is located. Due to the 6 month duration of the eruption, mainly in wintertime, daily monitoring was particularly challenging. During the eruption, the European volcanological project FutureVolc was ongoing, allowing collaboration of many European experts on volcano monitoring activities. Icelandic volcanoes are also a permanent Supersite within the Geohazard Supersites and Natural Laboratories (GSNL) initiative, with support from the Committee on Earth Observation Satellite (CEOS) in the form of a large collection of SAR images. SAR data were acquired by Cosmo-SkyMed (CSK) and TerraSAR-X (TSX) satellites and complemented by aerial SAR images. The large set of SAR satellite data significantly contributed to the daily monitoring during the unrest at Barðarbunga caldera, the Holuhraun eruption and the year following the eruption. Detection of surface changes using both SAR amplitude and phase information was conducted throughout the whole duration of the volcano-tectonic event, and in the following months, to quantify and track the evolution of volcanic processes at Holuhraun and geothermal activity at Barðarbunga volcano. Combination of SAR data with other data sets, e.g. satellite optical images and geodetic Global Positioning System (GPS) measurements, was essential for the evaluation of the volcanic hazard in the whole area. International collaboration within the Futurevolc project formed the basis for successful analyses and interpretation of the large SAR data set. Information was provided at Scientific Advisory Board meetings of the Icelandic Civil Protection and used in decision-making, as well as for supporting field-deployment and air-based surveys.

Automatic interpretation of otoliths using deep learning.

The age structure of a fish population has important implications for recruitment processes and population fluctuations, and is a key input to fisheries-assessment models. The current method of determining age structure relies on manually reading age from otoliths, and the process is labor intensive and dependent on specialist expertise. Recent advances in machine learning have provided methods that have been remarkably successful in a variety of settings, with potential to automate analysis that previously required manual curation. Machine learning models have previously been successfully applied to object recognition and similar image analysis tasks. Here we investigate whether deep learning models can also be used for estimating the age of otoliths from images. We adapt a pre-trained convolutional neural network designed for object recognition, to estimate the age of fish from otolith images. The model is trained and validated on a large collection of images of Greenland halibut otoliths. We show that the model works well, and that its precision is comparable to documented precision obtained by human experts. Automating this analysis may help to improve consistency, lower cost, and increase the extent of age estimation. Given that adequate data are available, this method could also be used to estimate age of other species using images of otoliths or fish scales.

Tool for automatic tuning of binarisation techniques

Most of the proposed binarisation methods include parameters that must be set correctly before use. The determination of the values of these parameters is made most of the time manually after several tests. However, the optimum parameter values differ from an image to another and therefore the parameterisation shall be carried out for each image separately. In fact, as this task is very difficult, even impossible for large collections of images, the tuning is usually done once for the entire image collection. In this study, the authors propose a tool for automatic and adaptive parameterisation of binarisation techniques for each image separately. The adopted methodology is based on the use of an artificial neural network (ANN) to learn the optimal parameter values of a binarisation method for a set of images (training set), based on their features, and to use the trained ANN to determine the optimal parameter values for other images not learned. Several experiments have been conducted on images of degraded documents and the obtained results are encouraging.

MAIA-A machine learning assisted image annotation method for environmental monitoring and exploration.

Digital imaging has become one of the most important techniques in environmental monitoring and exploration. In the case of the marine environment, mobile platforms such as autonomous underwater vehicles (AUVs) are now equipped with high-resolution cameras to capture huge collections of images from the seabed. However, the timely evaluation of all these images presents a bottleneck problem as tens of thousands or more images can be collected during a single dive. This makes computational support for marine image analysis essential. Computer-aided analysis of environmental images (and marine images in particular) with machine learning algorithms is promising, but challenging and different to other imaging domains because training data and class labels cannot be collected as efficiently and comprehensively as in other areas. In this paper, we present Machine learning Assisted Image Annotation (MAIA), a new image annotation method for environmental monitoring and exploration that overcomes the obstacle of missing training data. The method uses a combination of autoencoder networks and Mask Region-based Convolutional Neural Network (Mask R-CNN), which allows human observers to annotate large image collections much faster than before. We evaluated the method with three marine image datasets featuring different types of background, imaging equipment and object classes. Using MAIA, we were able to annotate objects of interest with an average recall of 84.1% more than twice as fast as compared to “traditional” annotation methods, which are purely based on software-supported direct visual inspection and manual annotation. The speed gain increases proportionally with the size of a dataset. The MAIA approach represents a substantial improvement on the path to greater efficiency in the annotation of large benthic image collections.

Integrating Wikipedia Articles and Images into an Information Resource for Radiology Patients.

Wikipedia-an open-access online encyclopedia-contains a large number of medically relevant articles and images that may help supplement glossaries of radiology terms. We sought to determine the extent to which concepts from a large online radiology glossary developed as part of the Patient-Oriented Radiology Reporter (PORTER) initiative could be mapped to relevant Wikipedia web pages and images using automated or semi-automated approaches. The glossary included 4090 concepts with their definitions; the concept's preferred name and lexical variants, such as plurals, adjectival forms, synonyms, and abbreviations, yielded a total of 13,030 terms. Of the 4090 concepts, 3063 (74.9%) had a corresponding English-language Wikipedia page identified by automated search with subsequent manual review. We applied the MediaWiki application programming interface (API) to generate web-service calls to identify the images from each concept's corresponding Wikipedia page; three reviewers selected relevant images to associate with the glossary's concepts. Licensing terms for the images were reviewed. For 800 randomly sampled concepts that had associated Wikipedia pages, 362 distinct images were identified from the MediaWiki library and matched to 404 concepts (51%). Three images (1%) had unspecified licensing terms; the rest were in the public domain or available via a Creative Commons license. Wikipedia and the MediaWiki library offer a large collection of medical articles and images that can be incorporated into an online lay-language glossary of radiology terms though a semi-automated approach.

Deep feature extraction and its application for hailstorm detection in a large collection of radar images

With the improvement of sensing and storing technologies, a large amount of weather data become available, and the data size will continue growing as radar imaging instruments continuously acquire data. In this work, we develop a deep convolutional neural network with a large collection of radar images as input to train and validate a classification model, and then we use the model to detect hailstorm events. This is interdisciplinary work between the disciplines of computer science and meteorology. We are primarily interested in what hailstorm features the network learns and how it learns as convolving into deeper iterations. The evaluation results show a high classification accuracy in comparison with existing hailstorm detection approaches. The proposed approach can also be used to detect other types of severe weather events with minimal efforts on variable or parameter changes.

Dataset and Metrics for Predicting Local Visible Differences

A large number of imaging and computer graphics applications require localized information on the visibility of image distortions. Existing image quality metrics are not suitable for this task as they provide a single quality value per image. Existing visibility metrics produce visual difference maps, and are specifically designed for detecting just noticeable distortions but their predictions are often inaccurate. In this work, we argue that the key reason for this problem is the lack of large image collections with a good coverage of possible distortions that occur in different applications. To address the problem, we collect an extensive dataset of reference and distorted image pairs together with user markings indicating whether distortions are visible or not. We propose a statistical model that is designed for the meaningful interpretation of such data, which is affected by visual search and imprecision of manual marking. We use our dataset for training existing metrics and we demonstrate that their performance significantly improves. We show that our dataset with the proposed statistical model can be used to train a new CNN-based metric, which outperforms the existing solutions. We demonstrate the utility of such a metric in visually lossless JPEG compression, super-resolution and watermarking.

Content-aware image resizing: An improved and shadow-preserving seam carving method

Abstract The performance of seam carving-based image resizing algorithms is strongly dependent on the quality of the importance map extracted from the image. To date, various approaches have been proposed to extract the importance map, however, none has considered to take the shadows within the image into account. In most cases, existing shadows in the images would imply important information and help in better and quick understanding of the content of the images. This fact motivates us to keep them during the image resizing as much as we can. Therefore, in this paper, a seam carving-based algorithm is presented where we introduce an efficient shadow extraction algorithm in the direction of our motivation. We also propose a saliency map for highlighting the salient objects in the images. In addition to these two maps, a gradient map is also extracted to portray the details of the background. These maps are then combined in order to produce the final importance map. Extensive experiments conducted on a large collection of images indicate that the proposed method, in terms of preserving the main content of the input image, the shadows within it, and the important structures of the image, is superior to state-of-the-art algorithms.

Learning a Convolutional Neural Network for Image Compact-Resolution.

We study the dual problem of image super-resolution (SR), which we term image compact-resolution (CR). Opposite to image SR that hallucinates a visually plausible high-resolution image given a low-resolution input, image CR provides a low-resolution version of a high-resolution image, such that the low-resolution version is both visually pleasing and as informative as possible compared to the high-resolution image. We propose a convolutional neural network (CNN) for image CR, namely, CNN-CR, inspired by the great success of CNN for image SR. Specifically, we translate the requirements of image CR into operable optimization targets for training CNN-CR: the visual quality of the compact resolved image is ensured by constraining its difference from a naively downsampled version and the information loss of image CR is measured by upsampling/super-resolving the compact-resolved image and comparing that to the original image. Accordingly, CNN-CR can be trained either separately or jointly with a CNN for image SR. We explore different training strategies as well as different network structures for CNN-CR. Our experimental results show that the proposed CNN-CR clearly outperforms simple bicubic downsampling and achieves on average 2.25 dB improvement in terms of the reconstruction quality on a large collection of natural images. We further investigate two applications of image CR, i.e., low-bit-rate image compression and image retargeting. Experimental results show that the proposed CNN-CR helps achieve significant bits saving than High Efficiency Video Coding when applied to image compression and produce visually pleasing results when applied to image retargeting.

A Jungle Computing approach to common image source identification in large collections of images

Analyzing digital images is an important investigation in forensics with the ever increasing number of images from computers and smartphones. In this article we aim to advance the state-of-the-art in common image source identification (which images originate from the same source camera). To this end, we present two types of applications for different goals that make use of a) a modern Desktop computer with a GPU and b) highly heterogeneous cluster computers with many different kinds of GPUs, something we call computing jungles. The first application targets medium-scale investigations, for example within a crime laboratory, the second application is targeted at large-scale investigations, for example within institutions. We advance the state-of-the-art by 1) explaining in detail how we obtain the performance to 2) support large databases of images in reasonable time while 3) not giving up accuracy. Moreover, we do not apply filtering ensuring that 4) our results are highly reproducible.

Imaging the unimaginable: Medical imaging in the realm of photography

There is an almost innate urge in human beings to represent reality in a visual form. From rock art in the Paleolithic to images of galaxies, the quotidian and the extraordinary have been visually represented through the ages. Medical and scientific disciplines are no exception. Accurate representation of the human body structures and anatomy based on cadaver dissections was almost not possible up to the Renaissance due to ethical, social, and religious beliefs and objections. The works of Leonardo da Vinci (1452-1519) and others and, later, Andreas Vesalius (1514-1564), who produced De Humanis Corporis Fabrica, are considered landmarks in the history of medicine. During the following centuries medical and scientific illustration relied upon the expertise of physician-artists and scientist-artists until a new paradigm appeared in the realm of scientific (medical) illustration: the invention of photography in the 19th century. Two of the medical disciplines most rapidly influenced by photography were dermatology and pathology, both macro- and microscopic. Physicians rapidly started to use photographs as a tool for consultation, documentation, and education, and large collections of images were amassed by individuals and institutions for these purposes. Photographic images are produced by visible light impressing a light-sensitive material such as a silver halide plate, and nowadays a silicon chip. But photons are reflected by nontransparent objects, including the human skin. Developments in science and technology allowed the use of other types of radiation to reveal internal structures in the human body and, most interestingly, noninvasively. Thus today much of the medical diagnosis and treatment is guided by the so-called medical imaging with the use of these techniques, that is, medical photography, endoscopy, x-ray radiography, computer-aided tomography, magnetic resonance imaging, ultrasonography, thermography, and nuclear medicine functional imaging techniques as positron emission tomography (PET) and single-photon emission computed tomography (SPECT). Some of these techniques are being applied at the microscopic level to study cell structure and even functional changes in real time. All these advancements in science and technology applied to medicine and other disciplines pose the question as to what extent physicians are trading their capabilities as clinicians. Ethics issues add to the complexity of this new era governed by constant changes in scientific paradigms.

The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions

Training of neural networks for automated diagnosis of pigmented skin lesions is hampered by the small size and lack of diversity of available datasets of dermatoscopic images. We tackle this problem by releasing the HAM10000 (“Human Against Machine with 10000 training images”) dataset. We collected dermatoscopic images from different populations acquired and stored by different modalities. Given this diversity we had to apply different acquisition and cleaning methods and developed semi-automatic workflows utilizing specifically trained neural networks. The final dataset consists of 10015 dermatoscopic images which are released as a training set for academic machine learning purposes and are publicly available through the ISIC archive. This benchmark dataset can be used for machine learning and for comparisons with human experts. Cases include a representative collection of all important diagnostic categories in the realm of pigmented lesions. More than 50% of lesions have been confirmed by pathology, while the ground truth for the rest of the cases was either follow-up, expert consensus, or confirmation by in-vivo confocal microscopy.