Large Image Collections Research Articles

Qatris iManager: a general purpose CBIR system

Content-based image retrieval (CBIR) has drawn much interest from the research community over the past decade, as a good number of CBIR techniques, methods and systems have emerged, contributing new solutions to the issue of storing, managing and retrieving images, as database management systems do with structured data. There is undoubtedly a crucial need to characterize image content as well as subjectivity in the interpretation of this content (for which the community has coined the term "semantic gap"). In this paper the CBIR system developed by our research group, Qatris iManager, is described as a positive proposal to cope with chief issues in the field, especially the semantic gap, from a novel and original perspective. Based on color, texture and shape features, our system provides a broad range of useful operations to facilitate the storage, management, retrieval and browsing of large image collections. Local and remote image loading processes enable the population of image collections. Classification methods allow users to organize the collections according to their own interests. A multidimensional access method contributes to the efficiency in similarity searches. Parameterized similarity functions give flexibility to the search by content processes. Finally, the integrated automatic learning methods for classification and search processes teach the system about the user's information needs. The proposed system is the result of a joint effort with different research tasks. This paper extensively describes all the system functionalities, techniques, processes and algorithms implemented.

Multi-modal Similarity Retrieval with Distributed Key-value Store

We propose a system architecture for large-scale similarity search in various types of digital data. The architecture combines contemporary highly-scalable distributed data stores with recent efficient similarity indexes and also with other types of search indexes. The system enables various types of data access by distance-based similarity queries, standard term and attribute queries, and advanced queries combining several search aspects (modalities). The first part of this work describes the generic architecture and similarity index PPP-Codes, which is suitable for our system. In the second part, we describe two specific instances of this architecture that manage two large collections of digital images and provide content-based visual search, keyword search, attribute-based access, and their combinations. The first collection is the CoPhIR benchmark with 106 million images accessed by MPEG7 visual descriptors and the second collection contains 20 million images with complex features obtained from deep convolutional neural network.

Automated content based image retrieval technology (CBIR) and ITS application on dermoscopic images of pigmented skin lesions

Dermoscopy (dermatoscopy, epiluminescence microscopy) is a non-invasive diagnostic technique for the in vivo observation of pigmented skin lesions (PSLs). This diagnostic tool permits the recognition of morphologic structures not visible by the naked eye, thus opening a new dimension in the analysis of the clinical morphologic features of PSLs. Due to the difficulty and variability of human interpretation, computerized analysis of dermoscopy images has become an important research area in the last years. Recently, numerous systems designed to provide computer-aided analysis of digital images have been reported in literature. Content-based image retrieval systems (CBIR) technology exploits the visual content in image data. It has been proposed to benefit the management of increasingly large biomedical image collections as well as to aid clinical medicine, research and education. Goal of this article is to describe the application of CBIR systems for dermoscopic images, dealing in detail with the model created in our laboratory, named FIDE. Function of FIDE is to documenting the image analysis side of the diagnostic process, focusing on accompanying and aiding it and on providing efficient digital atlas navigation aiming both at providing precision (cases most similar to the one under analysis) and clarifying context (similar cases in different categories). The FIDE system is effective in retrieving PSL images with known pathology visually similar to the image under evaluation giving a valuable and intuitive aid to the clinician in the decision making process. FIDE represents, indeed, the first CBIR system successfully applied to dermoscopic images. The real impact in clinical practice of this system will be revealed only by its application in a wide clinical setting.

Generating Representative Sets and Summaries for Large Collection of Images Using Image Cropping Techniques and Result Comparison

The collection of photos hosted on photo archives and social networking sites has been increasing exponentially. It is really hard to get the summary of a large image set without browsing through the entire collection. In this paper two different techniques of image cropping (random windows technique and sequential windows technique) have been proposed to generate effective representative sets. A ranking mechanism has been also proposed for finding the best representative set.

Effects of Presentation Mode and Pace Control on Performance in Image Classification

A common task in visualization is to quickly find interesting items in large sets. When appropriate metadata is missing, automatic queries are impossible and users have to inspect all elements visually. We compared two fundamentally different, but obvious display modes for this task and investigated the difference with respect to effectiveness, efficiency, and satisfaction. The static mode is based on the page metaphor and presents successive pages with a static grid of items. The moving mode is based on the conveyor belt metaphor and lets a grid of items slide though the screen in a continuous flow. In our evaluation, we applied both modes to the common task of browsing images. We performed two experiments where 18 participants had to search for certain target images in a large image collection. The number of shown images per second (pace) was predefined in the first experiment, and under user control in the second one. We conclude that at a fixed pace, the mode has no significant impact on the recall. The perceived pace is generally slower for moving mode, which causes users to systematically choose for a faster real pace than in static mode at the cost of recall, keeping the average number of target images found per second equal for both modes.

Image indexing using regularized online non-negative semantic embedding

This paper proposes a novel method for image indexing based on an online learning approach which can deal with large repositories of images. The proposed method is based on a semantic embedding strategy which models a mapping between visual and text representations. This method enhances the image representation by taking advantage of the text annotations associated to the images, which have a rich and clean semantic interpretation. Once the mapping is learned, a new (annotated or unannotated) image can be projected to the space defined by semantic annotations. In this manner, this method can be used to search into the collection using an image as query (query-by-example strategy) and to annotate new unannotated images. The main drawback of semantic embedding strategies is that the associated algorithms are computationally expensive, making them infeasible for large data collections. In order to address this issue, the proposed method is formulated as an online learning algorithm using the stochastic gradient descent approach, which can scale to deal with large image collections. According with the experimental evaluation, the proposed method, in comparison with several baseline methods, is faster and consumes less memory, without degradation in the performance in content-based image search.

Color Feature Extraction in Content based Image Retrieval based on Quaternion Space

Interest in the digital images has increased a lot over the last few years, but the process of locating a desired image in such a large and diverse image collection becomes very difficult. Traditionally text in different languages is used for efficient retrieval of images; it has several drawbacks such as language constraint and subjectivity of human perception. Content-based image retrieval is a technique which uses visual contents such as color texture and shape to search images from large image databases according to user’s desire. Color is the most commonly used feature for content based image retrieval. In many applications color histogram is used to represent extracted color features. The important drawback of usual color histogram based method is that, it does not take image color distribution into consideration and inflexibly partition the color spaces into a fixed number of bins. In this paper we propose a moment-preserving technique based on binary quaternion space for feature extraction. It aims to extract color features according to the image color distribution that effectively reduces the distortion incurred in the feature extraction process. We also propose an efficient clustering based algorithm to compare similarity between two histograms. It is observed that minimizing the distortion incurred in the extraction process can improve the accuracy of retrieval. Our experimental results show that the proposed extraction methods can improve the average retrieval precision rate by a factor of 25% over that of a color histogram based feature extraction method (binning method). It is also observed that, this technique effectively reduces the average retrieval time.

3-D Interfaces to Improve the Performance of Visual Known-Item Search

Most interfaces in the field of image and video search use a two-dimensional grid interface, which presents image thumbnails in a left-to-right arrangement that can be browsed from top to bottom. This grid interface, however, has several drawbacks that become particularly apparent when performing interactive search tasks for target items in large collections of images or videos. Therefore, we propose to use 3-D interfaces as an alternative to the grid interface for interactive known-item search in visual data as they can partially overcome these drawbacks. In this paper, we first summarize our ideas and discuss design aspects of a 3-D ring and a 3-D globe interface. Next, we present results from four different user studies, where we evaluated the performance of these interfaces for known-item search tasks in image collections. Our results from these studies show that the proposed 3-D interfaces allow for significantly faster visual target search on desktop computers with mouse interaction as well as on tablet devices. The interfaces also achieve better subjective ratings. However, our evaluation also shows that on smartphones with 3.5-in screens an improvement over the grid interface in terms of visual search time is only possible in collections with more than 200 images.

Fast Feature‐Oriented Visual Connection for Large Image Collections

AbstractDeriving the visual connectivity across large image collections is a computationally expensive task. Different from current image‐oriented match graph construction methods which build on pairwise image matching, we present a novel and scalable feature‐oriented image matching algorithm for large collections. Our method improves the match graph construction procedure in three ways. First, instead of building trees repeatedly, we put the feature points of the input image collection into a single kd‐tree and select the leaves as our anchor points. Then we construct an anchor graph from which each feature can intelligently find a small portion of related candidates to match. Finally, we design a new form of adjacency matrix for fast feature similarity measuring, and return all the matches in different photos across the whole dataset directly. Experiments show that our feature‐oriented correspondence algorithm can explore visual connectivity between images with significant improvement in speed.

Citizen Science in Archaeology

Copyright © 2014 by the Society for American Archaeology. Citizen science, as a process of volunteer participation through crowdsourcing, facilitates the creation of mass data sets needed to address subtle and large-scale patterns in complex phenomena. Citizen science efforts in other field disciplines such as biology, geography, and astronomy indicate how new web-based interfaces can enhance and expand upon archaeologists' existing platforms of volunteer engagement such as field schools, community archaeology, site stewardship, and professional-avocational partnerships. Archaeological research can benefit from the citizen science paradigm in four ways: fieldwork that makes use of widely available technologies such as mobile applications for photography and data upload; searches of large satellite image collections for site identification and monitoring; crowdfunding; and crowdsourced computer entry of heritage data.

Citizen Science in Archaeology

Abstract Citizen science, as a process of volunteer participation through crowdsourcing, facilitates the creation of mass data sets needed to address subtle and large-scale patterns in complex phenomena. Citizen science efforts in other field disciplines such as biology, geography, and astronomy indicate how new web-based interfaces can enhance and expand upon archaeologists’ existing platforms of volunteer engagement such as field schools, community archaeology, site stewardship, and professional–avocational partnerships. Archaeological research can benefit from the citizen science paradigm in four ways: fieldwork that makes use of widely available technologies such as mobile applications for photography and data upload; searches of large satellite image collections for site identification and monitoring; crowdfunding; and crowdsourced computer entry of heritage data.

Image re-ranking system based on closed frequent patterns

Text-based image retrieval is a popular and simple framework, which consists in using text annotations (e.g., image names, tags) to efficiently collect images relevant to a query word, from very large image collections. Even if the set of images retrieved using text annotations is noisy, it constitutes a reasonable initial set of images that can be considered as a bootstrap and improved further by analyzing image content. In this context, this paper introduces an approach for improving this initial set by re-ranking the so-obtained images, assuming that non-relevant images are scattered (i.e., they do not form clusters), unlike the relevant ones. More specifically, the approach consists in computing efficiently and on-the-fly closed frequent patterns, and in re-ranking images based on the number of patterns they contain. To do this, the paper introduces a simple but powerful new scoring function. Moreover, after the re-ranking process, we show how pattern mining techniques can also be applied for promoting diversity in the top-ranked images. The approach is validated on three different datasets for which state-of-the-art results are obtained.

Moving portraits

We present an approach for generating face animations from large image collections of the same person. Such collections, which we call photobios, are remarkable in that they summarize a person's life in photos; the photos sample the appearance of a person over changes in age, pose, facial expression, hairstyle, and other variations. Yet, browsing and exploring photobios is infeasible due to their large volume. By optimizing the quantity and order in which photos are displayed and cross dissolving between them, we can render smooth transitions between face pose (e.g., from frowning to smiling), and create moving portraits from collections of still photos. Used in this context, the cross dissolve produces a very strong motion effect; a key contribution of the paper is to explain this effect and analyze its operating range. We demonstrate results on a variety of datasets including time-lapse photography, personal photo collections, and images of celebrities downloaded from the Internet. Our approach is completely automatic and has been widely deployed as the "Face Movies" feature in Google's Picasa.

Roughness measurements over an agricultural soil surface with Structure from Motion

This paper presents an accessible and reliable method to measure surface roughness of agricultural soils with a setup designed to tackle some of the challenges posed by roughness to SAR remote sensing. The method relies on Structure from Motion (SfM). From a large collection of unconstrained images (∼700 images) acquired with a commercial-grade camera, digital elevation models (DEMs) are generated for a SAR-pixel-size plot (2×11m), with horizontal and vertical RMS errors of respectively 1.5mm and 3.1mm.Example results highlight the need for individually detrending all sampled sub-DEMs when studying the convergence of the roughness parameters for increasing DEM length. This point appears to be missing in previous publications. The efficiency of the Fourier-based method used to compute the roughness parameters allows investigating anisotropy at a 1° angular resolution. This could benefit investigations on the flashing fields phenomenon observed within narrow direction bands over tilled fields.The inclusion of permanent reference targets into the soil makes multitemporal measurements over the same plot straightforward. Ten acquisitions from April to July 2013 show noticeable natural changes in roughness with cracking during dry periods and smoothing during rainfalls. As expected, changes in RMS height and correlation length appear inversely correlated and can be related to in situ measurements of soil moisture, soil temperature, and rainfall. Analysis of changes in power spectral density indicates that the observed roughness changes only affect scales below 50cm, i.e. scales relevant for microwave scattering. Even though it seems that millimetric changes for horizontal scales below 1cm are not observable, measurement performance could be improved by adding more detailed pictures to the image set.This SfM-based method appears to be well-suited to study the dynamics and characterization of roughness for SAR and more generally for geosciences.

In the wild image retrieval and clustering for 3D cultural heritage landmarks reconstruction

One of the main characteristics of Internet era is the free and online availability of extremely large collections of images located on distributed and heterogeneous platforms over the web. The pro...

Shape annotation by semi-supervised fuzzy clustering

Image annotation is an important and challenging task when managing large image collections. In this paper, a fuzzy shape annotation approach for semi-automatic image annotation is presented. A fuzzy clustering process guided by partial supervision is applied to shapes represented by Fourier descriptors in order to derive a set of shape prototypes representative of a number of semantic categories. Next, prototypes are manually annotated by attaching textual labels related to semantic categories. Based on the labeled proto-types, a new shape is automatically labeled by associating a fuzzy set that provides membership degrees of the shape to all semantic categories. The proposed annotation approach provides an innovative indexing method for shape-based image retrieval. Indeed, shape prototypes represent an inter-mediate indexing level that allows a faster retrieval process since a query is matched against prototypes, instead of the whole shape database, resulting in a speed up of the retrieval. The proposed approach is tested on synthetic and real-word images in order to show its suitability.

AverageExplorer

This paper proposes an interactive framework that allows a user to rapidly explore and visualize a large image collection using the medium of average images . Average images have been gaining popularity as means of artistic expression and data visualization, but the creation of compelling examples is a surprisingly laborious and manual process. Our interactive, real-time system provides a way to summarize large amounts of visual data by weighted average(s) of an image collection, with the weights reflecting user-indicated importance. The aim is to capture not just the mean of the distribution, but a set of modes discovered via interactive exploration. We pose this exploration in terms of a user interactively "editing" the average image using various types of strokes, brushes and warps, similar to a normal image editor, with each user interaction providing a new constraint to update the average. New weighted averages can be spawned and edited either individually or jointly. Together, these tools allow the user to simultaneously perform two fundamental operations on visual data: user-guided clustering and user-guided alignment, within the same framework. We show that our system is useful for various computer vision and graphics applications.

Probabilistic hypergraph based hash codes for social image search

With the rapid development of the Internet, recent years have seen the explosive growth of social media. This brings great challenges in performing efficient and accurate image retrieval on a large scale. Recent work shows that using hashing methods to embed high-dimensional image features and tag information into Hamming space provides a powerful way to index large collections of social images. By learning hash codes through a spectral graph partitioning algorithm, spectral hashing (SH) has shown promising performance among various hashing approaches. However, it is incomplete to model the relations among images only by pairwise simple graphs which ignore the relationship in a higher order. In this paper, we utilize a probabilistic hypergraph model to learn hash codes for social image retrieval. A probabilistic hypergraph model offers a higher order representation among social images by connecting more than two images in one hyperedge. Unlike a normal hypergraph model, a probabilistic hypergraph model considers not only the grouping information, but also the similarities between vertices in hyperedges. Experiments on Flickr image datasets verify the performance of our proposed approach.

Assessing age-related gray matter decline with voxel-based morphometry depends significantly on segmentation and normalization procedures.

Healthy ageing coincides with a progressive decline of brain gray matter (GM) ultimately affecting the entire brain. For a long time, manual delineation-based volumetry within predefined regions of interest (ROI) has been the gold standard for assessing such degeneration. Voxel-Based Morphometry (VBM) offers an automated alternative approach that, however, relies critically on the segmentation and spatial normalization of a large collection of images from different subjects. This can be achieved via different algorithms, with SPM5/SPM8, DARTEL of SPM8 and FSL tools (FAST, FNIRT) being three of the most frequently used. We complemented these voxel based measurements with a ROI based approach, whereby the ROIs are defined by transforms of an atlas (containing different tissue probability maps as well as predefined anatomic labels) to the individual subject images in order to obtain volumetric information at the level of the whole brain or within separate ROIs. Comparing GM decline between 21 young subjects (mean age 23) and 18 elderly (mean age 66) revealed that volumetric measurements differed significantly between methods. The unified segmentation/normalization of SPM5/SPM8 revealed the largest age-related differences and DARTEL the smallest, with FSL being more similar to the DARTEL approach. Method specific differences were substantial after segmentation and most pronounced for the cortical structures in close vicinity to major sulci and fissures. Our findings suggest that algorithms that provide only limited degrees of freedom for local deformations (such as the unified segmentation and normalization of SPM5/SPM8) tend to overestimate between-group differences in VBM results when compared to methods providing more flexible warping. This difference seems to be most pronounced if the anatomy of one of the groups deviates from custom templates, a finding that is of particular importance when results are compared across studies using different VBM methods.

Modeling Image Patches with a Generic Dictionary of Mini-Epitomes.

The goal of this paper is to question the necessity of features like SIFT in categorical visual recognition tasks. As an alternative, we develop a generative model for the raw intensity of image patches and show that it can support image classification performance on par with optimized SIFT-based techniques in a bag-of-visual-words setting. Key ingredient of the proposed model is a compact dictionary of mini-epitomes, learned in an unsupervised fashion on a large collection of images. The use of epitomes allows us to explicitly account for photometric and position variability in image appearance. We show that this flexibility considerably increases the capacity of the dictionary to accurately approximate the appearance of image patches and support recognition tasks. For image classification, we develop histogram-based image encoding methods tailored to the epitomic representation, as well as an "epitomic footprint" encoding which is easy to visualize and highlights the generative nature of our model. We discuss in detail computational aspects and develop efficient algorithms to make the model scalable to large tasks. The proposed techniques are evaluated with experiments on the challenging PASCAL VOC 2007 image classification benchmark.