Multimedia Information Retrieval Research Articles

ACM ICMR 2014 best papers in image retrieval

The ACM International Conference onMultimedia Retrieval (ICMR) is the top multimedia retrieval conference worldwide. This year it was held in Glasgow, Scotland by the general chairs: Joemon Jose and Keith van Rijsbergen. It is my honor to present in this issue the extended versions of the best image retrieval papers which were selected on the basis of the evaluations from the program committee (special thanks to the program chairs: Mohan Kankanhalli, Stefan Rueger and R. Manmatha). One of the grand challenges for multimedia information retrieval is automatic image annotation because it can provide a direct bridge over the semantic gap from low-level features to human level tags and annotations. The paper “A Sparse Kernel RelevanceModel for Automatic Image Annotation” by SeanMoran andVictor Lavrenko introduces a new form of the Continuous RelevanceModel which can perform adaptive kernel selection per feature type. Also, they describe two novel data-adaptive kernels which improve the modeling of image feature distributions. Their findings show that the adaptive kernel assignment can yield significantly higher annotation accuracy. After the initial results have been found from an image retrieval query, a fundamental problem is to re-rank the images toward providing the user withmore accurate and relevant results. The paper “Image re-ranking system based on frequent closed patterns” by Winn Voravuthikunchai, Bruno

A Model of Media Information Management for Content Retrieval

The rapid development of digital library makes the multimedia information resources organization and management become a key issue to develop the digital library, and the multimedia information retrieval technology has become the core technology of the digital book data management, the existing multimedia retrieval model cannot effectively implement multimedia digital books data management. Therefore, this paper proposes a content-based multimedia retrieval data model and combined with the different modal data abstraction and decomposition classification thought, discusses the significance of this data model, also briefly introduces the building method of this model. Finally by combining this design it implements a multimedia digital books data retrieval system.

A hybrid cloud-P2P architecture for multimedia information retrieval on VoD services

Recent research in Cloud Computing and Peer-to-Peer systems for Video-on-Demand (VoD) has focused on multimedia information retrieval, using cloud nodes as video streaming servers and peers as a way to distribute and share the video segments. A key challenge faced by these systems is providing an efficient way to retrieve the information segments descriptor, composed of its metadata and video segments, distributed among the cloud nodes and the Peer-to-Peer (P2P) network. In this paper, we propose a novel Cloud Computing and P2P hybrid architecture for multimedia information retrieval on VoD services that supports random seeking while providing scalability and efficiency. The architecture comprises Cloud and P2P layers. The Cloud layer is responsible for video segment metadata retrieval, using ontologies to improve the relevance of the retrieved information, and for distributing the metadata structures among cloud nodes. The P2P layer is responsible for finding peers that have the physical location of a segment. In this layer, we use trackers, which manage and collect the segments shared among other peers. We also use two Distributed Hash Tables, one to find these trackers and the other to store the information collected in case the tracker leaves the network and another peer needs to replace it. Unlike previous work, our architecture separates cloud nodes and peers responsibilities to manage the video metadata and its segments, respectively. Also, we show via simulations, the possibility of converting any peer to act as a tracker, while maintaining system scalability and performance, avoiding using centralized and powerful servers.

An effective and economical architecture for semantic-based heterogeneous multimedia big data retrieval

Data variety has been one of the most critical features for multimedia big data. Some multimedia documents, although in different data formats and storage structures, often express similar semantic information. Therefore, the way to manage and retrieve multimedia documents reflecting users’ intent in heterogeneous big data environments has become an important issue. In this paper, we present an effective and economical architecture named SHMR (Semantic-based Heterogeneous Multimedia Retrieval), which uses low cost to store and retrieve semantic information from heterogeneous multimedia data. Firstly, the particularity of heterogeneous multimedia retrieval in big data environments is addressed. Secondly, an approach to extract and represent semantic information for heterogeneous multimedia documents is proposed. Thirdly, a NoSQL-based approach to semantic storage, in which multimedia can be parallel processed in distributed nodes is provided. Finally, a MapReduce-based retrieval algorithm is presented and a user feedback supported scheme to achieve high retrieval precision and good user experience is designed. The experimental results indicate that the retrieval performance and economic efficiency of SHMR are suitable for multimedia information retrieval in heterogeneous big data environments.

Geometric discriminative features for aerial image retrieval in social media

The aerial image recognition is an important problem in multimedia information retrieval in social media. In this paper, we propose a new approach by integrating aerial image's local features into a discriminative one which reflects both the geometric property and the color distribution of aerial image. Firstly, each aerial image is segmented into several regions in terms of their color intensities. And region connected graph (RCG), the links between the spatial neighboring regions, is presented to encode the spatial context of aerial images. Secondly, we mine frequent structures in the RCGs corresponding to training aerial images collected from social media. And a set of refined structures are selected among the frequent ones towards being more discriminative and less redundant. Finally, given a new aerial image, its sub-RCGs corresponding to all the refined structures are extracted and quantized into a discriminative feature for aerial image recognition. The experimental results validate the proposed method by providing a more accurate recognition result of the aerial images on different datasets from different social medias.

A Novel Local Features Based Salient Object Recognition Algorithm via Hybrid SVM-QPSO Model

As the salient objects extraction is of great importance in computer vision and multimedia information retrieval, this paper concentrates on the problem of salient object recognition using local features. Considering the rotational invariance performance of circular region is much better, we exploit a circular region to replace the rectangular region. To implement the salient object detection, the visual object classes should be constructed from training image dataset through SIFT features clustering. Furthermore, for a test image, the object class which the test image belonged to can be detected by interest points matching. Afterwards, the SIFT features clustering and local features matching process can be implemented through the proposed hybrid SVM-QPSO model. To promote the quality of parameter selection in SVM, we utilize the quantum behaved particle swarm optimization technique to select suitable SVM parameters. Finally, experiments are conducted to make performance evaluation using the MSRC dataset. Experimental results show that compared with other methods, the proposed algorithm can effectively detect salient objects in both object detecting precision and computing efficiency.

PIR

Multimedia Information Retrieval (MIR) is a problem domain that includes programming tasks such as salient feature extraction, machine learning, indexing, and retrieval. There are a variety of implementations and algorithms for these tasks in different languages and frameworks, which are difficult to compose and reuse due to the interface and language incompatibility. Due to this low reusability, researchers often have to implement their experiments from scratch and the resulting programs cannot be easily adapted to parallel and distributed executions, which is important for handling large data sets. In this paper, we present Pipeline Information Retrieval (PIR), a Domain Specific Language (DSL) for multi-modal feature manipulation. The goal of PIR is to unify the MIR programming tasks by hiding the programming details under a flexible layer of domain specific interface. PIR optimizes the MIR tasks by compiling the DSL programs into pipeline graphs, which can be executed using a variety of strategies (e.g. sequential, parallel, or distributed execution). The authors evaluated the performance of PIR applications on single machine with multiple cores, local cluster, and Amazon Elastic Compute Cloud (EC2) platform. The result shows that the PIR programs can greatly help MIR researchers and developers perform fast prototyping on single machine environment and achieve nice scalability on distributed platforms.

A Survey on Nearest Neighbor Search Methods

ABSTRACT Nowadays, the need to techniques, approaches, and algorithms to search on data is increased due to improvements in computer science and increasing amount of information. This ever increasing information volume has led to time and computation complexity. Recently, different methods to solve such problems are proposed. Among the others, nearest neighbor search is one of the best techniques to this end which is focused by many researchers. Different techniques are used for nearest neighbor search. In addition to put an end to some complexities, variety of these techniques has made them suitable for different applications such as pattern recognition, searching in multimedia data, information retrieval, databases, data mining, and computational geometry to name but a few. In this paper, by opening a new view to this problem, a comprehensive evaluation on structures, techniques and different algorithms in this field is done and a new categorization of techniques in NNS is presented. This categorization is consists of seven groups: Weighted, Reductional, Additive, Reverse, Continuous, Principal Axis and Other techniques which are studied, evaluated and compared in this paper. Complexity of used structures, techniques and their algorithms are discussed, as well.

Multi region based image retrieval system

Multimedia information retrieval systems continue to be an active research area in the world of huge and voluminous data. The paramount challenge is to translate or convert a visual query from a human and find similar images or videos in large digital collection. In this paper, a technique of region based image retrieval, a branch of Content Based Image Retrieval, is proposed. The proposed model does not need prior knowledge or full semantic understanding of image content. It identifies significant regions in an image based on feature-based attention model which mimic viewer’s attention. The Curvelet Transform in combination with colour descriptors are used to represent each significant region in an image. Experimental results are analysed and compared with the state-of-the-art Region Based Image Retrieval Technique.

Information extraction from multimedia web documents: an open-source platform and testbed

The LivingKnowledge project aimed to enhance the current state of the art in search, retrieval and knowledge management on the web by advancing the use of sentiment and opinion analysis within multimedia applications. To achieve this aim, a diverse set of novel and complementary analysis techniques have been integrated into a single, but extensible software platform on which such applications can be built. The platform combines state-of-the-art techniques for extracting facts, opinions and sentiment from multimedia documents, and unlike earlier platforms, it exploits both visual and textual techniques to support multimedia information retrieval. Foreseeing the usefulness of this software in the wider community, the platform has been made generally available as an open-source project. This paper describes the platform design, gives an overview of the analysis algorithms integrated into the system and describes two applications that utilise the system for multimedia information retrieval.

A data-driven study of image feature extraction and fusion

Feature analysis is the extraction and comparison of signals from multimedia data, which can subsequently be semantically analyzed. Feature analysis is the foundation of many multimedia computing tasks such as object recognition, image annotation, and multimedia information retrieval. In recent decades, considerable work has been devoted to the research of feature analysis. In this work, we use large-scale datasets to conduct a comparative study of four state-of-the-art, representative feature extraction algorithms: color-texture codebook (CT), SIFT codebook, HMAX, and convolutional networks (ConvNet). Our comparative evaluation demonstrates that different feature extraction algorithms enjoy their own advantages, and excel in different image categories. We provide key observations to explain where these algorithms excel and why. Based on these observations, we recommend feature extraction principles and identify several pitfalls for researchers and practitioners to avoid. Furthermore, we determine that in a large training dataset with more than 10,000 instances per image category, the four evaluated algorithms can converge to the same high level of category-prediction accuracy. This result supports the effectiveness of the data-driven approach. Finally, based on learned clues from each algorithm’s confusion matrix, we devise a fusion algorithm to harvest synergies between these four algorithms and further improve class-prediction accuracy.

Spaces, Trees, and Colors

Document retrieval is one of the best-established information retrieval activities since the ’60s, pervading all search engines. Its aim is to obtain, from a collection of text documents, those most relevant to a pattern query. Current technology is mostly oriented to “natural language” text collections, where inverted indexes are the preferred solution. As successful as this paradigm has been, it fails to properly handle various East Asian languages and other scenarios where the “natural language” assumptions do not hold. Inthis survey, we cover the recent research in extending the document retrieval techniques to a broader class of sequence collections, which has applications in bioinformatics, data and web mining, chemoinformatics, software engineering, multimedia information retrieval, and many other fields. We focus on the algorithmic aspects of the techniques, uncovering a rich world of relations between document retrieval challenges and fundamental problems on trees, strings, range queries, discrete geometry, and other areas.

RELIEF-MM: effective modality weighting for multimedia information retrieval

Fusing multimodal information in multimedia data usually improves the retrieval performance. One of the major issues in multimodal fusion is how to determine the best modalities. To combine the modalities more effectively, we propose a RELIEF-based modality weighting approach, named as RELIEF-MM. The original RELIEF algorithm is extended for weaknesses in several major issues: class-specific feature selection, complexities with multi-labeled data and noise, handling unbalanced datasets, and using the algorithm with classifier predictions. RELIEF-MM employs an improved weight estimation function, which exploits the representation and reliability capabilities of modalities, as well as the discrimination capability, without any increase in the computational complexity. The comprehensive experiments conducted on TRECVID 2007, TRECVID 2008 and CCV datasets validate RELIEF-MM as an efficient, accurate and robust way of modality weighting for multimedia data.

Semantic indexing of multimedia content using textual and visual information

The challenge in multimedia information retrieval remains in the indexing process, an active search area. There are three fundamental techniques for indexing multimedia content: using textual information, using low-level information and combining different information extracted from multimedia. Each approach has its advantages and disadvantages as well to improve multimedia retrieval systems. The recent works are oriented towards multimodal approaches. In this paper, we propose an approach that combines the surrounding text with the information extracted from the visual content of multimedia and represented in the same repository in order to allow querying multimedia content based on keywords or concepts. Each word contained in queries or in description of multimedia is disambiguated using the WordNet ontology in order to define its semantic concept. Support vector machines SVMs are used for image classification in one of the defined semantic concept based on SIFT scale invariant feature transform descriptors.

SPEECH/MUSIC CLASSIFICATION USING WAVELET BASED FEATURE EXTRACTION TECHNIQUES

Audio classification serves as the fundamental step towards the rapid growth in audio data volume. Due to the increasing size of the multimedia sources speec h and music classification is one of the most impor tant issues for multimedia information retrieval. In thi s work a speech/music discrimination system is deve loped which utilizes the Discrete Wavelet Transform (DWT) as the acoustic feature. Multi resolution analysis is the most significant statistical way to extract the features from the input signal and in this study, a method is deployed to model the extracted wavelet feature. Su pport Vector Machines (SVM) are based on the principle of structural risk minimization. SVM is a pplied to classify audio into their classes namely speech and music, by learning from training data. Then the proposed method extends the application of Gaussia n Mixture Models (GMM) to estimate the probability density function using maximum likelihood decision methods. The system shows significant results with an accuracy of 94.5%.

EnContRA: a generic multimedia information retrieval meta-framework

Over the last years, multimedia collections have largely increased as new items are produced every day, such as pictures, audio/music or video. In Multimedia Information Retrieval, this exponential growth leads content-based approaches to gain advantage over other solutions, not only because they take advantage of the intrinsic information contained in the objects, but also because they automatically process and extract it, reducing the burden taken by developers. Several domain specific frameworks have been developed to efficiently retrieve multimedia items empowering the creation of new content-based applications. However, these frameworks are attached to a specific media type, are too complex to be used in a fast prototyping environment, and are not very flexible nor extensible. To solve these issues, we developed EnContRA, an architectural meta-framework that provides generic building blocks for creating domain specific frameworks. Our meta-framework aims at being ready to be used for fast prototyping, with support for rich and multimodal queries, allowing validation of new descriptors, indexing structures or searching algorithms, while creating domain specific frameworks. In this paper we present the meta-framework architecture and describe in detail its modules and features. To validate the meta-framework, we created an image retrieval framework and a demo application that combines image descriptors with textual information, showing how the hierarchical design of EnContRA could be applied to a searching system and to empower the creation of queries.

Multimedia Information Retrieval Based on Late Semantic Fusion Approaches: Experiments on a Wikipedia Image Collection

Main goal of this work is to show the improvement of using a textual pre-filtering combined with an image re-ranking in a Multimedia Information Retrieval task. The defined three step-based retrieval processes and a well-selected combination of visual and textual techniques help the developed Multimedia Information Retrieval System to overcome the semantic gap in a given query. In the paper, five different late semantic fusion approaches are discussed and experimented in a realistic scenario for multimedia retrieval like the one provided by the publicly available ImageCLEF Wikipedia Collection.

Multimedia information retrieval and environmental monitoring: Shared perspectives on data fusion

Computer-based remote monitoring of our environment is increasingly based on combining data derived from in-situ-sensors with data derived from remote sources, such as satellite images or CCTV. In such deployments it is necessary to continuously monitor the accuracy of each of the sensor data streams so that we can account for sudden failures of sensors, or errors due to calibration drive or biofouling. In multimedia information retrieval (MMIR), we search through archives of multimedia artefacts like video programs, by implementing several independent retrieval systems or agents, and we combine the outputs of each retrieval agent in order to generate an overall ranking. In this paper we draw parallels between these seemingly very different applications and show how they share several similarities. In the case of environmental monitoring we also need some mechanism by which we can establish the trust and reputation of each contributing sensor, though this is something we do not need in MMIR. In this paper we present an outline of a trust and reputation framework we have developed and deployed for monitoring the performance of sensors in a heterogeneous sensor network.

A Robust and Scalable Visual Category and Action Recognition System Using Kernel Discriminant Analysis With Spectral Regression

Visual concept detection and action recognition are one of the most important tasks in content-based multimedia information retrieval (CBMIR) technology. It aims at annotating images using a vocabulary defined by a set of concepts of interest including scenes types (mountains, snow, etc.) or human actions (phoning, playing instrument). This paper describes our system in the ImageCLEF@ICPR10, Pascal VOC 08 Visual Concept Detection and Pascal VOC 10 Action Recognition Challenges. The proposed system ranked first in these large-scale tasks when evaluated independently by the organizers. The proposed system involves state-of-the-art local descriptor computation, vector quantization via clustering, structured scene or object representation via localized histograms of vector codes, similarity measure for kernel construction and classifier learning. The main novelty is the classifier-level and kernel-level fusion using Kernel Discriminant Analysis and Spectral Regression (SR-KDA) with RBF Chi-Squared kernels obtained from various image descriptors. The distinctiveness of the proposed method is also assessed experimentally using a video benchmark: the Mediamill Challenge along with benchmarks from ImageCLEF@ICPR10, Pascal VOC 10 and Pascal VOC 08. From the experimental results, it can be derived that the presented system consistently yields significant performance gains when compared with the state-of-the art methods. The other strong point is the introduction of SR-KDA in the classification stage where the time complexity scales linearly with respect to the number of concepts and the main computational complexity is independent of the number of categories.

Personalized Multimedia Information Retrieval based on User Profile Mining

This paper focuses on how to retrieve personalized multimedia information based on user interest which can be mined from user profile. After analyzing the related works, a general structure of the personalized multimedia information retrieval system is given, which combines online module and offline module. Firstly, we collect a large-sale of photos from multimedia information sharing websites. Then, we record the information of the users who upload the multimedia information. For a given user, we save his history data which could describe the multimedia data. Secondly, the relationship between contents of multimedia data and semantic information is analyzed and then the user interest model is constructed by a modified LDA model which can integrate all the influencing factors in the task of multimedia information retrieval. Thirdly, the query distributions of all the topics can be estimated by the proposed modified LDA model. Thirdly, based on the above offline computing process, the online personalized multimedia information ranking algorithm is given which utilize the user interest model and the query word. Fourthly, multimedia information retrieval results are obtained using the proposed personalized multimedia information ranking algorithm. Finally, performance evaluation is conducted by a series of experiments to test the performance of the proposed algorithm compared with other methods on different datasets.