Multimedia Information Retrieval Research Articles

Weighted subspace modeling for semantic concept retrieval using gaussian mixture models

At the era of digital revolution, social media data are growing at an explosive speed. Thanks to the prevailing popularity of mobile devices with cheap costs and high resolutions as well as the ubiquitous Internet access provided by mobile carriers, Wi-Fi, etc., numerous numbers of videos and pictures are generated and uploaded to social media websites such as Facebook, Flickr, and Twitter everyday. To efficiently and effectively search and retrieve information from the large amounts of multimedia data (structured, semi-structured, or unstructured), lots of algorithms and tools have been developed. Among them, a variety of data mining and machine learning methods have been explored and proposed and have shown their effectiveness and potentials in handling the growing requests to retrieve semantic information from those large-scale multimedia data. However, it is well-acknowledged that the performance of such multimedia semantic information retrieval is far from satisfactory, due to the challenges like rare events, data imbalance, etc. In this paper, a novel weighted subspace modeling framework is proposed that is based on the Gaussian Mixture Model (GMM) and is able to effectively retrieve semantic concepts, even from the highly imbalanced datasets. Experimental results performed on two public-available benchmark datasets against our previous GMM-based subspace modeling method and the other prevailing counterparts demonstrate the effectiveness of the proposed weighted GMM-based subspace modeling framework with the improved retrieval performance in terms of the mean average precision (MAP) values.

SimUSF: an efficient and effective similarity measure that is invariant to violations of the interval scale assumption

Similarity measures are central to many machine learning algorithms. There are many different similarity measures, each catering for different applications and data requirements. Most similarity measures used with numerical data assume that the attributes are interval scale. In the interval scale, it is assumed that a unit difference has the same meaning irrespective of the magnitudes of the values separated. When this assumption is violated, accuracy may be reduced. Our experiments show that removing the interval scale assumption by transforming data to ranks can improve the accuracy of distance-based similarity measures on some tasks. However the rank transform has high time and storage overheads. In this paper, we introduce an efficient similarity measure which does not consider the magnitudes of inter-instance distances. We compare the new similarity measure with popular similarity measures in two applications: DBScan clustering and content based multimedia information retrieval with real world datasets and different transform functions. The results show that the proposed similarity measure provides good performance on a range of tasks and is invariant to violations of the interval scale assumption.

“The most passionate cover I’ve seen”: emotional information in fan-created U2 music videos

Purpose– The purpose of this paper is to explore how both producers and consumers of user-created music videos on YouTube communicate emotional information.Design/methodology/approach– In total, 150 filmic documents containing fan-generated versions of U2’s “Song for Someone” were purposively collected. The author used discourse analysis to understand the types of videos created, the communication of emotional information from both the producers and the consumers, the social construction of emotion in the filmic documents, and elements of intertextuality that represented emotion.Findings– Fans created videos containing cover versions, original versions of the song with new visual content, and tutorials about how to play the song. Producers of cover versions communicated emotional information, especially tenderness, through facial expression, their surroundings, and corresponding musical elements. Producers’ visual content expressed emotion through meaningful photographs and sad stories. Producers’ descriptions revealed emotion as well. Emotions were individually experienced and socially constructed. Consumers conveyed emotion through likes, dislikes, and expressive positive comments. Intertextuality communicated passion for U2 through tour references, paraphernalia displays, band photographs, imitating the band, and musical mashups.Practical implications– Information science can work towards a new generation of multimedia information retrieval systems that incorporate emotion in order to help users discover documents in meaningful ways that move beyond keyword and bibliographic searches.Originality/value– This is one of the earliest research papers in the area of emotional information retrieval (EmIR).

Exploit Every Bit: Effective Caching for High-Dimensional Nearest Neighbor Search

High-dimensional $k$ nearest neighbor (kNN) search has a wide range of applications in multimedia information retrieval. Existing disk-based $k$ NN search methods incur significant I/O costs in the candidate refinement phase. In this paper, we propose to cache compact approximate representations of data points in main memory in order to reduce the candidate refinement time during $k$ NN search. This problem raises two challenging issues: (i) which is the most effective encoding scheme for data points to support $k$ NN search? and (ii) what is the optimal number of bits for encoding a data point? For (i), we formulate and solve a novel histogram optimization problem that decides the most effective encoding scheme. For (ii), we develop a cost model for automatically tuning the optimal number of bits for encoding points. In addition, our approach is generic and applicable to exact / approximate $k$ NN search methods. Extensive experimental results on real datasets demonstrate that our proposal can accelerate the candidate refinement time of $k$ NN search by at least an order of magnitude.

A survey on Flickr multimedia research challenges

Multimedia content sharing within social networks has become one of the most interesting and trending research fields over the last few years. This undoubted emerge of related research works is rather twofold, namely it includes both the analysis and management techniques of the content itself, as well as new ways for its accompanied meaningful interpretation and exploitation. In this paper, we review the recent advances in the above fields in the humanistic framework of the popular Flickr social network. In addition, the major research challenges in the area are demonstrated and discussed, which include current state-of-the-art approaches with respect to interesting humanistic data collection and interpretation research fields, such as multimedia information retrieval, (semi-) automatic tag manipulation, travel applications, semantic knowledge extraction, human activity tracking, as well as related benchmarking efforts. At the end of this survey, we also discuss the main challenges and propose a number of future research directions for interested fellow researchers to continue investigation in the field.

Learning contextualized semantics from co-occurring terms via a Siamese architecture

One of the biggest challenges in Multimedia information retrieval and understanding is to bridge the semantic gap by properly modeling concept semantics in context. The presence of out of vocabulary (OOV) concepts exacerbates this difficulty. To address the semantic gap issues, we formulate a problem on learning contextualized semantics from descriptive terms and propose a novel Siamese architecture to model the contextualized semantics from descriptive terms. By means of pattern aggregation and probabilistic topic models, our Siamese architecture captures contextualized semantics from the co-occurring descriptive terms via unsupervised learning, which leads to a concept embedding space of the terms in context. Furthermore, the co-occurring OOV concepts can be easily represented in the learnt concept embedding space. The main properties of the concept embedding space are demonstrated via visualization. Using various settings in semantic priming, we have carried out a thorough evaluation by comparing our approach to a number of state-of-the-art methods on six annotation corpora in different domains, i.e., MagTag5K, CAL500 and Million Song Dataset in the music domain as well as Corel5K, LabelMe and SUNDatabase in the image domain. Experimental results on semantic priming suggest that our approach outperforms those state-of-the-art methods considerably in various aspects.

Knowledge Organisation and its Role in Multimedia Information Retrieval

KO KNOWLEDGE ORGANIZATION , Seite 180 - 183

Hybrid operations for content-based Vietnamese agricultural multimedia information retrieval

Content-based multimedia information retrieval is never a trivial task even with state-of-the-art approaches. Its mandatory challenge, called “semantic gap,” requires much more understanding of the way human perceive things (i.e., visual and auditory information). Computer scientists have spent thousands of hours seeking optimal solutions, only ended up falling in the bound of this gap for both visual and spoken contexts. While an over-the-gap approach is unreachable, we insist on assembling current viable techniques from both contexts, aligned with a domain concept base (i.e., an ontology), to construct an info service for the retrieval of agricultural multimedia information. The development process spans over three packages: (1) building a Vietnamese agricultural thesaurus; (2) crafting a visual-auditory intertwined search engine; and (3) system deployment as an info service. We spring our the thesaurus in 2 sub-boughs: the aquaculture ontology consists of 3455 concepts and 5396 terms, with 28 relationships, covering about 2200 fish species and their related terms; and the plant production ontology comprises of 3437 concepts and 6874 terms, with 5 relationships, covering farming, plant production, pests, etc. These ontologies serve as a global linkage between keywords, visual, and spoken features, as well as providing the reinforcement for the system performances (e.g., through query expansion, knowledge indexing…). On the other hand, constructing a visual-auditory intertwined search engine is a bit trickier. Automatic transcriptions of audio channels are marked as the anchor points for the collection of visual features. These features, in turn, got clustered based on the referenced thesauri, and ultimately tracking out missing info induced by the speech recognizer’s word error rates. This compensation technique bought us back 14 % of loss recall and an increase of 9 % accuracy over the baseline system. Finally, wrapping the retrieval system as an info service guarantees its practical deployment, asour target audiences are the majority of farmers in developing countries who are unable to reach modern farming information and knowledge.

Kernel canonical correlation analysis via gradient descent

Kernel canonical correlation analysis (CCA) is a powerful statistical tool characterizing nonlinear relations between two sets of multidimensional variables. It has been widely used in many branches of science and technology, e.g. bioinfomatics, multi-media information retrieval, cross-language document retrieval, fMRI (functional magnetic resonance imaging). Previous algorithms focus on sparsity analysis of kernel CCA. In this paper, from another viewpoint, we address a new gradient descent kernel CCA algorithm, which is based on the relation between kernel CCA and linear systems of equations. Meanwhile, stability analysis of the algorithm is addressed by means of suitable error decomposition formula and compact operator theory. Theoretical analysis is elegantly investigated in terms of choices of regularization parameter and step size. Experimental results on real-world datasets demonstrate the effectiveness of the algorithm for content-based image retrieval task. The results indicate that the proposed algorithm is stable and the performance is comparable with several state-of-the-art CCA algorithms.

Learning of Multimodal Representations With Random Walks on the Click Graph.

In multimedia information retrieval, most classic approaches tend to represent different modalities of media in the same feature space. With the click data collected from the users' searching behavior, existing approaches take either one-to-one paired data (text-image pairs) or ranking examples (text-query-image and/or image-query-text ranking lists) as training examples, which do not make full use of the click data, particularly the implicit connections among the data objects. In this paper, we treat the click data as a large click graph, in which vertices are images/text queries and edges indicate the clicks between an image and a query. We consider learning a multimodal representation from the perspective of encoding the explicit/implicit relevance relationship between the vertices in the click graph. By minimizing both the truncated random walk loss as well as the distance between the learned representation of vertices and their corresponding deep neural network output, the proposed model which is named multimodal random walk neural network (MRW-NN) can be applied to not only learn robust representation of the existing multimodal data in the click graph, but also deal with the unseen queries and images to support cross-modal retrieval. We evaluate the latent representation learned by MRW-NN on a public large-scale click log data set Clickture and further show that MRW-NN achieves much better cross-modal retrieval performance on the unseen queries/images than the other state-of-the-art methods.

Late Semantic Fusion Approaches for Multimedia Information Retrieval with Automatic Tag Generation

retrieval in general and content based image retrieval in particular are well known research fields in information retrieval management. An image contains several types of visual information which are difficult to extract and combine manually by humans. The main goal of this paper is to show multimedia information retrieval task using the combination of textual pre-filtering and image re-ranking. The combination of textual and visual techniques and retrieval processes used to develop the multimedia information retrieval system by which we solves the problem of the semantic gap of the given query. Five late semantic fusion approaches are used for text based and content based image retrieval of any dataset. The logistic regression relevance feedback algorithm is used to determine the similarity between the images from the dataset to the query.

Joint Structural Learning to Rank with Deep Linear Feature Learning

Multimedia information retrieval usually involves two key modules including effective feature representation and ranking model construction. Most existing approaches are incapable of well modeling the inherent correlations and interactions between them, resulting in the loss of the latent consensus structure information. To alleviate this problem, we propose a learning to rank approach that simultaneously obtains a set of deep linear features and constructs structure-aware ranking models in a joint learning framework. Specifically, the deep linear feature learning corresponds to a series of matrix factorization tasks in a hierarchical manner, while the learning-to-rank part concentrates on building a ranking model that effectively encodes the intrinsic ranking information by structural SVM learning. Through a joint learning mechanism, the two parts are mutually reinforced in our approach, and meanwhile their underlying interaction relationships are implicitly reflected by solving an alternating optimization problem. Due to the intrinsic correlations among different queries (i.e., similar queries for similar ranking lists), we further formulate the learning-to-rank problem as a multi-task problem, which is associated with a set of mutually related query-specific learning-to-rank subproblems. For computational efficiency and scalability, we design a MapReduce-based parallelization approach to speed up the learning processes. Experimental results demonstrate the efficiency, effectiveness, and scalability of the proposed approach in multimedia information retrieval.

A Deep and Autoregressive Approach for Topic Modeling of Multimodal Data.

Topic modeling based on latent Dirichlet allocation (LDA) has been a framework of choice to deal with multimodal data, such as in image annotation tasks. Another popular approach to model the multimodal data is through deep neural networks, such as the deep Boltzmann machine (DBM). Recently, a new type of topic model called the Document Neural Autoregressive Distribution Estimator (DocNADE) was proposed and demonstrated state-of-the-art performance for text document modeling. In this work, we show how to successfully apply and extend this model to multimodal data, such as simultaneous image classification and annotation. First, we propose SupDocNADE, a supervised extension of DocNADE, that increases the discriminative power of the learned hidden topic features and show how to employ it to learn a joint representation from image visual words, annotation words and class label information. We test our model on the LabelMe and UIUC-Sports data sets and show that it compares favorably to other topic models. Second, we propose a deep extension of our model and provide an efficient way of training the deep model. Experimental results show that our deep model outperforms its shallow version and reaches state-of-the-art performance on the Multimedia Information Retrieval (MIR) Flickr data set.

Towards conscientious peers: Combining agents and peers for efficient and scalable video segment retrieval for VoD services

In the last years, Video-on-Demand (VoD) systems, such as Youtube, have become a very popular way to watch videos. Researches in Multiagent Systems (MAS) and Peer-to-Peer (P2P) for VoD have focused on using agents to assist peers to share video segments efficiently, by replicating them or creating the best path between peers to transfer such segments. A key challenge faced by these systems is to efficiently search and to retrieve which peers own certain segments in order to provide the streaming service. In this paper we propose an innovative architecture that combines MAS and P2P network for multimedia information retrieval in the VoD environment. This architecture, composed of a P2P and a MAS layer, supports streaming and random seeking while providing scalability and efficiency. The P2P layer is responsible for discovering the peers that own the video segments and share these segments among them. Nevertheless, if such search is time consuming, it is forwarded to the MAS layer. The MAS layer is responsible for monitoring groups of peers as well as for discovering agents that manage a group of peers sharing certain video segments that could not be found efficiently by peers. This is made for improving the sharing performance of its monitored peers. Experimental results, using several metrics, showed the viability of adopting our approach if compared with the most used approaches.

A comparative study for multiple visual concepts detection in images and videos

Automatic indexing of images and videos is a highly relevant and important research area in multimedia information retrieval. The difficulty of this task is no longer something to prove. Most efforts of the research community have been focusing, in the past, on the detection of single concepts in images/videos, which is already a hard task. With the evolution of information retrieval systems, users' needs become more abstract, and lead to a larger number of words composing the queries. It is important to think about indexing multimedia documents with more than just individual concepts, to help retrieval systems to answer such complex queries. Few studies addressed specifically the problem of detecting multiple concepts (multi-concept) in images and videos. Most of them concern the detection of concept pairs. These studies showed that such challenge is even greater than the one of single concept detection. In this work, we address the problem of multi-concept detection in images/videos by making a comparative and detailed study. Three types of approaches are considered: 1) building detectors for multi-concept, 2) fusing single concepts detectors and 3) exploiting detectors of a set of single concepts in a stacking scheme. We conducted our evaluations on PASCAL VOC'12 collection regarding the detection of pairs and triplets of concepts. We extended the evaluation process on TRECVid 2013 dataset for infrequent concept pairs' detection. Our results show that the three types of approaches give globally comparable results for images, but they differ for specific kinds of pairs/triplets. In the case of videos, late fusion of detectors seems to be more effective and efficient when single concept detectors have good performances. Otherwise, directly building bi-concept detectors remains the best alternative, especially if a well-annotated dataset is available. The third approach did not bring additional gain or efficiency.

ICMR 2014

ICMR was initially started as a workshop on challenges in image retrieval (in Newcastle in 1998 ) and later transformed into the Conference on Image and Video Retrieval (CIVR) series. In 2011 the CIVR and the ACM Workshop on Multimedia Information Retrieval were combined into a single conference that now forms the ICMR series. The 4th ACM International Conference on Multimedia Retrieval took place in Glasgow, Scotland, from 1 – 4 April 2014. This was the largest edition of ICMR to date with approximately 170 attendees from 25 different countries. ICMR is one of the premier scientific conference for multimedia retrieval held worldwide, with the stated mission “to illuminate the state of the art in multimedia retrieval by bringing together researchers and practitioners in the field of multimedia retrieval .” According to the Chinese Computing Federation Conference Ranking (2013), ACM ICMR is the number one multimedia retrieval conference worldwide and the number four conference in the category of multimedia and graphics. Although ICMR is about multimedia retrieval, in a wider sense, it is also about automated multimedia understanding. Much of the work in that area involves the analysis of media on a pixel, voxel, and wavelet level, but it also involves innovative retrieval, visualisation and interaction paradigms utilising the nature of the multimedia — be it video, images, speech, or more abstract (sensor) data. The conference aims to promote intellectual exchanges and interactions among scientists, engineers, students, and multimedia researchers in academia as well as industry through various events, including a keynote talk, oral, special and poster sessions focused on re search challenges and solutions, technical and industrial demonstrations of prototypes, tutorials, research, and an industrial panel. In the remainder of this report we will summarise the events that took place at the 4th ACM ICMR conference.

Sparse Linear Integration of Content and Context Modalities for Semantic Concept Retrieval

The semantic gap between low-level visual features and high-level semantics is a well-known challenge in content-based multimedia information retrieval. With the rapid popularization of social media, which allows users to assign tags to describe images and videos, attention is naturally drawn to take advantage of these metadata in order to bridge the semantic gap. This paper proposes a sparse linear integration (SLI) model that focuses on integrating visual content and its associated metadata, which are referred to as the content and the context modalities, respectively, for semantic concept retrieval. An optimization problem is formulated to approximate an instance using a sparse linear combination of other instances and minimize the difference between them. The prediction score of a concept for a test instance measures how well it can be reconstructed by the positive instances of that concept. Two benchmark image data sets and their associated tags are used to evaluate the SLI model. Experimental results show promising performance by comparing with the approaches based on a single modality and approaches based on popular fusion methods.

Optimization of Filter Combination for Query by Humming

In a multimedia information retrieval system, the response time to a query is crucial for user experience. For a practical QBH system, developers not only need to devise an effective method of melody match, but also need to consider the response speed to a query. Although the index-based methods can effectively reduce the latency, to meet the strict requirements in most real situations where, for example, the database contains hundreds of thousands of songs, further efforts are needed. To strike a balance between response speed and retrieval accuracy, this paper presents an optimal design for filter combination (FC) applied in a query by humming (QBH) system. In the design of FC, we first make use of a greedy algorithm to sort filters and then group filters to reduce the total number of levels of filters. To further accelerate the speed of retrieval and present results within a time limit, we adopt the dynamic programming to optimize the confidence threshold in each filter. The proposed methodology can greatly accelerate the retrieval process with the cost of sacrificing the least accuracy. A four-level filtering QBH system is evaluated on a large-scale database of 100,000 MIDI files. The experimental results demonstrate the feasibility of the proposed method.

Unsupervised Visual and Textual Information Fusion in CBMIR Using Graph-Based Methods

Multimedia collections are more than ever growing in size and diversity. Effective multimedia retrieval systems are thus critical to access these datasets from the end-user perspective and in a scalable way. We are interested in repositories of image/text multimedia objects and we study multimodal information fusion techniques in the context of content-based multimedia information retrieval. We focus on graph-based methods, which have proven to provide state-of-the-art performances. We particularly examine two such methods: cross-media similarities and random-walk-based scores. From a theoretical viewpoint, we propose a unifying graph-based framework, which encompasses the two aforementioned approaches. Our proposal allows us to highlight the core features one should consider when using a graph-based technique for the combination of visual and textual information. We compare cross-media and random-walk-based results using three different real-world datasets. From a practical standpoint, our extended empirical analyses allow us to provide insights and guidelines about the use of graph-based methods for multimodal information fusion in content-based multimedia information retrieval.

An interactive tool for manual, semi-automatic and automatic video annotation

The annotation of image and video data of large datasets is a fundamental task in multimedia information retrieval and computer vision applications. The aim of annotation tools is to relieve the user from the burden of the manual annotation as much as possible. To achieve this ideal goal, many different functionalities are required in order to make the annotations process as automatic as possible. Motivated by the limitations of existing tools, we have developed the iVAT: an interactive Video Annotation Tool. It supports manual, semi-automatic and automatic annotations through the interaction of the user with various detection algorithms. To the best of our knowledge, it is the first tool that integrates several computer vision algorithms working in an interactive and incremental learning framework. This makes the tool flexible and suitable to be used in different application domains. A quantitative and qualitative evaluation of the proposed tool on a challenging case study domain is presented and discussed. Results demonstrate that the use of the semi-automatic, as well as the automatic, modality drastically reduces the human effort while preserving the quality of the annotations.