Multimedia Retrieval Applications Research Articles

Spatial-aware data partition for distributed memory parallelization of ANN search in multimedia retrieval

Content-based multimedia retrieval (CBMR) applications are becoming very popular in several online services which handles large volumes of data and are submitted to high query rates. While these applications may be complex, finding the nearest neighboring objects (multimedia descriptors) is typically their most time consuming operation. In order to address this problem, several recent works have proposed distributed memory parallelization of approximate nearest neighbors (ANN) search. These solutions employ a variety of ANN algorithms and different parallelization strategies. In this paper, we have identified the currently used parallelization strategies (Data Equal Split (DES) and Bucket Equal Split (BES)) and systematically evaluated their performance. We have also developed a framework to simplify the deployment of ANN algorithms in distributed memory machines with customized parallelization or data partition strategies. We further proposed a novel class of data partition/parallelization strategies that takes into account the data spatial proximity. Our approaches (SABES and SABES++) improves data locality and the system efficiency as compared to DES and BES. For instance, SABES++ achieved speedups of 4.2× and 1.8× on top of DES and BES, respectively, in the baseline case (40 nodes). Further, SABES and SABES++ also attained higher multi-node scalability and the gains vs DES and BES increase a larger number of nodes. SABES++ is 14.5× faster than DES when 160 nodes are used.

A new design of multimedia big data retrieval enabled by deep feature learning and Adaptive Semantic Similarity Function.

Nowadays, multimedia big data have grown exponentially in diverse applications like social networks, transportation, health, and e-commerce, etc. Accessing preferred data in large-scale datasets needs efficient and sophisticated retrieval approaches. Multimedia big data consists of the most significant features with different types of data. Even though the multimedia supports various data formats with corresponding storage frameworks, similar semantic information is expressed by the multimedia. The overlap of semantic features is most efficient for theory and research related to semantic memory. Correspondingly, in recent years, deep multimodal hashing gets more attention owing to the efficient performance of huge-scale multimedia retrieval applications. On the other hand, the deep multimodal hashing has limited efforts for exploring the complex multilevel semantic structure. The main intention of this proposal is to develop enhanced deep multimedia big data retrieval with the Adaptive Semantic Similarity Function (A-SSF). The proposed model of this research covers several phases “(a) Data collection, (b) deep feature extraction, (c) semantic feature selection and (d) adaptive similarity function for retrieval. The two main processes of multimedia big data retrieval are training and testing. Once after collecting the dataset involved with video, text, images, and audio, the training phase starts. Here, the deep semantic feature extraction is performed by the Convolutional Neural Network (CNN), which is again subjected to the semantic feature selection process by the new hybrid algorithm termed Spider Monkey-Deer Hunting Optimization Algorithm (SM-DHOA). The final optimal semantic features are stored in the feature library. During testing, selected semantic features are added to the map-reduce framework in the Hadoop environment for handling the big data, thus ensuring the proper big data distribution. Here, the main contribution termed A-SSF is introduced to compute the correlation between the multimedia semantics of the testing data and training data, thus retrieving the data with minimum similarity. Extensive experiments on benchmark multimodal datasets demonstrate that the proposed method can outperform the state-of-the-art performance for all types of data.

Content placement in networks of similarity caches

Similarity caching systems have recently attracted the attention of the scientific community, as they can be profitably used in many application contexts, like multimedia retrieval, advertising, object recognition, recommender systems and online content-match applications. In such systems, a user request for an object o, which is not in the cache, can be (partially) satisfied by a similar stored object o’, at the cost of a loss of user utility. In this paper we make a first step into the novel area of similarity caching networks, where requests can be forwarded along a path of caches to get the best efficiency–accuracy tradeoff. The offline problem of content placement can be easily shown to be NP-hard, while different polynomial algorithms can be devised to approach the optimal solution in discrete cases. As the content space grows large, we propose a continuous problem formulation whose solution exhibits a simple structure in a class of tree topologies. We verify our findings using synthetic and realistic request traces.

Multi-Level Correlation Adversarial Hashing for Cross-Modal Retrieval

Cross-modal hashing (CMH) has been widely used for similarity search in multimedia retrieval applications, thanks to low storage cost and fast query speed. However, preserving the content similarities in finite-length hash codes between different data modalities is still challenging due to the existing heterogeneity gap. To further address the crucial bottleneck, we propose a Multi-Level Correlation Adversarial Hashing (MLCAH) algorithm to integrate the multi-level correlation information into hash codes. The proposed MLCAH model enjoys several merits. First, to the best of our knowledge, it is the early attempt of leveraging the multi-level correlation information for cross-modal hashing retrieval. Second, we propose global and local semantic alignment mechanisms, which can effectively encode multi-level correlation information, including global information, local information, and label information into hash codes. Third, a label-consistency attention mechanism with adversarial training is designed for exploiting the local cross-modality similarity from multi-modality data. Extensive evaluations on four benchmarks demonstrate that the proposed model brings significant improvements over several state-of-the-art cross-modal hashing methods.

Effective and efficient indexing in cross-modal hashing-based datasets

To overcome the barrier of storage and computation, the hashing technique has been widely used for nearest neighbor search in multimedia retrieval applications recently. Particularly, cross-modal retrieval that searches across different modalities becomes an active but challenging problem. Although numerous of cross-modal hashing algorithms are proposed to yield compact binary codes, exhaustive search is impractical for large-scale datasets, and Hamming distance computation suffers inaccurate results. In this paper, we propose a novel search method that utilizes a probability-based index scheme over binary hash codes in cross-modal retrieval. The proposed indexing scheme employs a few binary bits from the hash code as the index code. We construct an inverted index table based on the index codes, and train a neural network for ranking and indexing to improve the retrieval accuracy. Experiments are performed on two benchmark datasets for retrieval across image and text modalities, where hash codes are generated and compared with several state-of-the-art cross-modal hashing methods. Results show the proposed method effectively boosts the performance on search accuracy, computation cost, and memory consumption in these datasets and hashing methods. The source code is available on https://github.com/msarawut/HCI.

SADIH: Semantic-Aware DIscrete Hashing

Due to its low storage cost and fast query speed, hashing has been recognized to accomplish similarity search in largescale multimedia retrieval applications. Particularly, supervised hashing has recently received considerable research attention by leveraging the label information to preserve the pairwise similarities of data points in the Hamming space. However, there still remain two crucial bottlenecks: 1) the learning process of the full pairwise similarity preservation is computationally unaffordable and unscalable to deal with big data; 2) the available category information of data are not well-explored to learn discriminative hash functions. To overcome these challenges, we propose a unified Semantic-Aware DIscrete Hashing (SADIH) framework, which aims to directly embed the transformed semantic information into the asymmetric similarity approximation and discriminative hashing function learning. Specifically, a semantic-aware latent embedding is introduced to asymmetrically preserve the full pairwise similarities while skillfully handle the cumbersome n×n pairwise similarity matrix. Meanwhile, a semantic-aware autoencoder is developed to jointly preserve the data structures in the discriminative latent semantic space and perform data reconstruction. Moreover, an efficient alternating optimization algorithm is proposed to solve the resulting discrete optimization problem. Extensive experimental results on multiple large-scale datasets demonstrate that our SADIH can clearly outperform the state-of-the-art baselines with the additional benefit of lower computational costs.

Large-scale parallel similarity search with Product Quantization for online multimedia services

The similarity search in high-dimensional spaces is a core operation found in several online multimedia retrieval applications. With the popularity of these applications, they are required to handle very large and increasing datasets, while keeping the response time low. This problem is worsened in the context of online applications, mostly due to the fact that load on these systems vary during the execution according to the users demands. Those variations require the application to adapt during the execution in order to minimize the response times. In this paper, we address these challenges with an efficient parallelization of the Product Quantization Approximate Nearest Neighbor Search (PQANNS) indexing. This method is capable of answering queries with a reduced memory demand and, coupled with a distributed memory parallelization proposed here, can efficiently handle very large datasets. We have also proposed mechanisms to minimize the query response times in online scenarios in which the query rates vary at run-time. For this sake, our strategies tune the parallelism configurations and task granularity during the execution. The parallelism and granularity tuning approaches (ADAPT and ADAPT+G) have shown, for instance, to reduce the query response times by a factor of 6.4× in comparison with the best static configuration of parallelism and task granularity. Further, the distributed memory execution using 128 nodes/3584 CPU cores has attained a parallel efficiency of 0.97 with a dataset of 256 billion SIFT vectors.

Sparse Online Learning of Image Similarity

Learning image similarity plays a critical role in real-world multimedia information retrieval applications, especially in Content-Based Image Retrieval (CBIR) tasks, in which an accurate retrieval of visually similar objects largely relies on an effective image similarity function. Crafting a good similarity function is very challenging because visual contents of images are often represented as feature vectors in high-dimensional spaces, for example, via bag-of-words (BoW) representations, and traditional rigid similarity functions, for example, cosine similarity, are often suboptimal for CBIR tasks. In this article, we address this fundamental problem, that is, learning to optimize image similarity with sparse and high-dimensional representations from large-scale training data, and propose a novel scheme of Sparse Online Learning of Image Similarity (SOLIS). In contrast to many existing image-similarity learning algorithms that are designed to work with low-dimensional data, SOLIS is able to learn image similarity from large-scale image data in sparse and high-dimensional spaces. Our encouraging results showed that the proposed new technique achieves highly competitive accuracy as compared to the state-of-the-art approaches but enjoys significant advantages in computational efficiency, model sparsity, and retrieval scalability, making it more practical for real-world multimedia retrieval applications.

Noise robust sound event classification with convolutional neural network

Automatic sound recognition (ASR) is a remarkable field of research in recent years. The ability to automatically recognize sound events through computers in a complex audio environment is very useful for machine hearing, acoustic surveillance and multimedia retrieval applications. On the other hand, ASR task become highly difficult as the ambient noise levels increase and many traditional methods show very weak performance under noise. Recent studies has shown that spectrogram image features (SIF) have high performance under noise, while success rates in clean conditions are relatively lower than in the state-of-the-art approaches. In this study, after converting highly overlapped spectrograms into linear quantized images and reducing dimensions by applying various image resizing methods, feature extraction and classification are performed with convolutional neural networks (CNN), which have very high performance in image classification. In the mismatched case, the proposed method achieves a performance improvement of 4.5%, which is equivalent to a relative error reduction of 63.4%, with a classification success of 97.4%, while the multicondition training method achieves an average of 98.63% success rate.

Efficient multi-modal geometric mean metric learning

With the fast development of information acquisition, there is a rapid growth of multi-modality data, e.g., text, audio, image and video, in health care, multimedia retrieval and many other applications. Confronted with the challenges of clustering, classification or regression with multi-modality information, it is essential to effectively measure the distance or similarity between objects described with heterogeneous features. Metric learning, aimed at finding a task-oriented distance function, is a hot topic in machine learning. However, most existing algorithms lack efficiency for high-dimensional multi-modality tasks. In this work, we develop an effective and efficient metric learning algorithm for multi-modality data, i.e., Efficient Multi-modal Geometric Mean Metric Learning (EMGMML). The proposed algorithm learns a distinctive distance metric for each view by minimizing the distance between similar pairs while maximizing the distance between dissimilar pairs. To avoid overfitting, the optimization objective is regularized by symmetrized LogDet divergence. EMGMML is very efficient in that there is a closed-form solution for each distance metric. Experimental results show that the proposed algorithm outperforms the state-of-the-art metric learning methods in terms of both accuracy and efficiency.

Lanczos kernel based spectrogram image features for sound classification

Automatic sound recognition (ASR) is a prominently emerging research area in recent years Recognition of sound events automatically through the computers in the complex audio environment is quite useful for machine hearing, acoustic surveillance and multimedia retrieval applications. Although a lot of features such as mel-frequency cepstral coefficients in ASR tasks provide very good results in noiseless environments, noisy conditions in the real world reduce success rates in a remarkable way. On the other hand, it was reported that spectrogram image features showed much better classification performance at low signal noise ratio values in many studies. In this article, it was proposed the preparation of feature vector after the images are reduced in size by applying the resizing process to spectrogram images with Lanczos kernel. Classification performance was compared by using deep artificial neural networks in different noise levels and although the feature vector was reduced, parallel values with results in the literature were obtained in the noiseless environment. It has remained slightly below the current state-of-the-art techniques using spectrogram features while better results compared to other commonly used features such as MFCC were obtained under the noisy conditions.

Performance Analysis of Multimedia Retrieval Workloads Running on Multicores

Multimedia data has become a major data type in the Big Data era. The explosive volume of such data and the increasing real-time requirement to retrieve useful information from it have put significant pressure in processing such data in a timely fashion. However, while prior efforts have done in-depth analysis on architectural characteristics of traditional multimedia processing and text-based retrieval algorithms, there has been no systematic study towards the emerging multimedia retrieval applications. This may impede the architecture design and system evaluation of these applications. In this paper, we make the first attempt to construct a multimedia retrieval benchmark suite (MMRBench for short) that can be used to evaluate architectures and system designs for multimedia retrieval applications. MMRBench covers modern multimedia retrieval algorithms with different versions (sequential, parallel and distributed). MMRBench also provides a series of flexible interfaces as well as certain automation tools. With such a flexible design, the algorithms in MMRBench can be used both in individual kernel-level evaluation and in integration to form a complete multimedia data retrieval infrastructure for full system evaluation. Furthermore, we use performance counters to analyze a set of architecture characteristics of multimedia retrieval algorithms in MMRBench, including the characteristics of core level, chip level and inter-chip level. The study shows that micro-architecture design in current processor is inefficient (both in performance and power) for these multimedia retrieval workloads, especially in core resources and memory systems. We then derive some insights into the architecture design and system evaluation for such multimedia retrieval algorithms.

RGB-D Hand-Held Object Recognition Based on Heterogeneous Feature Fusion

Object recognition has many applications in human-machine interaction and multimedia retrieval. However, due to large intra-class variability and inter-class similarity, accurate recognition relying only on RGB data is still a big challenge. Recently, with the emergence of inexpensive RGB-D devices, this challenge can be better addressed by leveraging additional depth information. A very special yet important case of object recognition is hand-held object recognition, as manipulating objects with hands is common and intuitive in human-human and human-machine interactions. In this paper, we study this problem and introduce an effective framework to address it. This framework first detects and segments the hand-held object by exploiting skeleton information combined with depth information. In the object recognition stage, this work exploits heterogeneous features extracted from different modalities and fuses them to improve the recognition accuracy. In particular, we incorporate handcrafted and deep learned features and study several multi-step fusion variants. Experimental evaluations validate the effectiveness of the proposed method.

A Multimedia Retrieval Framework Based on Semi-Supervised Ranking and Relevance Feedback

We present a new framework for multimedia content analysis and retrieval which consists of two independent algorithms. First, we propose a new semi-supervised algorithm called ranking with Local Regression and Global Alignment (LRGA) to learn a robust Laplacian matrix for data ranking. In LRGA, for each data point, a local linear regression model is used to predict the ranking scores of its neighboring points. A unified objective function is then proposed to globally align the local models from all the data points so that an optimal ranking score can be assigned to each data point. Second, we propose a semi-supervised long-term Relevance Feedback (RF) algorithm to refine the multimedia data representation. The proposed long-term RF algorithm utilizes both the multimedia data distribution in multimedia feature space and the history RF information provided by users. A trace ratio optimization problem is then formulated and solved by an efficient algorithm. The algorithms have been applied to several content-based multimedia retrieval applications, including cross-media retrieval, image retrieval, and 3D motion/pose data retrieval. Comprehensive experiments on four data sets have demonstrated its advantages in precision, robustness, scalability, and computational efficiency.

An efficient algorithm for hyperspherical range query processing in high-dimensional data space

Recently, similarity search on high-dimensional feature vectors has become an important search paradigm for various multimedia retrieval applications. The technique used is to map the data items as points into a high-dimensional feature space. The feature space is usually indexed using a multi-dimensional index structure. Similarity search then corresponds to the hyperspherical range query that returns all objects within a threshold level of similarity to the query object [1]. The hyperspherical range query for similarity search can be defined as follows:

Spatial representation and retrieval. Getting an edge on what it takes to find matching pairs in 3D

Discusses two functions for retrieving 3D images from large collections. One proposed representation is referred to as the 3D spatial orientation graph (SOG). SOG is a fully-connected, undirected, weighted graph. The time complexity of this SOG-based spatial similarity (SIM/sub 3D/) algorithm is /spl Theta/(|E/sub q/|+|E/sub d/|), where |E/sub q/| and |E/sub d/| are the number of edges, respectively, in the query or database images. The other proposed representation is referred to as the 3D surface normal list (SNL). The surface normal is the unit vector perpendicular to the plane formed by three space objects. A list with surface normals is used to represent an image. This SNL-based SIM/sub 3D/ algorithm has linear time complexity based on the total number of objects in the query and the database images. We expect both the SOG- and SNL-based SIM/sub 3D/ algorithms to be quite useful in multimedia retrieval applications such as architectural design, interior design and real-estate marketing. The original SOG algorithm has been improved to recognize rotational variant images and arbitrary variants involving a composition of translation, scaling and rotation. For the SNL algorithm, the choice of the base point is significant. It cannot recognize the difference between objects unevenly scaled along the line to the base point of an image. To overcome these limitations, an improved SNL algorithm was worked on. The improved algorithm is very robust, but it has /spl Theta/(n/sup 2/) time complexity. It is effective when used for a small database or for refining the search result of the original SNL algorithm.