Content-based Multimedia Retrieval Research Articles

Spatial-aware data partition for distributed memory parallelization of ANN search in multimedia retrieval

Content-based multimedia retrieval (CBMR) applications are becoming very popular in several online services which handles large volumes of data and are submitted to high query rates. While these applications may be complex, finding the nearest neighboring objects (multimedia descriptors) is typically their most time consuming operation. In order to address this problem, several recent works have proposed distributed memory parallelization of approximate nearest neighbors (ANN) search. These solutions employ a variety of ANN algorithms and different parallelization strategies. In this paper, we have identified the currently used parallelization strategies (Data Equal Split (DES) and Bucket Equal Split (BES)) and systematically evaluated their performance. We have also developed a framework to simplify the deployment of ANN algorithms in distributed memory machines with customized parallelization or data partition strategies. We further proposed a novel class of data partition/parallelization strategies that takes into account the data spatial proximity. Our approaches (SABES and SABES++) improves data locality and the system efficiency as compared to DES and BES. For instance, SABES++ achieved speedups of 4.2× and 1.8× on top of DES and BES, respectively, in the baseline case (40 nodes). Further, SABES and SABES++ also attained higher multi-node scalability and the gains vs DES and BES increase a larger number of nodes. SABES++ is 14.5× faster than DES when 160 nodes are used.

Analysis of User Personalized Retrieval of Multimedia Digital Archives Dependent on BP Neural Network Algorithm

This paper briefly introduces the characteristics of content-based multimedia retrieval under the information background, analyzes the implementation process of these technologies in the multimedia archives retrieval system including video and image information of digital archives, and points out that the content-based multimedia retrieval technology is bound to be organically combined with the traditional text retrieval methods. The information retrieval technologies in the past can only comply with the specific requirements of customers. Due to their characteristics of universality, they can hardly meet the demands of different environments, various purposes, and different times at the same time yet. Researchers have put forward personalized retrieval of multimedia files based on the BP neural network computing. In this way, the interest model of customers can be analyzed based on the characteristics of the different classification areas of users. Subsequently, the corresponding calculations are carried out, and the model is updated accordingly. Through the experiments, it is verified that the probability model put forward in this paper is the optimal solution to express the interest of customers and its changes.

Semantics-preserving hashing based on multi-scale fusion for cross-modal retrieval

Research on hash-based cross-modal retrieval has been a hotspot in the field of content-based multimedia retrieval research. Most deep cross-modal hashing methods only consider inter-modal loss that can remain local information of training data, and ignore the loss within data samples of the same modality that can remain the global information of dataset. In addition, they also ignore the factor that different scales of single modal data contain different semantic information, which affects the representation of data features. In this paper, we propose a semantics-preserving hashing method based on multi-scale fusion. More concretely, a multi-scale fusion pooling model is proposed for both image feature training network and text feature training network. Therefore, we can extract the multi-scale features of image dataset and solve the sparsity problem of text BOW vectors. When constructing the loss function, we consider intra-modal loss while considering inter-modal loss. Therefore, the output hash code retains both global and local underlying semantic correlation when image and text feature training network are trained. Experiment results on NUS-WIDE and MIRFlickr-25 K prove that against other existing methods, our algorithm improves cross-modal retrieval accuracy.

Descriptor Optimization for Semantic Concept Detection Using Visual Content

Concept detection has been considered a difficult problem and has attracted the interest of the content-based multimedia retrieval community. This detection implies an association between the concept and the visual content. In other words, the visual characteristics extracted from the video. This includes taking knowledge about the concept itself and its context. This work focuses on the problem of concept detection. For that, several stages are elaborated: first, a method of extraction and semi-automatic annotation of the video plans for the training set is proposed. This new method is based on the genetic algorithm. Then, a preliminary concept detection is carried out to generate the visual dictionary (BoVS). This second step is improved thanks to a noise reduction mechanism. This article's contribution has proven its effectiveness by testing it on a large dataset.

Unsupervised Video Hashing via Deep Neural Network

Hashing is a common solution for content-based multimedia retrieval by encoding high-dimensional feature vectors into short binary codes. Previous works mainly focus on image hashing problem. However, these methods can not be directly used for video hashing, as videos contain not only spatial structure within each frame, but also temporal correlation between successive frames. Several researchers proposed to handle this by encoding the extracted key frames, but these frame-based methods are time-consuming in real applications. Other researchers proposed to characterize the video by averaging the spatial features of frames and then the existing hashing methods can be adopted. Unfortunately, the sort of “video” features does not take the correlation between frames into consideration and may lead to the loss of the temporal information. Therefore, in this paper, we propose a novel unsupervised video hashing framework via deep neural network, which performs video hashing by incorporating the temporal structure as well as the conventional spatial structure. Specially, the spatial features of videos are obtained by utilizing convolutional neural network, and the temporal features are established via long-short term memory. After that, the time series pooling strategy is employed to obtain the single feature vector for each video. The obtained spatio-temporal feature can be applied to many existing unsupervised hashing methods. Experimental results on two real datasets indicate that by employing the spatio-temporal features, our hashing method significantly improves the performance of existing methods which only deploy the spatial features, and meanwhile obtains higher mean average precision compared with the state-of-the-art video hashing methods.

Prior-based probabilistic latent semantic analysis for multimedia retrieval

Topic models have shown to be one of the most effective tools in Content-Based Multimedia Retrieval (CBMR). However, the high computational learning cost together with the huge expansion of multimedia collections limit the scalability of topic-based CBMR systems in real-life multimedia applications. The present work pursues a twofold objective. On the one hand, to study the effect of using clustering-based document reduction schemes over standard topic models pLSA (probabilistic Latent Semantic Analysis) and LDA (Latent Dirichlet Allocation). On the other hand, to develop a pLSA-based extension oriented to integrate this reduction scheme within the own model in order to improve the CBMR effectiveness. The experimental part of the work includes three different multimedia databases, three ranking functions, four retrieval scenarios, three different numbers of topics and ten document reduction levels. Experiments revealed that standard topic models are highly sensitive to the document reduction level whereas the proposed model is able to provide a competitive advantage within the content-based retrieval field.

Retrieval From and Understanding of Large-Scale Multi-modal Medical Datasets: A Review

Content-based multimedia retrieval (CBMR) has been an active research domain since the mid 1990s. In medicine visual retrieval started later and has mostly remained a research instrument and less a clinical tool. The limited size of data sets due to privacy constraints is often mentioned as reason for these limitations. Nevertheless, much work has been done in CBMR, including the availability of increasingly large data sets and scientific challenges. Annotated data sets and clinical data for images have now become available and can be combined for multi-modal retrieval. Much has been learned on user behavior and application scenarios. This text is motivated by the advances in medical image analysis and the availability of public large data sets that often include clinical data. It is a systematic review of recent work (concentrating on the period 2011–2017) on multi-modal CBMR and image understanding in the medical domain, where image understanding includes techniques such as detection, localization, and classification for leveraging visual content. With the objective of summarizing the current state of research for multimedia researchers outside the medical field, the text provides ways to get data sets and identifies current limitations and promising research directions. The text highlights advances in the past six years and a trend to use larger scale training data and deep learning approaches that can replace/complement hand-crafted features. Using images alone will likely only work in limited domains but combining multiple sources of data for multi-modal retrieval has the biggest chances of success, particularly for clinical impact.

Multilinear Kernel Mapping for Feature Dimension Reduction in Content Based Multimedia Retrieval System

In the process of content-based multimedia retrieval, multimedia information is processed in order to obtain descriptive features. Descriptive representation of features, results in a huge feature count, which results in processing overhead. To reduce this descriptive feature overhead, various approaches have been used to dimensional reduction, among which PCA and LDA are the most used methods. However, these methods do not reflect the significance of feature content in terms of inter-relation among all dataset features. To achieve a dimension reduction based on histogram transformation, features with low significance can be eliminated. In this paper, we propose a feature dimensional reduction approaches to achieve the dimension reduction approach based on a multi-linear kernel (MLK) modeling. A benchmark dataset for the experimental work is taken and the proposed work is observed to be improved in analysis in comparison to the conventional system.

Hybrid operations for content-based Vietnamese agricultural multimedia information retrieval

Content-based multimedia information retrieval is never a trivial task even with state-of-the-art approaches. Its mandatory challenge, called “semantic gap,” requires much more understanding of the way human perceive things (i.e., visual and auditory information). Computer scientists have spent thousands of hours seeking optimal solutions, only ended up falling in the bound of this gap for both visual and spoken contexts. While an over-the-gap approach is unreachable, we insist on assembling current viable techniques from both contexts, aligned with a domain concept base (i.e., an ontology), to construct an info service for the retrieval of agricultural multimedia information. The development process spans over three packages: (1) building a Vietnamese agricultural thesaurus; (2) crafting a visual-auditory intertwined search engine; and (3) system deployment as an info service. We spring our the thesaurus in 2 sub-boughs: the aquaculture ontology consists of 3455 concepts and 5396 terms, with 28 relationships, covering about 2200 fish species and their related terms; and the plant production ontology comprises of 3437 concepts and 6874 terms, with 5 relationships, covering farming, plant production, pests, etc. These ontologies serve as a global linkage between keywords, visual, and spoken features, as well as providing the reinforcement for the system performances (e.g., through query expansion, knowledge indexing…). On the other hand, constructing a visual-auditory intertwined search engine is a bit trickier. Automatic transcriptions of audio channels are marked as the anchor points for the collection of visual features. These features, in turn, got clustered based on the referenced thesauri, and ultimately tracking out missing info induced by the speech recognizer’s word error rates. This compensation technique bought us back 14 % of loss recall and an increase of 9 % accuracy over the baseline system. Finally, wrapping the retrieval system as an info service guarantees its practical deployment, asour target audiences are the majority of farmers in developing countries who are unable to reach modern farming information and knowledge.

A Study of Content based Multimedia Retrieval Systems

A Study of Content based Multimedia Retrieval Systems

Content based Multimedia Retrieval using Automatic Speech Recognition

Our system deals with retrieval of suitable video based on the user input and voice commands. The microphone takes user voice as input and processes it to convert it into the text. It further checks the repository database of videos to find a video that matches the spoken keyword by the user. A list of keywords is available for the user in order to aid him in searching and querying process. After the relevant video is played as per user’s wish, it is further given an option if the user wants to navigate to particular topic within the video. The system can do the same for the user. Our system basically encompasses and covers application in lecture video domain. The videos in system are based on a wide variety of topics. In general, navigation in videos is too time consuming as it performed by trial and error. However, with the help of this system, searching becomes faster and response time increases. Proper indexed query handling in database makes navigation easier and efficient. The system is now restricted to lecture videos but can be extended to various different domains too in industry. General Terms Fast retrieval, content based retrieval, repository of videos

A Model of Media Information Management for Content Retrieval

The rapid development of digital library makes the multimedia information resources organization and management become a key issue to develop the digital library, and the multimedia information retrieval technology has become the core technology of the digital book data management, the existing multimedia retrieval model cannot effectively implement multimedia digital books data management. Therefore, this paper proposes a content-based multimedia retrieval data model and combined with the different modal data abstraction and decomposition classification thought, discusses the significance of this data model, also briefly introduces the building method of this model. Finally by combining this design it implements a multimedia digital books data retrieval system.

Granular modeling and computing approaches for intelligent analysis of non-geometric data

Data analysis techniques have been traditionally conceived to cope with data described in terms of numeric vectors. The reason behind this fact is that numeric vectors have a well-defined and clear geometric interpretation, which facilitates the analysis from the mathematical viewpoint. However, the state-of-the-art research on current topics of fundamental importance, such as smart grids, networks of dynamical systems, biochemical and biophysical systems, intelligent trading systems, multimedia content-based retrieval systems, and social networks analysis, deal with structured and non-conventional information characterizing the data, providing richer and hence more complex patterns to be analyzed. As a consequence, representing patterns by complex (relational) structures and defining suitable, usually non-metric, dissimilarity measures is becoming a consolidated practice in related fields. However, as the data sources become more complex, the capability of judging over the data quality (or reliability) and related interpretability issues can be seriously compromised. For this purpose, automated methods able to synthesize relevant information, and at the same time rigorously describe the uncertainty in the available datasets, are very important: information granulation is the key aspect in the analysis of complex data. In this paper, we discuss our general viewpoint on the adoption of information granulation techniques in the general context of soft computing and pattern recognition, conceived as a fundamental approach towards the challenging problem of automatic modeling of complex systems. We focus on the specific setting of processing the so-called non-geometric data, which diverges significantly from what has been done so far in the related literature. We highlight the motivations, the founding concepts, and finally we provide the high-level conceptualization of the proposed data analysis framework.

An automatic clustering algorithm for probability density functions

We propose an intuitive and computationally simple algorithm for clustering the probability density functions (pdfs). A data-driven learning mechanism is incorporated in the algorithm in order to determine the suitable widths of the clusters. The clustering results prove that the proposed algorithm is able to automatically group the pdfs and provide the optimal cluster number without any a priori information. The performance study also shows that the proposed algorithm is more efficient than existing ones. In addition, the clustering can serve as the intermediate compression tool in content-based multimedia retrieval that we apply the proposed algorithm to categorize a subset of COREL image database. And the clustering results indicate that the proposed algorithm performs well in colour image categorization.

Enhance the Technique of Relevance Feedback for Content-based Multimedia Retrieval by using Mining Algorithm

Today images, multimedia are immensely important in information retrieval system. In existing relevance feedback technique , there is semantic gap between high level concepts and low level features of images as well as videos, another drawback is according to user requirement we cannot retrieve relevant multimedia data (images, videos) from multimedia database and image database. To overcome from this drawback in Content based Multimedia Retrieval (CBMR), using navigation pattern relevance feedback technique to retrieve most relevant videos, images from multimedia data according to user requirement. To provide efficient and effective retrieval of content based multimedia data and images from multimedia database like video data, images by using relevance feedback technique and mining algorithm. General Terms Data mining algorithm: K-means, Apriori algorithm.

CoBITs: a distributed indexing approach to collaborative content-based multimedia retrieval across digital archives

There's more and more precious content digitized in digital archives especially for cultural heritage. It could cost much effort in digitization and archiving. To meet the requirements in a digital archiving system, several issues must be addressed. First, it usually requires resources such as computation and storage for each individual digital archive to maintain its own service. Second, the archived content would be more useful if they can be easily utilized in providing services such as searching across multiple archives. Current approaches usually adopt metadata harvesting that would build a centralized index from separate digital libraries. They usually suffer from the problem of metadata inconsistency. In this paper, we propose a distributed indexing approach to collaborative content-based multimedia retrieval across digital archives. To reduce the loads in each archive, we dynamically distribute the tasks of crawling, indexing, and query processing depending on the response time. Distributed crawler-based approach can simplify the design of indexing and query processing steps by maintaining the data to be indexed local to the machine for crawling. It can facilitate efficient archiving and indexing by automatically following the link structure of contents published on the Web. Also, it enables simpler implementation and easier support for cross-archive applications such as search and copy detection. Experimental results show the potential of the proposed approach in load balancing with appropriate task distribution.

Personalized Interactive Storyboarding utilizing Content Based Multimedia Retrieval

Abstract An explosion of digital multimedia technologies that permit quick and easy uploading of any multimedia file to the web, coupled with the rapid advancement of hardware and software have resulted in petabytes of multimedia data being available on the internet. In this paper, we provide a framework that rapidly integrates these data for personalized storyboarding utilizing Content Based Multimedia Retrieval (CBMR) techniques. A storyboard is a narrative of audio, text, video and images of a particular topic that are linked systematically. People have different abilities to understand a story/topic according to their learning styles coupled with experience, age, knowledge, gender etc. Personalized storyboard multimedia learning help the user dictate the learning process in a creative way through multiple means of representation, expression and engagement. This process involves active learning, both behavioral as well as cognitive. Learning is constructive, and information learned is remembered at a deeper level. The use of multimedia promotes meaningful learning that can be transferred or generalized to other situations. In this paper, various learning styles are discussed. The goal of the research was to design and implement a novel approach that integrates human reasoning with computerized algorithms for multimedia storyboarding for learning. By systematically coupling human reasoning and computerized process, we attempt to minimize the barrier between the human's cognitive model of what they are trying to storyboard and the computers understanding of the user's task. We present our model which utilizes a fusion of content based image retrieval, content based video retrieval, content based audio retrieval, and text based retrieval techniques with human computer interaction based relevance feedback to enhance the learning process through personalized multimedia storyboarding. To illustrate the advantages of this model in a greater detail, we showcased the concepts of this model utilizing the game of cricket. We summarize the paper by discussing the future in this area.

Rule-Based Semantic Concept Classification from Large-Scale Video Collections

The explosive growth and increasing complexity of the multimedia data have created a high demand of multimedia services and applications in various areas so that people can access and distribute the data easily. Unfortunately, traditional keyword-based information retrieval is no longer suitable. Instead, multimedia data mining and content-based multimedia information retrieval have become the key technologies in modern societies. Among many data mining techniques, association rule mining (ARM) is considered one of the most popular approaches to extract useful information from multimedia data in terms of relationships between variables. In this paper, a novel rule-based semantic concept classification framework using weighted association rule mining (WARM), capturing the significance degrees of the feature-value pairs to improve the applicability of ARM, is proposed to deal with major issues and challenges in large-scale video semantic concept classification. Unlike traditional ARM that the rules are generated by frequency count and the items existing in one rule are equally important, our proposed WARM algorithm utilizes multiple correspondence analysis (MCA) to explore the relationships among features and concepts and to signify different contributions of the features in rule generation. To the authors best knowledge, this is one of the first WARM-based classifiers in the field of multimedia concept retrieval. The experimental results on the benchmark TRECVID data demonstrate that the proposed framework is able to handle large-scale and imbalanced video data with promising classification and retrieval performance.

Combining CPU and GPU architectures for fast similarity search

The Signature Quadratic Form Distance on feature signatures represents a flexible distance-based similarity model for effective content-based multimedia retrieval. Although metric indexing approaches are able to speed up query processing by two orders of magnitude, their applicability to large-scale multimedia databases containing billions of images is still a challenging issue. In this paper, we propose a parallel approach that balances the utilization of CPU and many-core GPUs for efficient similarity search with the Signature Quadratic Form Distance. In particular, we show how to process multiple distance computations and other parts of the search procedure in parallel, achieving maximal performance of the combined CPU/GPU system. The experimental evaluation demonstrates that our approach implemented on a common workstation with 2 GPU cards outperforms traditional parallel implementation on a high-end 48-core NUMA server in terms of efficiency almost by an order of magnitude. If we consider also the price of the high-end server that is ten times higher than that of the GPU workstation then, based on price/performance ratio, the GPU-based similarity search beats the CPU-based solution by almost two orders of magnitude. Although proposed for the SQFD, our approach of fast GPU-based similarity search is applicable for any distance function that is efficiently parallelizable in the SIMT execution model.

A Multimedia Retrieval Framework Based on Semi-Supervised Ranking and Relevance Feedback

We present a new framework for multimedia content analysis and retrieval which consists of two independent algorithms. First, we propose a new semi-supervised algorithm called ranking with Local Regression and Global Alignment (LRGA) to learn a robust Laplacian matrix for data ranking. In LRGA, for each data point, a local linear regression model is used to predict the ranking scores of its neighboring points. A unified objective function is then proposed to globally align the local models from all the data points so that an optimal ranking score can be assigned to each data point. Second, we propose a semi-supervised long-term Relevance Feedback (RF) algorithm to refine the multimedia data representation. The proposed long-term RF algorithm utilizes both the multimedia data distribution in multimedia feature space and the history RF information provided by users. A trace ratio optimization problem is then formulated and solved by an efficient algorithm. The algorithms have been applied to several content-based multimedia retrieval applications, including cross-media retrieval, image retrieval, and 3D motion/pose data retrieval. Comprehensive experiments on four data sets have demonstrated its advantages in precision, robustness, scalability, and computational efficiency.