Multimedia Retrieval Research Articles

One for more: Structured Multi-Modal Hashing for multiple multimedia retrieval tasks

Hashing has been widely studied in support of efficient multimedia retrieval due to its high computation and storage efficiency. However, existing multimedia hashing models are designed only for specific multimedia retrieval tasks, e.g., cross-modal retrieval and composite multi-modal retrieval. To simultaneously support multiple multimedia retrieval tasks, multiple different hashing models should be equipped in a multimedia retrieval system. This paper proposes a Structured Multi-Modal Hashing (SMMH) method, which can learn only a hash model for multiple multimedia retrieval tasks. Specifically, we generate structured hash codes by independently projecting the modality data into the Hamming space, where each part of hash codes preserves the specific characteristic of the specific modality data, and they are collaborated to represent the whole multi-modal data. To bridge the heterogeneous gap between multiple modalities, a fusion multi-modal space is constructed based on structured hash codes with label and pair-wise semantic supervision. Besides, we learn hash functions for composite multi-modal retrieval by projecting heterogeneous modalities into the collaborative multi-modal features. Finally, we develop a discrete hash optimization strategy to efficiently solve binary hash codes. Extensive experiments show that our model outperforms the strong baseline methods on multiple multimedia retrieval tasks. Source codes are available at https://github.com/ChaoqunZheng/SMMH.

Ascent Similarity Caching With Approximate Indexes

Similarity search is a key operation in multimedia retrieval systems and recommender systems, and it will play an important role also for future machine learning and augmented reality applications. When these systems need to serve large objects with tight delay constraints, edge servers close to the end-user can operate as similarity caches to speed up the retrieval. In this paper we present, a new similarity caching policy which improves on the state of the art by using (i) an (approximate) index for the whole catalog to decide which objects to serve locally and which to retrieve from the remote server, and (ii) a mirror ascent algorithm to update the set of local objects with strong guarantees even when the request process does not exhibit any statistical regularity.

Machine Learning in Pattern Recognition

Supervised or unsupervised classification is the main objective of pattern recognition. The statistical approach is the most popular approach that is practised among the several frameworks where pattern recognition is initially formulated. In the recent past, the neural network technique and the methodology scheme from the statistical learning theory have garnered the attention of people. It requires proper attention to deal with the design of the recognition system. There are several issues associated with the design of the recognition system. They are the pattern class definition, sensing environment and representation extraction and selection of features, cluster analysis, classifier design, learning, and choosing the training and test samples. There is no solution to the general issue of recognizing complex patterns associated with arbitrary patterns. Data mining, web searching, and retrieval of multimedia are the various emerging applications that require proper and effective regulation techniques. The main purpose of this paper is to give a detailed overview of the various methods that can be used in the different stages of the pattern recognition system. The paper also aims to figure out the research topics in the application that can be highlighted in this challenging field.

A tale of two interfaces: vitrivr at the lifelog search challenge

The past decades have seen an exponential growth in the amount of data which is produced by individuals. Smartphones which capture images, videos and sensor data have become commonplace, and wearables for fitness and health are growing in popularity. Lifelog retrieval systems aim to aid users in finding and exploring their personal history. We present two systems for lifelog retrieval: vitrivr and vitrivr-VR, which share a common retrieval model and backend for multi-modal multimedia retrieval. They differ in the user interface component, where vitrivr relies on a traditional desktop-based user interface and vitrivr-VR has a Virtual Reality user interface. Their effectiveness is evaluated at the Lifelog Search Challenge 2021, which offers an opportunity for interactive retrieval systems to compete with a focus on textual descriptions of past events. Our results show that the conventional user interface outperformed the VR user interface. However, the format of the evaluation campaign does not provide enough data for a thorough assessment and thus to make robust statements about the difference between the systems. Thus, we conclude by making suggestions for future interactive evaluation campaigns which would enable further insights.

Multimedia information retrieval using artificial neural network

<span lang="EN-US">The importance of the multimedia information retrieval (MIR) is highlighted by the extensive amount of the information on the internet. Image, audio, video, and text are all examples of the characteristics of the raw multimedia data. It is greatly challenging to represent a concept of human perception and how the machine-level language can grasp it (semantic gap of MIR). However, this paper aims to improve the information retrieval model that retrieves data from multimedia. This can be implemented by leveraging the use of variety of algorithms that go through training and testing to extract the model. One of these algorithms extracts text information based on the query language's nature as the vector space model (VSM) and the latent semantic index (LSI) were used. The other technique uses curvelet decomposition and statistic parameters like mean, standard deviation, and signal energy to recover these properties. Additionally, a discrete wavelet transforms (DWT) and signal characteristics-based method is used to retrieve audio signals. Finally, the neural network learning is modeled and trained on a collection of different multimedia images. The learned features have been utilized for presenting a highly sufficient system of multimedia retrieval which operates for a large set of multi-modal datasets. </span>

Deep debiased contrastive hashing

Hashing has achieved great success in multimedia retrieval due to its high computing efficiency and low storage cost. Recently, contrastive-learning-based hashing methods have achieved decent retrieval performance in label-free scenarios by learning distortion-invariant representations with Siamese networks. Their learning principle, i.e., instance discrimination, maximizes the correlation between self-augmented views and treats all others as negative samples. However, it may learn with false negative samples that are naturally similar, resulting in biased hash learning. To bridge this flaw, we reveal the between-instance similarity of naturally similar samples by exploring the latent structure of the training data. As a result, we propose the Deep Debiased Contrastive Hashing (DDCH) algorithm, using the neighborhood discovery module to explore the intrinsic similarity relationship that can help contrastive hashing reduce false negatives for superior discriminatory ability. Furthermore, we elucidate the rationale for incorporating the module into the contrastive hashing framework and explain our hashing process from an Expectation-Maximization (EM) perspective. Extensive experimental results on three benchmark image datasets demonstrate that DDCH significantly outperforms the state-of-the-art unsupervised hashing methods for image retrieval.

Comparing Interactive Retrieval Approaches at the Lifelog Search Challenge 2021

The Lifelog Search Challenge (LSC) is an interactive benchmarking evaluation workshop for lifelog retrieval systems. The challenge was first organised in 2018 aiming to find the system that can quickly retrieve relevant lifelog images for a given semantic query. This paper provides an analysis of the performance of all 17 systems participating in the 4th LSC workshop held at the 2021 Annual ACM International Conference on Multimedia Retrieval (ICMR). LSC’21 was the largest effort at comparing different approaches to interactive lifelog retrieval systems seen thus far. Findings from the challenge suggest that many different interactive factors contribute to the success (or otherwise) of participating teams. In this paper, we provide an overview of the LSC’21 challenge, introduce each team’s approach and explore these factors in depth and offer clues on how to develop a high-performing interactive lifelog search engine.

Unsupervised Cross-Modal Hashing via Semantic Text Mining

Cross-modal hashing has been widely used in multimedia retrieval tasks due to its fast retrieval speed and low storage cost. Recently, many deep unsupervised cross-modal hashing methods have been proposed to deal the unlabeled datasets. These methods usually construct an instance similarity matrix by fusing the image and text modality-specific similarity matrices as the guiding information to train the hashing networks. However, most of them directly use cosine similarities between the bag-of-words (BoW) vectors of text datapoints to define the text modality-specific similarity matrix, which fails to mine the semantic similarity information contained in the text modal datapoints and leads to the poor quality of the instance similarity matrix. To tackle the aforementioned problem, in this paper, we propose a novel Unsupervised Cross-modal Hashing via Semantic Text Mining, called UCHSTM. Specifically, UCHSTM first mines the correlations between the words of text datapoints. Then, UCHSTM constructs the text modality-specific similarity matrix for the training instances based on the mined correlations between their words. Next, UCHSTM fuses the image and text modality-specific similarity matrices as the final instance similarity matrix to guide the training of hashing model. Furthermore, during the process of training the hashing networks, a novel self-redefined-similarity loss is proposed to further correct some wrong defined similarities in the constructed instance similarity matrix, thereby further enhancing the retrieval performance. Extensive experiments on two widely used datasets show that the proposed UCHSTM outperforms state-of-the-art baselines on cross-modal retrieval tasks. We provide our source codes at: <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/rongchengtu1/UCHTIM.</uri>

Hash Bit Selection With Reinforcement Learning for Image Retrieval

In recent years, binary hashing methods have been widely used in large-scale multimedia retrieval because of the low computational complexity and memory cost. Generally, better retrieval accuracy can be achieved with a longer hash code, which, however, may suffer redundancy. In this paper, we propose a novel hash bit selection method, called Hash Bit Selection with Reinforcement Learning (HBS-RL), which aims to adaptively select the most informative bits from the database binary codes. In our approach, the hash bit selection problem is firstly modeled as a Markov Decision Process (MDP), which is solved with reinforcement learning. HBS-RL learns a policy for bit selection, which effectively identifies the most informative bits by directly maximizing mean Average Precision (mAP) during training. Specially, given a generated bit pool, our HBS-RL can sequentially select bits with different code lengths with a very lightweight fully-connected policy network. The proposed method is evaluated on the MNIST, CIFAR-10, ImageNet and NUS-WIDE datasets, and the results show that it significantly improves the retrieval performance of the existing unsupervised and deep supervised hashing methods. It also outperforms the state-of-the-art bit selection methods. For convenience of repeating our results, we release our source code at: <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/xyez/HBS-RL</uri> .

Partial Multi-Modal Hashing via Neighbor-Aware Completion Learning

Multi-modal hashing technology can support large-scale multimedia retrieval well, because of its fast query speed and low storage consumption. Although many multi-modal hashing methods have been developed in the literature, they still suffer from three important problems: 1) Most multi-modal hashing methods assume that multi-modal data are complete. This ideal assumption limits their application to practical retrieval scenarios, where the modality-missing is common. 2) Existing partial multi-modal hashing methods directly model incomplete multi-modal data for hash learning, which may result in partial multi-modal semantics in the learned hash codes. 3) Most of the methods are based on the shallow learning framework, which inevitably suffers from limited representation capability. To solve the above problems, we propose a flexible deep partial multi-modal hash learning framework, named Neighbor-aware Completion Hashing (NCH). Our framework jointly performs the cross-modal completion learning for incomplete multi-modal data and the multi-modal hash learning. It can not only support model training with incomplete multi-modal data but also handle incomplete multi-modal queries. Besides, we design a neighbor-aware completion learning module to capture neighbor semantics and generate distribution-consistent completed features. Finally, we conduct extensive experiments to evaluate our method on both fully-paired and partial multi-modal retrieval scenarios. The experimental results verify the superiority of our proposed method over the state-of-the-art baselines. The experimental codes of our method are released at: <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/FutureTwT/NCH.</uri>

Review on Human Activity Recognition for Military Restricted Areas

Abstract: Human Activity Recognition(HAR) is an active field of research and scientific development in which various models have been proposed using different methods for identification and categorization of activities using Machine Learning. HAR has reached a remarkable milestone in the area of computer vision. Except for applications in human-computer interactions, surveillance systems and robotics, lately it has extended its applicability in the fields of healthcare, multimedia retrieval, social networking, and education as well. It aims to bring latest technologies together to develop complex assistive system with adaptive capability and learning behaviour. HAR interprets human motion using computer and machine vision technologies to identify and detect simple and complex actions in real-world. This paper presents research made for surveillance of restricted military areas. Our scope is to develop a live monitoring system for tracking the illegal activities done in the restricted area for border security, which is an issue of concern since decades. In this we have introduced a deep learning model that learns to classify human actions without having prior knowledge. The features of image or video set are extracted and detected for detected for classifying whether the activity is illegal or not. Many harmful actions can be avoided or at least have their negative effects reduced as a result of the adoption of this concept. Finally, the activity recognition rate showed a good performance as a result of these findings

Deep continual hashing with gradient-aware memory for cross-modal retrieval

Cross-modal hashing (CMH) has become widely used for large-scale multimedia retrieval. However, most current CMH methods focus on the closed retrieval scenario, not the real-world environments, i.e., complex and changing semantics. When data containing new class objects emerge, the current CMH has to retrain the model on all history training data, not the new data, to accommodate new semantics, but the never-stop upload of data on the Internet makes this impractical. In this paper, we devise a deep hashing method called Continual Cross-Modal Hashing with Gradient Aware Memory (CCMH-GAM) for learning binary codes of multi-label cross-modal data with increasing categories. CCMH-GAM is a two-step hashing architecture, one hashing network learns to hash the increasing semantics of data, i.e., label, into the semantic codes, and other modality-specific hashing networks learn to map data into the corresponding semantic codes. Specifically, to keep the encoding ability for old semantics, a regularization based on accumulating low-storage label-code pairs is designed for the former network. For the modality-specific networks, we propose a memory construction method via approximating the full episodic gradients of all data by some exemplars and derive its fast implementation with the upper bound of approximation error. Based on this memory, we propose a gradient projection method to theoretically improve the probability of old data’s code being unchanged after updating the model. Extensive experiments on three datasets demonstrate that CCMH-GAM can continually learn hash functions and yield state-of-the-art retrieval performance.

Spatial-aware data partition for distributed memory parallelization of ANN search in multimedia retrieval

Content-based multimedia retrieval (CBMR) applications are becoming very popular in several online services which handles large volumes of data and are submitted to high query rates. While these applications may be complex, finding the nearest neighboring objects (multimedia descriptors) is typically their most time consuming operation. In order to address this problem, several recent works have proposed distributed memory parallelization of approximate nearest neighbors (ANN) search. These solutions employ a variety of ANN algorithms and different parallelization strategies. In this paper, we have identified the currently used parallelization strategies (Data Equal Split (DES) and Bucket Equal Split (BES)) and systematically evaluated their performance. We have also developed a framework to simplify the deployment of ANN algorithms in distributed memory machines with customized parallelization or data partition strategies. We further proposed a novel class of data partition/parallelization strategies that takes into account the data spatial proximity. Our approaches (SABES and SABES++) improves data locality and the system efficiency as compared to DES and BES. For instance, SABES++ achieved speedups of 4.2× and 1.8× on top of DES and BES, respectively, in the baseline case (40 nodes). Further, SABES and SABES++ also attained higher multi-node scalability and the gains vs DES and BES increase a larger number of nodes. SABES++ is 14.5× faster than DES when 160 nodes are used.

Fast unsupervised consistent and modality-specific hashing for multimedia retrieval

Fast unsupervised consistent and modality-specific hashing for multimedia retrieval

Unifying knowledge iterative dissemination and relational reconstruction network for image–text matching

Image–text matching is a crucial branch in multimedia retrieval which relies on learning inter-modal correspondences. Most existing methods focus on global or local correspondence and fail to explore fine-grained global–local alignment. Moreover, the issue of how to infer more accurate similarity scores remains unresolved. In this study, we propose a novel unifying knowledge iterative dissemination and relational reconstruction (KIDRR) network for image–text matching. Particularly, the knowledge graph iterative dissemination module is designed to iteratively broadcast global semantic knowledge, enabling relevant nodes to be associated, resulting in fine-grained intra-modal correlations and features. Hence, vector-based similarity representations are learned from multiple perspectives to model multi-level alignments comprehensively. The relation graph reconstruction module is further developed to enhance cross-modal correspondences by constructing similarity relation graphs and adaptively reconstructing them. We conducted experiments on the datasets Flickr30K and MSCOCO, which have 31,783 and 123,287 images, respectively. Experiments show that KIDRR achieves improvements of nearly 2.2% and 1.6% relative to Recall@1 on Flicr30K and MSCOCO, respectively, compared to the current state-of-the-art baselines.

Dual-Level Representation Enhancement on Characteristic and Context for Image-Text Retrieval

Image-text retrieval is a fundamental and vital task in multi-media retrieval and has received growing attention since it connects heterogeneous data. Previous methods that perform well on image-text retrieval mainly focus on the interaction between image regions and text words. But these approaches lack joint exploration of characteristics and contexts of regions and words, which will cause semantic confusion of similar objects and loss of contextual understanding. To address these issues, a dual-level representation enhancement network (DREN) is proposed to strength the characteristic and contextual representations by innovative block-level and instance-level representation enhancement modules, respectively. The block-level module focuses on mining the potential relations between multiple blocks within each instance representation, while the instance-level module concentrates on learning the contextual relations between different instances. To facilitate the accurate matching of image-text pairs, we propose the graph correlation inference and weighted adaptive filtering to conduct the local and global matching between image-text pairs. Extensive experiments on two challenging datasets (i.e., Flickr30K and MSCOCO) verify the superiority of our method for image-text retrieval.

Semi‐Paired Semi‐Supervised Deep Hashing for cross‐view retrieval

AbstractDue to its fast computational speed and low storage cost, hashing has been effectively applied to large‐scale multimedia retrieval tasks, such as medical video and security video retrieval. Most existing cross‐view hashing methods require good matching information, however, this exact pairing relationship is difficult to fully realise in practice. The association between views is incomplete, as is the label information. This task of missing paired and labelled information is very challenging, but less explored in research. In this study, a semi‐supervised semi‐paired deep hashing for large‐scale data is proposed, named Semi‐Paired Semi‐Supervised Deep Hashing (SPSDH) to solve this challenging task. SPSDH is a novel end‐to‐end deep neural network model with high‐order affinity. A non‐local higher‐order affinity measure that better considers the multimodal neighbourhood structure is proposed. A common representation to associate different modalities is introduced, which combined with the labelled information greatly maintains the consistency within the modalities. SPSDH is evaluated on three benchmark datasets for large‐scale cross‐view approximate nearest neighbour search and compared with several state‐of‐the‐art hashing methods. Extensive experimental results demonstrate the superior performance of our proposed SPSDH in semi‐supervised semi‐paired retrieval tasks.

Deep learning model based multimedia retrieval and its optimization in augmented reality applications.

With the uproar of touchless technology, the Virtual Continuum has seen some spark in the upcoming products. Today numerous gadgets support the use of Mixed Reality / Augmented Reality (AR)/ Virtual Reality. The Head Mounted Displays (HMDs) like that of Hololens, Google Lens, Jio Glass manifested reality into virtuality. Other than the HMDs many organizations tend to develop mobile AR applications to support umpteen number of industries like medicine, education, construction. Currently, the major issue lies in the performance parameters of these applications, while deploying for mobile application’s graphics performance, latency, and CPU functioning. Many industries pose real-time computation requirements in AR but do not implement an efficient algorithm in their frameworks. Offloading the computation of deep learning models involved in the application to the cloud servers will highly affect the processing parameters. For our use case, we will be using Multi-Task Cascaded Convolutional Neural Network (MTCNN) which is a modern tool for face detection, using a 3-stage neural network detector. Therefore, the optimization of communication between local application and cloud computing frameworks needs to be optimized. The proposed framework defines how the parameters involving the complete deployment of a mobile AR application can be optimized in terms of retrieval of multimedia, its processing, and augmentation of graphics, eventually enhancing the performance. To implement the proposed algorithm a mobile application is created in Unity3D. The mobile application virtually augments a 3D model of a skeleton on a target face. After the mentioned experimentation, it is found that average Media Retrieval Time (1.1471 μ s) and Client Time (1.1207 μ s) in the local application are extremely low than the average API process time (288.934ms). The highest time latency is achieved at the frame rate higher than 80fps.

Retrieval and Characteristic Analysis of Multimedia Tester Based on Bragg Equation

Abstract In order to solve the problem of Bragg equation, a research on retrieval and feature of multimedia testing machine is proposed. Firstly, this paper investigates the various stages of multimedia retrieval system and some mainstream multimedia retrieval algorithms, in order to cover all aspects of relevant algorithms. Secondly, on the basis of investigation, eight representative algorithms are selected. These algorithms cover three stages of multimedia retrieval system, including feature extraction stage, feature matching stage and geometric verification stage. Based on the above observed characteristics, finally, some relevant suggestions on architecture design and system optimization are put forward, including the design of single node processor, the design in parallel environment and the suggestions for simulation and evaluation using this test set.

Multimodal Based Audio-Visual Speech Recognition for Hard-of-Hearing: State of the Art Techniques and Challenges

Multimodal Integration (MI) is the study of merging the knowledge acquired by the nervous system using sensory modalities such as speech, vision, touch, and gesture. The applications of MI expand over the areas of Audio-Visual Speech Recognition (AVSR), Sign Language Recognition (SLR), Emotion Recognition (ER), Bio Metrics Applications (BMA), Affect Recognition (AR), Multimedia Retrieval (MR), etc. The fusion of modalities such as hand gestures- facial, lip- hand position, etc., are mainly used sensory modalities for the development of hearing-impaired multimodal systems. This paper encapsulates an overview of multimodal systems available within literature towards hearing impaired studies. This paper also discusses some of the studies related to hearing-impaired acoustic analysis. It is observed that very less algorithms have been developed for hearing impaired AVSR as compared to normal hearing. Thus, the study of audio-visual based speech recognition systems for the hearing impaired is highly demanded for the people who are trying to communicate with natively speaking languages. This paper also highlights the state-of-the-art techniques in AVSR and the challenges faced by the researchers for the development of AVSR systems.