Multimedia Retrieval Research Articles

Performance Evaluation in Multimedia Retrieval

Performance evaluation in multimedia retrieval, as in the information retrieval domain at large, relies heavily on retrieval experiments, employing a broad range of techniques and metrics. These can involve human-in-the-loop and machine-only settings for the retrieval process itself and the subsequent verification of results. Such experiments can be elaborate and use-case-specific, which can make them difficult to compare or replicate. In this paper, we present a formal model to express all relevant aspects of such retrieval experiments, as well as a flexible open-source evaluation infrastructure that implements the model. These contributions intend to make a step towards lowering the hurdles for conducting retrieval experiments and improving their reproducibility.

Application of Multimedia Information Retrieval Technology in Japanese Text Content Information Query Platform

The research and development of Japanese operating platform has become a new industry. At the same time, it is necessary to strengthen the research on query and feedback technology in MIR(Multimedia Information Retrieval). Based on this, this paper introduces the research status and progress of query and feedback technology in multimedia retrieval, and introduces the overall framework and technical links of MIR. This paper discusses the application of MIR technology in Japanese text content information query platform. A new multimodal retrieval mechanism is proposed, which combines cross-reference index and underlying features in retrieval and related feedback to realize unified processing of multimodal data. The time consumption of the algorithm is lower than that of the two comparative algorithms, which also verifies the superiority. It is feasible to apply the MIR technology proposed in this paper to the Japanese text content information query platform. It provides a new path for Japanese text content information retrieval.

Spatial-Aware and Load Balancing Distributed Data Partitioning Strategies for Content-Based Multimedia Retrieval.

Content-Based Multimedia Retrieval (CBMR) has become very popular in several applications, driven by the growing routine use of multimedia data. Since the datasets used in real-world applications are very large and descriptor's dimensionality is high, querying is an expensive, albeit important functionality. Further, exact search is prohibitive in most cases, motivating the use of Approximate Nearest Neighbour Search (ANNS) algorithms, trading accuracy for performance. These have been mainly developed targeting a sequential execution in a single node. However, the large and increasing datasets used and the high query loads submitted to those systems typically surpass the memory and computing resources available in a single node. This motivated the development of parallel distributed memory ANNS solutions to meet the computing capabilities required by those applications. A common problem that must be handled when using distributed memory systems is data partitioning and its impact on load imbalance. Several data partitioning approaches have already been proposed, including elaborated spatial-aware strategies. However, little effort has been put into carefully analyzing the performance of those strategies at scale. Here, we evaluated the commonly used data partitioning strategies in ANNS and identified their limitations to propose a novel class of partitioning algorithms that can minimize load imbalance while improving data locality to attain high performance on the distributed memory search. Experimentally, we found that our proposed algorithms (SABBS and SABBSR) improved search performance by up to 1.64× compared to the best previous solution. In a distributed memory weak scaling evaluation, with up to 12 billion 128-dimensional descriptors and 60 compute nodes, the gains were maintained as the system scaled with our novel approaches. These results demonstrate the efficiency of our new algorithms for billion-scale ANNS and the importance of considering not only data locality but also data and load imbalance in the data partitioning.

Privacy-Enhanced Prototype-Based Federated Cross-Modal Hashing for Cross-Modal Retrieval

Cross-modal hashing is widely used for efficient similarity searches, improving data processing efficiency, and reducing storage costs. Existing cross-modal hashing methods primarily focus on centralized training scenarios, where fixed-scale and fixed-category multi-modal data is collected beforehand. However, these methods often face challenges associated with the potential risk of privacy breaches and high data communication costs during data transmission in real-world multimedia retrieval tasks. To tackle these challenges, in this article, we propose an efficient privacy-enhanced prototype-based federated cross-modal hashing (PEPFCH). In PEPFCH, we integrate local and global prototypes in order to effectively capture the distinctive traits of individual clients, while also harnessing the collective intelligence of the entire federated learning system. Moreover, to ensure the security of prototype information and prevent its disclosure during the aggregation process, we use a prototype encryption transmission mechanism to encrypt the prototype information before transmission, making it challenging for attackers to gain access to sensitive data. Additionally, to facilitate personalized federated learning and alleviate the issue of parametric catastrophic forgetting, we establish the image and text hyper-networks for each client and adopt a hyper-network extension strategy to selectively preserve and update previously acquired knowledge when acquiring new concepts or categories. Comprehensive experiments highlight the efficiency and superiority of our proposed method. To enhance research and accessibility, we have publicly released our source codes at: https://github.com/vindahi/PEPFCH .

VEDANet: A dense blocked network for visual emotion analysis in multimedia retrieval

VEDANet: A dense blocked network for visual emotion analysis in multimedia retrieval

An Efficient Search on Cloud Images with Blockchain Technology

A key task in computer vision is image retrieval, which has wide-ranging applications across multiple domains. Using query picture features, this abstract proposes an image retrieval approach that focuses on the widely used Scale-Invariant Feature Transform (SIFT) algorithm for feature extraction and distance calculation. The suggested method starts by extracting SIFT features from a set of photos, building a keypoints and descriptor database. By capturing the unique qualities of nearby image regions, these features enable reliable matching and retrieval. The security of private cloud data, which includes query queries, the search tree, and outsourced photographs, is another major issue. Use a feature extraction method first for integrated picture features, which are composed of fundamental components like colour and shape. In particular, because the proposed method uses a balancing index tree, it can achieve logarithmic search time. Second, the picture and query feature are encrypted using the secure inner product. Include a system for determining duplicate picture material as well. When given a query image, SIFT feature extraction is applied to it, producing keypoints and descriptors that correspond to its visual characteristics. Next, a distance computation method like Euclidean distance is used to compare the features of the database photos with the query image features. The similarity between the query image and the database images is measured through this comparison. The retrieved photos are ranked in order of similarity to the query image based on the calculated distances. Search results with lesser distances between images are deemed more similar and are displayed first. To locate visually related images in huge databases, the image retrieval system that uses SIFT feature extraction and distance calculation provides a reliable and effective solution. It contributes to developments in multimedia retrieval, visual analytics, and picture understanding by enabling applications like content-based image search, image recommendation systems, and image clustering.

Research on Wireless Multimedia Big Data Transmission Model Based on Non-linear Algebraic Equation with the Adaptive Scheduling Algorithm

With the rapid development of computer technology, image processing and automatic book recognition has become a very important and practical field, and this discipline has a broad prospect and great potential. The digital vision system is a multi-plane information analysis method based on the human brain, which is applied to library identification problems by studying several regular changes in the distribution of objects themselves in two-dimensional space and three-dimensional position, in order to improve book quality and reduce costs. In addition, digital reading devices have become an essential part of people's daily learning life, and multimedia retrieval methods have also undergone radical changes, image processing is becoming more and more complex, and algorithms must become more efficient and reliable as the amount of computing increases to a certain level, so as to adapt to this fast and powerful data storage and query needs. Therefore, the trend is to improve the speed and reliability of computer systems. In this research, we provide a nonlinear model of optimal allocation employing an ant colony management strategy, a genetic operator, and an adaptive scheduling algorithm based on topology (dB) to discover the best scheduling scheme and test its feasibility via simulation. It proposes a viable idea for tackling the practical wireless multimedia massive data transmission model problem by applying the method to the real-time processing process in other sectors with high importance and resilience.

Research on wireless multimedia big data transmission model based on non-linear algebraic equation with the adaptive scheduling algorithm

With the rapid development of computer technology, image processing and automatic book recognition has become a very important and practical field, and this discipline has a broad prospect and great potential. The digital visión system is a multi-plane information análisis method basad on the human brain, which is applied to library identification problems by studying several regular changes in the distribution of objects themselves in two-dimensional space and three-dimensional position, in order to improve book quality and reduce costs. In addition, digital Reading devices have become an essential part of people’s daily learning life, and multimedia retrieval methods have also undergone radical changes, image processing is becoming more and more complex, and algorithms must become more efficient and reliable as the amount of computing increases to a certain level, so as to adapt to this fast and powerful data storage and query needs. Therefore, the tren is to improve the speed and reliability of computer systems. This paper discusses and proposes measures for the analysis and extraction, classification and sorting of library images.

TTL model for an LRU-based similarity caching policy

Similarity caching allows requests for an item to be served by a similar item. Applications include recommendation systems, multimedia retrieval, and machine learning. Recently, many similarity caching policies have been proposed, like SIM-LRU and its generalization RND-LRU, but the performance analysis of their hit ratio is still wanting. In this paper, we show how to extend the popular time-to-live approximation in classic caching to similarity caching. In particular, we propose a method to estimate the hit ratio of the similarity caching policy RND-LRU. Our method, the RND-TTL approximation, introduces the RND-TTL cache model and then tunes its parameters in such a way as to mimic the behavior of RND-LRU. The parameter tuning involves solving a fixed point system of equations for which we provide an algorithm for numerical resolution and sufficient conditions for its convergence. Our approach for approximating the hit ratio of RND-LRU is evaluated on both synthetic and real-world traces.

LCEMH: Label Correlation Enhanced Multi-modal Hashing for efficient multi-modal retrieval

Supervised multi-modal hashing can effectively improve the discriminative power of hash codes by leveraging semantic label information. However, most existing supervised multi-modal hashing models generally rely on discrete labels for supervision. They only provide limited guidance and may not fully capture the potential semantic similarity between multi-modal data. Moreover, the binary constraints of hash codes further restrict their ability to fully capture the rich semantic information embedded in the labels. To address these limitations, we propose a Label Correlation Enhanced Multi-modal Hashing (LCEMH) approach for efficient multi-modal retrieval. Specifically, our proposed model can effectively expand the inter-class boundaries of discrete labels to further enrich the supervision of similarity. Additionally, we directly transform the acquired richer label information into hash codes simply and effectively, thus retaining more comprehensive information in the labels with minimal quantization loss, and enhancing the discriminant ability of the generated hash codes. Experimental results on three public multimedia retrieval datasets demonstrate the superiority of LCEMH in performance. The source code of LCEMH is available online at: https://github.com/ChaoqunZheng/LCEMH.

Multi-Modal Hashing for Efficient Multimedia Retrieval: A Survey

With the explosive growth of multimedia contents, multimedia retrieval is facing unprecedented challenges on both storage cost and retrieval speed. Hashing technique can project the high-dimensional data into compact binary hash codes. With it, the most time-consuming semantic similarity computation during the multimedia retrieval process can be significantly accelerated with fast Hamming distance computation, and meanwhile the storage cost can be reduced greatly by the binary embedding. In the light of this, multi-modal hashing has recently received considerable attention to support large-scale multimedia retrieval. Different from uni-modal hashing, the multi-modal hashing focuses on modeling the multi-modal semantics and further preserving them into binary hash codes with hash learning. In this paper, we first systematically review the existing learning to hash methods for efficient multimedia retrieval, categorizing them according to the multimedia retrieval tasks, the specific multi-modal semantic modeling techniques, and hash learning strategies. Thereafter, we present the performance comparison results. We ultimately discuss the challenges and potential research directions that may require further investigation in multi-modal hash learning. To facilitate the research on multi-modal hashing, we develop an open-source performance comparison tool at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/BMC-SDNU/Hashing-Retrieval</uri> .

Deep Adaptive Quadruplet Hashing With Probability Sampling for Large-Scale Image Retrieval

With the preferable efficiency in storage and computation, hashing has shown potential application in large-scale multimedia retrieval. Compared with traditional hashing algorithms using hand-crafted characteristics, deep hashing inherits the representational capacity of deep neural networks to jointly learn semantic features and hash functions, encoding raw data into compact binary codes with significant discrimination. Generally, most of the current multi-wise hashing methods view the similarity margins between image pairs as constant values in training process. When the distance between sample pairs exceeds the fixed margin, the hashing network would not learn anything. Besides, available hashing methods commonly introduce the random sampling strategy to build training batches and ignore the sample distribution, which is harmful to parameter optimization. In this paper, we propose a novel Deep Adaptive Quadruplet Hashing with probability sampling (DAQH) for discriminative binary code learning. Specifically, with exploring the distribution relationship of raw samples, a non-uniform probability sampling strategy is proposed to build more informative and representative training batches, while maintaining the diversity of training samples. By introducing the prior similarity of sample pairs to calculate corresponding margins, an adaptive margin quadruplet loss is designed to dynamically preserve the underlying semantic relationships with its neighbors. To tune the attributes of binary codes, by combining quadruple regularization and orthogonality optimization, binary code constraint is developed to make the learned embedding with significant discrimination. Extensive experimental results on various benchmark datasets demonstrate our proposed DAQH framework achieves state-of-the-art visual similarity search performance.

Fast metric multi-view hashing for multimedia retrieval

The acquisition of multi-view hash representation for heterogeneous data holds paramount importance in the domain of multimedia retrieval. The limited retrieval precision observed in current approaches stems from their inadequate integration of multi-view features and their failure to effectively leverage the metric information available from diverse samples. Commonly employed fusion methods, such as concatenation or weighted sum, are insufficient in capturing the complementarity among multiple view features. Furthermore, these methods neglect the valuable information contributed by dissimilar samples. To address these challenges, we propose an innovative method termed Fast Metric Multi-View Hashing (FMMVH). Our approach showcases the superiority of gate-based fusion over traditional methods, as substantiated by extensive empirical evidence. Additionally, this paper proposes a novel deep metric loss function to enable the utilization of metric information from dissimilar samples. We exclusively train our method using this single loss function. To enhance practical applicability in industrial production environments, we employ model compression techniques to optimize the proposed method. On benchmark datasets such as MIR-Flickr25K, NUS-WIDE, and MS COCO, the performance of our FMMVH method significantly surpasses that of existing state-of-the-art methods, demonstrating improvements of up to 7.47% in mean Average Precision (mAP).

Development and research of deep neural network fusion computer vision technology

Abstract Deep learning (DL) has revolutionized advanced digital picture processing, enabling significant advancements in computer vision (CV). However, it is important to note that older CV techniques, developed prior to the emergence of DL, still hold value and relevance. Particularly in the realm of more complex, three-dimensional (3D) data such as video and 3D models, CV and multimedia retrieval remain at the forefront of technological advancements. We provide critical insights into the progress made in developing higher-dimensional qualities through the application of DL, and also discuss the advantages and strategies employed in DL. With the widespread use of 3D sensor data and 3D modeling, the analysis and representation of the world in three dimensions have become commonplace. This progress has been facilitated by the development of additional sensors, driven by advancements in areas such as 3D gaming and self-driving vehicles. These advancements have enabled researchers to create feature description models that surpass traditional two-dimensional approaches. This study reveals the current state of advanced digital picture processing, highlighting the role of DL in pushing the boundaries of CV and multimedia retrieval in handling complex, 3D data.

Adaptive semi-paired query hashing for multi-modal retrieval

Multi-modal hashing has attracted enormous attention in large-scale multimedia retrieval, owing to its advantages of low storage cost and fast Hamming distance computation. Existing multi-modal hashing methods assume that all multi-modal data are well paired and then encode the paired multiple modalities into joint binary codes. However, it is not ensured that all data are fully paired in practical applications. In this paper, we present an adaptive semi-paired query hashing method, which facilitates learning the hash codes for semi-paired query samples. The proposed method performs projection learning and cross-modal reconstruction learning to maintain the semantic consistency between multi-modal data. Meanwhile, the semantic similarity structure and the complementary multi-modal information are preserved by hash codes to obtain a discriminative hash function. In the encoding stage, the missing modality features are completed via the learned cross-modal reconstruction matrices. In addition, the multimodal fusion weights are fine-tuned adaptively for the new query data to capture the modality difference. The extensive experiment results on three benchmark datasets show that our proposed algorithm outperforms state-of-the-art multi-modal hashing methods.

Efficient Supervised Graph Embedding Hashing for large-scale cross-media retrieval

Recently, graph based hashing has gained much attention due to its effectiveness in multi-media retrieval. Although several graph embedding based works have been designed and achieved promising performance, there are still some issues that need to be further studied, including, (1) one significant drawback of graph embedding is its expensive memory and computation cost caused by the graph Laplacian matrix; (2) most pioneer works fail to fully explore the available class labels in training procedure, which generally makes them suffer from unsatisfactory retrieval performance. To overcome these drawbacks, we propose a simple yet effective supervised cross-media hashing scheme, termed Efficient Supervised Graph Embedding Hashing (ESGEH), which can simultaneously learn hash functions and discrete binary codes efficiently. Specifically, ESGEH leverages both class label based semantic embedding and graph embedding to generate a sharing semantic subspace, and class labels are also incorporated to minimize the quantization error for better approximating the generated binary codes. In order to reduce the computational sources, a well-designed intermediate terms decomposition is proposed to avoid explicitly computing the graph Laplacian matrix. Finally, an iterative discrete optimal algorithm is derived to solve above problem, and each subproblem can yield a closed-form solution. Extensive experimental results on four public datasets demonstrate the superiority of the proposed approach over several existing cross-media hashing methods in terms of both accuracy and efficiency.

Time frequency domain deep CNN for automatic background classification in speech signals

Many application areas, such as background identification, predictive maintenance in industrial applications, smart home applications, assisting deaf people with their daily activities and indexing and retrieval of content-based multimedia, etc., use automatic background classification using speech signals. It is challenging to predict the background environment accurately from speech signal information. Thus, a novel synchrosqueezed wavelet transform (SWT)-based deep learning (DL) approach is proposed in this paper for automatically classifying background information embedded in speech signals. Here, SWT is incorporated to obtain the time-frequency plot from the speech signals. These time-frequency signals are then fed to a deep convolutional neural network (DCNN) to classify background information embedded in speech signals. The proposed DCNN model consists of three convolution layers, one batch-normalization layer, three max-pooling layers, one dropout layer, and one fully connected layer. The proposed method is tested using various background signals embedded in speech signals, such as airport, airplane, drone, street, babble, car, helicopter, exhibition, station, restaurant, and train sounds. According to the results, the proposed SWT-based DCNN approach has an overall classification accuracy of 97.96 (± 0.53)% to classify background information embedded in speech signals. Finally, the performance of the proposed approach is compared to the existing methods.

Gesture retrieval and its application to the study of multimodal communication

Comprehending communication is dependent on analyzing the different modalities of conversation, including audio, visual, and others. This is a natural process for humans, but in digital libraries, where preservation and dissemination of digital information are crucial, it is a complex task. A rich conversational model, encompassing all modalities and their co-occurrences, is required to effectively analyze and interact with digital information. Currently, the analysis of co-speech gestures in videos is done through manual annotation by linguistic experts based on textual searches. However, this approach is limited and does not fully utilize the visual modality of gestures. This paper proposes a visual gesture retrieval method using a deep learning architecture to extend current research in this area. The method is based on body keypoints and uses an attention mechanism to focus on specific groups. Experiments were conducted on a subset of the NewsScape dataset, which presents challenges such as multiple people, camera perspective changes, and occlusions. A user study was conducted to assess the usability of the results, establishing a baseline for future gesture retrieval methods in real-world video collections. The results of the experiment demonstrate the high potential of the proposed method in multimodal communication research and highlight the significance of visual gesture retrieval in enhancing interaction with video content. The integration of visual similarity search for gestures in the open-source multimedia retrieval stack, vitrivr, can greatly contribute to the field of computational linguistics. This research advances the understanding of the role of the visual modality in co-speech gestures and highlights the need for further development in this area.

Construction of law and economics litigation service platform based on multimedia retrieval

Abstract Legal economics uses empirical research to analyze and predict the effects of various legal system arrangements to better explain whether the actual effect of the law is consistent with the expected effect of the law. Artificial Intelligence (AI) has become the core driving force of the new round of scientific and technological revolutions, and the trend of cross-border integration is becoming more and more obvious. Therefore, the intelligent construction of a legal litigation platform is the only way for social development. Litigation service management is a platform that uses big data technology to process and facilitate the information development of legal litigation services. In the face of the rapid changes in legal litigation services, an intelligent litigation service management platform is needed to serve the public. Given the above defects, this paper proposes a law and economics litigation service management platform based on multimedia retrieval, including a litigation service management system, a queuing system, and an autonomous operating system. The litigation service management system is connected with the queuing system and autonomous operating system, respectively. We provide a litigation service management platform to provide efficient operations with humanized functions to assist and improve the service quality of legal litigation services. At the same time, we combined the current application status of AI in grassroots courts and the actual work to analyze the difficulties and problems faced, draw lessons from the successful experience of some courts in China, and find out the direction and path for the application and promotion of AI in grassroots courts.

Unsupervised Hashing Retrieval via Efficient Correlation Distillation

Deep hashing has been widely used in multimedia retrieval systems due to its storage and computation efficiency. Unsupervised hashing has received a lot of attention in recent years because it does not rely on label information. However, existing deep unsupervised hashing methods usually use rough pairwise relations to constrain the similarity between hash codes locally, which is insufficient and inefficient to reconstruct accurate correlations across samples. To address this issue, we propose a generic distillation framework for the preservation of the similarity relationship. Specifically, we design a distillation loss to reconstruct the batchwise similarity distribution between feature space and hash code space, allowing us to capture the global correlation knowledge contained in features and propagate it into hash codes efficiently. This framework can apply to both intra-modal and inter-modal scenarios. Furthermore, we design a new quantization method that quantizes the continuous values to a clipping value instead of ±1 to reduce the inconsistency between continuous features and hash codes. This method can also avoid the vanishing gradient problem during training. Finally, extensive experiments for image hashing retrieval and cross-modal hashing retrieval on public datasets demonstrate that the proposed method can yield compact hash codes and outperforms the state-of-the-art baselines.