Multimedia Information Retrieval Research Articles

Correlation-aware Cross-modal Attention Network for Fashion Compatibility Modeling in UGC Systems

Empowered by the continuous integration of social multimedia and artificial intelligence, the application scenarios of information retrieval (IR) progressively tend to be diversified and personalized. Currently, User-Generated Content (UGC) systems have great potential to handle the interactions between large-scale users and massive media contents. As an emerging multimedia IR, Fashion Compatibility Modeling (FCM) aims to predict the matching degree of each given outfit and provide complementary item recommendation for user queries. Although existing studies attempt to explore the FCM task from a multimodal perspective with promising progress, they still fail to fully leverage the interactions between multimodal information or ignore the item-item contextual connectivities of intra-outfit. In this paper, a novel fashion compatibility modeling scheme is proposed based on Correlation-aware Cross-modal Attention Network. To better tackle these issues, our work mainly focuses on enhancing comprehensive multimodal representations of fashion items by integrating the cross-modal collaborative contents and uncovering the contextual correlations. Since the multimodal information of fashion items can deliver various semantic clues from multiple aspects, a modality-driven collaborative learning module is presented to explicitly model the interactions of modal consistency and complementarity via a co-attention mechanism. Considering the rich connections among numerous items in each outfit as contextual cues, a correlation-aware information aggregation module is further designed to adaptively capture significant intra-correlations of item-item for characterizing the content-aware outfit representations. Experiments conducted on two real-world fashion datasets demonstrate the superiority of our approach over state-of-the-art methods.

Application of Multimedia Information Retrieval Technology in Japanese Text Content Information Query Platform

The research and development of Japanese operating platform has become a new industry. At the same time, it is necessary to strengthen the research on query and feedback technology in MIR(Multimedia Information Retrieval). Based on this, this paper introduces the research status and progress of query and feedback technology in multimedia retrieval, and introduces the overall framework and technical links of MIR. This paper discusses the application of MIR technology in Japanese text content information query platform. A new multimodal retrieval mechanism is proposed, which combines cross-reference index and underlying features in retrieval and related feedback to realize unified processing of multimodal data. The time consumption of the algorithm is lower than that of the two comparative algorithms, which also verifies the superiority. It is feasible to apply the MIR technology proposed in this paper to the Japanese text content information query platform. It provides a new path for Japanese text content information retrieval.

Evaluation Method of Intelligent Teaching in Physical Training Classrooms Based on Multimedia Information Retrieval Technology

In different historical periods, people have different demands for Physical Education (PE), and the educational goal of school PE will also change, and the teaching mode and requirements of school PE will also be different. Multimedia information retrieval (IR) technology refers to searching in a large-scale multimedia database according to the content and context of media and media objects. In this paper, the Back Propagation instructional level of physical training class, and model is established. The BPNN is used to analyze the Multimedia IR data, and the simulation and experiment show the effectiveness of the assessment model, which provides a new method for comprehensively, fairly and scientifically evaluating the teaching work. This method is improved by 11.85% compared with the comparative assessment algorithms. The method of using BPNN to evaluate instructional level is easy to operate, which can overcome the shortcomings of traditional assessment methods, and avoid the complicated process of traditional assessment of instructional level.

Application of Interactive Object Model in Sports Teaching

Traditional physical education is just repeated training without quantitative teaching and training data results. Multimedia information analysis and retrieval technology can make repeated research on the content of competition or training, and increase the quantitative analysis of training indicators. Therefore, this paper proposes an interactive compressed domain key video object model to realize the design and production of physical education video, so as to quantify the effect of physical education. This paper discusses the interactive design and application of PE teaching video from three aspects: static, dynamic, and semantic. The interactive object model proposed in this paper can formulate corresponding strategies for different key sports video objects. The distortion rate of the complete key video object is 2.3%, which is 0.09% lower than the distortion rate of the Kim2002 scheme. From a usage perspective, through the scheme proposed in this article, coaches can obtain teaching videos with simple initial extreme values on an interactive interface, and the distortion rate of the video object is lower than that of traditional schemes.

Stacked collaborative transformer network with contrastive learning for video moment localization

Video grounding intends to perform temporal localization in multimedia information retrieval. The temporal bounds of the target video span are determined for the given input query. A novel interactive multi-head self-attention (IMSA) transformer is proposed to localize an unseen moment in the untrimmed video for the given image. A new semantic-trained self-supervised approach is considered in this paper to perform cross-domain learning to match the image query – video segment. It normalizes the convolution function enabling efficient correlation and collecting of semantically related video segments across time based on the image query. A double hostile Contrastive learning with Gaussian distribution parameters method is advanced to learn the representations of video. The proposed approach performs dynamically on various video components to achieve exact semantic synchronization and localization among queries and video. In the proposed approach, the IMSA model localizes frames greatly compared to other approaches. Experiments on benchmark datasets show that the proposed model can significantly increase temporal grounding accuracy. The moment occurrence is identified in the video with a start and end boundary ascertains an average recall of 86.45% and a mAP of 59.3%.

Continuous Hidden Markov Algorithm-Based Multimedia Melody Retrieval System for Music Education

Education professionals receive instruction in Music Education (ME) to prepare for prospective jobs like secondary or primary music teachers, schools ensembles executives, or ensembles directors at music institutions. In the discipline of music education, educators do original research on different approaches to teaching and studying music. The most accurate and effective method of extracting music from huge music databases has become one of the most frequently discussed participants in contemporary multimedia information retrieval development. The essence of multimedia material is presented within a range of techniques since it is not bound to a single side. These several categories could consist of the song’s audio components and lyrics for musical information. Retrieving melodic information, subsequently, becomes the main focus of most recent studies. Aside from being an expensive deviate from academics, music programs are neither a viable profession neither a valid pastime. Therefore, in this study, we offer a Continuous Hidden Markov Algorithm (CHMA) related a novel method for recovering melodies from musical multimedia recordings. CHMA is considered to be the most basic dynamic Bayesian network. Two various types of audio frame features and audio example features are extracted throughout the feature extraction procedure from the audio signal according to unit length. Every music clip receives a unique approach that we implement with concurrently using various CHMA. The initial music gets processed using a trained CHMA that monitors fundamental frequencies, maps states, and generates retrieval outcomes. The training time for Traditional opera reached 455.76 minutes, the testing time for Narration achieved 56.10 minutes, and the recognition accuracy for advertisement reached an impressive 98.02%. A subsequently experimental result validates the applicability of the proposed approach.

A study of the role of new feature fusion based on multimedia analysis on accounting investment decision methods in economic models

Accounting investing analysis has been expanding at a steady clip, and some of the findings suggest that investors' restricted reasoning and self-psychological sentiments may not always lead them to make their negative emotions influence their financial decisions, leading to a loss. Digital multimedia fusion and display are made possible, and numerous terminals may now communicate with one another in a seamless, real-time manner thanks to the growth of e-commerce and multimedia. This paper proposes a Multimedia Analysis (MA) feature fusion method for understanding the psychological emotions associated with accounting investments in the online retail environment, which can then be used to guide the development of an investment strategy that is both appropriate and successful for the target demographic. The primary goal of this study is to show how multimedia information retrieval tasks may benefit from combining text pre-filtering with image sorting. For this investigation; they used information from the reliable China Stock Market and Accounting Research (CSMAR) Database. The information fusion technology that supports this paper's investigation is used to dissect experiment outcomes, examine issues with the emotional effect of financial investment clients, and assess the paper's intended study topic. In experiments, we found a 97% accuracy rate in terms of accuracy.

Multimedia information retrieval using content-based image retrieval and context link for Chinese cultural artifacts

PurposeAn increasing number of images are generated daily, and images are gradually becoming a search target. Content-based image retrieval (CBIR) is helpful for users to express their requirements using an image query. Nevertheless, determining whether the retrieval system can provide convenient operation and relevant retrieval results is challenging. A CBIR system based on deep learning features was proposed in this study to effectively search and navigate images in digital articles.Design/methodology/approachConvolutional neural networks (CNNs) were used as the feature extractors in the author's experiments. Using pretrained parameters, the training time and retrieval time were reduced. Different CNN features were extracted from the constructed image databases consisting of images taken from the National Palace Museum Journals Archive and were compared in the CBIR system.FindingsDenseNet201 achieved the best performance, with a top-10 mAP of 89% and a query time of 0.14 s.Practical implicationsThe CBIR homepage displayed image categories showing the content of the database and provided the default query images. After retrieval, the result showed the metadata of the retrieved images and links back to the original pages.Originality/valueWith the interface and retrieval demonstration, a novel image-based reading mode can be established via the CBIR and links to the original images and contextual descriptions.

Deep Learning for Approximate Nearest Neighbour Search: A Survey and Future Directions

Approximate nearest neighbour search (ANNS) in high-dimensional space is an essential and fundamental operation in many applications from many domains such as multimedia database, information retrieval and computer vision. With the rapidly growing volume of data and the dramatically increasing demands of users, traditional heuristic-based ANNS solutions have been facing great challenges in terms of both efficiency and accuracy. Inspired by the recent successes of deep learning in many fields, substantial efforts have been devoted to applying deep learning techniques to ANNS for learning to index and learning to search, resulting in numerous algorithms that achieve state-of-the-art performance compared with conventional methods. In this survey paper, we comprehensively review the different types of deep learning-based ANNS methods according to two learning paradigms: <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">learning to index</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">learning to search</i> . We provide a comprehensive overview and analysis of these methods in a systematic manner. Based on the overview, we point out that <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">end-to-end learning</i> will be a new and promising research direction for deep learning-based ANNS, i.e., applying deep learning techniques to jointly learn the indexing and searching together, such that the underlying knowledge learned from data can directly contribute to the final searching performance. Finally, we conduct experiments and provide general performance analyses for the representative deep learning-based ANNS algorithms.

Using Multimedia Information Retrieval Technology to Optimize the Strategies of College English Teaching

Abstract In the design of college ELT(English Language Teaching), it is the most important link to create situations conducive to learners’ meaning construction. At present, the starting point of college ELT is no longer simply and mechanically teaching words, sentences, and grammar, but based on language tools, which improves students’ cultural quality, personality, and professional knowledge. Therefore, this paper makes a research on college ELT through multimedia information retrieval technology. The research shows that 38.40% of the students think that the use of multimedia in teaching can stimulate their interest in learning. 14.66% of the students believe that they can effectively stimulate their interest in learning through extracurricular activities in English. Only 7.38% of the students felt that teachers should encourage and help them. Some students think that emphasizing the importance of English learning can stimulate their interest in learning. At present, the English classroom is more scientific, orderly, diversified, and systematic, which provides convenience for the new curriculum reform being implemented. This paper discusses ELT under multimedia information retrieval technology and puts forward predictive strategies for the problems that may arise now and in the future.

Analysis on the Multimedia Information Retrieval Algorithm Based on Enterprise Correlation Financial Analysis under the Background of Big Data

At present, based on the background of the big data era, the operation of enterprises has a great impact, and relevance is a principle set to meet the needs of enterprise operation and management. In today’s digital and network era, multimedia has become the main part of the Internet information transmission, and multidisciplinary art and science communion has promoted the development of the multimedia information retrieval. As a result, this paper examines the correlation of big data against the background of financial and multimedia information retrieval algorithms.

Multimedia information retrieval using artificial neural network

&lt;span lang="EN-US"&gt;The importance of the multimedia information retrieval (MIR) is highlighted by the extensive amount of the information on the internet. Image, audio, video, and text are all examples of the characteristics of the raw multimedia data. It is greatly challenging to represent a concept of human perception and how the machine-level language can grasp it (semantic gap of MIR). However, this paper aims to improve the information retrieval model that retrieves data from multimedia. This can be implemented by leveraging the use of variety of algorithms that go through training and testing to extract the model. One of these algorithms extracts text information based on the query language's nature as the vector space model (VSM) and the latent semantic index (LSI) were used. The other technique uses curvelet decomposition and statistic parameters like mean, standard deviation, and signal energy to recover these properties. Additionally, a discrete wavelet transforms (DWT) and signal characteristics-based method is used to retrieve audio signals. Finally, the neural network learning is modeled and trained on a collection of different multimedia images. The learned features have been utilized for presenting a highly sufficient system of multimedia retrieval which operates for a large set of multi-modal datasets. &lt;/span&gt;

Smart Multimedia Information Retrieval

The area of multimedia information retrieval (MMIR) faces two major challenges: the enormously growing number of multimedia objects (i.e., images, videos, audio, and text files), and the fast increasing level of detail of these objects (e.g., the number of pixels in images). Both challenges lead to a high demand of scalability, semantic representations, and explainability of MMIR processes. Smart MMIR solves these challenges by employing graph codes as an indexing structure, attaching semantic annotations for explainability, and employing application profiling for scaling, which results in human-understandable, expressive, and interoperable MMIR. The mathematical foundation, the modeling, implementation detail, and experimental results are shown in this paper, which confirm that Smart MMIR improves MMIR in the area of efficiency, effectiveness, and human understandability.

Multi-Modal Cross-Domain Alignment Network for Video Moment Retrieval

As an increasingly popular task in multimedia information retrieval, video moment retrieval (VMR) aims to localize the target moment from an untrimmed video according to a given language query. Most previous methods depend heavily on numerous manual annotations ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e.</i> , moment boundaries), which are extremely expensive to acquire in practice. In addition, due to the domain gap between different datasets, directly applying these pre-trained models to an unseen domain leads to a significant performance drop. In this paper, we focus on a novel task: cross-domain VMR, where fully-annotated datasets are available in one domain (“source domain”), but the domain of interest (“target domain”) only contains unannotated datasets. As far as we know, we present the first study on cross-domain VMR. To address this new task, we propose a novel Multi-Modal Cross-Domain Alignment (MMCDA) network to transfer the annotation knowledge from the source domain to the target domain. However, due to the domain discrepancy between the source and target domains and the semantic gap between videos and queries, directly applying trained models to the target domain generally leads to a performance drop. To solve this problem, we develop three novel modules: (i) a domain alignment module is designed to align the feature distributions between different domains of each modality; (ii) a cross-modal alignment module aims to map both video and query features into a joint embedding space and to align the feature distributions between different modalities in the target domain; (iii) a specific alignment module tries to obtain the fine-grained similarity between a specific frame and the given query for optimal localization. By jointly training these three modules, our MMCDA can learn domain-invariant and semantic-aligned cross-modal representations. Extensive experiments on three challenging benchmarks (ActivityNet Captions, Charades-STA and TACoS) illustrate that our cross-domain method MMCDA outperforms all state-of-the-art single-domain methods. Impressively, MMCDA raises the performance by more than 7% in representative cases, which demonstrates its effectiveness.

Recent Query Reformulation Approaches for Information Retrieval System - A Survey

Abstract:Around trillions of data are uploaded to the internet every year. Extracting useful information using only a few keywords has become a major challenge. The field of Query Reformulation (QR) is focused on the efficient retrieval of information to overcome this. It is widely used in the domain of information retrieval (IR) and related fields such as search engines, multimedia IR, cross-language IR, recommender systems, and so on. Query reformulation techniques incur extra computational costs. Due to this reason, the use of query reformulation techniques is sometimes prohibited in internet searches as searching over the internet requires a fast response time. But due to the success of NLP (Natural Language Processing) using machine learning/deep learning in recent years, there has been a boom of study in this area. In this literature, a variety of term selection, term extraction, and query reformulation strategies based on recent technologies used by researchers have been presented, necessitating a wide survey to focus research in this promising area. Recent QR approaches and the datasets, techniques, and evaluation metrics used in this paper will help researchers understand and focus more on research in this promising area so that a better solution will be proposed. From the survey, it may be observed that one of the hottest subjects in the field of IR right now is applying deep learning to IR systems for query reformulation.

An NLP-guided ontology development and refinement approach to represent and query visual information

The ubiquitous presence of surveillance systems generates massive amounts of video data. Storage and analysis of this data in real-time is a substantial challenge. There is huge potential in representing data in machine-readable and machine-interpretable format due to the presence of hidden semantics in images and videos. However, such representation requires ontology, which calls for expert domain knowledge. In this paper, a novel NLP-guided approach to generate an ontology for multimedia representation and information retrieval is proposed. A semi-automatic NLP-guided framework, which extracts all possible relations among objects is presented. This framework leverages the textual data of the domain to generate possible descriptions and actions within the domain. Relations among objects get embedded as object properties, whereas the category of an object as a class. Features and attributes of objects encode the data properties of the ontology. The proposed ontology is compared with existing multimedia ontologies and evaluated with regard to its capability to represent relations occurring in benchmark datasets, demonstrating the completeness and thorough coverage of the domain concepts. Spatial reasoning rules are established using Semantic Web Rule Language (SWRL) rules, and information retrieval is demonstrated using Description Logic (DL) and SPARQL queries. The proposed NLP-guided ontology generation approach is general enough to help in the development of ontologies for other domains as well, by providing video and textual data of the domain of interest, with limited human involvement.

Explainable Multimedia Feature Fusion for Medical Applications.

Due to the exponential growth of medical information in the form of, e.g., text, images, Electrocardiograms (ECGs), X-rays, and multimedia, the management of a patient’s data has become a huge challenge. In particular, the extraction of features from various different formats and their representation in a homogeneous way are areas of interest in medical applications. Multimedia Information Retrieval (MMIR) frameworks, like the Generic Multimedia Analysis Framework (GMAF), can contribute to solving this problem, when adapted to special requirements and modalities of medical applications. In this paper, we demonstrate how typical multimedia processing techniques can be extended and adapted to medical applications and how these applications benefit from employing a Multimedia Feature Graph (MMFG) and specialized, efficient indexing structures in the form of Graph Codes. These Graph Codes are transformed to feature relevant Graph Codes by employing a modified Term Frequency Inverse Document Frequency (TFIDF) algorithm, which further supports value ranges and Boolean operations required in the medical context. On this basis, various metrics for the calculation of similarity, recommendations, and automated inferencing and reasoning can be applied supporting the field of diagnostics. Finally, the presentation of these new facilities in the form of explainability is introduced and demonstrated. Thus, in this paper, we show how Graph Codes contribute new querying options for diagnosis and how Explainable Graph Codes can help to readily understand medical multimedia formats.

Dual Face Alignment Learning Network for NIR-VIS Face Recognition

As the most important topic in Heterogeneous Face Recognition (HFR), Near-InfraRed and VISual (NIR-VIS) face recognition has attracted increasing research attentions owing to its potential application in the field of criminal detective cases and multimedia information retrieval. However, due to its dramatic intra-class variations, including modality, pose, occlusion, blurry, lighting, distance, expression, etc, it is very challenging to retain inherent identity information. To address the above issue, we propose a novel Dual Face Alignment Learning (DFAL) algorithm to explore the potential domain-invariant neutral face representations of the cross-modal images. Our model contains three effective components including Feature-level Face Alignment (FFA), Image-level Face Alignment (IFA) and Cross-domain compact Representation (CdR). Firstly, Teacher-Encoder CNNs (TeEn-CNNs) and Student-Encoder CNNs (StEn-CNNs) are designed to encode features for VIS neutral face images and non-neutral face images, and the FFA is introduced to learn neutral face representations by performing feature-level alignment between non-neutral face and VIS neutral face. Secondly, Student-Decoder CNNs (StDe-CNNs) is developed to decode features to restore face images, and the IFA is designed to reconstruct neutral face image by imposing image-level alignment. Notably, the FFA acts as the primary target to learn VIS neutral face representations for cross-view data, while the IFA plays a role in the icing on the cake, i.e., further disentangling domain and residual information through the synthesis process. Finally, the CdR dispels modality features and distills identity features by mining inter-class information, inter-domain information and inter-semantic relationship. The joint scheme enables the elimination of intra-class variations and the purification of identity information. We carry out comprehensive experiments to illustrate the effectiveness of the DFAL approach on three challenging NIR-VIS databases.

A Supervised Video Hashing Method Based on a Deep 3D Convolutional Neural Network for Large-Scale Video Retrieval.

Recently, with the popularization of camera tools such as mobile phones and the rise of various short video platforms, a lot of videos are being uploaded to the Internet at all times, for which a video retrieval system with fast retrieval speed and high precision is very necessary. Therefore, content-based video retrieval (CBVR) has aroused the interest of many researchers. A typical CBVR system mainly contains the following two essential parts: video feature extraction and similarity comparison. Feature extraction of video is very challenging, previous video retrieval methods are mostly based on extracting features from single video frames, while resulting the loss of temporal information in the videos. Hashing methods are extensively used in multimedia information retrieval due to its retrieval efficiency, but most of them are currently only applied to image retrieval. In order to solve these problems in video retrieval, we build an end-to-end framework called deep supervised video hashing (DSVH), which employs a 3D convolutional neural network (CNN) to obtain spatial-temporal features of videos, then train a set of hash functions by supervised hashing to transfer the video features into binary space and get the compact binary codes of videos. Finally, we use triplet loss for network training. We conduct a lot of experiments on three public video datasets UCF-101, JHMDB and HMDB-51, and the results show that the proposed method has advantages over many state-of-the-art video retrieval methods. Compared with the DVH method, the mAP value of UCF-101 dataset is improved by 9.3%, and the minimum improvement on JHMDB dataset is also increased by 0.3%. At the same time, we also demonstrate the stability of the algorithm in the HMDB-51 dataset.

Approximate k-NN Graph Construction: A Generic Online Approach

Nearest neighbor search and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">k</i> -nearest neighbor graph construction are two fundamental issues that arise from many disciplines such as multimedia information retrieval, data-mining, and machine learning. They become more and more imminent given the big data emerge in various fields in recent years. In this paper, a simple but effective solution both for approximate <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">k</i> -nearest neighbor search and approximate <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">k</i> -nearest neighbor graph construction is presented. These two issues are addressed jointly in our solution. On one hand, the approximate <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">k</i> -nearest neighbor graph construction is treated as a search task. Each sample along with its <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">k</i> -nearest neighbors is joined into the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">k</i> -nearest neighbor graph by performing the nearest neighbor search sequentially on the graph under construction. On the other hand, the built <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">k</i> -nearest neighbor graph is used to support <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">k</i> -nearest neighbor search. Since the graph is built online, the dynamic update on the graph, which is not possible for most of the existing solutions, is supported. This solution is feasible for various distance measures. Its effectiveness both as <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">k</i> -nearest neighbor construction and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">k</i> -nearest neighbor search approaches is verified across different types of data in different scales, various dimensions, and under different metrics.