Multimedia Search Research Articles

Improving semantic video retrieval models by training with a relevance-aware online mining strategy

To retrieve a video via a multimedia search engine, a textual query is usually created by the user and then used to perform the search. Recent state-of-the-art cross-modal retrieval methods learn a joint text–video embedding space by using contrastive loss functions, which maximize the similarity of positive pairs while decreasing that of the negative pairs. Although the choice of these pairs is fundamental for the construction of the joint embedding space, the selection procedure is usually driven by the relationships found within the dataset: a positive pair is commonly formed by a video and its own caption, whereas unrelated video-caption pairs represent the negative ones. We hypothesize that this choice results in a retrieval system with limited semantics understanding, as the standard training procedure requires the system to discriminate between groundtruth and negative even though there is no difference in their semantics. Therefore, differently from the previous approaches, in this paper we propose a novel strategy for the selection of both positive and negative pairs which takes into account both the annotations and the semantic contents of the captions. By doing so, the selected negatives do not share semantic concepts with the positive pair anymore, and it is also possible to discover new positives within the dataset. Based on our hypothesis, we provide a novel design of two popular contrastive loss functions, and explore their effectiveness on four heterogeneous state-of-the-art approaches. The extensive experimental analysis conducted on four datasets, EPIC-Kitchens-100, MSR-VTT, MSVD, and Charades, validates the effectiveness of the proposed strategy, observing, e.g., more than +20% nDCG on EPIC-Kitchens-100. Furthermore, these results are corroborated with qualitative evidence both supporting our hypothesis and explaining why the proposed strategy effectively overcomes it.

Implementation of multimedia search & management system based on remote education

In order for remote education using multimedia to be effective, an efficient management technique for video information needs to be developed. Therefore, for real-time processing of moving images, it is necessary to manage and search image data in a compressed state. MPEG-4 is the most widely used video compression technology. In order to process video in real time in distance education using multimedia, it is very important to develop a technique for managing and retrieving video information compressed with MPEG-4. Therefore, in this paper, a multimedia information management system and search technology were developed using MPEG-4 compression technology used for real-time distance education.

ViSTORY: Effective Video Storyboard Generation with Visual Keyframes using Discrete Cosine Transform

Nowadays, multimedia content utility is increasing rapidly. Multimedia search engines like Google, Yahoo, Bing, etc., are available just a click away to all users. There are around 500-600 hours of video uploads per unit of time to the Internet. So, among other types of multimedia content, such as text and images, video is the most complicated content for indexing, browsing, and retrieval. Videos give more scope for implementation because of their complex and unstructured nature. This paper proposes a new method of video storyboard generation with keyframe extraction in spatial and frequency domains using Discrete Cosine Transform (DCT) for video summarization. It discusses the empirical appraisal of video visual keyframes with t-test analysis in comparison with spatial and frequency domains, resulting in a quick response to customer demands by providing static storyboards. This study proposes a new performance measure as matching frames by analyzing input videos and the standard benchmarks video dataset, i.e., Open Video Project (OVP) and SumMe. Among all the keyframe extraction techniques, DCT gives higher accuracy and a better matching rate.

Deep Learning Approaches To Image Classification: Exploring The Future Of Visual Data Analysis

Over the past decade, deep learning has emerged as a revolutionary technology for the analysis of visual data, particularly images. This master's thesis focuses on deep learning approaches to image classification, which is a key task in many applications using visual data analysis. A state-of-the-art deep learning model, namely the Vision Transformer (ViT), is explored for image classification. ViT is trained using transfer-learning techniques on a new dataset of over 350,000 photographs of European buildings in eight cities, obtained across two separate flights from a drone-mounted camera. Initial results demonstrate that models pre-trained on large datasets such as JFT-300M can achieve performance competitively with the fine-tuning of models trained from scratch on smaller datasets and that ViT outperforms convolutional neural networks for drone-captured images. Further, the prospects of deep learning for image classification are discussed, highlighting the potential impact of new research directions within the architectural vision transformer domain (e.g., Swin-Transformer, CrossViTs, T2T-vision Transformer) and new training techniques (e.g., Vision-Language Pre-training models, multi-modality input). The exponential increase in data generated by cameras, mobile devices, and Internet-of-Things (IoT) sensors has escalated the need for automated processing and analysis of visual data. Furthermore, images and video frames are a popular medium for data collection across various domains, including commercial and industrial. Image classification, or finding the most relevant label for a given photograph, is one key task in many applications using visual data analysis. Popular applications include multimedia search engines, mobile applications navigating to points of interest (POI), and anomaly detection in industrial cameras. As a consequence, many datasets have been assembled, containing millions of photographs collected and labeled according to city, object, or scene. Deep neural networks trained end-to-end directly on pixels have become state-of-the-art image classification technology. More recently, architectures based solely on attention mechanisms, eschewing convolutions, have challenged the long-standing dominance of convolutional neural networks.

Intelligent Auxiliary Artificial Wood Plank Pattern Design Based on the Subject Search Algorithm of Multimedia Resources

Regarding the restriction of the wood processing enterprises in the market, intelligent artificial wood materials are mainly based on the demand for pattern quality levels, and the calculation method of multimedia resource theme search is used to achieve the pattern design of intelligent auxiliary artificial wood materials. First, analyze the pattern characteristics of intelligent auxiliary artificial wood materials. After analyzing the characteristics, use the multimedia resource subject search calculation method to carry out the binarization design. At the same time, use the self-learning method to optimize the convergence efficiency and reduce the design time. Finally, pass the softmax designer extracts design schemes for patterns and straight lines.

Robust Video Hashing Based on Multidimensional Scaling and Ordinal Measures

Multimedia hashing is a useful technology of multimedia management, e.g., multimedia search and multimedia security. This paper proposes a robust multimedia hashing for processing videos. The proposed video hashing constructs a high-dimensional matrix via gradient features in the discrete wavelet transform (DWT) domain of preprocessed video, learns low-dimensional features from high-dimensional matrix via multidimensional scaling, and calculates video hash by ordinal measures of the learned low-dimensional features. Extensive experiments on 8300 videos are performed to examine the proposed video hashing. Performance comparisons reveal that the proposed scheme is better than several state-of-the-art schemes in balancing the performances of robustness and discrimination.

An architecture for non-linear discovery of aggregated multimedia document web search results.

The recent proliferation of multimedia information on the web enhances user information need from simple textual lookup to multi-modal exploration activities. The current search engines act as major gateways to access the immense amount of multimedia data. However, access to the multimedia content is provided by aggregating disjoint multimedia search verticals. The aggregation of the multimedia search results cannot consider relationships in them and are partially blended. Additionally, the search results’ presentation is via linear lists, which cannot support the users’ non-linear navigation patterns to explore the multimedia search results. Contrarily, users’ are demanding more services from search engines. It includes adequate access to navigate, explore, and discover multimedia information. Our discovery approach allow users to explore and discover multimedia information by semantically aggregating disjoint verticals using sentence embeddings and transforming snippets into conceptually similar multimedia document groups. The proposed aggregation approach retains the relationship in the retrieved multimedia search results. A non-linear graph is instantiated to augment the users’ non-linear information navigation and exploration patterns, which leads to discovering new and interesting search results at various aggregated granularity levels. Our method’s empirical evaluation results achieve 99% accuracy in the aggregation of disjoint search results at different aggregated search granularity levels. Our approach provides a standard baseline for the exploration of multimedia aggregation search results.

An Effectual Video Indexing and Retrieval Model: A Comparative Study

Video indexing and retrieval are considered to be the most crucial factor that has to be concentrated in multimedia search engine based investigations. The analysis with multimedia content may enhance the accuracy of the indexing and retrieval system, which is performed with two different phases. Former is content-based, and the latter is context-based annotation. The previous annotation includes both higher level and lower level features, which is acquired from pixel intensity; similarly, the latter is based on semantic video details. Here, an extensive analysis is done with three different methods known as Fuzzy based SVM classifier, logit boost ensemble classifier, and Naive Bayes classifier, where feature extraction is concentrated on various factors. These diverse methods pretend to carry of retrieval of the video effectually with specific advantages and disadvantages. Here, numerical analysis is done with different factors like accuracy, sensitivity, specificity, F-measure, Recall, and so on. The total amount of frames considered for computation is also determined. The videos are attained from the available online dataset, and initial pre-processing is carried out to make the quality of image frames to be superior. The overall analysis shows the performance of classifiers with improved prediction accuracy for video retrieval.

ASTS: attention based spatio-temporal sequential framework for movie trailer genre classification

Automatic movie trailer genre classification is a challenging task because trailers have more diverse content and high-level sequential semantic concepts within the movie storyline, which can help for multimedia search and personalized movie recommendation. Traditional methods generally extract the low-level features or consider the local sequential dependencies among trailer frames, ignoring the global high-level sequential semantic concepts. In this manuscript, we propose a novel and effective Attention based Spatio-temporal Sequential Framework (ASTS) for movie trailer genre classification. The proposed framework mainly consists of two modules, respectively the spatio-temporal descriptive module and the attention-based sequential module. The spatio-temporal descriptive module adopts some advanced convolution neural networks to extract the spatio-temporal features of key trailer frames, which can capture the local spatio-temporal semantic features. The attention-based sequential module is designed to process the extracted spatio-temporal feature representation sequence for capturing the global high-level sequential semantic concepts within the movie storyline. We crawl 14,415 labeled movie trailers from YouTube and integrate them into the public dataset MovieLens. Experiment results show that our proposed framework is superior to state-of-the-art methods.

Multimedia Intelligence: When Multimedia Meets Artificial Intelligence

Owing to the rich emerging multimedia applications and services in the past decade, super large amount of multimedia data has been produced for the purpose of advanced research in multimedia. Furthermore, multimedia research has made great progress on image/video content analysis, multimedia search and recommendation, multimedia streaming, multimedia content delivery etc. At the same time, Artificial Intelligence (AI) has undergone a “new” wave of development since being officially regarded as an academic discipline in 1950s, which should give credits to the extreme success of deep learning. Thus, one question naturally arises: What happens when multimedia meets Artificial Intelligence? To answer this question, we introduce the concept of Multimedia Intelligence through investigating the mutual-influence between multimedia and Artificial Intelligence. We explore the mutual influences between multimedia and Artificial Intelligence from two aspects: i) multimedia drives Artificial Intelligence to experience a paradigm shift towards more explainability and ii) Artificial Intelligence in turn injects new ways of thinking for multimedia research. As such, these two aspects form a loop in which multimedia and Artificial Intelligence interactively enhance each other. In this paper, we discuss what and how efforts have been done in literature and share our insights on research directions that deserve further study to produce potentially profound impact on multimedia intelligence.

Evaluating Multimedia and Language Tasks.

Evaluating information access tasks, including textual and multimedia search, question answering, and understanding has been the core mission of NIST's Retrieval Group since 1989. The TRECVID Evaluations of Multimedia Access began in 2001 with a goal of driving content-based search technology for multimedia just as its progenitor, the Text Retrieval Conference (TREC) did for text and web1.

Knowledge expansion of metadata using script mining analysis in multimedia recommendation

In this paper, a method for knowledge expansion of metadata using script mining analysis for multimedia recommendation systems is proposed. The method allows the extraction of new metadata and knowledge expansion through the mining analysis of multimedia scripts, which include a large amount of information. The scripts are collected by a Web crawler based on Python. From the collected scripts, hidden information is extracted through keyword analysis and sentiment analysis. In keyword analysis, scripts, unlike general documents, show a high frequency of names of characters or proper nouns. Such names or proper nouns are not frequently used in other media content, and therefore, their importance is high. Frequently, they are already offered in the conventional metadata, and consequently cause information duplication. Accordingly, term frequency–inverse document and metadata frequency (TF–IDMF), which considers the frequency of metadata in general term frequency–inverse document frequency (TF–IDF), is used. Thus, the importance of the names of characters or proper nouns in scripts can be decreased. Because the keywords for the extracted scripts are in fact included in the scripts, they can be used for precise multimedia search and recommendation. In sentiment analysis, the AFINN lexicon and the Bing lexicon are utilized to scan words in a script. The Bing lexicon is used to examine whether the words in the entire script are positive or negative. Then, the total numbers of positive words and negative words are used to calculate the representative sentiment of the script. The AFINN lexicon includes approximately 170 sentiment words, the negative or positive sentiment of which is presented in the range − 5 to +5. One script is divided into 100 sentences, and then, the representative sentiment in each sentence is evaluated as either positive or negative. Through script scanning, the flow of sentiment in multimedia streams can be discovered. The Bing lexicon categorizes words into positive, negative, and neutral sentiments. Through script scanning, the words included in each category can be quantified. Depending on the result of the script sentiment analysis, a different sentence embedding method based on inter-sentence similarity is used to cluster similar media. The results of the keyword analysis and sentiment analysis of a script are added to the metadata in a new column in a knowledge base to expand knowledge. To evaluate the significance of multimedia recommendations, keywords and sentiment information are used, and then, the similarity and clustering of the extracted media are assessed. As a result, script mining analysis based on the attributes that include actual information of media is considerably better than that based on types or a range of metadata attributes. Therefore, the proposed knowledge expansion method achieves significant results and shows an excellent performance in multimedia recommendation.

Multi-hash chain based multimedia search technology optimized for distributed environments using IoT devices

Recently, research has been active in various fields in relation to multimedia data retrieval to reduce the accuracy and calculation costs of multimedia retrieval. In particular, various search methods using multimedia has been developed. However, in many applications currently operating in the system, large multimedia data is distributed across servers or sites at different locations, so assembling multimedia data only as a hash presents huge computational, communication, and storage costs. In this paper, when searching for multimedia information stored in distributed environments using IoT devices, multiple hash chain-based multimedia retrieval techniques are proposed to minimize accuracy and calculation costs. The proposed technique is aimed at improving the accuracy of multimedia by using IoT devices to divide-and-conquer multimedia information that is collected and stored distributed. The proposed technique provides probability values for each segmented multimedia information by disaggregating the various attribute information that is composed of multimedia as much as possible, thereby increasing the accuracy of the multimedia that users want. Suggestion techniques determine the group size of multimedia data according to the priority of layered multimedia data. Multimedia group data include multimedia data contained in higher layers and subdivides the various attribute information constituting multimedia data. In addition, the proposed techniques improve the accessibility of multimedia data over existing ones by linking priority information between multimedia data.

Artificial Intelligence Recognition Simulation of 3D Multimedia Visual Image Based on Sparse Representation Algorithm

With the rapid development of computer networks and multimedia technologies, images, which are important carriers of information dissemination, have made human cognition of things easier. Image recognition is a basic research task in computer vision, multimedia search, image understanding and other fields. This paper proposes a hierarchical feature learning structure that is completely automatically based on the original pixels of the image, and uses the K-SVD (K-Singular Value Decomposition) algorithm with label consistency constraints to train the discriminant dictionary. For different types of image data sets, the algorithm only extracts image blocks. After dense sampling, an efficient OMP (Orthogonal Matching Pursuit) encoder is used to obtain a layered sparse representation. The improved SIFT (Scale Invariant Feature Transform) algorithm is used to solve the difficult problem of multimedia visual image stereo matching. The feature point extraction and stereo matching of multimedia visual images, different scales and different viewpoint images are analyzed separately. Aiming at a large number of low-dimensional geometric features of 3D images, this paper studies the extraction and sorting strategies of low-dimensional geometric features of 3D images. A sparse representation method for 3D images is proposed, and the sparseness of image features is evaluated. This further improves the accuracy of 3D image representation and the robustness of 3D image recognition algorithms.

ПОШУК МУЛЬТИМЕДІЙНОЇ ІНФОРМАЦІЇ НА ОСНОВІ НЕЙРОННИХ МЕРЕЖ

The article considers approaches to the use of neural networks in multimedia information retrieval. The develop ment of methods for multimedia information retrieval is necessary due to the large amount of such information. Tradi tional methods of multimedia information retrieval have a high speed of data processing, but low accuracy due to the inability of semantic search. The use of neural networks allows for semantic search, which increases its accuracy and completeness. Approaches to the use of neural networks at the stages of indexing and retrieval of multimedia infor mation are considered. With the help of a neural network, a multimedia file is analyzed and classified. The result of classifying a file is used to create its textual description - an annotation that is compared to the search query to deter mine relevance. There are many ready-made classification networks that can be used to speed up the process of creat ing a multimedia search system, but it is not possible to create a neural network to classify all real-world objects, so multiple neural networks should be used. Neural networks are also use to build feature vectors for a media file and a search query. Similarity functions, such as cosine of similarity, are applied to constructed vectors to determine the semantic similarity of a query and a media file. In this case, the search query can be both in text form and in the form of the appropriate format of the desired media file. This approach allows to build an optimal neural network for a specific task. Neural networks are used to compare the constructed annotation of a file and a query, which increases the accu racy and completeness of the search, compared to traditional methods, due to the ability of neural networks to take into account the semantic meaning of the text.

Automatic text location of multimedia video for subtitle frame

The traditional multi-media video text auto-positioning is too dependent on man-made function, and its disadvantages are mainly embodied in the aspects of strong subjectivity, large quantity of work, slow processing speed and the like. Through the establishment of the basic framework of multimedia video text search, the feature vectors of multimedia video text are calculated. On the basis of the feature of the independent video captioning (ICA), the method of automatic positioning of the multimedia video text is studied. The method comprises the following steps of: extracting a plurality of ICA image features of an independent multimedia video text sub-block by adopting an independent video caption-based ICA feature extraction method; performing image candidate character line automatic detection based on a plurality of ICA image features, adopting an adaptive iterative positioning algorithm to quickly obtain a character candidate line, In the process of automatic text block detection, the flaw of text candidate line detection is checked, and the text candidate block is obtained by threshold calculation. The experimental results show that the proposed method has the highest accuracy in recognizing video subtitles with independent caption based ICA features, and the recall and precision of the text lines and blocks of the video image are located by the proposed method, and good automatic positioning effect.

Efficient Supervised Discrete Multi-View Hashing for Large-Scale Multimedia Search

Hashing has recently received substantial attention in large-scale multimedia search for its extremely low-cost storage cost and high retrieval efficiency. However, most existing hashing techniques focus on learning hash codes for single-view or cross-view retrieval. It is still an unsolved problem that how to efficiently learn discriminative binary codes for multi-view data that is common in real world multimedia search. In this paper, we propose an efficient Supervised Discrete Multi-view Hashing (SDMH) to solve the problem. SDMH first properly detects the shared binary hash codes, with an integrated multi-view feature mapping and latent hash coding, by exploiting the complementarity of different view-specific features and removing the involved inter-view redundancy. To further enhance the discriminative capability of hash codes, SDMH directly represses the explicit semantic labels of data samples with their corresponding binary codes. Different from most existing multi-view hashing methods that adopt “relaxing+rounding” hash optimization strategy or the discrete optimization method based on discrete cyclic coordinate descent, an efficient augmented Lagrangian multiplier (ALM) based discrete hash optimization method is developed in this paper to optimize the hash codes within a single step. Experimental results on four benchmark datasets demonstrate the superior performance of the proposed approach over state-of-the-art hashing techniques, in terms of both learning efficiency and retrieval accuracy.

On Machine Learning and Knowledge Organization in Multimedia Information Retrieval

Recent technological developments have increased the use of machine learning to solve many problems, including many in information retrieval. Multimedia information retrieval as a problem represents a significant challenge to machine learning as a technological solution, but some problems can still be addressed by using appropriate AI techniques. We review the technological developments and provide a perspective on the use of machine learning in conjunction with knowledge organization to address multimedia IR needs. The semantic gap in multimedia IR remains a significant problem in the field, and solutions to them are many years off. However, new technological developments allow the use of knowledge organization and machine learning in multimedia search systems and services. Specifically, we argue that, the improvement of detection of some classes of low-level features in images music and video can be used in conjunction with knowledge organization to tag or label multimedia content for better retrieval performance. We provide an overview of the use of knowledge organization schemes in machine learning and make recommendations to information professionals on the use of this technology with knowledge organization techniques to solve multimedia IR problems. We introduce a five-step process model that extracts features from multimedia objects (Step 1) from both knowledge organization (Step 1a) and machine learning (Step 1b), merging them together (Step 2) to create an index of those multimedia objects (Step 3). We also overview further steps in creating an application to utilize the multimedia objects (Step 4) and maintaining and updating the database of features on those objects (Step 5).

A personalized cloud engine for multimedia search based on binary ant colony algorithm

At present, the search content and results of common multimedia search engines have gradually failed to meet the personalized search requirements of users. The paper focuses on personalized cloud search engines for multimedia. First, the binary ant colony algorithm is optimized by the cloud model. Then, the binary ant colony algorithm is used to improve the multimedia search engine. Detailedly, a binary directed graph for ant colony traversal is designed in which each ant traverses its own path, and the solution traverses by each ant is integrated to solve the problem. Experiments show that the proposed method effectively improves the query speed of personalized multimedia search engines, reduces redundant information, and improves the user search experience.

Deep Attention-Guided Hashing

With the rapid growth of multimedia data (e.g., image, audio and video etc.) on the web, learning-based hashing techniques such as Deep Supervised Hashing (DSH) have proven to be very efficient for large-scale multimedia search. The recent successes seen in Learning-based hashing methods are largely due to the success of deep learning-based hashing methods. However, there are some limitations to previous learning-based hashing methods (e.g., the learned hash codes containing repetitive and highly correlated information). In this paper, we propose a novel learning-based hashing method, named Deep Attention-guided Hashing (DAgH). DAgH is implemented using two stream frameworks. The core idea is to use guided hash codes which are generated by the hashing network of the first stream framework (called first hashing network) to guide the training of the hashing network of the second stream framework (called second hashing network). Specifically, in the first network, it leverages an attention network and hashing network to generate the attention-guided hash codes from the original images. The loss function we propose contains two components: the semantic loss and the attention loss. The attention loss is used to punish the attention network to obtain the salient region from pairs of images; in the second network, these attention-guided hash codes are used to guide the training of the second hashing network (i.e., these codes are treated as supervised labels to train the second network). By doing this, DAgH can make full use of the most critical information contained in images to guide the second hashing network in order to learn efficient hash codes in a true end-to-end fashion. Results from our experiments demonstrate that DAgH can generate high quality hash codes and it outperforms current state-of-the-art methods on three benchmark datasets, CIFAR-10, NUS-WIDE, and ImageNet.