Multimedia Retrieval Research Articles

Joint Generative Image Deblurring Aided by Edge Attention Prior and Dynamic Kernel Selection

Image deblurring is a classic and important problem in industrial fields, such as aviation photo restoration, object recognition in robotics, and autonomous vehicles. Blurry images in real‐world scenarios consist of mixed blurring types, such as a natural motion blurring owing to shaking of the camera. Fast deblurring does not deblur the entire image because it is not the best option. Considering the computational costs, it is also better to have an alternative kernel to deblur different objects at a high‐semantic level. To achieve better image restoration quality, it is also beneficial to combine the blurring category location and important structural information in terms of specific artifacts and degree of blurring. The goal of blind image deblurring is to restore sharpness from the unknown blurring kernel of an image. Recent deblurring methods tend to reconstruct prior knowledge, neglecting the influence of blur estimation and visual fidelity on image details and structure. Generative adversarial networks(GANs) have recently been attracting considerable attention from both academia and industry because GAN can perfectly generate new data with the same statistics as the training set. Therefore, this study proposes a generative neural architecture and an edge attention algorithm developed to restore vivid multimedia patches. Joint edge generation and image restoration techniques are designed to solve the low‐level multimedia retrieval. This multipath refinement fusion network (MRFNet) can not only perform deblurring of images directly but also individual the frames separately from videos. Ablation experiments validate that our generative adversarial network MRFNet performs better in joint training than in multimodel. Compared to other GAN methods, our two‐phase method exhibited state‐of‐the‐art performance in terms of speed and accuracy as well as has a significant visual improvement.

Semantic-Driven Interpretable Deep Multi-Modal Hashing for Large-Scale Multimedia Retrieval

Multi-modal hashing focuses on fusing different modalities and exploring the complementarity of heterogeneous multi-modal data for compact hash learning. However, existing multi-modal hashing methods still suffer from several problems, including: 1) Almost all existing methods generate unexplainable hash codes. They roughly assume that the contribution of each hash code bit to the retrieval results is the same, ignoring the discriminative information embedded in hash learning and semantic similarity in hash retrieval. Moreover, the length of hash code is empirically set, which will cause bit redundancy and affect retrieval accuracy. 2) Most existing methods exploit shallow models which fail to fully capture higher-level correlation of multi-modal data. 3) Most existing methods adopt online hashing strategy based on immutable direct projection, which generates query codes for new samples without considering the differences of semantic categories. In this paper, we propose a Semantic-driven Interpretable Deep Multi-modal Hashing (SIDMH) method to generate interpretable hash codes driven by semantic categories within a deep hashing architecture, which can solve all these three problems in an integrated model. The main contributions are: 1) A novel deep multi-modal hashing network is developed to progressively extract hidden representations of heterogeneous modality features and deeply exploit the complementarity of multi-modal data. 2) Learning interpretable hash codes, with discriminant information of different categories distinctively embedded into hash codes and their different impacts on hash retrieval intuitively explained. Besides, the code length depends on the number of categories in the dataset, which can reduce the bit redundancy and improve the retrieval accuracy. 3) The semantic-driven online hashing strategy encodes the significant branches and discards the negligible branches of each query sample according to the semantics contained in it, therefore it could capture different semantics in dynamic queries. Finally, we consider both the nearest neighbor similarity and semantic similarity of hash codes. Experiments on several public multimedia retrieval datasets validate the superiority of the proposed method.

WITHDRAWN: Electronic Medical Record Design and Glaucoma Surgery Rehabilitation Nursing Based on Embedded System

Abstract The health sector of developing countries, India, such as mobile computing and cloud computing, has begun to use the latest new technology. Glaucoma is a chronic neurodegenerative disease of the optic nerve. The purpose of therapy is to prevent optic nerve damage's onset and progression by lowering intraocular pressure. The current treatment options, local/systemic drugs, reduced the production of increasing or aqueous humor, including laser treatment targeting the trabecular meshwork and ciliary body and the incision surgery. The collection of enormous capacity, storage, processing, and digital multimedia retrieval is referred to as electronic medical records. Electronic medical records have been constructed with high patient safety attention images. Surgery, documentation of the management process, despite the poor compliance with mandatory tolerated dose therapy, and the case is the largest in glaucomatous optic neuropathy. Please provide an overview of the surgeon to the available treatment options for glaucoma. The possible surgical method, change over an improved time to improve the patient's prognosis and the visual recovery, is full of intraoperative, postoperative complications. Each possible surgical treatment plan's risks and benefits are strictly evaluated; will need to adjust to meet a particular patient's needs. Here is the current evidence is insufficient to establish any of the advantages of these surgeries on top of the other person. The possible surgical method, change over an improved time to improve the patient's prognosis and the visual recovery, is full of intraoperative, postoperative complications.

Semantics-preserving hashing based on multi-scale fusion for cross-modal retrieval

Research on hash-based cross-modal retrieval has been a hotspot in the field of content-based multimedia retrieval research. Most deep cross-modal hashing methods only consider inter-modal loss that can remain local information of training data, and ignore the loss within data samples of the same modality that can remain the global information of dataset. In addition, they also ignore the factor that different scales of single modal data contain different semantic information, which affects the representation of data features. In this paper, we propose a semantics-preserving hashing method based on multi-scale fusion. More concretely, a multi-scale fusion pooling model is proposed for both image feature training network and text feature training network. Therefore, we can extract the multi-scale features of image dataset and solve the sparsity problem of text BOW vectors. When constructing the loss function, we consider intra-modal loss while considering inter-modal loss. Therefore, the output hash code retains both global and local underlying semantic correlation when image and text feature training network are trained. Experiment results on NUS-WIDE and MIRFlickr-25 K prove that against other existing methods, our algorithm improves cross-modal retrieval accuracy.

Fast Discrete Collaborative Multi-Modal Hashing for Large-Scale Multimedia Retrieval

Many achievements have been made on learning to hash for uni-modal and cross-modal retrieval. However, it is still an unsolved problem that how to directly and efficiently learn discriminative discrete hash codes for the multimedia retrieval, where both query and database samples are represented with heterogeneous multi-modal features. With this motivation, we propose a Fast Discrete Collaborative Multi-modal Hashing (FDCMH) method in this paper. We first propose an efficient collaborative multi-modal mapping that first transforms heterogeneous multi-modal features into the unified factors to exploit the complementarity of multi-modal features and preserve the semantic correlations in multiple modalities with linear computation and space complexity. Such shared factors also bridge the heterogeneous modality gap and remove the inter-modality redundancy. Further, we develop an asymmetric hashing learning module to simultaneously correlate the learned hash codes with low-level data distribution and high-level semantics. In particular, this design could avoid the challenging symmetric semantic matrix factorization and $O(n^2)$ O ( n 2 ) memory cost ( $n$ n is the number of training samples). It can support both computation and memory efficient discrete hash optimization. Experiments on several public multimedia retrieval datasets demonstrate the superiority of the proposed approach compared with state-of-the-art hashing techniques, in terms of both model learning efficiency and retrieval accuracy.

Joint Versus Independent Multiview Hashing for Cross-View Retrieval.

Thanks to the low storage cost and high query speed, cross-view hashing (CVH) has been successfully used for similarity search in multimedia retrieval. However, most existing CVH methods use all views to learn a common Hamming space, thus making it difficult to handle the data with increasing views or a large number of views. To overcome these difficulties, we propose a decoupled CVH network (DCHN) approach which consists of a semantic hashing autoencoder module (SHAM) and multiple multiview hashing networks (MHNs). To be specific, SHAM adopts a hashing encoder and decoder to learn a discriminative Hamming space using either a few labels or the number of classes, that is, the so-called flexible inputs. After that, MHN independently projects all samples into the discriminative Hamming space that is treated as an alternative ground truth. In brief, the Hamming space is learned from the semantic space induced from the flexible inputs, which is further used to guide view-specific hashing in an independent fashion. Thanks to such an independent/decoupled paradigm, our method could enjoy high computational efficiency and the capacity of handling the increasing number of views by only using a few labels or the number of classes. For a newly coming view, we only need to add a view-specific network into our model and avoid retraining the entire model using the new and previous views. Extensive experiments are carried out on five widely used multiview databases compared with 15 state-of-the-art approaches. The results show that the proposed independent hashing paradigm is superior to the common joint ones while enjoying high efficiency and the capacity of handling newly coming views.

Online multimedia retrieval on CPU–GPU platforms with adaptive work partition

Nearest neighbors search is a core operation found in several online multimedia services. These services have to handle very large databases, while, at the same time, they must minimize the query response times observed by users. This is specially complex because those services deal with fluctuating query workloads (rates). Consequently, they must adapt at run-time to minimize the response times as the load varies. In this paper, we address the aforementioned challenges with a distributed memory parallelization of the product quantization nearest neighbor search, also known as IVFADC, for hybrid CPU–GPU machines. Our parallel IVFADC implements an out-of-GPU memory execution scheme to use the GPU for databases in which the index does not fit in its memory, which is crucial for searching in very large databases. The careful use of CPU and GPU with work stealing led to an average response time reduction of 2.4× as compared to using the GPU only. Also, our approach to adapt the system to fluctuating loads, called Dynamic Query Processing Policy (DQPP), attained a response time reduction of up to 5× vs. the best static (BS) policy for moderate loads. The system has attained high query processing rates and near-linear scalability in all experiments. We have evaluated our system on a machine with up to 256 NVIDIA V100 GPUs processing a database of 256 billion SIFT features vectors.

Inductive Structure Consistent Hashing via Flexible Semantic Calibration.

Semantic-preserving hashing establishes efficient multimedia retrieval by transferring knowledge from original data to hash codes so that the latter can preserve the underlying visual and semantic similarities. However, it becomes a crucial bottleneck: how to effectively bridge the trilateral domain gaps (i.e., the visual, semantic, and hashing spaces) to further improve the retrieval accuracy. In this article, we propose an inductive structure consistent hashing (ISCH) method, which can interactively coordinate the semantic correlations between the visual feature space, the binary class space, and the discrete hashing space. Specifically, an inductive semantic space is formulated by a simple multilayer stacking class-encoder, which transforms the naive class information into flexible semantic embeddings. Meanwhile, we design a semantic dictionary learning model to facilitate the bilateral visual-semantic bridging and guide the class-encoder toward reliable semantics, which could well alleviate the visual-semantic bias problem. In particular, the visual descriptors and respective semantic class representations are regularized with a coinciding alignment module. In order to generate privileged hash codes, we further explore semantic and prototype binary code learning to jointly quantify the semantic and latent visual representations into unified discrete hash codes. Moreover, an efficient optimization algorithm is developed to address the resulting discrete programming problem. Comprehensive experiments conducted on four large-scale data sets, i.e., CIFAR-10, NUSWIDE, ImageNet, and MSCOCO, demonstrate the superiority of our method over the state-of-the-art alternatives against different evaluation protocols.

Human motion recognition based on limit learning machine

With the development of technology, human motion capture data have been widely used in the fields of human–computer interaction, interactive entertainment, education, and medical treatment. As a problem in the field of computer vision, human motion recognition has become a key technology in somatosensory games, security protection, and multimedia information retrieval. Therefore, it is important to improve the recognition rate of human motion. Based on the above background, the purpose of this article is human motion recognition based on extreme learning machine. Based on the existing action feature descriptors, this article makes improvements to features and classifiers and performs experiments on the Microsoft model specific register (MSR)-Action3D data set and the Bonn University high density metal (HDM05) motion capture data set. Based on displacement covariance descriptor and direction histogram descriptor, this article described both combine to produce a new combination; the description can statically reflect the joint position relevant information and at the same time, the change information dynamically reflects the joint position, uses the extreme learning machine for classification, and gets better recognition result. The experimental results show that the combined descriptor and extreme learning machine recognition rate on these two data sets is significantly improved by about 3% compared with the existing methods.

HMR-vid: a comparative analytical survey on human motion recognition in video data

According to the rapid spread of multimedia data and online observations by users, the importance of researching on machine vision also, analyzing and automatic understanding of video data content is progressively increasing. Human motion recognition in video data is a crucial research subject in machine vision science that has plenty of applications, for instance, video surveillance, video indexing, robotics, human-computer interface and multimedia retrieval. Despite a high number of researches conducted on this topic, there is a necessity to achieve a more in-depth understanding, complete classification, and evaluation of existing human motion recognition stages. The novelty of this paper, our comparative analytical framework includes three major parts. Firstly, three different stages are introduced in recognizing human motion consisting of background subtraction, feature extraction, and machine learning classification. Secondly, five essential criteria are defined for evaluating the proposed human motion recognition methods. Finally, our comparative analysis of human motion recognition stages comprises two models. The analysis of background subtraction methods is based on applying the criteria for a qualitative comparison. Next, the feature extraction and machine learning classification methods are examined by specifying their main idea, benefits and challenges. Our comparative analytical framework can be beneficial for every researcher in this field by simplifying accurate selection and development of human motion recognition methods in future works.

Automatic Identification of Individual Primates with Deep Learning Techniques.

SummaryThe difficulty of obtaining reliable individual identification of animals has limited researcher's ability to obtain quantitative data to address important ecological, behavioral, and conservation questions. Traditional marking methods placed animals at undue risk. Machine learning approaches for identifying species through analysis of animal images has been proved to be successful. But for many questions, there needs a tool to identify not only species but also individuals. Here, we introduce a system developed specifically for automated face detection and individual identification with deep learning methods using both videos and still-framed images that can be reliably used for multiple species. The system was trained and tested with a dataset containing 102,399 images of 1,040 individuals across 41 primate species whose individual identity was known and 6,562 images of 91 individuals across four carnivore species. For primates, the system correctly identified individuals 94.1% of the time and could process 31 facial images per second.

Neural relational inference for disaster multimedia retrieval

Events around the world are increasingly documented on social media, especially by the people experiencing them, as these platforms become more popular over time. As a consequence, social media turns into a valuable source of data for understanding those events. Due to their destructive potential, natural disasters are among events of particular interest to response operations and environmental monitoring agencies. However, this amount of information also makes it challenging to identify relevant content pertaining to those events. In this paper, we use a relational neural network model for identifying this type of content. The model is particularly suitable for unstructured text, that is, text with no particular arrangement of words, such as tags, which is commonplace in social media data. In addition, our method can be combined with a CNN for handling multimodal data where text and visual data are available. We perform experiments in three different scenarios, where different modalities are evaluated: visual, textual, and both. Our method achieves competitive performance in both modalities by themselves, while significantly outperforms the baseline on the multimodal scenario. We also demonstrate the behavior of the proposed method in different applications by performing additional experiments in the CUB-200-2011 multimodal dataset.

Semi-supervised discrete hashing for efficient cross-modal retrieval

Cross-modal hashing has recently gained significant popularity to facilitate multimedia retrieval across different modalities. Since the acquisition of large-scale labeled training data are very labor intensive, most supervised cross-modal hashing methods are uncompetitive for real applications. With limited label available, this paper presents a novel S emi-S upervised D iscrete H ashing (SSDH) for efficient cross-modal retrieval. In contrast to most semi-supervised cross-modal hashing works that need to predict the label of unlabeled data, our proposed approach groups the labeled and unlabeled data together, and exploits the informative unlabeled data to promote hashing code learning directly. Specifically, the proposed SSDH approach utilizes the relaxed hash representations to characterize each modality, and learns the semi-supervised semantic-preserving regularization to correlate the semantic consistency between the heterogeneous modalities. Accordingly, an efficient objective function is proposed to learn the hash representation, while designing an efficient optimization algorithm to optimize the hash codes for both labeled and unlabeled data. Without sacrificing the retrieval performance, the proposed SSDH method is adaptive to benefit various kinds of retrieval tasks, i.e., unsupervised, semi-supervised and supervised. Experimental results compared with several competitive algorithms show the effectiveness of the proposed method and its superiority over state-of-the-arts.

Robust Unsupervised Cross-modal Hashing for Multimedia Retrieval

With the quick development of social websites, there are more opportunities to have different media types (such as text, image, video, etc.) describing the same topic from large-scale heterogeneous data sources. To efficiently identify the inter-media correlations for multimedia retrieval, unsupervised cross-modal hashing (UCMH) has gained increased interest due to the significant reduction in computation and storage. However, most UCMH methods assume that the data from different modalities are well paired. As a result, existing UCMH methods may not achieve satisfactory performance when partially paired data are given only. In this article, we propose a new-type of UCMH method called robust unsupervised cross-modal hashing ( RUCMH ). The major contribution lies in jointly learning modal-specific hash function, exploring the correlations among modalities with partial or even without any pairwise correspondence, and preserving the information of original features as much as possible. The learning process can be modeled via a joint minimization problem, and the corresponding optimization algorithm is presented. A series of experiments is conducted on four real-world datasets (Wiki, MIRFlickr, NUS-WIDE, and MS-COCO). The results demonstrate that RUCMH can significantly outperform the state-of-the-art unsupervised cross-modal hashing methods, especially for the partially paired case, which validates the effectiveness of RUCMH.

MediaEval multimedia evaluation benchmark

MediaEval is a benchmarking initiative that offers challenges in multimedia retrieval, analysis and exploration. The tasks offered by MediaEval concentrate specifically on the human and social aspects of multimedia. They encourage researchers to bring together multiple modalities (visual, text, audio) and to think in terms of systems that serve users. Our larger aim is to promote reproducible research that makes multimedia a positive force for society. In order to provide an impression of the topical scope of MediaEval, we describe a few examples of typical tasks.

Deep Collaborative Discrete Hashing With Semantic-Invariant Structure Construction

While deep hashing has made great progress in large-scale multimedia retrieval, most of the existing approaches under-explore the semantic correlations and neglect the effect of context-aware visual learning. In this paper, we propose a dual-stream learning framework, termed as Deep Collaborative Discrete Hashing (DCDH), which constructs a discriminative common discrete space by collaboratively incorporating the shared and individual semantics deduced from visual features and semantics. Specifically, DCDH generates context-aware representations by employing the outer product of visual embeddings and semantic encodings. To further preserve the original semantics and alleviate the class imbalance problem, we introduce the focal loss to take advantage of frequent and rare concepts. Furthermore, a common binary code space is constructed based on the joint learning of the visual representations, the context-aware representations, and the label distribution calibration. Three losses, i.e., the pairwise similarity loss, the quantization loss, and the balanced classification loss, are collaboratively optimized in the general learning framework of DCDH. Extensive experiments conducted on three large-scale benchmark datasets demonstrate the superiority of the proposed method, yielding the state-of-the-art image retrieval performance.

Video Human Behavior Recognition Based on ISA Deep Network Model

Vision-based behavior recognition is the analysis and recognition of human behavior in video. It has been widely used in many aspects such as multimedia information retrieval, behavior monitoring, and robot perception. This paper uses the Independent Subspace Analysis (ISA) deep network model feature extraction method, which is based on the ISA model and neural network theory, and combines data preprocessing methods, [Formula: see text]-means clustering methods, and Support Vector Machine (SVM) classifiers to achieve video classification and identification of human behavior. The ISA-based deep network model feature extraction method is an unsupervised learning method that can obtain behavior characteristics with good invariance and characterization capabilities in video human behavior. The experiment was conducted on the basis of the Hollywood2 human behavior data set. This experiment was compared with other commonly used human behavior feature extraction and recognition methods. The experimental results validated the effectiveness and advantages of this method in the classification and recognition of human behavior.

Deep Loss Driven Multi-Scale Hashing Based on Pyramid Connected Network

Thanks to the great success of the deep learning, deep hashing for large-scale multimedia retrieval has made significant progress recently. However, most existing deep hashing algorithms suffer from slow convergence due to the gradient vanishing problem, caused by deep network structures and saturated activation functions. Moreover, a single convolution layer is often followed by down-sampling such as max pooling, resulting in local information loss that might affect the overall system robustness and performance. In this work, we propose a novel deep supervised hashing, Deep Loss Driven Multi-Scale Hashing (DLDMSH), which learns the high-quality approximate binary codes through an end-to-end network and improves the representative capacity of hash codes for large-scale image retrieval. Specifically, we design a Loss Driven Multi-Scale (LDMS) feature which is aggregated from convolutional feature maps. Moreover, a Pyramid Connected Convolutional Neural Network (PCNet) architecture is devised to generate LDMS feature, which inputs pairs of images during the training and outputs an image to approximate discrete values. In particular, 1 × 1 convolution kernels are applied to make a linear combination of features for realizing feature reduction, and the reduced features are fused in the fusion layer. This effectively improves the performance of deep features. A novel loss function preserving semantic information is integrated into an end-to-end learning scheme, which enhances the representative capacity of binary codes. Extensive experiments over four benchmark datasets show that DLDMSH significantly outperforms several other state-of-the-art hashing methods.

Sharing emotions: determining films’ evoked emotional experience from their online reviews

Online reviews are broadly believed to reflect consumers’ opinions towards the reviewed items. In this work, we postulate that online reviews for experience goods also reflect something very different, the reviewer’s emotions while experiencing the item. We study the case of films, which are made with the intent of evoking an emotional response. We postulate that the emotions that the viewer experienced while watching the movie are reflected in their online review, and can be reliably extracted from it. We aggregate these emotional experiences to create an emotional signature for the film—the emotions that it tends to evoke in viewers. We conduct a series of validations, each designed to offer evidence that the emotional signature indeed reflects the emotions that the film evokes in viewers. Our results demonstrate that experience goods’ online reviews can be utilized for extracting and understanding users’ emotional experience. Items’ induced emotions can be used in affective recommender systems and affective multimedia retrieval. In addition, users’ ability to easily see in advance the emotions that the item has induced in others, is relevant for research on the effect of expected emotions on decision-making.

Deep balanced discrete hashing for image retrieval

Hashing has been widely used for large-scale multimedia retrieval because of its advantages in storage and retrieval efficiency. Traditional supervised hash methods represent an image as a feature vector and then perform a separate quantization step to generate a binary code. Due to the difficulty of discrete optimization of hash codes, continuous relaxation is generally used to replace discrete optimization. However, the process of continuous relaxation leads to inevitable quantization error. To avoid this drawback, a deep balanced discrete hashing method is proposed, which uses discrete gradient propagation with the straight-through estimator. The proposed method does not use the traditional continuous relaxation strategy, thereby reducing the quantization error caused by continuous relaxation. And the proposed method uses supervised information to directly guide the discrete coding and deep feature learning process. In the proposed method, the last layer of the Convolutional Neural Network (CNN) outputs the binary code directly. In the loss function, discrete values are calculated by combining the pairwise loss and a balance controlling term. The learned binary hash code maintains the similar relationship and label consistency at the same time. While maintaining the pairwise similarity, the proposed method keeps the balance of hash codes to improve retrieval performance. Extensive experiments show that the proposed method outperforms the state-of-the-art hashing methods on four image retrieval benchmark datasets.