NUS-WIDE Datasets Research Articles

Double Attention for Multi-Label Image Classification

Multi-label image classification is an essential task in image processing. How to improve the correlation between labels by learning multi-scale features from images is a very challenging problem. We propose a Double Attention Network (DAN) to improve the correlation between image feature regions and labels, as well as between labels and labels. Firstly, the dynamic learning strategy is used to extract the multi-scale features of the image to solve the problem of inconsistent scale of objects in the image. Secondly, in order to improve the correlation between the image feature regions and the labels, we use the spatial attention module to focus on the important regions of the image to learn their salient features, while we use the channel attention module to model the correlation between the channels to improve the correlation between the labels. Finally, the output features of two attention modules are fused as one multi-label image classification model. Experiments on MS-COCO 2014 dataset, Pascal VOC 2007 dataset and NUS-WIDE dataset demonstrate that our model is significantly better than the state-of-the-art models. Besides, visualization analyses show that our model has a strong ability for image salient feature learning and label correlation capturing.

Personalized Recommendation of Social Images by Constructing a User Interest Tree With Deep Features and Tag Trees

In view of the great diversity and complexity of social images, it is of great significance to improve the performance of personalized recommendation by learning a user interest from large-scale social images. Deep learning, as the latest research in the field of artificial intelligence, provides a new personalized recommendation solution of social images for learning a user's interest. Moreover, social image sharing websites (such as Flickr) allow users to tag uploaded images with tags. As an important image semantic cue, effective tags not only represent the latent image information but also show personalized user interest. Therefore, a personalized recommendation method of social image is proposed by constructing a user-interest tree with deep features and tag trees in this paper. The main contributions of our paper are as follows: first, to efficiently make use of tags, a tag tree of social images is created by the re-ranked tags; second, for compactly representing the image content, deep features are learned by training the AlexNet network; third, a user-interest tree is constructed with deep features and tag trees that include the user-interest tree of social images and the user-interest tree of tags, respectively, and finally, a personalized recommendation system of social images is built based on a user-interest tree. Experiments on the NUS-WIDE dataset have shown that our method outperforms state-of-the-art methods in terms of both precision and recall of personalized recommendations.

Multiview Uncorrelated Locality Preserving Projection.

Canonical Correlation Analysis (CCA) is a popular multiview dimension reduction method, which aims to maximize the correlation between two views to find the common subspace shared by these two views. However, it can only deal with two-view data, while the number of views frequently exceeds two in many real applications. To handle data with more than two views, in the previous studies, either the pairwise correlation or the high-order correlation was employed. These two types of correlation define the relation of multiview data from different viewpoints, and both have special effects for view consistency. To obtain flexible view consistency, in this article, we propose multiview uncorrelated locality preserving projection (MULPP), which considers two types of correlation simultaneously. The MULPP also considers the complementary property of different views by preserving the local structures of all the views. To obtain multiple projections and minimize the redundancy of low-dimensional features, for each view, the MULPP makes the features extracted by different projections uncorrelated. The MULPP is solved by an iteration algorithm, and the convergence of the algorithm is proven. The experiments on Multiple Feature, Coil-100, 3Sources, and NUS-WIDE data sets demonstrate the effectiveness of MULPP.

Multi-view adaptive semi-supervised feature selection with the self-paced learning

In recent decades, semi-supervised feature selection (SSFS) has gained extensive research in the field of machine learning and computer vision. Most SSFS algorithms are based on graph-based semi-supervised learning (GSSL), and their performance depend heavily on the quality of the Laplacian weight graph. However, the Laplacian weight graph can't be changed once it is constructed, which greatly restricts the performance of SSFS. To address this defect, in this paper we propose a novel Multi-view Adaptive Semi-supervised Feature Selection (MASFS) algorithm, which introduces the self-paced learning (SPL) into SSFS to make the Laplacian weight graph adaptively change according to the current predicted information. Meanwhile, the MASFS algorithm utilizes the multi-view learning to effectively explore the complementary and related information contained in different views to enhance SSFS performance. We propose a valid iterative algorithm for optimizing the objective function, followed by the convergence analysis and the complexity analysis. To illustrate the effectiveness of the MASFS algorithm, some experiments are carried out on NUS-WIDE dataset and MSRA-MM2.0 dataset and the experimental results indicate that MASFS has better performance than other SSFS algorithms based on GSSL.

Self-supervised learning-based weight adaptive hashing for fast cross-modal retrieval

Due to the low storage cost and fast search speed, hashing is widely used in cross-modal retrieval. However, there still remain some crucial bottlenecks: Firstly, there are not suitable big datasets for multimodal data. Secondly, imbalance instances will affect the accuracy of the retrieval system. In this paper, we propose an end-to-end self-supervised learning-based weight adaptive hashing method for cross-modal retrieval. For the restriction of datasets, we use the self-supervised fashion to directly extract fine-grained features from labels and use them to supervise the hashing learning of other modalities. To overcome the problem of imbalance instances, we design an adaptive weight loss to flexibly adjust the weight of training samples according to their proportions. Besides these, we also use a binary approximation regularization term to reduce the regularization error. Experiments on MIRFLICKR-25K and NUS-WIDE datasets show that our method can improve 3% performance compared to other methods.

Result diversification in image retrieval based on semantic distance

User requirements for result diversification in image retrieval have been increasing with the explosion of image resources. Result diversification requires that image retrieval systems are made capable of handling semantic gaps between image visual features and semantic concepts, and providing both relevant and diversified image results. Context information, such as captions, descriptions, and tags, provides opportunities for image retrieval systems to improve their result diversification. This study explores a mechanism for improving result diversification using the semantic distance of image social tags. We design and compare nine strategies that combine three different semantic distance algorithms (WordNet, Google Distance, and Explicit Semantic Analysis) with three re-ranking algorithms (MMR, xQuAD, and Score Difference) for result diversification. In order to better prove the effectiveness of our strategy of applying semantic information, we also make use of visual features of images for result diversification experiment and make comparison. Our data for experimentation were extracted from 269,648 images selected from the NUS-WIDE datasets with manually annotated subtopics. Experimental results affirm the effectiveness of applying semantic information for improving result diversification in image retrieval. In particular, WordNet-based semantic distance combined with the Score Difference (WordNet-DivScore) outperformed other strategies in diversifying image retrieval results.

Multimodal Multiclass Boosting and its Application to Cross-modal Retrieval

Although Boosting approach has been proved to be a very successful ensemble learning technology, the conventional ones are limited to two classes or single modality. In this paper, to deal with multiclass setting and heterogeneous modalities, we propose a multimodal multiclass Boosting framework called MMBoost, in which the intra-modal semantic information and inter-modal semantic correlation can be captured at the same time. By utilizing the multiclass exponential and logistic loss functions, we further acquire two new versions of MMBoost, namely MMBoost_exp and MMBoost_log. The empirical risk, which simultaneously considers the intra-modal and inter-modal losses, is designed and then minimized by gradient descent in the multidimensional functional spaces. More concretely, the optimization problem is solved in turn for each modality. The posterior probability of semantic category can be naturally attained by applying sigmoid function to the multiclass margin. A series of experiments on the Wiki and NUS-WIDE datasets demonstrate that the performance of our proposed method significantly outperforms those of existing Boosting approaches for cross-modal retrieval.

Structured Label Inference for Visual Understanding

Visual data such as images and videos contain a rich source of structured semantic labels as well as a wide range of interacting components. Visual content could be assigned with fine-grained labels describing major components, coarse-grained labels depicting high level abstractions, or a set of labels revealing attributes. Such categorization over different, interacting layers of labels evinces the potential for a graph-based encoding of label information. In this paper, we exploit this rich structure for performing graph-based inference in label space for a number of tasks: multi-label image and video classification and action detection in untrimmed videos. We consider the use of the Bidirectional Inference Neural Network (BINN) and Structured Inference Neural Network (SINN) for performing graph-based inference in label space and propose a Long Short-Term Memory (LSTM) based extension for exploiting activity progression on untrimmed videos. The methods were evaluated on (i) the Animal with Attributes (AwA), Scene Understanding (SUN) and NUS-WIDE datasets for multi-label image classification, (ii) the first two releases of the YouTube-8M large scale dataset for multi-label video classification, and (iii) the THUMOS'14 and MultiTHUMOS video datasets for action detection. Our results demonstrate the effectiveness of structured label inference in these challenging tasks, achieving significant improvements against baselines.

VSE-ens: Visual-Semantic Embeddings with Efficient Negative Sampling

Jointing visual-semantic embeddings (VSE) have become a research hotpot for the task of image annotation, which suffers from the issue of semantic gap, i.e., the gap between images' visual features (low-level) and labels' semantic features (high-level). This issue will be even more challenging if visual features cannot be retrieved from images, that is, when images are only denoted by numerical IDs as given in some real datasets. The typical way of existing VSE methods is to perform a uniform sampling method for negative examples that violate the ranking order against positive examples, which requires a time-consuming search in the whole label space. In this paper, we propose a fast adaptive negative sampler that can work well in the settings of no figure pixels available. Our sampling strategy is to choose the negative examples that are most likely to meet the requirements of violation according to the latent factors of images. In this way, our approach can linearly scale up to large datasets. The experiments demonstrate that our approach converges 5.02x faster than the state-of-the-art approaches on OpenImages, 2.5x on IAPR-TCI2 and 2.06x on NUS-WIDE datasets, as well as better ranking accuracy across datasets.

Multimedia automatic annotation by mining label set correlation

Organizing and retrieving multimedia data heavily rely on the relevant textual descriptions. Multimedia automatic annotation, which assigns text labels to multimedia samples, has been widely studied. Among others, search-based annotation methods are well suited for annotation tasks on large-scale datasets and are studied in depth because of their simplicity and scalability. However, classical search based annotation methods address this problem by treating each label independently, which ignores the correlation between different labels in the assigned label set. This paper aims to integrate the relevant information of the label set with respect to the multimedia content and the inner correlated information of the label set into a joint learning framework. We evaluate the performance of the proposed method on MIRFLICKR-25000 and NUS-WIDE datasets. Experimental results show that the proposed annotation method achieves excellent performance.

Large scale automatic image annotation based on convolutional neural network

Automatic image annotation is one of the most important challenges in computer vision, which is critical to many real-world researches and applications. In this paper, we focus on the issue of large scale image annotation with deep learning. Firstly, considering the existing image data, especially the network images, most of the labels of themselves are inaccurate or imprecise. We propose a Multitask Voting (MV) method, which can improve the accuracy of original annotation to a certain extent, thereby enhancing the training effect of the model. Secondly, the MV method can also achieve the adaptive label, whereas most existing methods pre-specify the number of tags to be selected. Additionally, based on convolutional neural network, a large scale image annotation model MVAIACNN is constructed. Finally, we evaluate the performance with experiments on the MIRFlickr25K and NUS-WIDE datasets, and compare with other methods, demonstrating the effectiveness of the MVAIACNN.

Global similarity preserving hashing

Hashing learning has attracted increasing attention these years with the explosive increase in data volume. Most existing hashing learning methods can be divided into two stages. Firstly, obtain low-dimensional representation of the original data. Secondly, quantize the low-dimensional representation of each sample and map them to binary codes. This two-stage hashing framework separates projection operation and quantization operation apart, and the original data structure cannot be well preserved after this kind of two-stage operation. Considering this, global similarity preserving hashing (GSPH) is proposed, which utilizes a joint hashing framework to directly project the original data to hamming space, and reduces the projection error and the quantization loss simultaneously. Moreover, GSPH presents a global similarity-based data sample reconstruction method, which describes the intrinsic manifold structure of original data more precisely. The image retrieval experimental results on Corel, CIFAR, LabelMe and NUS-WIDE datasets illustrate that our algorithm outperforms several other state-of-the-art methods.

Uni- and multimodal methods for single- and multi-label recognition

The multimodal approach is becoming more and more attractive and common method in multimedia information retrieval and description. It often shows better content recognition results than using only unimodal methods, but depending on the used data, this is not always the case. Most of the current multimodal media content classification methods still depend on unimodal recognition results. For both uni- and multimodal approaches it is important to choose the best features and classification models. In addition, in the case of unimodal models, the final multimodal recognitions still need to be produced with an appropriate late fusion technique. In this article, we study several multi- and unimodal recognition methods, features for them and their combination techniques, in the application setup of concept detection in image–text data. We consider both single- and multi-label recognition tasks. As the image features, we use GoogLeNet deep convolutional neural network (DCNN) activation features and semantic concept or classeme vectors. For text features, we use simple binary vectors for tags and the word2vec embedding vectors. The Multimodal Deep Boltzmann Machine (DBM) model is used as the multimodal model and the Support Vector Machine (SVM) with both linear and non-linear radial basis function (RBF) kernels as the unimodal one. The experiments are performed with the MIRFLICKR-1M and the NUS-WIDE datasets. The results show that the two models have equally good performance in the single-label recognition task of the former database, while the Multimodal DBM produces clearly better results in the multi-label task of the latter database. Compared with the results in the literature, we exceed the state of the art in both datasets, mostly due to the use of DCNN features and semantic concept vectors based on them.

Image Re-Ranking Based on Topic Diversity.

Social media sharing Websites allow users to annotate images with free tags, which significantly contribute to the development of the web image retrieval. Tag-based image search is an important method to find images shared by users in social networks. However, how to make the top ranked result relevant and with diversity is challenging. In this paper, we propose a topic diverse ranking approach for tag-based image retrieval with the consideration of promoting the topic coverage performance. First, we construct a tag graph based on the similarity between each tag. Then, the community detection method is conducted to mine the topic community of each tag. After that, inter-community and intra-community ranking are introduced to obtain the final retrieved results. In the inter-community ranking process, an adaptive random walk model is employed to rank the community based on the multi-information of each topic community. Besides, we build an inverted index structure for images to accelerate the searching process. Experimental results on Flickr data set and NUS-Wide data sets show the effectiveness of the proposed approach.

Supervised Coarse-to-Fine Semantic Hashing for cross-media retrieval

Due to its storage efficiency and fast query speed, cross-media hashing methods have attracted much attention for retrieving semantically similar data over heterogeneous datasets. Supervised hashing methods, which utilize the labeled information to promote the quality of hashing functions, achieve promising performance. However, the existing supervised methods generally focus on utilizing coarse semantic information between samples (e.g. similar or dissimilar), and ignore fine semantic information between samples which may degrade the quality of hashing functions. Accordingly, in this paper, we propose a supervised hashing method for cross-media retrieval which utilizes the coarse-to-fine semantic similarity to learn a sharing space. The inter-category and intra-category semantic similarity are effectively preserved in the sharing space. Then an iterative descent scheme is proposed to achieve an optimal relaxed solution, and hashing codes can be generated by quantizing the relaxed solution. At last, to further improve the discrimination of hashing codes, an orthogonal rotation matrix is learned by minimizing the quantization loss while preserving the optimality of the relaxed solution. Extensive experiments on widely used Wiki and NUS-WIDE datasets demonstrate that the proposed method outperforms the existing methods.

Salient object detection of social images based on semantic tag context

Salient object detection is an important process for machines to understand visual contents as humans. Typically, most previous studies on salient object detection infer salient map by only using the visual features. In this paper, we propose a new paradigm on salient object detection, which aims at producing more reliable results by mining the context information from user's annotated tags. To address this problem, we firstly construct a large scale salient object dataset, which includes 5429 images from the NUSWIDE dataset (a real world web image database from National University of Singapore) with tag information and accurate human-labeled masks. Moreover, a specialised conditional random field (CRF) model is also proposed which takes account of both tag contexts and appearance cues. Extensive experiments show the effectiveness and superiority of our model which can accurately locate the salient objects.

Image multi-label annotation based on supervised nonnegative matrix factorization with new matching measurement

Nonnegative Matrix Factorization (NMF) has been attracting many scholars in the fields of pattern recognition and data mining to study it since its inception. To date, a large number of variant methods have been proposed and applied in image retrieval and image Single-Label Annotation (SLA) successfully. However, the effectiveness of NMF for Multi-Label Annotation (MLA) encounters difficulties and is still an open topic. To meet this goal, this paper proposes a supervised NMF with new matching measurement to enhance MLA accuracy. In contrast with other NMF algorithms with sparse or discriminant constraints, the proposed NMF algorithm implements a supervised training method while integrates feature dimension reduction. What’s more, we improve a novel matching measurement function by considering positive and negative samples respectively, which is proved to be more suitable for MLA. In addition, the proposed NMF object function is solved by using the projected gradient method, and image annotation can be achieved. Experiments results on NUSWIDE dataset showed that the proposed algorithm can achieve strong performance compared with existing algorithms in terms of False Rejection Rate (FRR) and False Acceptance Rate (FAR).

Visual Understanding via Multi-Feature Shared Learning With Global Consistency

Image/video data is usually represented with multiple visual features. Fusion of multi-source information for establishing the attributes has been widely recognized. Multi-feature visual recognition has recently received much attention in multimedia applications. This paper studies visual understanding via a newly proposed l_2-norm based multi-feature shared learning framework, which can simultaneously learn a global label matrix and multiple sub-classifiers with the labeled multi-feature data. Additionally, a group graph manifold regularizer composed of the Laplacian and Hessian graph is proposed for better preserving the manifold structure of each feature, such that the label prediction power is much improved through the semi-supervised learning with global label consistency. For convenience, we call the proposed approach Global-Label-Consistent Classifier (GLCC). The merits of the proposed method include: 1) the manifold structure information of each feature is exploited in learning, resulting in a more faithful classification owing to the global label consistency; 2) a group graph manifold regularizer based on the Laplacian and Hessian regularization is constructed; 3) an efficient alternative optimization method is introduced as a fast solver owing to the convex sub-problems. Experiments on several benchmark visual datasets for multimedia understanding, such as the 17-category Oxford Flower dataset, the challenging 101-category Caltech dataset, the YouTube & Consumer Videos dataset and the large-scale NUS-WIDE dataset, demonstrate that the proposed approach compares favorably with the state-of-the-art algorithms. An extensive experiment on the deep convolutional activation features also show the effectiveness of the proposed approach. The code is available on http://www.escience.cn/people/lei/index.html

Learning Visual Semantic Relationships for Efficient Visual Retrieval

In this paper, we investigate how to establish the relationship between semantic concepts based on the large-scale realworld click data from image commercial engine, which is a challenging topic because the click data suffers from the noise such as typos, the same concept with different queries, etc. We first define five specific relationships between concepts. We then extract some concept relationship features in textual and visual domain to train the concept relationship models. The relationship of each pair of concepts will thus be classified into one of the five special relationships. We study the efficacy of the conceptual relationships by applying them to augment imperfect image tags, i.e., improve representative power. We further employ a sophisticated hashing approach to transform augmented image tags into binary codes, which are subsequently used for content-based image retrieval task. Experimental results on NUS-WIDE dataset demonstrate the superiority of our proposed approach as compared to state-of-the-art methods.

Co-Labeling for Multi-View Weakly Labeled Learning.

It is often expensive and time consuming to collect labeled training samples in many real-world applications. To reduce human effort on annotating training samples, many machine learning techniques (e.g., semi-supervised learning (SSL), multi-instance learning (MIL), etc.) have been studied to exploit weakly labeled training samples. Meanwhile, when the training data is represented with multiple types of features, many multi-view learning methods have shown that classifiers trained on different views can help each other to better utilize the unlabeled training samples for the SSL task. In this paper, we study a new learning problem called multi-view weakly labeled learning, in which we aim to develop a unified approach to learn robust classifiers by effectively utilizing different types of weakly labeled multi-view data from a broad range of tasks including SSL, MIL and relative outlier detection (ROD). We propose an effective approach called co-labeling to solve the multi-view weakly labeled learning problem. Specifically, we model the learning problem on each view as a weakly labeled learning problem, which aims to learn an optimal classifier from a set of pseudo-label vectors generated by using the classifiers trained from other views. Unlike traditional co-training approaches using a single pseudo-label vector for training each classifier, our co-labeling approach explores different strategies to utilize the predictions from different views, biases and iterations for generating the pseudo-label vectors, making our approach more robust for real-world applications. Moreover, to further improve the weakly labeled learning on each view, we also exploit the inherent group structure in the pseudo-label vectors generated from different strategies, which leads to a new multi-layer multiple kernel learning problem. Promising results for text-based image retrieval on the NUS-WIDE dataset as well as news classification and text categorization on several real-world multi-view datasets clearly demonstrate that our proposed co-labeling approach achieves state-of-the-art performance for various multi-view weakly labeled learning problems including multi-view SSL, multi-view MIL and multi-view ROD.