Multi-label Image Research Articles

Discrete semi-supervised learning for multi-label image classification and large-scale image retrieval

Multi-label image classification is a critical problem in image semantic learning. Traditional semi-supervised multi-label learning methods are mainly based on continuous learning of both labelled and unlabelled data. They usually learn classification functions from continuous label space. And the neglect of discrete constraint of labels impedes the classification performance. In this paper, we specifically consider the discrete constraint and propose Discrete Semi-supervised Multi-label Learning (DSML) for image classification. In DSML, we propose a semi-supervised framework with discrete constraint. Then we introduce anchor graph learning to improve the scalability, and derive an ADMM based alternating optimization process to solve our framework. The main experimental results on two real-world image datasets MIR Flickr and NUS-WIDE demonstrate the superiority of DSML compared with several advanced multi-label methods. Furthermore, additional experiments of image retrieval show the potential advantages of DSML in other image applications.

Multi-Label Image Classification by Feature Attention Network

Learning the correlation among labels is a standing-problem in the multi-label image recognition task. The label correlation is the key to solve the multi-label classification but it is too abstract to model. Most solutions try to learn image label dependencies to improve multi-label classification performance. However, they have ignored two more realistic problems: object scale inconsistent and label tail (category imbalance). These two problems will impact the bad influence on the classification model. To tackle these two problems and learn the label correlations, we propose feature attention network (FAN) which contains feature refinement network and correlation learning network. FAN builds top-down feature fusion mechanism to refine more important features and learn the correlations among convolutional features from FAN to indirect learn the label dependencies. Following our proposed solution, we achieve performed classification accuracy on MSCOCO 2014 and VOC 2007 dataset.

Structured Label Inference for Visual Understanding

Visual data such as images and videos contain a rich source of structured semantic labels as well as a wide range of interacting components. Visual content could be assigned with fine-grained labels describing major components, coarse-grained labels depicting high level abstractions, or a set of labels revealing attributes. Such categorization over different, interacting layers of labels evinces the potential for a graph-based encoding of label information. In this paper, we exploit this rich structure for performing graph-based inference in label space for a number of tasks: multi-label image and video classification and action detection in untrimmed videos. We consider the use of the Bidirectional Inference Neural Network (BINN) and Structured Inference Neural Network (SINN) for performing graph-based inference in label space and propose a Long Short-Term Memory (LSTM) based extension for exploiting activity progression on untrimmed videos. The methods were evaluated on (i) the Animal with Attributes (AwA), Scene Understanding (SUN) and NUS-WIDE datasets for multi-label image classification, (ii) the first two releases of the YouTube-8M large scale dataset for multi-label video classification, and (iii) the THUMOS'14 and MultiTHUMOS video datasets for action detection. Our results demonstrate the effectiveness of structured label inference in these challenging tasks, achieving significant improvements against baselines.

Reconstruction Regularized Deep Metric Learning for Multi-Label Image Classification.

In this paper, we present a novel deep metric learning method to tackle the multi-label image classification problem. In order to better learn the correlations among images features, as well as labels, we attempt to explore a latent space, where images and labels are embedded via two unique deep neural networks, respectively. To capture the relationships between image features and labels, we aim to learn a two-way deep distance metric over the embedding space from two different views, i.e., the distance between one image and its labels is not only smaller than those distances between the image and its labels' nearest neighbors but also smaller than the distances between the labels and other images corresponding to the labels' nearest neighbors. Moreover, a reconstruction module for recovering correct labels is incorporated into the whole framework as a regularization term, such that the label embedding space is more representative. Our model can be trained in an end-to-end manner. Experimental results on publicly available image data sets corroborate the efficacy of our method compared with the state of the arts.

Cograph Regularized Collective Nonnegative Matrix Factorization for Multilabel Image Annotation

Cograph Regularized Collective Nonnegative Matrix Factorization for Multilabel Image Annotation

Trigraph Regularized Collective Matrix Tri-Factorization Framework on Multiview Features for Multilabel Image Annotation

Due to the explosive growth of image data, image annotation has been one of the most popular research directions in computer vision. It has been widely used in image retrieval, image analysis and understanding. Because traditional manual image annotation is time consuming, more advanced automatic annotation methods are needed. A major challenge in developing an efficient image annotation method is how to effectively use all available information contained in the data. To this end, this paper proposes a novel image annotation framework that uses multiple information from data. It employs nonnegative matrix tri-factorization (NMTF) to simultaneously factorize image-to-label, image-to-feature, and feature-to-label relation matrices using their intertype relationships and incorporates the intratype information through manifold regularizations. This method can be referred to as the trigraph regularized collective matrix tri-factorization framework (TG-CMTF). TG-CMTF captures the correlations among different labels, different images and different features. By taking advantage of these relations from images, features and labels, TG-CMTF can achieve better annotation performance than most state-of-the-art methods. The promising experimental results on three standard benchmarks have shown the effectiveness of this information. Furthermore, we show the annotation process as a precise optimization problem and solve it by an iterative algorithm, which proves the correctness of the proposed method from the mathematical theory.

Facial Attribute Recognition With Feature Decoupling and Graph Convolutional Networks

The objective of facial attribute recognition is to predict a set of labels for each face image. Similar to general multi-label image classification task, facial attribute recognition suffers from several typical problems: discriminative feature learning, handling the correlations among attributes, and imbalanced training data. In this paper, we propose a unified facial attribute recognition solution via feature decoupling and graph convolutional networks (GCN): 1) We utilize the orthonormal regularizer to constrain that each dimension of the general facial representation represents a certain visual pattern. Meanwhile, a learnable matrix is unveiled, which can convert general facial feature into attribute-specific representation through modeling the importance of different visual patterns. 2) To handle the correlations between facial attributes, we build a GCN to capture the label dependencies and map the nodes in the proposed GCN to a set of inter-dependent attribute classifiers. Besides, we normalize the weights in all classifiers to alleviate the influence of data imbalance. We have conducted extensive experiments on two benchmarks, and both the qualitative and quantitative evaluation results have demonstrated the effectiveness of the proposed method.

Tencent ML-Images: A Large-Scale Multi-Label Image Database for Visual Representation Learning

In existing visual representation learning tasks, deep convolutional neural networks (CNNs) are often trained on images annotated with single tags, such as ImageNet. However, a single tag cannot describe all important contents of one image, and some useful visual information may be wasted during training. In this work, we propose to train CNNs from images annotated with multiple tags, to enhance the quality of visual representation of the trained CNN model. To this end, we build a large-scale multi-label image database with 18M images and 11K categories, dubbed Tencent ML-Images. We efficiently train the ResNet-101 model with multi-label outputs on Tencent ML-Images, taking 90 hours for 60 epochs, based on a large-scale distributed deep learning framework,i.e.,TFplus. The good quality of the visual representation of the Tencent ML-Images checkpoint is verified through three transfer learning tasks, including single-label image classification on ImageNet and Caltech-256, object detection on PASCAL VOC 2007, and semantic segmentation on PASCAL VOC 2012. The Tencent ML-Images database, the checkpoints of ResNet-101, and all the training codehave been released at https://github.com/Tencent/tencent-ml-images. It is expected to promote other vision tasks in the research and industry community.

Multi-label image classification with recurrently learning semantic dependencies

Recognizing multi-label images is a significant but challenging task toward high-level visual understanding. Remarkable success has been achieved by applying CNN–RNN design-based models to capture the underlying semantic dependencies of labels and predict the label distributions over the global-level features output by CNNs. However, such global-level features often fuse the information of multiple objects, leading to the difficulty in recognizing small object and capturing the label co-relation. To better solve this problem, in this paper, we propose a novel multi-label image classification framework which is an improvement to the CNN–RNN design pattern. By introducing the attention network module in the CNN–RNN architecture, the objects features of the attention map are separated by the channels which are further send to the LSTM network to capture dependencies and predict labels sequentially. A category-wise max-pooling operation is then performed to integrate these labels into the final prediction. Experimental results on PASCAL2007 and MS-COCO datasets demonstrate that our model can effectively exploit the correlation between tags to improve the classification performance as well as better recognize the small targets.

Non-Linear Machine Learning Techniques for Multi-Label Image Data Classification

Non-Linear Machine Learning Techniques for Multi-Label Image Data Classification

Multilabel Image Classification With Regional Latent Semantic Dependencies

Deep convolution neural networks (CNN) have demonstrated advanced performance on single-label image classification, and various progress also have been made to apply CNN methods on multi-label image classification, which requires to annotate objects, attributes, scene categories etc. in a single shot. Recent state-of-the-art approaches to multi-label image classification exploit the label dependencies in an image, at global level, largely improving the labeling capacity. However, predicting small objects and visual concepts is still challenging due to the limited discrimination of the global visual features. In this paper, we propose a Regional Latent Semantic Dependencies model (RLSD) to address this problem. The utilized model includes a fully convolutional localization architecture to localize the regions that may contain multiple highly-dependent labels. The localized regions are further sent to the recurrent neural networks (RNN) to characterize the latent semantic dependencies at the regional level. Experimental results on several benchmark datasets show that our proposed model achieves the best performance compared to the state-of-the-art models, especially for predicting small objects occurred in the images. In addition, we set up an upper bound model (RLSD+ft-RPN) using bounding box coordinates during training, the experimental results also show that our RLSD can approach the upper bound without using the bounding-box annotations, which is more realistic in the real world.

Object-Location-Aware Hashing for Multi-Label Image Retrieval via Automatic Mask Learning.

Learning-based hashing is a leading approach of approximate nearest neighbor search for large-scale image retrieval. In this paper, we develop a deep supervised hashing method for multi-label image retrieval, in which we propose to learn a binary "mask" map that can identify the approximate locations of objects in an image, so that we use this binary "mask" map to obtain length-limited hash codes which mainly focus on an image's objects but ignore the background. The proposed deep architecture consists of four parts: 1) a convolutional sub-network to generate effective image features; 2) a binary "mask" sub-network to identify image objects' approximate locations; 3) a weighted average pooling operation based on the binary "mask" to obtain feature representations and hash codes that pay most attention to foreground objects but ignore the background; and 4) the combination of a triplet ranking loss designed to preserve relative similarities among images and a cross entropy loss defined on image labels. We conduct comprehensive evaluations on four multi-label image data sets. The results indicate that the proposed hashing method achieves superior performance gains over the state-of-the-art supervised or unsupervised hashing baselines.

A Deep Multi-Modal CNN for Multi-Instance Multi-Label Image Classification.

Deep convolutional neural networks (CNNs) have shown superior performance on the task of single-label image classification. However, the applicability of CNNs to multi-label images still remains an open problem, mainly because of two reasons. First, each image is usually treated as an inseparable entity and represented as one instance, which mixes the visual information corresponding to different labels. Second, the correlations amongst labels are often overlooked. To address these limitations, we propose a deep multi-modal CNN for multi-instance multi-label image classification, called MMCNN-MIML. By combining CNNs with multi-instance multi-label (MIML) learning, our model represents each image as a bag of instances for image classification and inherits the merits of both CNNs and MIML. In particular, MMCNN-MIML has three main appealing properties: 1) it can automatically generate instance representations for MIML by exploiting the architecture of CNNs; 2) it takes advantage of the label correlations by grouping labels in its later layers; and 3) it incorporates the textual context of label groups to generate multi-modal instances, which are effective in discriminating visually similar objects belonging to different groups. Empirical studies on several benchmark multi-label image data sets show that MMCNN-MIML significantly outperforms the state-of-the-art baselines on multi-label image classification tasks.

A Semisupervised Framework for Automatic Image Annotation Based on Graph Embedding and Multiview Nonnegative Matrix Factorization

Automatic image annotation is for more accurate image retrieval and classification by assigning labels to images. This paper proposes a semisupervised framework based on graph embedding and multiview nonnegative matrix factorization (GENMF) for automatic image annotation with multilabel images. First, we construct a graph embedding term in the multiview NMF based on the association diagrams between labels for semantic constraints. Then, the multiview features are fused and dimensions are reduced based on multiview NMF algorithm. Finally, image annotation is achieved by using the new features through a KNN-based approach. Experiments validate that the proposed algorithm has achieved competitive performance in terms of accuracy and efficiency.

Tree-based Ensemble Classifier Learning for Automatic Brain Glioma Segmentation

Abstract We introduce a dynamic multiscale tree (DMT) architecture that learns how to leverage the strengths of different state-of-the-art classifiers for supervised multi-label image segmentation. Unlike previous works that simply aggregate or cascade classifiers for addressing image segmentation and labeling tasks, we propose to embed strong classifiers into a tree structure that allows bi-directional flow of information between its classifier nodes to gradually improve their performances. Our DMT is a generic classification model that inherently embeds different cascades of classifiers while enhancing learning transfer between them to boost up their classification accuracies. Specifically, each node in our DMT can nest a Structured Random Forest (SRF) classifier or a Bayesian Network (BN) classifier. The proposed SRF-BN DMT architecture has several appealing properties. First, while SRF operates at a patch-level (regular image region), BN operates at the super-pixel level (irregular image region), thereby enabling the DMT to integrate multi-level image knowledge in the learning process. Second, although BN is powerful in modeling dependencies between image elements (superpixels, edges) and their features, the learning of its structure and parameters is challenging. On the other hand, SRF may fail to accurately detect very irregular object boundaries. The proposed DMT robustly overcomes these limitations for both classifiers through the ascending and descending flow of contextual information between each parent node and its children nodes. Third, we train DMT using different scales for input patches and superpixels. Basically, as we go deeper along the tree edges nearing its leaf nodes, we progressively decrease the patch and superpixel sizes, producing segmentation maps that capture a coarse-to-fine image details. Last, DMT demonstrates its outperformance in comparison to several state-of-the-art segmentation methods for multi-labeling of brain images with gliomas.

Joint Learning of Set Cardinality and State Distribution

We present a novel approach for learning to predict sets using deep learning. In recent years, deep neural networks have shown remarkable results in computer vision, natural language processing and other related problems. Despite their success,traditional architectures suffer from a serious limitation in that they are built to deal with structured input and output data,i.e. vectors or matrices. Many real-world problems, however, are naturally described as sets, rather than vectors. Existing techniques that allow for sequential data, such as recurrent neural networks, typically heavily depend on the input and output order and do not guarantee a valid solution. Here, we derive in a principled way, a mathematical formulation for set prediction where the output is permutation invariant. In particular, our approach jointly learns both the cardinality and the state distribution of the target set. We demonstrate the validity of our method on the task of multi-label image classification and achieve a new state of the art on the PASCAL VOC and MS COCO datasets.

Reinforced Multi-Label Image Classification by Exploring Curriculum

Humans and animals learn much better when the examples are not randomly presented but organized in a meaningful order which illustrates gradually more concepts, and gradually more complex ones. Inspired by this curriculum learning mechanism, we propose a reinforced multi-label image classification approach imitating human behavior to label image from easy to complex. This approach allows a reinforcement learning agent to sequentially predict labels by fully exploiting image feature and previously predicted labels. The agent discovers the optimal policies through maximizing the long-term reward which reflects prediction accuracies. Experimental results on PASCAL VOC2007 and 2012 demonstrate the necessity of reinforcement multi-label learning and the algorithm’s effectiveness in real-world multi-label image classification tasks.

Recurrent Attentional Reinforcement Learning for Multi-Label Image Recognition

Recognizing multiple labels of images is a fundamental but challenging task in computer vision, and remarkable progress has been attained by localizing semantic-aware image regions and predicting their labels with deep convolutional neural networks. The step of hypothesis regions (region proposals) localization in these existing multi-label image recognition pipelines, however, usually takes redundant computation cost, e.g., generating hundreds of meaningless proposals with non-discriminative information and extracting their features, and the spatial contextual dependency modeling among the localized regions are often ignored or over-simplified. To resolve these issues, this paper proposes a recurrent attention reinforcement learning framework to iteratively discover a sequence of attentional and informative regions that are related to different semantic objects and further predict label scores conditioned on these regions. Besides, our method explicitly models long-term dependencies among these attentional regions that help to capture semantic label co-occurrence and thus facilitate multi-label recognition. Extensive experiments and comparisons on two large-scale benchmarks (i.e., PASCAL VOC and MS-COCO) show that our model achieves superior performance over existing state-of-the-art methods in both performance and efficiency as well as explicitly identifying image-level semantic labels to specific object regions.

An Auto-Segmentation Algorithm for Multi-Label Image Based on Graph Cut

The image segmentation algorithm based on graph cut guarantees a globally optimal solution for energy solution, which is usually with the aid of user’s interactive operation. For the multi-label image segmentation application, the graph cut algorithm has two drawbacks. Firstly, it has a higher computational complexity of segment multi-label images. Secondly, it is prone to be trapped in local minima when solves the energy formulation. For the two drawbacks, this paper presents an auto-segmentation algorithm based on graph cut to segment multi-label images. The number of the labels is obtained via the main colors of the image, then the main colors are employed as pre-specified nodes feature, rather than select seeds with the aid of prior knowledge or initialization operation by the user. The seeds can be selected automatically without complex mathematical formulations to computerize, and it reduces the computational complexity successfully, and avoids falling into local minima effectively. In additional, we use a fast α-expansion move algorithm to optimize the energy function, which can improve the speed of segmentation. Comparing the proposed algorithm with the state-of-the-art segmentation methods, the experimental results show that the proposed algorithm has superior performance.

Joint graph regularized extreme learning machine for multi-label image classification

Joint graph regularized extreme learning machine for multi-label image classification