MIRFLICKR Datasets Research Articles

Adaptive optimized residual convolutional image annotation model with bionic feature selection model

The advancements in Internet and Web 2.0 technologies resulted in the easier generation and distribution of digital images. Many images are stored without any form of significant labelling. This makes searching for desired images tedious and time-consuming, resulting in an increased demand for an effective image retrieval technique. Image annotation provides one or more labels to an image that describes its content. These annotations greatly help to retrieve images from large databases using different related queries. Automatic Image Annotation (AIA) applications in computer vision, data mining and web search are increasing progressively. This work proposes a hybrid Deep Learning (DL) based optimized image annotation model for AIA. Initially, the pre-processing of an input image is carried out to remove the image noise using the triple Gaussian guided filter. In the second stage, the image's features like shape, colour and texture are extracted using Slantlet transform, YCrCb Colour space and Local Binary Pattern (LBP). The feature dimensionality is reduced by selecting features based on the adaptive red colobuses monkey bionic model. Finally, the adaptive image annotation is obtained using the deep-optimized convolutional residual image annotation model. The loss function reduction and updating of weight parameters are facilitated using the hunger game bionic model. The experiments are performed in Python using Corel-5 K, ESP-Game and MIR FLICKR datasets. The proposed model attains an overall precision of 56, 99.2 and 90%, respectively. Thus, the experimental evaluation results demonstrate the excellent performance of a proposed model over existing approaches.

Deep Supervised Dual Cycle Adversarial Network for Cross-Modal Retrieval

Cross-modal retrieval tasks, which are more natural and challenging than traditional retrieval tasks, have attracted increasing interest from researchers in recent years. Although different modalities with the same semantics have some potential relevance, the feature space heterogeneity still seriously weakens the performance of cross-modal retrieval models. To solve this problem, common space-based methods in which multimodal data is projected into a learned common space for similarity measurement have become the mainstream approach for cross-modal retrieval tasks. However, current methods entangle the modality style and semantic content in the common space and neglect to fully explore the semantic and discriminative representation/reconstruction of the semantic content. This often results in an unsatisfactory retrieval performance. To solve these issues, this paper proposes a new Deep Supervised Dual Cycle Adversarial Network (DSDCAN) model based on common space learning. It is composed of two cross-modal cycle GANs, one for the image and one for the text. The proposed cycle GAN model disentangles the semantic content and modality style features by making the data of one modality well reconstructed from the extracted modal style feature and the content feature of the other modality. Then, a discriminative semantic and label loss is proposed by fully considering the category, sample contrast, and label supervision to enhance the semantic discrimination of the common space representation. Besides this, to make the data distribution between two modalities similar, a second-order similarity is presented as a distance measurement of the cross-modal representation in the common space. Extensive experiments have been conducted on the Wikipedia, Pascal Sentence, NUS-WIDE-10k, PKU XMedia, MSCOCO, NUS-WIDE, Flickr30k and MIRFlickr datasets. The results demonstrate that the proposed method can achieve a higher performance than the state-of-the-art methods.

Cross-Modal Discrimination Hashing Retrieval Using Variable Length

Fast cross-modal retrieval technology based on hash coding has become a hot topic for the rich multimodal data (text, image, audio, etc.), especially security and privacy challenges in the Internet of Things and mobile edge computing. However, most methods based on hash coding are only mapped to the common hash coding space, and it relaxes the two value constraints of hash coding. Therefore, the learning of the multimodal hash coding may not be sufficient and effective to express the original multimodal data and cause the hash encoding category to be less discriminatory. For the sake of solving these problems, this paper proposes a method of mapping each modal data to the optimal length of hash coding space, respectively, and then the hash encoding of each modal data is solved by the discrete cross-modal hash algorithm of two value constraints. Finally, the similarity of multimodal data is compared in the potential space. The experimental results of the cross-model retrieval based on variable hash coding are better than that of the relative comparison methods in the WIKI data set, NUS-WIDE data set, as well as MIRFlickr data set, and the method we proposed is proved to be feasible and effective.

Performance of Stego Images Calibration Using Advanced Denoising Methods

Reliable detection of messages that are unauthorized embedding into cover files, namely digital images, is topical task today. Design of effective stegdetectors is complicated by limited prior information of used adaptive embedding methods that minimize cover image alteration during data hiding. Modern approaches for improving detection accuracy are based on usage either ensembles of stegdetectors, or advanced neural networks, which characterize by high-computation complexity. This makes them inappropriate for real scenarios where fast re-tuning for detection of unknown embedding methods is needed. The alternative approach for stegdetector’s design is based on modification of image pre-processing (calibration) methods, namely selection of methods for detection and extraction weak alteration of cover image features caused by message hiding. Special interest is taken to advanced methods for image denoising due to theirs high “adaptiveness” to statistical features of processed images that provide local processing of image’s regions instead of global ones. The work is devoted to performance analysis of Total Variation Minimization (TVM) technique usage of calibration of stego images formed by adaptive embedding methods. According to obtained results we may conclude that applying of advanced TVM-based denoising methods does not allow considerably improving stegdetector detection accuracy in comparison with usage of standard SPAM model. On the other hand, applying of component analysis, namely Principal Component Analysis, allows considerably improve Matthews Correlation Coefficient (up to 0.3) for stegdetector in case of processing real images from MIRFlickr dataset. This can be explained by “grouping” of cover image alterations caused by message hiding into single component with the smallest energy (singular value) that makes possible its easy removal.

Multi-Datasets Evaluation of GB-RAS Network Based Stegdetectors Robustness to Domain Adaptation Problem

Protection of state and commercial critical infrastructures is topical task today. Among known threats, special interest is taken to early detection of hidden (steganographic) channels usage by attackers for unauthorized transmission of sensitive data. These channels are created by modification of the innocuous files, such as digital images, for message hiding and further transmission of modified files. Detection of formed stego files is non-trivial task due to the wide usage by attackers of adaptive embedding methods that preserve low level of cover image distortion during embedding. Modern solutions for stego images detection are based on utilization the novel convolution neural networks for revealing weak alterations of cover’s features during message hiding. Training of such networks is performed on fixed set of standard image databases, such as BOSS, ALASKA etc. Despite high detection accuracy of pre-trained stegdetectors, theirs performance “in the wild” on natural images remains an open question. The work is devoted to performance analysis of advanced GB-Ras convolutional network based stegdetector for adaptive embedding methods detection on natural images presented in various datasets. Obtained results proved the effectiveness of neural network applying for mitigation with domain adaptation problem for modern stegdetectors. The GB-Ras model allows improving detection accuracy of stego images for standard ALASKA (up to 6% for for S-UNIWARD embedding methods) and VISION (up to 13% for S-UNIWARD and to 9% for MG embedding methods) datasets in comparison with cover rich models based stegdetectors. Nevertheless, the performance of considered GB-Ras based stegdetector for high quality images from MIRFlickr dataset is conceded to maxSRMd2 rich model based stegdetector (up to 11% for S-UNIWARD and 12% for MG embedding methods). Therefore, considered GB-Ras network may be used as a promising candidate for further design of novel stegdetectors that are robust to domain adaptation problem.

A new framework of multi-objective evolutionary algorithms for feature selection and multi-label classification of video data

There are few studies in the literature to address the multi-objective multi-label feature selection for the classification of video data using evolutionary algorithms. Selecting the most appropriate subset of features is a significant problem while maintaining/improving the accuracy of the prediction results. This study proposes a framework of parallel multi-objective Non-dominated Sorting Genetic Algorithms (NSGA-II) for exploring a Pareto set of non-dominated solutions. The subsets of non-dominated features are extracted and validated by multi-label classification techniques, Binary Relevance (BR), Classifier Chains (CC), Pruned Sets (PS), and Random k-Labelset (RAkEL). Base classifiers such as Support Vector Machines (SVM), J48-Decision Tree (J48), and Logistic Regression (LR) are performed in the classification phase of the algorithms. Comprehensive experiments are carried out with local feature descriptors extracted from two multi-label data sets, the well-known MIR-Flickr dataset and a Wireless Multimedia Sensor (WMS) dataset that we have generated from our video recordings. The prediction accuracy levels are improved by 6.36% and 25.7% for the MIR-Flickr and WMS datasets respectively while the number of features is significantly reduced. The results verify that the algorithms presented in this new framework outperform the state-of-the-art algorithms.

Deep Multi-Modal Metric Learning with Multi-Scale Correlation for Image-Text Retrieval

Multi-modal retrieval is a challenge due to heterogeneous gap and a complex semantic relationship between different modal data. Typical research map different modalities into a common subspace with a one-to-one correspondence or similarity/dissimilarity relationship of inter-modal data, in which the distances of heterogeneous data can be compared directly; thus, inter-modal retrieval can be achieved by the nearest neighboring search. However, most of them ignore intra-modal relations and complicated semantics between multi-modal data. In this paper, we propose a deep multi-modal metric learning method with multi-scale semantic correlation to deal with the retrieval tasks between image and text modalities. A deep model with two branches is designed to nonlinearly map raw heterogeneous data into comparable representations. In contrast to binary similarity, we formulate semantic relationship with multi-scale similarity to learn fine-grained multi-modal distances. Inter-modal and intra-modal correlations constructed on multi-scale semantic similarity are incorporated to train the deep model in an end-to-end way. Experiments validate the effectiveness of our proposed method on multi-modal retrieval tasks, and our method outperforms state-of-the-art methods on NUS-WIDE, MIR Flickr, and Wikipedia datasets.

Analysis of Multiobjective Algorithms for the Classification of Multi-Label Video Datasets

It is of great importance to extract and validate an optimal subset of non-dominated features for effective multi-label classification. However, deciding on the best subset of features is an NP-Hard problem and plays a key role in improving the prediction accuracy and the processing time of video datasets. In this study, we propose autoencoders for dimensionality reduction of video data sets and ensemble the features extracted by the multi-objective evolutionary Non-dominated Sorting Genetic Algorithm and the autoencoder. We explore the performance of well-known multi-label classification algorithms for video datasets in terms of prediction accuracy and the number of features used. More specifically, we evaluate Non-dominated Sorting Genetic Algorithm-II, autoencoders, ensemble learning algorithms, Principal Component Analysis, Information Gain, and Correlation Based Feature Selection. Some of these algorithms use feature selection techniques to improve the accuracy of the classification. Experiments are carried out with local feature descriptors extracted from two multi-label datasets, the MIR-Flickr dataset which consists of images and the Wireless Multimedia Sensor dataset that we have generated from our video recordings. Significant improvements in the accuracy performance of the algorithms are observed while the number of features is being reduced.

An algorithmic approach to estimate cognitive aesthetics of images relative to ground truth of human psychology through a large user study

ABSTRACTThis research introduces a learning model that estimates the cognitive perception of aesthetics. Taking psychology into account, this bridges the gap between human and machine. The goal is to build a machine-learning model that can estimate beauty in images perceived by human eyes. We have summand our research [Firoze, A., Osman, T., Psyche, S. S., & Rahman, R. M. (2018). Scoring photographic rule of thirds in a large MIRFLICKR dataset: A showdown between machine perception and human perception of image aesthetics. Asian Conference on Intelligent Information and Database Systems (pp. 466–475), Springer; Osman, T., Psyche, S. S., Deb, T., Firoze, A., & Rahman, R. M. (2018). Differential color harmony: A robust approach for extracting Harmonic Color features and perceive aesthetics in a large dataset. International Conference on Big Data and Cloud Computing, Springer] together with the idea of humans’ personal preferences and achieved higher than state of the art performances. An extensive user study (374 participants) has been conducted to support claims. Several photographical compositional metrics have been used. Colour gradient, rule of thirds and human subject’s psychology has been picked as features. The consideration of user’s perspective or psychology is one of the key contributions of this research.

Deep-FS: A feature selection algorithm for Deep Boltzmann Machines

A Deep Boltzmann Machine is a model of a Deep Neural Network formed from multiple layers of neurons with nonlinear activation functions. The structure of a Deep Boltzmann Machine enables it to learn very complex relationships between features and facilitates advanced performance in learning of high-level representation of features, compared to conventional Artificial Neural Networks. Feature selection at the input level of Deep Neural Networks has not been well studied, despite its importance in reducing the input features processed by the deep learning model, which facilitates understanding of the data. This paper proposes a novel algorithm, Deep Feature Selection (Deep-FS), which is capable of removing irrelevant features from large datasets in order to reduce the number of inputs which are modelled during the learning process. The proposed Deep-FS algorithm utilizes a Deep Boltzmann Machine, and uses knowledge which is acquired during training to remove features at the beginning of the learning process. Reducing inputs is important because it prevents the network from learning the associations between the irrelevant features which negatively impact on the acquired knowledge of the network about the overall distribution of the data. The Deep-FS method embeds feature selection in a Restricted Boltzmann Machine which is used for training a Deep Boltzmann Machine. The generative property of the Restricted Boltzmann Machine is used to reconstruct eliminated features and calculate reconstructed errors, in order to evaluate the impact of eliminating features. The performance of the proposed approach was evaluated with experiments conducted using the MNIST, MIR-Flickr, GISETTE, MADELON and PANCAN datasets. The results revealed that the proposed Deep-FS method enables improved feature selection without loss of accuracy on the MIR-Flickr dataset, where Deep-FS reduced the number of input features by removing 775 features without reduction in performance. With regards to the MNIST dataset, Deep-FS reduced the number of input features by more than 45%; it reduced the network error from 0.97% to 0.90%, and also reduced processing and classification time by more than 5.5%. Additionally, when compared to classical feature selection methods, Deep-FS returned higher accuracy. The experimental results on GISETTE, MADELON and PANCAN showed that Deep-FS reduced 81%, 57% and 77% of the number of input features, respectively. Moreover, the proposed feature selection method reduced the classifier training time by 82%, 70% and 85% on GISETTE, MADELON and PANCAN datasets, respectively. Experiments with various datasets, comprising a large number of features and samples, revealed that the proposed Deep-FS algorithm overcomes the main limitations of classical feature selection algorithms. More specifically, most classical methods require, as a prerequisite, a pre-specified number of features to retain, however in Deep-FS this number is identified automatically. Deep-FS performs the feature selection task faster than classical feature selection algorithms which makes it suitable for deep learning tasks. In addition, Deep-FS is suitable for finding features in large and big datasets which are normally stored in data batches for faster and more efficient processing.

Social Image Tag Relevance Learning based on Pixel Voting

Social image platforms allow their users sharing and searching their photos based on images’ tags. These tags are provided by different users. Inevitably, the tags are spontaneously ambiguous, and personalized. So, learning the relevance between tags and images is playing an important role in tag-based retrieval systems. Choosing visual neighbors for seed images as voters is a widely used method for learning tag relevance. However, most existing methods of choosing visual neighbors for seed images are based on the global features of the whole images, ignoring the local features. In this paper we propose a pixel voting method to choose the visual neighbors for seed images. Experiment shows that this method is a more natural way to measure the similarity of images. Based the selected neighbors we learn the tag relevance, and the experiment on the MIR Flickr dataset shows that our algorithm is effective in tag de-noising and tag ranking.

Automatic Image Annotation Based on Multi-Auxiliary Information

This paper introduces an automatic image annotation framework based on multi-auxiliary information which aims at improving the annotation performance. We propose three novel ideas in the framework of annotation: 1) multi-information extraction: besides various visual features, tag co-occurrence, and user interest vector are added to enrich the multi-auxiliary information; 2) initial labeling: based on the traditional term frequency—inverse document frequency model—we utilize the visibility of words and extended tag set to enhance the result of initial labeling and propose a more efficient model, TF-IDF, visibility and extended tag set model; and 3) tag refinement: by considering multi-auxiliary information, including multi-visual content, tag co-occurrence, and user interest similarity, we propose the multi-information all-labels model for tag refinement. The tag refinement process is formalized as an optimization problem by adjusting confidence score set by the initial labeling model. Experimental results demonstrate that, compared with the state-of-the-art methods, our method achieves the best performance on MIR-Flickr data sets, outperforming the second best by 2%.

Manifold regularized kernel logistic regression for web image annotation

With the rapid advance of Internet technology and smart devices, users often need to manage large amounts of multimedia information using smart devices, such as personal image and video accessing and browsing. These requirements heavily rely on the success of image (video) annotation, and thus large scale image annotation through innovative machine learning methods has attracted intensive attention in recent years. One representative work is support vector machine (SVM). Although it works well in binary classification, SVM has a non-smooth loss function and can not naturally cover multi-class case. In this paper, we propose manifold regularized kernel logistic regression (KLR) for web image annotation. Compared to SVM, KLR has the following advantages: (1) the KLR has a smooth loss function; (2) the KLR produces an explicit estimate of the probability instead of class label; and (3) the KLR can naturally be generalized to the multi-class case. We carefully conduct experiments on MIR FLICKR dataset and demonstrate the effectiveness of manifold regularized kernel logistic regression for image annotation.

Actively constructing an effective training set by expected gain maximization criterion

We address the problem of active learning which aims to select the most representative points. Out of many existing active learning techniques, close-to-boundary criterion has received considerable attention recently. The goal of typically uncertain-based sampling is to minimize the distance between the examples and the classification boundary. However, these methods only adapt to the independent identically distributed (i.i.d) data, while the non-i.i.d is ignored. In this paper, we propose an active scheme which takes into account of the situation that unlabeled data and training data may come from different distribution, and we approximately obtain the information of every example by expected gain maximization. Given the parameters of a classification model and a pool of unlabeled data, the real risk on the unknown distribution is estimated. The expected gain of every candidate example is obtained, and then the most representative examples of all are thus defined as those who can maximize the expected gain of the classification model. At last, a sampling sequence is acquired by solving the optimization problem. Moreover, we also give an active scheme of two-stage sampling strategy based on the criterion of expected gain maximization in order to obtain an effective training set. The experimental results on MIRFLICKR and Caltech-256 datasets have demonstrated the effectiveness of our proposed methods.

A Fuzzy Associative Classification Approach to Visual Concept Detection

In this paper we present a novel associative classification algorithm for visual concept detection using interest points (color descriptors). Our algorithm relies on fuzzy (soft) partitions to generate a per-descriptor record model which is used for training associative classifiers. In this paper, we use the first 10000 images of the MIR Flickr dataset (with 38 visual concepts — 14 regular annotation topics and 24 pre-annotations) for evaluating our algorithm. Because of its fuzzy nature, our approach also handles polysemy and synonymy (common problems in most crisp (non-fuzzy) image classifiers) very well. As associative classification leverages frequent patterns mined from a given dataset, its performance as adjudged from average area under curve (AUC) values over three-fold cross validation is very good, especially for regular annotation topics and for “scenic” concepts. Our approach has the added advantage that the rules used for classification have clear semantics, and can be comprehended easily, unlike other classifiers, such as SVM, which act as black-boxes. From an empirical perspective (on the MIR Flickr dataset), the performance of our algorithm is much better, than that of either bag-of-words (Linear SVM and SVM RBF Kernel) or Linear SVM based on fuzzy (soft) partitions—both benchmark approaches are based on interest points (color descriptors).

A GPU-accelerated non-negative sparse latent semantic analysis algorithm for social tagging data

Nowadays large-scale social tagging data have become very valuable in organizing and indexing multimedia resources. In this paper, we apply Non-negative Sparse Latent Semantic Analysis (NN-Sparse LSA) to discover the latent semantic space behind associations between multimedia resources and tagging data. Based on the traditional coordinate-descent algorithm, column-orthogonality and non-negative constraints, we derive a much faster optimization algorithm in theory for solving the NN-Sparse LSA model. Furthermore, we implement the parallel version of our fast NN-Sparse LSA algorithm using the NVIDIA CUDA (Compute Unified Device Architecture) parallel programming framework and a data partitioning scheme that effectively reduces the memory traffic between the global memory of the Graphic Processing Unit (GPU) and the host memory. The experimental results on image classification and tag recommendation tasks on MIRFLICKR and NUS-WIDE datasets show that our parallelized fast optimization algorithm can achieve comparable or even better performance than the other examined methods, while speeds up the original optimization algorithm 20–110 times.

Improving image annotation via ranking‐oriented neighbor search and learning‐based keyword propagation

Automatic image annotation plays a critical role in modern keyword‐based image retrieval systems. For this task, the nearest‐neighbor–based scheme works in two phases: first, it finds the most similar neighbors of a new image from the set of labeled images; then, it propagates the keywords associated with the neighbors to the new image. In this article, we propose a novel approach for image annotation, which simultaneously improves both phases of the nearest‐neighbor–based scheme. In the phase of neighbor search, different from existing work discovering the nearest neighbors with the predicted distance, we introduce a ranking‐oriented neighbor search mechanism (RNSM), where the ordering of labeled images is optimized directly without going through the intermediate step of distance prediction. In the phase of keyword propagation, different from existing work using simple heuristic rules to select the propagated keywords, we present a learning‐based keyword propagation strategy (LKPS), where a scoring function is learned to evaluate the relevance of keywords based on their multiple relations with the nearest neighbors. Extensive experiments on the Corel 5K data set and the MIR Flickr data set demonstrate the effectiveness of our approach.