Representation Of Image Content Research Articles

Manifold information through neighbor embedding projection for image retrieval

Although studied for decades, constructing effective image retrieval remains an open problem in a wide range of relevant applications. Impressive advances have been made to represent image content, mainly supported by the development of Convolution Neural Networks (CNNs) and Transformer-based models. On the other hand, effectively computing the similarity between such representations is still challenging, especially in collections in which images are structured in manifolds. This paper introduces a novel solution to this problem based on dimensionality reduction techniques, often used for data visualization. The key idea consists in exploiting the spatial relationships defined by neighbor embedding data visualization methods, such as t-SNE and UMAP, to compute a more effective distance/similarity measure between images. Experiments were conducted on several widely-used datasets. Obtained results indicate that the proposed approach leads to significant gains in comparison to the original feature representations. Experiments also indicate competitive results in comparison with state-of-the-art image retrieval approaches.

Image Style Transfer based on DeepLabV3+ Semantic Segmentation

Image style transfer has always been a popular research method in the computer vision community, which aims to learn the style of a given image and the content distribution of other images to generate new images with both the above style and content. Thanks to the rapid development of convolutional neural networks, the accuracy, and visualization of image segmentation and migration have been continuously improved, but there is still the problem of local content distortion. Some recent works introduce the segmentation branch to obtain pixel-level content information in order to obtain a more perfect transfer effect. In this paper, we use the VGG19 convolutional neural network model to extract high-level feature maps representing image content information and perform image style transfer based on the DeepLabV3+ semantic segmentation network, which can move images to the same or similar semantic regions during the entire transmission process. In order to prevent content distortion after image migration, we also introduce an affine function to control the content change of the image during image migration. Extensive experimental results show that the method in this paper can further clarify the segmentation boundary and improve the semantic accuracy of the transferred image. In addition, we also evaluated the expectations of different individuals on the degree of style transfer. By distributing the survey, this paper demonstrates that the method described in this study results in better image quality, more in line with the expectations of the majority of respondents.

Experimental evaluation of unsupervised image retrieval application using hybrid feature extraction by integrating deep learning and handcrafted techniques

Abstract Content Based Image Retrieval is ever growing technology for many applications such as medical, remote sensing, social media search engines and surveillance monitoring, etc,. Representing image content with appropriate features is tedious task using traditional low level feature extraction methods. Deep learning models achieved high precision in classification and object detection algorithms by extracting automated high level feature extraction process. This paper proposed a hybrid feature extraction technique by combining the high level features and low level features to improve the robustness of the feature vector. The proposed model used pre-trained Googlenet model as feature extractor and combined with Gabor multiscale texture features. The final feature vector will be used for retrieving the relevant image data from the large scale image dataset. It has achieved the precision of 91 percent which shows better than state of art methods.

Self-decisive algorithm for unconstrained optimization problems as in biomedical image analysis.

This study describes the construction of a new algorithm where image processing along with the two-step quasi-Newton methods is used in biomedical image analysis. It is a well-known fact that medical informatics is an essential component in the perspective of health care. Image processing and imaging technology are the recent advances in medical informatics, which include image content representation, image interpretation, and image acquisition, and focus on image information in the medical field. For this purpose, an algorithm was developed based on the image processing method that uses principle component analysis to find the image value of a particular test function and then direct the function toward its best method for evaluation. To validate the proposed algorithm, two functions, namely, the modified trigonometric and rosenbrock functions, are tested on variable space.

Design of deep convolution feature extraction for multimedia information retrieval

PurposeThe analysis of multimedia content is being applied in various real-time computer vision applications. In multimedia content, digital images constitute a significant part. The representation of digital images interpreted by humans is subjective in nature and complex. Hence, searching for relevant images from the archives is difficult. Thus, electronic image analysis strategies have become effective tools in the process of image interpretation.Design/methodology/approachThe traditional approach used is text-based, i.e. searching images using textual annotations. It consumes time in the manual process of annotating images and is difficult to reduce the dependency in textual annotations if the archive consists of large number of samples. Therefore, content-based image retrieval (CBIR) is adopted in which the high-level visuals of images are represented in terms of feature vectors, which contain numerical values. It is a commonly used approach to understand the content of query images in retrieving relevant images. Still, the performance is less than optimal due to the presence of semantic gap among the image content representation and human visual understanding perspective because of the image content photometric, geometric variations and occlusions in search environments.FindingsThe authors proposed an image retrieval framework to generate semantic response through the feature extraction with convolution network and optimization of extracted features using adaptive moment estimation algorithm towards enhancing the retrieval performance.Originality/valueThe proposed framework is tested on Corel-1k and ImageNet datasets resulted in an accuracy of 98 and 96%, respectively, compared to the state-of-the-art approaches.

Neural Encoding and Interpretation of High-Level Visual Cortices Based on fMRI Using Image Caption Features

On basis of functional magnetic resonance imaging (fMRI), researchers are devoted to designing visual encoding models to predict the neuron activity of human in response to presented image stimuli and analyze inner mechanism of human visual cortices. Deep network structure composed of hierarchical processing layers forms deep network models by learning features of data on specific task through big dataset. Deep network models have powerful and hierarchical representation of data, and have brought about breakthroughs for visual encoding, while revealing hierarchical structural similarity with the manner of information processing in human visual cortices. However, previous studies almost used image features of those deep network models pre-trained on classification task to construct visual encoding models. Except for deep network structure, the task or corresponding big dataset is also important for deep network models, but neglected by previous studies. Because image classification is a relatively fundamental task, it is difficult to guide deep network models to master high-level semantic representations of data, which causes into that encoding performance for high-level visual cortices is limited. In this study, we introduced one higher-level vision task: image caption (IC) task and proposed the visual encoding model based on IC features (ICFVEM) to encode voxels of high-level visual cortices. Experiment demonstrated that ICFVEM obtained better encoding performance than previous deep network models pre-trained on classification task. In addition, the interpretation of voxels was realized to explore the detailed characteristics of voxels based on the visualization of semantic words, and comparative analysis implied that high-level visual cortices behaved the correlative representation of image content.

Filtering Deep Convolutional Features for Image Retrieval

In image retrieval, highlighting target object and reducing the influence of background noise remains challenging. To address this problem, we propose a novel weighting method that aggregates deep convolutional features based on filtering, called filtering on spatial channel weighting (FSCW) factors, to represent image contents, and utilize it for image retrieval. There are three main contributions of this study. First, the designed filter can effectively remove the influence of background noise. Second, we propose a new channel selection and spatial weighting method, which can accurately distinguish target object from the background noise. Finally, we designed a new channel weighting strategy to suppress intra-image visual burstiness. Experimental results on benchmark datasets demonstrate that the proposed method effectively enhances discriminative power and outperforms some existing state-of-the-art methods in terms of the mAP metric. Furthermore, the proposed method is superior to some existing algorithms in distinguishing background noise and target object.

Y-Net: Learning Domain Robust Feature Representation for ground camera image and large-scale image-based point cloud registration

Registering the 2D images (2D space) with the 3D model of the environment (3D space) provides a promising solution to outdoor Augmented Reality (AR) virtual-real registration. In this work, we use the position and orientation of the ground camera image to synthesize a corresponding rendered image from the outdoor large-scale 3D image-based point cloud. To achieve the virtual-real registration, we indirectly establish the spatial relationship between 2D and 3D space by matching the above two kinds (2D/3D space) of cross-domain images. However, matching cross-domain images goes beyond the capability of handcrafted descriptors and existing deep neural networks. To address this issue, we propose an end-to-end network, Y-Net, to learn Domain Robust Feature Representations (DRFRs) for the cross-domain images. Besides, we introduce a cross-domain-constrained loss function that balances the loss in image content and cross-domain consistency of the feature representations. Experimental results show that the DRFRs simultaneously preserve the representation of image content and suppress the influence of independent domains. Furthermore, Y-Net outperforms the existing algorithms on extracting feature representations and achieves state-of-the-art performance in cross-domain image retrieval. Finally, we validate the Y-Net-based registration approach on campus to demonstrate its possible applicability.

KVL-BERT: Knowledge Enhanced Visual-and-Linguistic BERT for visual commonsense reasoning

Reasoning is a critical ability towards complete visual understanding. To develop machine with cognition-level visual understanding and reasoning abilities, the visual commonsense reasoning (VCR) task has been introduced. In VCR, given a challenging question about an image, a machine must answer correctly and then provide a rationale justifying its answer. The methods adopting the powerful BERT model as the backbone for learning joint representation of image content and natural language have shown promising improvements on VCR. However, none of the existing methods have utilized commonsense knowledge in visual commonsense reasoning, which we believe will be greatly helpful in this task. Therefore, we incorporate commonsense knowledge into the cross-modal BERT, and propose a novel Knowledge Enhanced Visual-and-Linguistic BERT (KVL-BERT for short) model. Besides taking visual and linguistic contents as input, external commonsense knowledge is integrated into the multi-layer Transformer. In order to preserve the structural information and semantic representation of the original sentence, we propose an algorithm called RMGSR (Relative-position-embedding and Mask-self-attention Guided Semantic Representations). Compared to other task-specific models and general task-agnostic pre-training models, our KVL-BERT outperforms them.

Visual Question Answering with External Knowledge

Visual Question Answering (VQA) system is an algorithm that takes an image, and a natural language question about it as input and produces an answer as the output. One of the most intriguing features of the VQA task is the uncertainty of the questions asked. To extract the knowledge required in answering them needs a spread of image manipulations. A significant part of the ongoing advancement in vision to language issues is accomplished by merging Recurrent Neural Networks and Convolutional Neural Networks. We propose a fully trainable neural network based on attribute extraction, established on the CNN and RNN design, which will apply to different vision to language issues. We do that by embedding a particular attribute representation of the scene interpretable by users. Each semantic feature corresponds to a word strip-mined from the training image description and signifies high-level information regarding the image context. We train an RCNN-based classifier for every attribute, and therefore the set of attribute likelihoods for a picture that forms a high-level representation of image content are learned. We then train an LSTM to get the captions, and answers, on the idea of these likelihoods. In light of the proposed feature-based vision to language model, the subsequent contribution is to present a means of incorporating information outside to the picture into the VQA task.

Automatic Identification of Local Features Representing Image Content with the Use of Convolutional Neural Networks

Image analysis has many practical applications and proper representation of image content is its crucial element. In this work, a novel type of representation is proposed where an image is reduced to a set of highly sparse matrices. Equivalently, it can be viewed as a set of local features of different types, as precise coordinates of detected keypoints are given. Additionally, every keypoint has a value expressing feature intensity at a given location. These features are extracted from a dedicated convolutional neural network autoencoder. This kind of representation has many advantages. First of all, local features are not manually designed but are automatically trained for a given class of images. Second, as they are trained in a network that restores its input on the output, they may be expected to minimize information loss. Consequently, they can be used to solve similar tasks replacing original images; such an ability was illustrated with image classification task. Third, the generated features, although automatically synthesized, are relatively easy to interpret. Taking a decoder part of our network, one can easily generate a visual building block connected with a specific feature. As the proposed method is entirely new, a detailed analysis of its properties for a relatively simple data set was conducted and is described in this work. Moreover, to present the quality of trained features, it is compared with results of convolutional neural networks having a similar working principle (sparse coding).

Deep Fuzzy Hashing Network for Efficient Image Retrieval

Hashing methods for efficient image retrieval aim at learning hash functions that map similar images to semantically correlated binary codes in the Hamming space with similarity well preserved. The traditional hashing methods usually represent image content by hand-crafted features. Deep hashing methods based on deep neural network (DNN) architectures can generate more effective image features and obtain better retrieval performance. However, the underlying data structure is hardly captured by existing DNN models. Moreover, the similarity (either visually or semantically) between pairwise images is ambiguous, even uncertain, to be measured in the existing deep hashing methods. In this article, we propose a novel hashing method termed deep fuzzy hashing network (DFHN) to overcome the shortcomings of existing deep hashing approaches. Our DFHN method combines the fuzzy logic technique and the DNN to learn more effective binary codes, which can leverage fuzzy rules to model the uncertainties underlying the data. Derived from fuzzy logic theory, the generalized hamming distance is devised in the convolutional layers and fully connected layers in our DFHN to model their outputs, which come from an efficient <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">xor</small> operation on given inputs and weights. Extensive experiments show that our DFHN method obtains competitive retrieval accuracy with highly efficient training speed compared with several state-of-the-art deep hashing approaches on two large-scale image datasets: CIFAR-10 and NUS-WIDE.

Ear Detection Using Convolutional Neural Network on Graphs with Filter Rotation.

Geometric deep learning (GDL) generalizes convolutional neural networks (CNNs) to non-Euclidean domains. In this work, a GDL technique, allowing the application of CNN on graphs, is examined. It defines convolutional filters with the use of the Gaussian mixture model (GMM). As those filters are defined in continuous space, they can be easily rotated without the need for some additional interpolation. This, in turn, allows constructing systems having rotation equivariance property. The characteristic of the proposed approach is illustrated with the problem of ear detection, which is of great importance in biometric systems enabling image based, discrete human identification. The analyzed graphs were constructed taking into account superpixels representing image content. This kind of representation has several advantages. On the one hand, it significantly reduces the amount of processed data, allowing building simpler and more effective models. On the other hand, it seems to be closer to the conscious process of human image understanding as it does not operate on millions of pixels. The contributions of the paper lie both in GDL application area extension (semantic segmentation of the images) and in the novel concept of trained filter transformations. We show that even significantly reduced information about image content and a relatively simple, in comparison with classic CNN, model (smaller number of parameters and significantly faster processing) allows obtaining detection results on the quality level similar to those reported in the literature on the UBEAR dataset. Moreover, we show experimentally that the proposed approach possesses in fact the rotation equivariance property allowing detecting rotated structures without the need for labor consuming training on all rotated and non-rotated images.

Remote sensing image retrieval method based on improved fast search and find of density peaks—bag of visual words model

Bag of visual words (BoVW) model has been widely used in content-based image retrieval for its good performance. The k-means clustering algorithm plays an important role in BoVW, but there are still some problems that need to be solved. Many methods are proposed to improve the clustering performance, such as fast search and find of density peaks (FSFDP), which has been successfully applied in image classification or image retrieval for its high efficiency but suffers from some limitations when it is applied to remote sensing image retrieval. A remote sensing image retrieval method based on an improved FSFDP-BoVW model is proposed. A feature selection method combining a chi-square test and a term frequency-inverse document frequency is proposed to select the features that can best represent image content and reduce the redundancy of remote sensing image features; a natural nearest neighbor algorithm is utilized to redefine the density of data points to reduce the interference of human subjectivity in FSFDP. Experimental results show that the proposed method can effectively reduce the time spent in constructing a codebook and improve the retrieval accuracy.

Saliency detection via multi-view graph based saliency optimization

Saliency detection is an important problem in computer vision and pattern recognition area. Many works have been proposed for addressing the saliency detection task. As a popular method, graph based saliency optimization has been widely studied. However, previous works have universally focussed on single graph optimization which fails to consider multi-view feature representation of image content. In this paper, we first provide a general framework for traditional graph based saliency optimization models. Then, we extend the general framework to the multi-view case and propose our general multi-view graph based saliency optimization model. Finally, we present a particular implementation of our general model and derive an effective updating algorithm to solve it. Experimental results using several benchmark datasets demonstrate the effectiveness of our proposed saliency model.

Camera Identification Based on Domain Knowledge-Driven Deep Multi-Task Learning

Camera identification has recently attracted considerable attention in the image forensic field of research. Several algorithms have been established based on the hand-crafted features and deep learning, through analysis of the traces achieved by the digital image acquisition process. Although these approaches have led to a breakthrough in the image forensics, some important problems still remain unsolved. For instance, extracting the hand-crafted features with human efforts is a difficult and time-consuming process, while data-driven deep learning methods tend to learn features that represent image contents rather than cameras’ characteristics. To fully take advantages of both hand-crafted and data-driven technologies, we propose a domain knowledge-driven method, which consists of one pre-processing module, one feature extractor, and one hierarchical multi-task learning procedure. The pre-processing module can introduce the domain knowledge to the subsequent deep learning network. Moreover, for device-level identification, hierarchical multi-task learning can provide more supervise information from the brand and model. The proposed framework is evaluated on three different tasks, i.e., the brand, model, and device-level identification using original and manipulated images. Our classification results demonstrate that the proposed method is effective and robust. To evaluate the robustness of the proposed method, we create a new database for the cell-phone identification and evaluate the proposed method. It is found that the accuracy of the cell-phone device identification can reach 84.3%, which is much higher than that of the camera identification. Moreover, the t-distributed stochastic neighbor embedding visualization results confirm that the features of different cell-phone devices are visually more separable than cameras.

Semantic-Enhanced Query Expansion System for Retrieving Medical Image Notes.

Most current image retrieval methods require constructing semantic metadata for representing image content. To manually create semantic metadata for medical images is time-consuming, yet it is a crucial component for query expansion. We proposed a new method for searching medical image notes that uses semantic metadata to improve query expansion and leverages a knowledge model developed specifically for the medical image domain to create relevant metadata. We used a syntactic parser and the Unified Medical Language System to analyze the corpus and store text information as semantic metadata in a knowledge model. Our new method has an interactive interface that allows users to provide relevance feedback and construct new queries more efficiently. Sixteen medical professionals evaluated the query expansion module, and each evaluator had prior experience searching for medical images. When using the initial query as the baseline standard, expanded queries achieved a performance boost of 22.6% in terms of the relevance score on first ten results (P-value<0.05). When using Google as another baseline, our system performed 24.6% better in terms of relevance score on the first ten results (P-value<0.05). Overall, 75% of the evaluators said the semantic-enhanced query expansion workflow is logical, easy to follow, and comfortable to use. In addition, 62% of the evaluators preferred using our system instead of Google. Evaluators who were positive about our system found the knowledge map-based visualization of candidate medical search terms helpful in refining cases from the initial search results.

Understanding visual scenes

AbstractA growing body of recent work focuses on the challenging problem of scene understanding using a variety of cross-modal methods which fuse techniques from image and text processing. In this paper, we develop representations for the semantics of scenes by explicitly encoding the objects detected in them and their spatial relations. We represent image content via two well-known types of tree representations, namely constituents and dependencies. Our representations are created deterministically, can be applied to any image dataset irrespective of the task at hand, and are amenable to standard NLP tools developed for tree-based structures. We show that we can apply syntax-based SMT and tree kernel methods in order to build models for image description generation and image-based retrieval. Experimental results on real-world images demonstrate the effectiveness of the framework.

Photo2clipart

We present a method to create vector cliparts from photographs. Our approach aims at reproducing two key properties of cliparts: they should be easily editable, and they should represent image content in a clean, simplified way. We observe that vector artists satisfy both of these properties by modeling cliparts with linear color gradients, which have a small number of parameters and approximate well smooth color variations. In addition, skilled artists produce intricate yet editable artworks by stacking multiple gradients using opaque and semi-transparent layers. Motivated by these observations, our goal is to decompose a bitmap photograph into a stack of layers, each layer containing a vector path filled with a linear color gradient. We cast this problem as an optimization that jointly assigns each pixel to one or more layer and finds the gradient parameters of each layer that best reproduce the input. Since a trivial solution would consist in assigning each pixel to a different, opaque layer, we complement our objective with a simplicity term that favors decompositions made of few, semi-transparent layers. However, this formulation results in a complex combinatorial problem combining discrete unknowns (the pixel assignments) and continuous unknowns (the layer parameters). We propose a Monte Carlo Tree Search algorithm that efficiently explores this solution space by leveraging layering cues at image junctions. We demonstrate the effectiveness of our method by reverse-engineering existing cliparts and by creating original cliparts from studio photographs.

Improving Large-Scale Image Retrieval Through Robust Aggregation of Local Descriptors

Visual search and image retrieval underpin numerous applications, however the task is still challenging predominantly due to the variability of object appearance and ever increasing size of the databases, often exceeding billions of images. Prior art methods rely on aggregation of local scale-invariant descriptors, such as SIFT, via mechanisms including Bag of Visual Words (BoW), Vector of Locally Aggregated Descriptors (VLAD) and Fisher Vectors (FV). However, their performance is still short of what is required. This paper presents a novel method for deriving a compact and distinctive representation of image content called Robust Visual Descriptor with Whitening (RVD-W). It significantly advances the state of the art and delivers world-class performance. In our approach local descriptors are rank-assigned to multiple clusters. Residual vectors are then computed in each cluster, normalized using a direction-preserving normalization function and aggregated based on the neighborhood rank. Importantly, the residual vectors are de-correlated and whitened in each cluster before aggregation, leading to a balanced energy distribution in each dimension and significantly improved performance. We also propose a new post-PCA normalization approach which improves separability between the matching and non-matching global descriptors. This new normalization benefits not only our RVD-W descriptor but also improves existing approaches based on FV and VLAD aggregation. Furthermore, we show that the aggregation framework developed using hand-crafted SIFT features also performs exceptionally well with Convolutional Neural Network (CNN) based features. The RVD-W pipeline outperforms state-of-the-art global descriptors on both the Holidays and Oxford datasets. On the large scale datasets, Holidays1M and Oxford1M, SIFT-based RVD-W representation obtains a mAP of 45.1 and 35.1 percent, while CNN-based RVD-W achieve a mAP of 63.5 and 44.8 percent, all yielding superior performance to the state-of-the-art.