Image Understanding Problem Research Articles

A Framework for Extensive Content-Based Image Retrieval System Incorporating Relevance Feedback and Query Suggestion

Over the last decade, there is a huge amount of images in which visual information has become increasingly important. If the images are to be utilized, they must be organized into databases, from there it can be searched based on different criteria. Content-based Image retrieval (CBIR) frameworks have turned into a mainstream subject of exploration; for their ability to retrieve images based on the real visual content rather than by manually connected textual descriptions. In the CBIR framework, analysis and interpretation of image information in large and diverse image databases is evidently complex because there is no prior information on the size or scale of individual structures within the images to be analyzed. In CBIR, retrieval is based on visual image features, which can be extracted automatically from the images with the help of human intervention, namely a technique called relevance feedback. Nonetheless, an efficient way of differentiating the visual content of images is complex to produce. Therefore, rather than a perfect solution CBIR systems must be able to exploit a partial solution to the problem of image understanding. In this paper, there is implementation of a CBIR framework is introduced that not only tries to efficiently capture the user intent based on the feedback but also provide query suggestions that can help its users to pose better queries in order to retrieve desired results in an efficient manner. The proposed technique is straightforward to implement and scopes efficiently to huge datasets. Extensive experiments on diverse real datasets with image similarity measures have revealed the dominance of the proposed method over original algorithms.

Generative Reasoning Integrated Label Noise Robust Deep Image Representation Learning.

The development of deep learning based image representation learning (IRL) methods has attracted great attention for various image understanding problems. Most of these methods require the availability of a set of high quantity and quality of annotated training images, which can be time-consuming, complex and costly to gather. To reduce labeling costs, crowdsourced data, automatic labeling procedures or citizen science projects can be considered. However, such approaches increase the risk of including label noise in training data. It may result in overfitting on noisy labels when discriminative reasoning is employed as in most of the existing methods. This leads to sub-optimal learning procedures, and thus inaccurate characterization of images. To address this issue, in this paper, we introduce a generative reasoning integrated label noise robust deep representation learning (GRID) approach. The proposed GRID approach aims to model the complementary characteristics of discriminative and generative reasoning for IRL under noisy labels. To this end, we first integrate generative reasoning into discriminative reasoning through a supervised variational autoencoder. This allows the proposed GRID approach to automatically detect training samples with noisy labels. Then, through our label noise robust hybrid representation learning strategy, GRID adjusts the whole learning procedure for IRL of these samples through generative reasoning and that of the other samples through discriminative reasoning. Our approach learns discriminative image representations while preventing interference of noisy labels during training independently from the IRL method being selected. Thus, unlike the existing label noise robust methods, GRID does not depend on the type of annotation, label noise, neural network architecture, loss function or learning task, and thus can be directly utilized for various image understanding problems. Experimental results show the effectiveness of the proposed GRID approach compared to the state-of-the-art methods. The code of the proposed approach is publicly available at https://github.com/gencersumbul/GRID.

Object–Part Registration–Fusion Net for Fine-Grained Image Classification

Classifying fine-grained categories (e.g., bird species, car, and aircraft types) is a crucial problem in image understanding and is difficult due to intra-class and inter-class variance. Most of the existing fine-grained approaches individually utilize various parts and local information of objects to improve the classification accuracy but neglect the mechanism of the feature fusion between the object (global) and object’s parts (local) to reinforce fine-grained features. In this paper, we present a novel framework, namely object–part registration–fusion Net (OR-Net), which considers the mechanism of registration and fusion between an object (global) and its parts’ (local) features for fine-grained classification. Our model learns the fine-grained features from the object of global and local regions and fuses these features with the registration mechanism to reinforce each region’s characteristics in the feature maps. Precisely, OR-Net consists of: (1) a multi-stream feature extraction net, which generates features with global and various local regions of objects; (2) a registration–fusion feature module calculates the dimension and location relationships between global (object) regions and local (parts) regions to generate the registration information and fuses the local features into the global features with registration information to generate the fine-grained feature. Experiments execute symmetric GPU devices with symmetric mini-batch to verify that OR-Net surpasses the state-of-the-art approaches on CUB-200-2011 (Birds), Stanford-Cars, and Stanford-Aircraft datasets.

Can deep learning solve a preschool image understanding problem?

Automatic assessment of learning is a process where the computer system automatically generates test items and evaluates the responses. Image is one of the major media to assess learning capabilities. In this article, we have proposed a system with a dataset containing images used to evaluate the kid’s ability to answer pictorial questions. The system tests a set of skills such as counting ability, color concept, and knowledge of objects and geometric shapes. It utilized a pipeline of RCNN and LSTM bridging with an object knowledge layer for generating question–answer pairs and achieved promising results. We benchmarked our dataset with state-of-the-art deep learning methods and assessed the performance of generating question–answer pairs from a given image. The source code and the dataset are available at https://github.com/skarifahmed/pic2question (will be publicly opened after the acceptance of the manuscript).

A NEW TECHNIQUE FOR OBJECT DETECTION AND TRACKING AND ITS APPLICATION TO ANALYSIS OF ROAD SCENE

Abstract. In this paper, a new technique for real-time object detection and tracking is presented. This technique is based on the geometrized histograms method (GHM) for segmenting and describing color images (frames of video sequences) and on the facilities for global image analysis provided by this method. Basic elements of the technique that make it possible to solve image understanding problems almost without using the pixel arrays of images are introduced and discussed.A real-time parallel software implementation of the developed technique is briefly discussed. This technique is applied to solving problems of road scene analysis. The application to finding small contrast objects in images, like traffic signals and signal zones of vehicles is given. The developed technique is applied also to detecting other vehicles in the frame. The results of processing different frame of videos of road scenes are presented and discussed.

Video Action Understanding

Many believe that the successes of deep learning on image understanding problems can be replicated in the realm of video understanding. However, due to the scale and temporal nature of video, the span of video understanding problems and the set of proposed deep learning solutions is arguably wider and more diverse than those of their 2D image siblings. Finding, identifying, and predicting actions are a few of the most salient tasks in this emerging and rapidly evolving field. With a pedagogical emphasis, this tutorial introduces and systematizes fundamental topics, basic concepts, and notable examples in supervised video action understanding. Specifically, we clarify a taxonomy of action problems, catalog and highlight video datasets, describe common video data preparation methods, present the building blocks of state-of-the art deep learning model architectures, and formalize domain-specific metrics to baseline proposed solutions. This tutorial is intended to be accessible to a general computer science audience and assumes a conceptual understanding of supervised learning.

Integrated Artificial Intelligence Systems

From Shakey the Robot to self‐driving cars, from the personal computer to personal assistants on our phones, the Defense Advanced Research Projects Agency (DARPA) has led the development of integrated artificial intelligence (AI) systems for more than half a century. From the earliest days of AI, it was apparent that a robust, generally intelligent system should include a complete set of capabilities: perception, memory, reasoning, learning, planning, and action; and when DARPA initiated AI research in the 1960s, ambitious projects such as Shakey the Robot went after the complete package. As DARPA realized the challenges, they backed away from the ultimate goal of integrated AI and tried to make progress on the individual problems of image understanding, speech and language understanding, knowledge representation and reasoning, planning and decision aids, machine learning, and robotic manipulation. Yet, even as researchers struggled to make progress in these subdisciplines, DARPA periodically resurrected the challenge of integrated intelligent systems and pushed the community to try again. In the 1980s, DARPA's Strategic Computing Initiative took on challenges of integrated AI projects such as the Autonomous Land Vehicle and the Pilot's Associate. These did not succeed, but instead set the stage for the several decades of more siloed research that followed, until it was time to try again. In the 2000s, DARPA took on the integrated AI problem again with its Grand Challenges, which led to the first self‐driving cars, and projects such as the Personalized Assistant that Learns, which produced Apple's Siri. These efforts created complex, richly‐integrated systems that represented quantum leaps ahead in machine intelligence. The integration of sophisticated capabilities in a fundamental way is the key to general intelligence. This is the story of DARPA's persistent long‐term support for this essential premise of AI.

Image Description using Attention Mechanism

Image Description involves generating a textual description of images which is essential for the problem of image understanding. The variable and ambiguous nature of possible image descriptions make this task challenging. There are different approaches for automated image captioning which explain the image contents along with a complete understanding of the image, rather than just simply classifying it into a particular object type. However, learning image contexts from the text and generating image descriptions similar to human's description requires to focus on important features of the image using attention mechanism. We provide an outline of the various recent works in image description models employing various attention mechanism. We present an analysis of the various approaches, datasets and evaluation metrics that are utilized for image description. We showcase a model using the encoder-decoder attention mechanism based on Flickr dataset and evaluate the performance using BLEU metrics.

Comparative Efficiency Analysis for Various Neuroarchitectures for Semantic Segmentation of Images in Remote Sensing Applications

The problem of image understanding currently attracts considerable attention of researchers, since its solution is critically important for a significant number of applied problems. Among the most critical components of this problem is the semantic segmentation of images, which provides a classification of objects on the image at the pixel level. One of the applied problems in which semantic segmentation is an essential element of the process of solving them is the information support of the behavior control systems for robotic UAVs. Among the various types of images that are used to solve such problems, it should be noted images obtained by remote sensing of the Earth’s surface. A significant number of variants of neuroarchitectures based on convolutional neural networks have been proposed to solve the semantic image segmentation problem, However, for various reasons, not all of them are suitable for working with images of the Earth’s surface obtained using remote sensing. Neuroarchitectures that are potentially suitable for solving the problem of semantic segmentation of images of the Earth’s surface are identified, a comparative analysis of their effectiveness concerning this task is carried out.

Sparse autoencoder for social image understanding

The rapid increase of social media images has made organizing these resources effectively a huge problem. Labeling unlabeled images becomes the crucial division of social image understanding. However, the enhancement of social image sharpness leads to the increase of surface feature dimension. These multidimensional complex features leads to the curse of dimensionality and the difficulty of feature extraction. In this paper, sparse autoencoder is studied to solve the problem of social image understanding, because sparse autoencoder can make these features represent the original data in a refined way, thus avoiding curse of dimensionality as much as possible and significantly improve the understanding effect. First, we explore the dimensional reduction capability of sparse autoencoder, and use sparse autoencoder to get low-dimensional features. Second, for low-dimensional features, an enhanced multi-label classifier is utilized to assign labels with the help of cosine similarity about tags correlation. The ability of dimensionality reduction of sparse autoencoder is proved by mapping matrix of image-label. Finally, we test our approach on several publicly available social media datasets. The results demonstrate that our proposed method is superior to lots of non-deep learning method among three evaluation indexes of social image understanding.

Stacked Autoencoder Based Weak Supervision for Social Image Understanding

Many studies in recent years have focused on social image understanding due to the increasing number of shared images from social networks and online communities. However, previous work in social image understanding fails to learn an effective feature representation because of a large amount of missing and irrelevant tags, though matrix completion techniques are frequently utilized for this purpose. Autoencoder models have been validated to be effective in learning latent low-dimensional representations in unsupervised learning. In this paper, we propose a new social image understanding model based on deep autoencoders, which can learn the shared latent codes of social images and tags as supervision information in the deep autoencoders. First, social images are extracted with multi-modal features, which provide a comprehensive characterization to image semantic understanding. And, the social image understanding problems are transformed into the problem of minimizing an optimization objective. Second, multi-layered autoencoders with weak supervision integration are employed to learn an efficient low-dimensional representation from the multi-view feature sources that can make up the semantic gap between image features and tags through minimizing the problem formulation. Finally, we design a new balanced loss function based on binary cross entropy, in which we address highly sparse inputs for a better optimization performance. The extensive experiments on several real-world social image datasets confirm the effectiveness and robustness of the proposed model compared with the state-of-the-art methods.

3D Reconstruction using artificial bee colony based iterative closest point algorithm

3D Reconstruction has been an enduring problem in Image understanding and computer vision. There is an increasing interest on 3D information in the general public due to rapid development of 3D imaging techniques, marketing of 3D movies and games, and low cost of depth cameras. Numerous algorithms that have been proposed for performing 3D reconstruction using different variants of Iterative Closest Point(ICP) algorithm focusses mainly on reducing the computation time. The accuracy of 3D reconstruction is not taken in to consideration. An efficient, accurate, real time and active 3D reconstruction method using Kinect sensor is developed in this paper focusing on improving the accuracy of 3D reconstruction in less computation time. An Artificial Bee colony based ICP algorithm is proposed by incorporating several efficient variants of ICP algorithm. The proposed algorithm is intended to improve the accuracy and stability of the standard ICP algorithm. The performance of the proposed algorithm is satisfactory when compared with structured light technique and several ICP variants with respect to accuracy, complexity and computation speed.

Dictionary Learning Inspired Deep Network for Scene Recognition

Scene recognition remains one of the most challenging problems in image understanding. With the help of fully connected layers (FCL) and rectified linear units (ReLu), deep networks can extract the moderately sparse and discriminative feature representation required for scene recognition. However, few methods consider exploiting a sparsity model for learning the feature representation in order to provide enhanced discriminative capability. In this paper, we replace the conventional FCL and ReLu with a new dictionary learning layer, that is composed of a finite number of recurrent units to simultaneously enhance the sparse representation and discriminative abilities of features via the determination of optimal dictionaries. In addition, with the help of the structure of the dictionary, we propose a new label discriminative regressor to boost the discrimination ability. We also propose new constraints to prevent overfitting by incorporating the advantage of the Mahalanobis and Euclidean distances to balance the recognition accuracy and generalization performance. Our proposed approach is evaluated using various scene datasets and shows superior performance to many state-of-the-art approaches.

Beyond Eleven Color Names for Image Understanding

Color description is one of the fundamental problems of image understanding. One of the popular ways to represent colors is by means of color names. Most existing work on color names focuses on only the eleven basic color terms of the English language. This could be limiting the discriminative power of these representations, and representations based on more color names are expected to perform better. However, there exists no clear strategy to choose additional color names. We collect a dataset of 28 additional color names. To ensure that the resulting color representation has high discriminative power we propose a method to order the additional color names according to their complementary nature with the basic color names. This allows us to compute color name representations with high discriminative power of arbitrary length. In the experiments we show that these new color name descriptors outperform the existing color name descriptor on the task of visual tracking, person re-identification and image classification.

Image Caption Generation with Part of Speech Guidance

As a fundamental problem in image understanding, image caption generation has attracted much attention from both computer vision and natural language processing communities. In this paper, we focus on how to exploit the structure information of a natural sentence, which is used to describe the content of an image. We discover that the Part of Speech (PoS) tags of a sentence, are very effective cues for guiding the Long Short-Term Memory (LSTM) based word generator. More specifically, given a sentence, the PoS tag of each word is utilized to determine whether it is essential to input image representation into the word generator. Benefiting from such a strategy, our model can closely connect the visual attributes of an image to the word concepts in the natural language space. Experimental results on the most popular benchmark datasets, e.g., Flickr30k and MS COCO, consistently demonstrate that our method can significantly enhance the performance of a standard image caption generation model, and achieve the conpetitive results.

Texture Characterization Using Shape Co-Occurrence Patterns.

Texture characterization is a key problem in image understanding and pattern recognition. In this paper, we present a flexible shape-based texture representation using shape co-occurrence patterns. More precisely, texture images are first represented by a tree of shapes, each of which is associated with several geometrical and radiometric attributes. Then, four typical kinds of shape co-occurrence patterns based on the hierarchical relationships among the shapes in the tree are learned as codewords. Three different coding methods are investigated for learning the codewords, which can be used to encode any given texture image into a descriptive vector. In contrast with existing works, the proposed approach not only inherits the shape-based method's strong ability to capture geometrical aspects of textures and high robustness to variations in imaging conditions but also provides a flexible way to consider shape relationships and to compute high-order statistics on the tree. To the best of our knowledge, this is the first time that co-occurrence patterns of explicit shapes have been used as a tool for texture analysis. Experiments on various texture and scene data sets demonstrate the efficiency of the proposed approach.

Weakly Supervised Deep Matrix Factorization for Social Image Understanding.

The number of images associated with weakly supervised user-provided tags has increased dramatically in recent years. User-provided tags are incomplete, subjective and noisy. In this paper, we focus on the problem of social image understanding, i.e., tag refinement, tag assignment, and image retrieval. Different from previous work, we propose a novel weakly supervised deep matrix factorization algorithm, which uncovers the latent image representations and tag representations embedded in the latent subspace by collaboratively exploring the weakly supervised tagging information, the visual structure, and the semantic structure. Due to the well-known semantic gap, the hidden representations of images are learned by a hierarchical model, which are progressively transformed from the visual feature space. It can naturally embed new images into the subspace using the learned deep architecture. The semantic and visual structures are jointly incorporated to learn a semantic subspace without overfitting the noisy, incomplete, or subjective tags. Besides, to remove the noisy or redundant visual features, a sparse model is imposed on the transformation matrix of the first layer in the deep architecture. Finally, a unified optimization problem with a well-defined objective function is developed to formulate the proposed problem and solved by a gradient descent procedure with curvilinear search. Extensive experiments on real-world social image databases are conducted on the tasks of image understanding: image tag refinement, assignment, and retrieval. Encouraging results are achieved with comparison with the state-of-the-art algorithms, which demonstrates the effectiveness of the proposed method.

Relativistic Psychometrics in Subjective Scaling

The article announces the possibilities of semantic modeling in the development of feedback tools in social sciences. A new approach to the computational theory of perceptions (CTP) for analysis of mental object is proposed. The article demonstrates the implementation of relativistic psychometrics for the study of mental response (opinions, expectations and attitudes). The problem of image understanding and its significance is considered in combination of soft and hard computing. It is shown that the modeling of object (its coding and decoding in ‘mental map’) obeys the semiotic and mathematical logic. Computing with perceptions for the rules of mental representation proves their identity to the laws of conservation. The article demonstrates the versatility of the semiotic description of objects in Minkowski space. It also confirms by mathematical solution C. S. Peirce's metaphor, according to which the semiology of language is a truly universal algebra of relations.

Edge detection for pattern recognition: a survey

This review provides an overview of the literature on the edge detection methods for pattern recognition that inspire from the understanding of human vision. We note that edge detection is one of the most fundamental processes within the low level vision and provides the basis for the higher level visual intelligence in primates. The recognition of the patterns within the images relates closely to the spatiotemporal processes of edge formations, and its implementation needs a cross-disciplinary approach in neuroscience, computing and pattern recognition. In this review, the edge detectors are grouped in as edge features, gradients and sketch models, and some example applications are provided for reference. We note a significant increase in the amount of published research in the last decade that utilises edge features in a wide range of problems in computer vision and image understanding having a direct implication to pattern recognition with images.

A novel low false alarm rate pedestrian detection framework based on single depth images

Pedestrian detection is an important image understanding problem with many potential applications. There has been little success in creating an algorithm which exhibits a high detection rate while keeping the false alarm in a relatively low rate. This paper presents a method designed to resolve this problem. The proposed method uses the Kinect or any similar type of sensors which facilitate the extraction of a distinct foreground. Then potential regions, which are candidates for the presence of human(s), are detected by employing the widely used Histogram of Oriented Gradients (HOG) technique, which performs well in terms of good detection rates but suffers from significantly high false alarm rates. Our method applies a sequence of operations to eliminate the false alarms produced by the HOG detector based on investigating the fine details of local shape information. Local shape information can be identified by efficient utilization of the edge points which, in this work, are used to formulate the so called Shape Context (SC) model. The proposed detection framework is divided in four sequential stages, with each stage aiming at refining the detection results of the previous stage. In addition, our approach employs a pre-evaluation stage to pre-screen and restrict further detection results. Extensive experimental results on the dataset created by the authors, involves 673 images collected from 11 different scenes, demonstrate that the proposed method eliminates a large percentage of the false alarms produced by the HOG pedestrian detector.