Object Patches Research Articles

Image segmentation and density clustering for moving object patches extraction in remote sensing image

Image segmentation and density clustering for moving object patches extraction in remote sensing image

Robust eye detection using deeply-learned gaze shifting path

Eye detection is a very useful technique in many intelligent applications. Since the importance of eyes to human beings, eye detection technique is an indispensable component in intelligent systems, e.g., emotional analysis, iris detection and gaze estimation. Recently, there have been proposed a large number of methods for eye detection, wherein good performances have been achieved. But these methods cannot take human visual perception into account, that is to say, human beings will first pay attention to the eyes when they are communicating with each other, and then nose, then mouth. In addition, their geometric positions are almost fixed, i.e., eyes are above the nose and mouth, and eyes are on both sides of the nose. So in our work, a novel method for eye detection is proposed using human visual perception. More specifically, we first derive object patches from a large quantity of training images. Then, a geometry-preserved object patches ranking method is designed to effectively mimic human visual mechanism when human beings are communicating with each other. After that, these ordered object patches will be fed into CNN to extract patch-level deep features, then patch-level deep features will be represented by deep representations. Finally, eye detection can be achieved using learned deep representation. Experimental results on different database show that our method can achieve high efficiency and accuracy of eye detection.

A sound-based video clipping framework toward sports scenes

Video clipping system is very important in many intelligent applications. In order to shorten the time of video and extract the framework of the video, many methods have been proposed. But these methods just considered videos without taking sound into account. As we know, sound is also an important information for image processing. For example, many sport match videos include rich sound of audience and commentator such as NBA. In addition, human pay more attention to some video clips of interest (VCOI) such as scoring time instead of pause. So in this paper, we propose a sound-based video clipping framework toward specific sports scenes. First, we convert sound of sport videos to sonogram. For some aesthetically-pleasing images (APIM) such as slam dunk or jump shot, a set of object patches are selected using BING feature. Then, these object patches are ordered by our active object patches ranking algorithm. After that, ordered object patches and sonogram are fed into CNN respectively to obtain patch-level deep feature. In order to obtain image-level deep representation, deep feature extracted from ordered object patches are aggregated statistically into a deep representation. Finally, probabilistic model is used to select VCOI and APIM. Experiments on some NBA basketball matches have shown the effectiveness of our video clipping framework.

Improved Hough transform by modeling context with conditional random fields for partially occluded pedestrian detection

Traditional Hough transform-based methods detect objects by casting votes to object centroids from object patches. It is difficult to disambiguate object patches from the background by a classifier without contextual information, as an image patch only carries partial information about the object. To leverage the contextual information among image patches, we capture the contextual relationships on image patches through a conditional random field (CRF) with latent variables denoted by locality-constrained linear coding (LLC). The strength of the pairwise energy in the CRF is measured using a Gaussian kernel. In the training stage, we modulate the visual codebook by learning the CRF model iteratively. In the test stage, the binary labels of image patches are jointly estimated by the CRF model. Image patches labeled as the object category cast weighted votes for object centroids in an image according to the LLC coefficients. Experimental results on the INRIA pedestrian, TUD Brussels, and Caltech pedestrian datasets demonstrate the effectiveness of the proposed method compared with other Hough transform-based methods.

Perceptually Aware Image Retargeting for Mobile Devices.

Retargeting aims at adapting an original high-resolution photograph/video to a low-resolution screen with an arbitrary aspect ratio. Conventional approaches are generally based on desktop PCs, since the computation might be intolerable for mobile platforms (especially when retargeting videos). Typically, only low-level visual features are exploited, and human visual perception is not well encoded. In this paper, we propose a novel retargeting framework that rapidly shrinks a photograph/video by leveraging human gaze behavior. Specifically, we first derive a geometry-preserving graph ranking algorithm, which efficiently selects a few salient object patches to mimic the human gaze shifting path (GSP) when viewing a scene. Afterward, an aggregation-based CNN is developed to hierarchically learn the deep representation for each GSP. Based on this, a probabilistic model is developed to learn the priors of the training photographs that are marked as aesthetically pleasing by professional photographers. We utilize the learned priors to efficiently shrink the corresponding GSP of a retargeted photograph/video to maximize its similarity to those from the training photographs. Extensive experiments have demonstrated that: 1) our method requires less than 35 ms to retarget a photograph (or a video frame) on popular iOS/Android devices, which is orders of magnitude faster than the conventional retargeting algorithms; 2) the retargeted photographs/videos produced by our method significantly outperform those of its competitors based on a paired-comparison-based user study; and 3) the learned GSPs are highly indicative of human visual attention according to the human eye tracking experiments.

Deep Active Learning with Contaminated Tags for Image Aesthetics Assessment.

Image aesthetic quality assessment has becoming an indispensable technique that facilitates a variety of image applications, e.g., photo retargeting and non-realistic rendering. Conventional approaches suffer from the following limitations: 1) the inefficiency of semantically describing images due to the inherent tag noise and incompletion, 2) the difficulty of accurately reflecting how humans actively perceive various regions inside each image, and 3) the challenge of incorporating the aesthetic experiences of multiple users. To solve these problems, we propose a novel semi-supervised deep active learning (SDAL) algorithm, which discovers how humans perceive semantically important regions from a large quantity of images partially assigned with contaminated tags. More specifically, as humans usually attend to the foreground objects before understanding them, we extract a succinct set of BING (binarized normed gradients) [60]-based object patches from each image. To simulate human visual perception, we propose SDAL which hierarchically learns human gaze shifting path (GSP) by sequentially linking semantically important object patches from each scenery. Noticeably, SDLA unifies the semantically important regions discovery and deep GSP feature learning into a principled framework, wherein only a small proportion of tagged images are adopted. Moreover, based on the sparsity penalty, SDLA can optimally abandon the noisy or redundant low-level image features. Finally, by leveraging the deeply-learned GSP features, a probabilistic model is developed for image aesthetics assessment, where the experience of multiple professional photographers can be encoded. Besides, auxiliary quality-related features can be conveniently integrated into our probabilistic model. Comprehensive experiments on a series of benchmark image sets have demonstrated the superiority of our method. As a byproduct, eye tracking experiments have shown that GSPs generated by our SDAL are about 93% consistent with real human gaze shifting paths.

Visual object tracking with multi-scale superpixels and color-feature guided kernelized correlation filters

While discriminative correlation filter based tracking algorithms have achieved competitive results and demonstrated successfully, there still remain some challenges, such as handling the scenarios of scale variation, fast motion, etc. A region of interest with a fixed size, which is usually used by discriminative correlation filter based tracking algorithms to train correlation filter and track object, makes the trackers hard to deal with the challenges of fast motion and scale variation. It also restricts the use of object structure information. In this paper, we propose a Multi-Scale Superpixels and Color Feature Guided Kernelized Correlation Filters (MSSCF-KCF) to deal with the problems mentioned above. Firstly, we treat tracking procedure as optimizing the combination of components of an object, and propose a multi-scale superpixel method to segment object image based on a proposed global confidence mask which determines the center and size of object patches automatically. Then, KCF is embedded into Bayesian filter framework, in which the proposed color guided confidence map is viewed as an observation model, to expand the tracking range and improve the performance when handling fast motion. Finally, we propose a center distance matrix of each object patch to use structure information of object. A novel min–max criterion as well as drop-out strategy is used to search the optimal combination and estimate the state of object, which also helps the tracker to cope with scale variation. Besides, monitoring update strategy is proposed to monitor tracking operation and adjust parameters dynamically. The proposed algorithm is tested on a self-made dataset and some challenging sequences with a novel evaluation criterion. The results demonstrate the feasibility and effectiveness compared with the state-of-the-art methods.

Multi‐image matching for object recognition

One of the central problems in object recognition is to develop appropriate representations for the objects in images. The authors present a novel approach for image representation that is based on graphs. In the proposed image graph, each node represents a patch and edges are added between neighbouring nodes. First, class‐specific match‐set graphs are generated by matching the image graphs that are in the same categories, and the multi‐image matching problem is solved by applying a seed‐expansion strategy. Then, the matches between the match‐set graphs and an image graph are considered to be the object patches in the image. Finally, the features extracted from these patches are used for the image representation. Extensive experiments are conducted to demonstrate that their approach can obtain state‐of‐the‐art results on several challenging datasets.

Fast Fourier Transform Networks for Object Tracking Based on Correlation Filter

In this paper, we propose fast Fourier transform networks for object tracking, called FFTNet . FFTNet is a correlation filter (CF)-based tracker that integrates two main components of CF, i.e., auto correlation and cross correlation between the features of two images. Thus, FFTNet takes full advantage of CF: 1) Auto correlation and cross correlation of CF are efficiently computed by FFT so that FFTNet achieves high computational efficiency and 2) FFTNet successfully performs displacement detection even in similar signals so that it achieves good tracking performance. Moreover, FFTNet combines the advantage of CF with convolutional neural networks so that it has strong capabilities of learning feature representation and matching function. FFTNet is trained end-to-end in an off-line manner. First, we input two-input patches of target object and search region into shared convolutional layers to get their features. Then, we calculate auto correlation and cross correlation from two features. Next, we concatenate the results of auto correlation and cross correlation, and put them into another subnetwork to learn a matching function. Finally, we get a response map whose values represent the probability of each pixel belonging to the target. Experimental results demonstrate that FFTNet outperforms state-of-the-arts in both tracking accuracy and computational efficiency.

An Efficient Marine Oil Spillage Identification Scheme Based on an Improved Active Contour Model Using Fully Polarimetric SAR Imagery

Oil spillage pollution results in damages to marine and coastal environments and ecosystems at all levels. Due to the distinct advantages of fully polarimetric synthetic aperture radar (Pol-SAR), it has been widely employed to improve oil spillage event observations. In this paper, with the aim of enhancing ocean oil spillage identification, an improved scheme was proposed, which was mainly based on an active contour model (ACM), with a special emphases on choosing an optimal initial box boundary for the ACM model, and a noise suppressing process. In this proposed scheme, the ACM initial box boundary was chosen to be the median level contour of the imagined mountain if regarding image grey values as elevations. Meanwhile, following Lee filter operator, a Gaussian smooth operator was hired to further weaken the speckle noise, thereby making the evolution of ACM more stable. Three sets of RADARSAT-2 fully Pol-SAR data were used to test of the proposed method. In addition, the current study achieved the first complete comparison of the identification abilities of the ten most common recently used fully Pol-SAR features for oil spills. The experimental results show: 1) The features of H; v; span; P; ${\text{M}}_{33}$ ; and $\rho $ possessed good distinguishable abilities for oil spillage detection in sea water. Specially, features span and v outperformed other Pol-SAR features with regard to the elaborate distinguishing identifications of small patches of oil spill objects from other small objects, such as oil platforms or ships; 2) An appropriate setting of the initial box boundary of the ACM was able to accelerate the global search, and eventually led to improvements in the accuracy of the oil spill identification, while simultaneously reducing the computational time; 3) The effective noise suppression was confirmed to be very necessary, guaranteeing stability of the ACM evolution. When compared with other traditional methods, the proposed improved scheme was found to be more effective in the overall identification accuracy (99.8%).

Categorizing scenes by exploring scene part information without constructing explicit models

Abstract Approaches based on scene parts are deemed to be one of the main streams for scene categorization. In previous methods, before one can utilize scene parts, models need to be constructed for them first. The quality of part models has a great influence on the final results. However, building high-quality scene part models is still an open question. To perform scene categorization based on parts effectively, in this paper we propose to explore scene part information without constructing explicit models for them. For this purpose, a cascading framework is used, at each of whose stages we aim to process image patches potentially corresponding to scene parts from different perspectives. Specifically, the first stage of the framework uses the selective search algorithm to extract possible object patches from images and represents obtained patches based on convolutional neural networks. Then, spectral clustering and linear support vector machines are adopted to select representative visual patterns for images in the second stage. In the third stage, random forest and multi-class support vector machines are combined to mine and classify image features for determining the categories of the images. Through using the cascading framework, we can explore scene part information step by step without needing to construct explicit models for them. Finally, extensive experiments are conducted to evaluate the proposed method on three well-known benchmark scene datasets, i.e. MIT Indoor 67, SUN397 and Places. Experiment results demonstrated the effectiveness of the proposed method.

Mapping Spatial Preferences in Face and Object Patches in the Rhesus Macaque Using fMRI

Mapping Spatial Preferences in Face and Object Patches in the Rhesus Macaque Using fMRI

Attention driven multi-modal similarity learning

To learn a function for measuring similarity or relevance between objects is an important machine learning task, referred to as similarity learning. Conventional methods are usually insufficient for processing complex patterns, while more sophisticated methods produce results supported by parameters and mathematical operations that are hard to interpret. To improve both model robustness and interpretability, we propose a novel attention driven multi-modal algorithm, which learns a distributed similarity score over different relation modalities and develops an interaction-oriented dynamic attention mechanism to selectively focus on salient patches of objects of interest. Neural networks are used to generate a set of high-level representation vectors for both the entire object and its segmented patches. Multi-view local neighboring structures between objects are encoded in the high-level object representation through an unsupervised pre-training procedure. By initializing the relation embeddings with object cluster centers, each relation modality can be reasonably interpreted as a semantic topic. A layer-wise training scheme based on a mixture of unsupervised and supervised training is proposed to improve generalization. The effectiveness of the proposed method and its superior performance compared against state-of-the-art algorithms are demonstrated through evaluations based on different image retrieval tasks.

Experimental setting affects the performance of guppies in a numerical discrimination task.

A recent study found that guppies (Poecilia reticulata) can be trained to discriminate 4 versus 5 objects, a numerical discrimination typically achieved only by some mammals and birds. In that study, guppies were required to discriminate between two patches of small objects on the bottom of the tank that they could remove to find a food reward. It is not clear whether this species possesses exceptional numerical accuracy compared with the other ectothermic vertebrates or whether its remarkable performance was due to a specific predisposition to discriminate between differences in the quality of patches while foraging. To disentangle these possibilities, we trained guppies to the same numerical discriminations with a more conventional two-choice discrimination task. Stimuli were sets of dots presented on a computer screen, and the subjects received a food reward upon approaching the set with the larger numerosity. Though the cognitive problem was identical in the two experiments, the change in the experimental setting led to a much poorer performance as most fish failed even the 2 versus 3 discrimination. In four additional experiments, we varied the duration of the decision time, the type of stimuli, the length of training, and whether correction was allowed in order to identify the factors responsible for the difference. None of these parameters succeeded in increasing the performance to the level of the previous study, although the group trained with three-dimensional stimuli learned the easiest numerical task. We suggest that the different results with the two experimental settings might be due to constraints on learning and that guppies might be prepared to accurately estimate patch quality during foraging but not to learn an abstract stimulus-reward association.

Human articulated body recognition method in high-resolution monitoring images

Building recognition is a crucial and still challenging problem in computer vision and pattern recognition field, especially in human articulate body recognition. In this paper, we present a novel formulation for detecting articulated human bodies from high-resolution monitoring images. Firstly, we use a coarse detection process to detect the human object patches from monitoring images. To build a descriptor which contain color frequency information based on histograms of oriented gradient method. Then, a linear support vector machine was used to speed up detecting large image patches which contain the object joints. Secondly, towards an efficient detection, a refinement classification method is proposed to determine the patches that actually contain objects. Finally, to increase the size of image patches in order that the whole human objects can be included, an accelerated and improved salient mask is used to improve the performance of the dense scale-invariant feature transform descriptor. Experimental results on three datasets demonstrated our method effectiveness.

Types, Locations, and Scales from Cluttered Natural Video and Actions

We model the autonomous development of brain-inspired circuits through two modalities—video stream and action stream that are synchronized in time. We assume that such multimodal streams are available to a baby through inborn reflexes, self-supervision, and caretaker’s supervision, when the baby interacts with the real world. By autonomous development, we mean that not only that the internal (inside the “skull”) self-organization is fully autonomous, but the developmental program (DP) that regulates the computation of the network is also task nonspecific. In this work, the task-nonspecificity is reflected by the fact that the actions associated with an attended object in a cluttered, natural, and dynamic scene is taught after the DP is finished and the “life” has begun. The actions correspond to neuronal firing patterns representing object type, object location and object scale, but learning is directly from unsegmented cluttered scenes. Along the line of where–what networks (WWN), this is the first one that explicitly models multiple “brain” areas—each for a different range of object scales. Among experiments, large natural video experiments were conducted. To show the power of automatic attention in unknown cluttered backgrounds, the last experimental group demonstrated disjoint tests in the presence of large within-class variations (object 3-D-rotations in very different unknown backgrounds), but small between-class variations (small object patches in large similar and different unknown backgrounds), in contrast with global classification tests such as ImageNet and Atari Games.

Detection of engineering vehicles in high-resolution monitoring images

This paper presents a novel formulation for detecting objects with articulated rigid bodies from high-resolution monitoring images, particularly engineering vehicles. There are many pixels in high-resolution monitoring images, and most of them represent the background. Our method first detects object patches from monitoring images using a coarse detection process. In this phase, we build a descriptor based on histograms of oriented gradient, which contain color frequency information. Then we use a linear support vector machine to rapidly detect many image patches that may contain object parts, with a low false negative rate and a high false positive rate. In the second phase, we apply a refinement classification to determine the patches that actually contain objects. In this stage, we increase the size of the image patches so that they include the complete object using models of the object parts. Then an accelerated and improved salient mask is used to improve the performance of the dense scale-invariant feature transform descriptor. The detection process returns the absolute position of positive objects in the original images. We have applied our methods to three datasets to demonstrate their effectiveness.

Global Coupled Learning and Local Consistencies Ensuring for sparse-based tracking

This paper presents a robust tracking algorithm by sparsely representing the object at both global and local levels. Accordingly, the algorithm is constructed by two complementary parts: Global Coupled Learning (GCL) part and Local Consistencies Ensuring (LCE) part. The global part is a discriminative model which aims to utilize the holistic features of the object via an over-complete global dictionary and classifier, and the dictionary and classifier are coupled learning to construct an adaptive GCL part. While in LCE part, we explore the object׳s local features by sparsely coding the object patches via a local dictionary, then both temporal and spatial consistencies of the local patches are ensured to refine the tracking results. Moreover, the GCL and LCE parts are integrated into a Bayesian framework for constructing the final tracker. Experiments on fifteen benchmark challenging sequences demonstrate that the proposed algorithm has more effectiveness and robustness than the alternative ten state-of-the-art trackers.

Object recognition via contextual color attention

Visual attention is effective in differentiating an object from its surroundings. Color is used to guide attention via a top-down category-specific attention map in the top-down color attention (CA) method. To uniformly highlight the entire object, our color attention map is reconstructed based on the estimated object patches. The object patches consist of strong patches and false weak patches whose contextual color attention values are beyond the optimal threshold of class-specific contextual color attention. The color attention map constructed by the object color histogram is then used to weight the local shape for object recognition. Extensive experiments are conducted to show that our method can provide state-of-the-art results on several challenging datasets.

Image Relevance Prediction Using Query-Context Bag-of-Object Retrieval Model

Image search reranking and image research result summarization are two effective approaches which enhance text-based image search results using visual information. Since the existing approaches optimize search relevance in terms of average performance, they usually cannot achieve satisfactory results for some particular classes of queries, like “object queries,” which is defined as the queries with the intent of searching for some kinds of objects. One possible reason is that the generic approaches such as , , are mostly built based on the global statistics of images as features while ignoring the fact that the relevance between the image and the query sometimes depends on an image patch instead of the whole image. In this paper, we therefore design a novel bag-of-object retrieval model to predict image relevance, which is particularly effective for object queries. First, we construct an object vocabulary containing query-relative objects by mining frequent object patches from the result image collection of the expanded query set. After representing each image as a bag of objects, our retrieval model can be derived from a risk-minimization framework for language modeling. To demonstrate the effectiveness of the proposed model, this paper also present two related applications: for image search reranking, we adopt a supervised framework to combine multiple ranking features from different assumptions; for image search result summarization, we propose a two-step ranking process which optimizes not only representativeness but also image attractiveness. The experimental results show that the proposed methods can significantly outperform the existing approaches.