Global Appearance Model Research Articles

Semi-supervised video object segmentation via an edge attention gated graph convolutional network

Video object segmentation (VOS) exhibits heavy occlusions, large deformation, and severe motion blur. While many remarkable convolutional neural networks are devoted to the VOS task, they often mis-identify background noise as the target or output coarse object boundaries, due to the failure of mining detail information and high-order correlations of pixels within the whole video. In this work, we propose an edge attention gated graph convolutional network (GCN) for VOS. The seed point initialization and graph construction stages construct a spatio-temporal graph of the video by exploring the spatial intra-frame correlation and the temporal inter-frame correlation of superpixels. The node classification stage identifies foreground superpixels by using an edge attention gated GCN which mines higher-order correlations between superpixels and propagates features among different nodes. The segmentation optimization stage optimizes the classification of foreground superpixels and reduces segmentation errors by using a global appearance model which captures the long-term stable feature of objects. In summary, the key contribution of our framework is twofold: (a) the spatio-temporal graph representation can propagate the seed points of the first frame to subsequent frames and facilitate our framework for the semi-supervised VOS task; (b) the edge attention gated GCN can learn the importance of each node with respect to both the neighboring nodes and the whole task with small number of layers. Experiments on Davis 2016 and Davis 2017 datasets show that our framework achieves the excellent performance with only small training samples (45 video sequences).

Effective Multi-Object Tracking via Global Object Models and Object Constraint Learning.

Effective multi-object tracking is still challenging due to the trade-off between tracking accuracy and speed. Because the recent multi-object tracking (MOT) methods leverage object appearance and motion models so as to associate detections between consecutive frames, the key for effective multi-object tracking is to reduce the computational complexity of learning both models. To this end, this work proposes global appearance and motion models to discriminate multiple objects instead of learning local object-specific models. In concrete detail, it learns a global appearance model using contrastive learning between object appearances. In addition, we learn a global relation motion model using relative motion learning between objects. Moreover, this paper proposes object constraint learning for improving tracking efficiency. This study considers the discriminability of the models as a constraint, and learns both models when inconsistency with the constraint occurs. Therefore, object constraint learning differs from the conventional online learning for multi-object tracking which updates learnable parameters per frame. This work incorporates global models and object constraint learning into the confidence-based association method, and compare our tracker with the state-of-the-art methods on public available MOT Challenge datasets. As a result, we achieve 64.5% MOTA (multi-object tracking accuracy) and 6.54 Hz tracking speed on the MOT16 test dataset. The comparison results show that our methods can contribute to improve tracking accuracy and tracking speed together.

Investigating the impact of supervoxel segmentation for unsupervised abnormal brain asymmetry detection

Several brain disorders are associated with abnormal brain asymmetries (asymmetric anomalies). Several computer-based methods aim to detect such anomalies automatically. Recent advances in this area use automatic unsupervised techniques that extract pairs of symmetric supervoxels in the hemispheres, model normal brain asymmetries for each pair from healthy subjects, and treat outliers as anomalies. Yet, there is no deep understanding of the impact of the supervoxel segmentation quality for abnormal asymmetry detection, especially for small anomalies, nor of the added value of using a specialized model for each supervoxel pair instead of a single global appearance model. We aim to answer these questions by a detailed evaluation of different scenarios for supervoxel segmentation and classification for detecting abnormal brain asymmetries. Experimental results on 3D MR-T1 brain images of stroke patients confirm the importance of high-quality supervoxels fit anomalies and the use of a specific classifier for each supervoxel. Next, we present a refinement of the detection method that reduces the number of false-positive supervoxels, thereby making the detection method easier to use for visual inspection and analysis of the found anomalies.

ROAM: A Rich Object Appearance Model with Application to Rotoscoping.

Rotoscoping, the detailed delineation of scene elements through a video shot, is a painstaking task of tremendous importance in professional post-production pipelines. While pixel-wise segmentation techniques can help for this task, professional rotoscoping tools rely on parametric curves that offer the artists a much better interactive control on the definition, editing and manipulation of the segments of interest. Sticking to this prevalent rotoscoping paradigm, we propose a novel framework to capture and track the visual aspect of an arbitrary object in a scene, given an initial closed outline of this object. This model combines a collection of local foreground/background appearance models spread along the outline, a global appearance model of the enclosed object and a set of distinctive foreground landmarks. The structure of this rich appearance model allows simple initialization, efficient iterative optimization with exact minimization at each step, and on-line adaptation in videos. We further extend this model by so-called trimaps which serve as an input to alpha-matting algorithms to allow truly seamless compositing. To this end, we leverage local classifiers attached to the roto-curves to define a confidence measure that is well-suited to define trimaps with adaptive band-widths. The resulting trimaps are parametric, temporally consistent and remain fully editable by the artist. We demonstrate qualitatively and quantitatively the merit of this framework through comparisons with tools based on either dynamic segmentation with a closed curve or pixel-wise binary labelling.

RETRACTED: Online contour extraction and texture analysis – A IoT based case study

Abstract We put forward a method of drawing effect analysis in drawing simulation education on the basis of the combination of contour extraction and texture elements in order to improve the effectiveness of drawing effect analysis algorithm in painting simulation education. The multi-scale representation of the image is created by iterative bilateral scale space decomposition, and then the local texture features are extracted by image spots and random projection to make random texture features. The occurrence probability based on texture primitives in the image region can be obtained in the texture appearance model of a region based on the global texture appearance model. Finally, we put forward a random region merging algorithm based on texture features of complex regions and it realizes the analysis of painting effect in painting simulation education based on regional texture appearance model.

Coupled-Region Visual Tracking Formulation Based on a Discriminative Correlation Filter Bank

The visual tracking algorithm based on discriminative correlation filter (DCF) has shown excellent performance in recent years, especially as the higher tracking speed meets the real-time requirement of object tracking. However, when the target is partially occluded, the traditional single discriminative correlation filter will not be able to effectively learn information reliability, resulting in tracker drift and even failure. To address this issue, this paper proposes a novel tracking-by-detection framework, which uses multiple discriminative correlation filters called discriminative correlation filter bank (DCFB), corresponding to different target sub-regions and global region patches to combine and optimize the final correlation output in the frequency domain. In tracking, the sub-region patches are zero-padded to the same size as the global target region, which can effectively avoid noise aliasing during correlation operation, thereby improving the robustness of the discriminative correlation filter. Considering that the sub-region target motion model is constrained by the global target region, adding the global region appearance model to our framework will completely preserve the intrinsic structure of the target, thus effectively utilizing the discriminative information of the visible sub-region to mitigate tracker drift when partial occlusion occurs. In addition, an adaptive scale estimation scheme is incorporated into our algorithm to make the tracker more robust against potential challenging attributes. The experimental results from the OTB-2015 and VOT-2015 datasets demonstrate that our method performs favorably compared with several state-of-the-art trackers.

Real-Time Object Tracking via Fusion of Global and Local Appearance Models

Real-Time Object Tracking via Fusion of Global and Local Appearance Models

Visual tracking via shallow and deep collaborative model

In this paper, we propose a robust tracking method based on the collaboration of a generative model and a discriminative classifier, where features are learned by shallow and deep architectures, respectively. For the generative model, we introduce a block-based incremental learning scheme, in which a local binary mask is constructed to deal with occlusion. The similarity degrees between the local patches and their corresponding subspace are integrated to formulate a more accurate global appearance model. In the discriminative model, we exploit the advances of deep learning architectures to learn generic features which are robust to both background clutters and foreground appearance variations. To this end, we first construct a discriminative training set from auxiliary video sequences. A deep classification neural network is then trained offline on this training set. Through online fine-tuning, both the hierarchical feature extractor and the classifier can be adapted to the appearance change of the target for effective online tracking. The collaboration of these two models achieves a good balance in handling occlusion and target appearance change, which are two contradictory challenging factors in visual tracking. Both quantitative and qualitative evaluations against several state-of-the-art algorithms on challenging image sequences demonstrate the accuracy and the robustness of the proposed tracker.

A joint appearance model of SRC and MFH for multi-objects tracking

This paper extends sparse representation based classification (SRC) and multi-feature hashing (MFH) into multi-object tracking task, and proposes a joint appearance model of SRC and MFH, which aims at discriminating different objects effectively. Unlike most previous approaches which only focus on producing appearance models for all targets, we further consider discriminative features for distinguishing difficult pairs of targets. Firstly, an SRC based global discriminative appearance model is designed for discriminating all targets. It formulates tracklets association as an SRC problem. A discriminative dictionary learning approach is introduced, which improves the SRC classification performance. By this way, the global discriminative appearance model can distinguish different targets more effectively. Secondly, an MFH based pairwise appearance model is designed. This pairwise appearance model focuses on distinguishable features from two targets without considering other targets or backgrounds, therefore it is more effective for differentiating specific close-by tracklets pairs. Data association framework is employed to generate final tracks. Considerable performance improvements are shown on challenging data sets, particularly in metrics of identity switches.

Discovering Primary Objects in Videos by Saliency Fusion and Iterative Appearance Estimation

In this paper, we propose a new method for detecting primary objects in unconstrained videos in a completely automatic setting. Here, we define the primary object in a video as the object that presents saliently in most of the frames. Unlike previous works considering only local saliency detection or common pattern discovery, the proposed method integrates the local visual/motion saliency extracted from each frame, global appearance consistency throughout the video, and spatiotemporal smoothness constraint on object trajectories. We first identify a temporal coherent salient region throughout the whole video, and then explicitly learn a global appearance model to distinguish the primary object against the background. In order to obtain high-quality saliency estimations from both appearance and motion cues, we propose a novel self-adaptive saliency map fusion method by learning the reliability of saliency maps from labeled data. As a whole, our method can robustly localize and track primary objects in diverse video content, and handle the challenges such as fast object and camera motion, large scale and appearance variation, background clutter, and pose deformation. Moreover, compared with some existing approaches that assume the object is present in all the frames, our approach can naturally handle the case where the object is present only in part of the frames, e.g., the object enters the scene in the middle of the video or leaves the scene before the video ends. We also propose a new video data set containing 51 videos for primary object detection with per-frame ground-truth labeling. Quantitative experiments on several challenging video data sets demonstrate the superiority of our method compared with the recent state of the arts.

Combined discriminative global and generative local models for visual tracking

It is a challenging task to develop an effective visual tracking algorithm due to factors such as pose variation, rotation, and so on. Combined discriminative global and generative local appearance models are proposed to address this problem. Specifically, we develop a compact global object representation by extracting the low-frequency coefficients of the color and texture of the object based on two-dimensional discrete cosine transform. Then, with the global appearance representation, we learn a discriminative metric classifier in an online fashion to differentiate the target object from its background, which is very important to robustly indicate the changes in appearance. Second, we develop a new generative local model that exploits the scale invariant feature transform and its spatial geometric information. To make use of the advantages of the global discriminative model and the generative local model, we incorporate them into Bayesian inference framework. In this framework, the complementary models help the tracker locate the target more accurately. Furthermore, we use different mechanisms to update global and local templates to capture appearance changes. The experimental results demonstrate that the proposed approach performs favorably against state-of-the-art methods in terms of accuracy.

Multi-camera handoff for person re-identification

During recent years, the demand for automated video surveillance has become the subject of much attention. Currently, video surveillance is an essential part of security and monitoring in banks, streets, buildings, department stores, stadiums, highways, railway stations, and crowded gatherings. One important reason for using video surveillance is to counter terrorism. In this context, a multi-camera handoff system for person re-identification using a color-based global appearance model is presented. We address two main factors that affect re-identification performance in this paper: (1) robust features (viewpoint orientation and indoor/outdoor cinematography) and (2) re-identification or classification. A hexagonal-superpixel method is introduced to minimize the false recognition rate due to the above-mentioned challenges. First we detect the human or object in a video and use a mixture of Gaussian models extended with a trajectory-learning algorithm to solve the trajectory dilemma. Second, a novel feature descriptor is proposed in which sensitive features are extracted using the proposed Color Hexagonal-SIFT and Color Histogram Features (CHF) methods. Third, we generate a combined appearance feature descriptor for re-identifying the human in different scenes. Finally, we introduce a Time-Tree-based learning method to minimize the gallery set׳s overhead and increase the accuracy of the re-identification results. The proposed framework is tested on videos taken using three cameras placed in three different locations. We also tested this framework on publically available data from the i-LIDS and VIPeR datasets. The results show outstanding speed and re-identification accuracy in dealing with the above-mentioned issues.

2D-3D Pose Estimation of Heterogeneous Objects Using a Region Based Approach

Recently, region based methods for estimating the 3D pose of an object from a 2D image have gained increasing popularity. They do not require prior knowledge of the object's texture, making them particularity attractive when the object's texture is unknown a priori. Region based methods estimate the 3D pose of an object by finding the pose which maximizes the image segmentation in to foreground and background regions. Typically the foreground and background regions are described using global appearance models, and an energy function measuring their fit quality is optimized with respect to the pose parameters. Applying a region based approach on standard 2D-3D pose estimation databases shows its performance is strongly dependent on the scene complexity. In simple scenes, where the statistical properties of the foreground and background do not spatially vary, it performs well. However, in more complex scenes, where the statistical properties of the foreground or background vary, the performance strongly degrades. The global appearance models used to segment the image do not sufficiently capture the spatial variation. Inspired by ideas from local active contours, we propose a framework for simultaneous image segmentation and pose estimation using multiple local appearance models. The local appearance models are capable of capturing spatial variation in statistical properties, where global appearance models are limited. We derive an energy function, measuring the image segmentation, using multiple local regions and optimize it with respect to the pose parameters. Our experiments show a substantially higher probability of estimating the correct pose for heterogeneous objects, whereas for homogeneous objects there is minor improvement.

Image segmentation via multi-scale stochastic regional texture appearance models

An ongoing challenge in the area of image segmentation is in dealing with scenes exhibiting complex textural characteristics. While many approaches have been proposed to tackle this particular challenge, a related topic of interest that has not been fully explored for dealing with this challenge is stochastic texture models, particularly for characterizing textural characteristics within regions of varying sizes and shapes. Therefore, this paper presents a novel method for image segmentation based on the concept of multi-scale stochastic regional texture appearance models. In the proposed method, a multi-scale representation of the image is constructed using an iterative bilateral scale space decomposition. Local texture features are then extracted via image patches and random projections to generate stochastic texture features. A texton dictionary is built from the stochastic features, and used to represent the global texture appearance model. Based on this global texture appearance model, a regional texture appearance model can then be obtained based on the texton occurrence probability given a region within an image. Finally, a stochastic region merging algorithm that allows the computation of complex features is presented to perform image segmentation based on proposed regional texture appearance model. Experimental results using the BDSD300 segmentation dataset showed that the proposed method achieves a Probabilistic Rand Index (PRI) of 0.83 and an F-measure of 0.77@(0.92, 0.68), and provides improved handling of color and luminance variation, as well as strong segmentation performance for images with highly textured regions when compared to a number of previous methods. These results suggest that the proposed stochastic regional texture appearance model is better suited for handling the texture variations of natural scenes, leading to more accurate segmentations, particularly in situations characterized by complex textural characteristics.

Ortho-diffusion decompositions of graph-based representation of images

This paper proposes a new feature representation methodology for graph-based data. Initially, random walks on matrices of pairwise data similarities are considered. A diffusion process is embedded into orthonormal decompositions of such matrices at various scales while enabling a data reduction mechanism as well. At each scale, the QR orthonormal decomposition algorithm, alternating with diffusions and data reduction stages is applied recursively on the given graph-based data representations. The proposed methodology is used in extracting complex feature representations from images, which are then used for image matching and in face recognition. In the face recognition application, both global appearance models and semantic representations of biometric features are considered. Both the correlation and the covariance of images of human faces are considered for the training stage when using global appearance models. The proposed data representation is shown to be robust in face recognition applications, when face images are represented in low resolution and when they are corrupted by noise.

Non-linear factorised dynamic shape and appearance models for facial expression analysis and tracking

Facial expressions exhibit non-linear shape and appearance deformations with variations in different people and expressions. The authors present a non-linear factorised shape and appearance model for facial expression analysis and tracking. The novel non-linear factorised generative model of facial expressions, using conceptual manifold embedding and empirical kernel maps, provides accurate facial expression shape and appearance. It preserves non-linear facial deformations based on the configuration, face style and expression type. The proposed model supports tasks, such as facial expression recognition, person identification and global and local facial motion tracking. Given a sequence of images, temporal embedding, expression type and person identification parameters are iteratively estimated for facial expression analysis. The authors combine global facial motion estimation and local facial deformation estimation for large global and subtle local facial motion tracking. The authors employ local facial motion deformation estimation using a thin-plate spline for subtle facial motion tracking. The global shape and appearance model provides appearance templates for the estimation of local deformation. Experimental results using Cohen-Kanade AU-coded facial expressions demonstrate facial expression recognition using estimated personal style parameter, and facial deformation tracking using global and local facial motion estimation.

Graph-based transductive learning for robust visual tracking

In object tracking problem, most methods assume brightness constancy or subspace constancy, which are violated in practice. In this paper, the object tracking problem is considered as a transductive learning problem and a robust tracking method is proposed under intrinsic and extrinsic varieties. The object not only fits the object model, but also has the same cluster with the previous objects, which are the labeled data. By constraining the global and local information, the cost function is constructed firstly. The solution for minimizing the cost function can be solved by a simple linear algebra with graph Laplacian. Moreover, a novel graph is constructed over the positive samples and candidate patches, which can simultaneously learn the object's global appearance model and the local intrinsic geometric structure of all the patches. Furthermore, a heuristic positive samples selection scheme is adopted to make the method more effective. The proposed method is tested on different videos, which undergo large pose, expression, illumination and partial occlusion, and compared with state-of-the-art algorithms. Experimental results and comparative studies are provided to demonstrate the efficiency of the proposed method.

Bilinear factorisation for facial expression analysis and synthesis

This paper addresses the issue of face representations for facial expression recognition and synthesis. In this context, a global active appearance model is used in conjunction with two bilinear factorisation models to separate expression and identity factors from the global appearance parameters. Although active appearance models and bilinear modelling are not new concepts, the main contribution of this paper consists in combining both techniques to improve facial expression recognition and synthesis (control). Indeed, facial expression recognition is performed through linear discriminant analysis of the global appearance parameters extracted by active appearance model search. Results are compared to the ones obtained for the same training and test images using classification of the expression factors extracted by bilinear factorisation. This experiment highlights the advantages of bilinear factorisation. Finally, it is proposed to exploit bilinear factorisation to synthesise facial expressions through replacement of the extracted expression factors. This yields very interesting synthesis performances in terms of visual quality of the synthetic faces. Indeed, synthetic open mouths reconstruction either with or without appearing teeth is of better quality than with classical linear-regression-based synthesis.