Object Appearance Variations Research Articles

Online Multiple Object Tracking Based on Open-Set Few-Shot Learning

How to make an online tracking model effectively adapt to newly appearing objects and object disappearance as well as appearance variations of target objects from few examples is an essential issue in multiple object tracking (MOT). Learning target appearances from few examples is a few-shot classification problem, while identifications of newly appearing objects and object disappearance has the aspect of open-set classification. In this work, we regard online MOT as open-set few-show classification to address both learning from few examples (few-shot classification) and unknown classes such as new objects (open-set classification). Specifically, we develop an embedding neural network, called VOFNet, consisting of convolutional and recurrent parts, to perform open-set few-shot classification. The convolutional part constructs a feature from an example of a target object and the recurrent part determines a representative feature of a target object from few examples. Then VOFNet is trained to provide effective features for open-set few-shot classification. Finally, we develop an online multiple object tracker based on the combination of VOFNet and the bipartite matching. The proposed tracker achieves 49.2 multiple object tracking accuracy (MOTA) with 28.9 frames per second on MOT17 dataset, which shows a significantly better trade-off between the accuracy and the speed than the existing algorithms. For example, the proposed algorithm yields about 3.17 times faster speed with 0.99 times lower accuracy than recent existing MOT algorithm [1].

Object-adaptive LSTM network for real-time visual tracking with adversarial data augmentation

In recent years, deep learning based visual tracking methods have obtained great success owing to the powerful feature representation ability of Convolutional Neural Networks (CNNs). Among these methods, classification-based tracking methods exhibit excellent performance while their speeds are heavily limited by the expensive computation for massive proposal feature extraction. In contrast, matching-based tracking methods (such as Siamese networks) possess remarkable speed superiority. However, the absence of online updating renders these methods unadaptable to significant object appearance variations. In this paper, we propose a novel real-time visual tracking method, which adopts an object-adaptive LSTM network to effectively capture the video sequential dependencies and adaptively learn the object appearance variations. For high computational efficiency, we also present a fast proposal selection strategy, which utilizes the matching-based tracking method to pre-estimate dense proposals and selects high-quality ones to feed to the LSTM network for classification. This strategy efficiently filters out some irrelevant proposals and avoids the redundant computation for feature extraction, which enables our method to operate faster than conventional classification-based tracking methods. In addition, to handle the problems of sample inadequacy and class imbalance during online tracking, we adopt a data augmentation technique based on the Generative Adversarial Network (GAN) to facilitate the training of the LSTM network. Extensive experiments on four visual tracking benchmarks demonstrate the state-of-the-art performance of our method in terms of both tracking accuracy and speed, which exhibits great potentials of recurrent structures for visual tracking.

Deep Regionlets: Blended Representation and Deep Learning for Generic Object Detection.

In this article, we propose a novel object detection algorithm named "Deep Regionlets" by integrating deep neural networks and a conventional detection schema for accurate generic object detection. Motivated by the effectiveness of regionlets for modeling object deformations and multiple aspect ratios, we incorporate regionlets into an end-to-end trainable deep learning framework. The deep regionlets framework consists of a region selection network and a deep regionlet learning module. Specifically, given a detection bounding box proposal, the region selection network provides guidance on where to select sub-regions from which features can be learned from. An object proposal typically contains three - 16 sub-regions. The regionlet learning module focuses on local feature selection and transformations to alleviate the effects of appearance variations. To this end, we first realize non-rectangular region selection within the detection framework to accommodate variations in object appearance. Moreover, we design a "gating network" within the regionlet leaning module to enable instance dependent soft feature selection and pooling. The Deep Regionlets framework is trained end-to-end without additional efforts. We present ablation studies and extensive experiments on the PASCAL VOC dataset and the Microsoft COCO dataset. The proposed method yields competitive performance over state-of-the-art algorithms, such as RetinaNet and Mask R-CNN, even without additional segmentation labels.

REMAP: Multi-layer entropy-guided pooling of dense CNN features for image retrieval.

This paper addresses the problem of very large-scale image retrieval, focusing on improving its accuracy and robustness. We target enhanced robustness of search to factors such as variations in illumination, object appearance and scale, partial occlusions, and cluttered backgrounds -particularly important when search is performed across very large datasets with significant variability. We propose a novel CNN-based global descriptor, called REMAP, which learns and aggregates a hierarchy of deep features from multiple CNN layers, and is trained end-to-end with a triplet loss. REMAP explicitly learns discriminative features which are mutually-supportive and complementary at various semantic levels of visual abstraction. These dense local features are max-pooled spatially at each layer, within multi-scale overlapping regions, before aggregation into a single image-level descriptor. To identify the semantically useful regions and layers for retrieval, we propose to measure the information gain of each region and layer using KL-divergence. Our system effectively learns during training how useful various regions and layers are and weights them accordingly. We show that such relative entropy-guided aggregation outperforms classical CNN-based aggregation controlled by SGD. The entire framework is trained in an end-to-end fashion, outperforming the latest state-of-the-art results. On image retrieval datasets Holidays, Oxford and MPEG, the REMAP descriptor achieves mAP of 95.5%, 91.5% and 80.1% respectively, outperforming any results published to date. REMAP also formed the core of the winning submission to the Google Landmark Retrieval Challenge on Kaggle.

Recognizing Human Activities From Video Using Weakly Supervised Contextual Features

Human activity recognition from the video is an important problem due to its potential applications in remote surveillance, content-based video retrieval, and in humanoid robots. Most of the visual activity recognition research in the last decade focus on recognizing basic human actions from well-constrained laboratory videos and depend upon fully annotated video dataset for model training. Whereas activity recognition from unconstrained videos is still very challenging due to large variations in object appearance and pose, occlusion, and inter and intraclass variations. It is an extremely laborious task to prepare large scale realistic video activity dataset with detailed annotations of the human, object, and their mutual interactions in each frame. Although it is intuitive to model contextual relationships form fully annotated dataset, however, it is unknown as to how reliably multilevel contextual features can be extracted in the absence of annotated dataset. To mitigate these challenges, we propose a weakly supervised approach for complex human activity recognition from realistic videos. The proposed approach requires only activity labels for each video to train the model. A novel multilevel contextual features and context estimation procedure from the un-annotated dataset is also introduced. Restricted Boltzman machine is used to systematically integrate multilevel contextual features. We evaluate the proposed approach on benchmark realistic surveillance video datasets for human-human and human-object interaction activity recognition. The experimental results show improved accuracies on benchmark datasets without using fully annotated datasets.

Weighted Multiple Instance-Based Deep Correlation Filter for Video Tracking Processing

With the development of internet technology, the video data has been widely used in multimedia devices, such as video surveillance, webcast, and so on. Lots of visual processing algorithms are developed to handle the corresponding visual task, but the challenging problems still exist. In this paper, we propose a weighted multiple instances based deep correlation filter for visual tracking processing, which utilizes the importance of instances for training of deep learning model and correlation filter. First, the initial object appearance is modeled based on the confidence of the object and background at the first frame. During the tracking, the superpixel is used to capture the object appearance variations. Most importantly, our tracker can enhance the discriminative ability of the object using deep residual network and improve the tracking efficiency with correlation filter. Second, we introduce the sample importance into residual deep learning model to improve the training performance. We define the importance of each instance by computing the sore of all the pixels within the corresponding instance. Third, we update the parameters of deep learning network and correlation filter in a fixed interval frames to reduce the object drift. Extensive experiments on the OTB2015 benchmark and VOT2018 dataset demonstrate that the proposed object tracking algorithm outperforms the state-of-the-art tracking algorithms.

Egocentric Video Summarization Based on People Interaction Using Deep Learning

The availability of wearable cameras in the consumer market has motivated the users to record their daily life activities and post them on the social media. This exponential growth of egocentric videos demand to develop automated techniques to effectively summarizes the first-person video data. Egocentric videos are commonly used to record lifelogs these days due to the availability of low cost wearable cameras. However, egocentric videos are challenging to process due to the fact that placement of camera results in a video which presents great deal of variation in object appearance, illumination conditions, and movement. This paper presents an egocentric video summarization framework based on detecting important people in the video. The proposed method generates a compact summary of egocentric videos that contains information of the people whom the camera wearer interacts with. Our proposed approach focuses on identifying the interaction of camera wearer with important people. We have used AlexNet convolutional neural network to filter the key-frames (frames where camera wearer interacts closely with the people). We used five convolutional layers and two completely connected hidden layers and an output layer. Dropout regularization method is used to reduce the overfitting problem in completely connected layers. Performance of the proposed method is evaluated onUT Egostandard dataset. Experimental results signify the effectiveness of the proposed method in terms of summarizing the egocentric videos.

Robust Visual Tracking via Smooth Manifold Kernel Sparse Learning

Various sparse-representation-based tracking meth-ods have been proposed to tackle visual tracking problems, and most of them use simple intensity feature as the observation of target object. Moreover, most of them only take into account either global or local image representation and only exploit the underlying relationship among target candidates in a single frame. All of these may make their appearance models less robust to deal with complex scenes. To overcome these problems, we propose a smooth manifold kernel sparse tracker under the framework of particle filter. The proposed method characterizes targets and candidates with region covariance matrix descriptors, and constructs object tracking as a kernel sparse learning model based on symmetric positive-definite (SPD) manifolds. The spatial-temporal interdependencies among candidates and global-local representations of candidates are jointly considered and unified via the kernel sparse learning model. Moreover, in order to make the model more robust, the detection of outlier tasks is also taken into account. To handle the variation of object appearance, we develop a robust and efficient online dictionary learning algorithm on SPD manifolds. Extensive experiments on multiple benchmark datasets demonstrate that our tracker performs favorably against state-of-the-art trackers.

Trajectories as Topics: Multi-Object Tracking by Topic Discovery.

This paper proposes a new approach to multi-object tracking by semantic topic discovery. We dynamically cluster frame-by-frame detections and treat objects as topics, allowing the application of the Dirichlet process mixture model. The tracking problem is cast as a topic-discovery task, where the video sequence is treated analogously to a document. It addresses tracking issues such as object exclusivity constraints as well as tracking management without the need for heuristic thresholds. Variation of object appearance is modeled as the dynamics of word co-occurrence and handled by updating the cluster parameters across the sequence in the dynamical clustering procedure. We develop two kinds of visual representation based on super-pixel and deformable part model and integrate them into the model of automatic topic discovery for tracking rigid and non-rigid objects, respectively. In experiments on public data sets, we demonstrate the effectiveness of the proposed algorithm.

Visual tracking using spatio-temporally nonlocally regularized correlation filter

Due to the factors like rapidly fast motion, cluttered backgrounds, arbitrary object appearance variation and shape deformation, an effective target representation plays a key role in robust visual tracking. Existing methods often employ bounding boxes for target representations, which are easily polluted by noisy clutter backgrounds that may cause drifting problem when the target undergoes large-scale non-rigid or articulated motions. To address this issue, in this paper, motivated by the spatio-temporal nonlocality of target appearance reoccurrence in a video, we explore the nonlocal information to accurately represent and segment the target, yielding an object likelihood map to regularize a correlation filter (CF) for visual tracking. Specifically, given a set of tracked target bounding boxes, we first generate a set of superpixels to represent the foreground and background, and then update the appearance of each superpixel with its long-term spatio-temporally nonlocal counterparts. Then, with the updated appearances, we formulate a spatio-temporally graphical model comprised of the superpixel label consistency potentials. Afterwards, we generate segmentation by optimizing the graphical model via iteratively updating the appearance model and estimating the labels. Finally, with the segmentation mask, we obtain an object likelihood map that is employed to adaptively regularize the CF learning by suppressing the clutter background noises while making full use of the long-term stable target appearance information. Extensive evaluations on the OTB50, SegTrack, Youtube-Objects datasets demonstrate the effectiveness of the proposed method, which performs favorably against some state-of-art methods.

A deep features based generative model for visual tracking

Abstract In this work, we propose a novel visual tracking algorithm based on a framework of generative model. In order to make the algorithm robust to various challenging appearance changes, we adopt the powerful deep features in the description of tracking object appearance. The features are extracted from a Convolutional Neural Network (CNN), which is a modified one based on the VGG-M nets but constructed with fewer convolution layers and sequences exclusively full connection layers. In the pretraining process, we add a special convolution layer called coefficients layer before the full connection layers. In the tracking process after the network being pretrained, we remove the coefficients layer and just update the full connection layers conditionally. To decide the new target’s positions, we compute the compositive similarity scores containing three kinds of similarities with different weights. The first kind is similarities between candidates and the target in the first frame, and the second kind is between candidates and tracking results in the last frame. The third kind is related to the important object appearance variations in the tracking process. We design a simple mechanism to produce a collection to record those historical templates when the object appearance changed largely. With similarities between candidates and the historical templates, the drift problem can be alleviated to some extent, because similar historical appearances sometimes appear repeatedly and the recorded historical templates can provide important information. We use the outputs of the convolution part before the full connection layers as features and weight them with the coefficients layer’s filter weights to compute all similarities. Finally, candidates with the highest scores will be regarded as new targets in the current frame. The evaluated results on CVPR2013 Online Object Tracking Benchmark show that our algorithm can achieve outstanding performance compared with state-of-the-art trackers.

Auto-contouring via Automatic Anatomy Recognition of Organs at Risk in Head and Neck Cancer on CT images.

Contouring of the organs at risk is a vital part of routine radiation therapy planning. For the head and neck (H&N) region, this is more challenging due to the complexity of anatomy, the presence of streak artifacts, and the variations of object appearance. In this paper, we describe the latest advances in our Automatic Anatomy Recognition (AAR) approach, which aims to automatically contour multiple objects in the head and neck region on planning CT images. Our method has three major steps: model building, object recognition, and object delineation. First, the better-quality images from our cohort of H&N CT studies are used to build fuzzy models and find the optimal hierarchy for arranging objects based on the relationship between objects. Then, the object recognition step exploits the rich prior anatomic information encoded in the hierarchy to derive the location and pose for each object, which leads to generalizable and robust methods and mitigation of object localization challenges. Finally, the delineation algorithms employ local features to contour the boundary based on object recognition results. We make several improvements within the AAR framework, including finding recognition-error-driven optimal hierarchy, modeling boundary relationships, combining texture and intensity, and evaluating object quality. Experiments were conducted on the largest ensemble of clinical data sets reported to date, including 216 planning CT studies and over 2,600 object samples. The preliminary results show that on data sets with minimal (<4 slices) streak artifacts and other deviations, overall recognition accuracy reaches 2 voxels, with overall delineation Dice coefficient close to 0.8 and Hausdorff Distance within 1 voxel.

Robust kernelised correlation filter tracking with least absolute deviations regression

Kernelised correlation filter (KCF) has demonstrated its superior performance in visual tracking. The strength of the approach stems from its ability to efficiently learn the object appearance variations over time. A fundamental drawback to KCF, however, is that the objective function is modelled as a ridge regression problem that is sensitive to outliers, thereby resulting in suboptimal performance when the target appearance during tracking suffers from severe variations due to illumination variation, partial occlusion, background clutter, and so on. To address this issue, the KCF as a least absolute deviations regression (LADR) that is robust to outliers is reformulated, and then an efficient and effective approach to solve it via the alternating direction method of multipliers (ADMM) is presented. The ADMM splits the LADR into a ridge regression problem as the conventional KCF and a soft-thresholding one, and hence the proposed approach is still extremely efficient as the conventional KCF. Evaluations are conducted on the CVPR2013 tracking benchmark dataset with 50 sequences, which demonstrate the superior performance of the proposed method over the state-of-the-art trackers.

A Hybrid Background Subtraction and Artificial Neural Networks for Movement Recognition in Memorizing Quran

Movement change beyond the duration of time and the variations of object appearance becomes an interesting topic for research in computer vision. Object behavior can be recognized through movement change on video. During the recognition of object behavior, the target and the trace of an object in a video must be determined in the sequence of frames. To date, the existence of object on a video has been widely used in different areas such as supervision, robotics, agriculture, health, sports, education, and traffic. This research focuses on the field of education by recognizing the movement of Quantum Maki Quran memorization through a video. The purpose of this study is to enhance the existing computer vision technique in detecting the Quantum Maki Quran memorization movement on a video. It combines the Background Subtraction method and Artificial Neural Networks; and evaluates the combination to optimize the system accuracy. Background Subtraction is used as object detection method and Back propagation in Artificial Neural Networks is used as object classification. Nine videos are obtained by three different volunteers. These nine videos are divided into six training and three testing data. The experimental result shows that the percentage of accuracy system is 91.67%. It can be concluded that there are several factors influencing the accuracy, such as video capturing factors, video improvements, the models, feature extraction and parameter definitions during the Artificial Neural Networks training.

Gated Convolutional Neural Network for Semantic Segmentation in High-Resolution Images

Semantic segmentation is a fundamental task in remote sensing image processing. The large appearance variations of ground objects make this task quite challenging. Recently, deep convolutional neural networks (DCNNs) have shown outstanding performance in this task. A common strategy of these methods (e.g., SegNet) for performance improvement is to combine the feature maps learned at different DCNN layers. However, such a combination is usually implemented via feature map summation or concatenation, indicating that the features are considered indiscriminately. In fact, features at different positions contribute differently to the final performance. It is advantageous to automatically select adaptive features when merging different-layer feature maps. To achieve this goal, we propose a gated convolutional neural network to fulfill this task. Specifically, we explore the relationship between the information entropy of the feature maps and the label-error map, and then a gate mechanism is embedded to integrate the feature maps more effectively. The gate is implemented by the entropy maps, which are generated to assign adaptive weights to different feature maps as their relative importance. Generally, the entropy maps, i.e., the gates, guide the network to focus on the highly-uncertain pixels, where detailed information from lower layers is required to improve the separability of these pixels. The selected features are finally combined to feed into the classifier layer, which predicts the semantic label of each pixel. The proposed method achieves competitive segmentation accuracy on the public ISPRS 2D Semantic Labeling benchmark, which is challenging for segmentation by only using the RGB images.

Non-parametric context-based object classification in images

Segmentation and classification of objects in images is one of the most important and yet one of the most complex problems in computer vision. Many different things contribute to the complexity of this problem, but amongst those that are most significant are variations in object appearance due to various perspectives, changes in scene illumination and typically a large number of partially occluded objects in a scene. Humans have the ability to successfully deal with these complications and to recognize objects in scenes without seemingly much effort. However, to aid their recognition process, humans use different sources of information such as previous experiences, context of objects placed in a scene and rules learned in the past that suggest how the physical world should appear and behave. Computer vision is trying to replicate the same strategies used by human vision and apply them to computer systems with the aim of improving typical tasks of detection, localization and classification of image objects. In this work we propose a new model for natural image object classification using contextual information at the level of image segments. Context modeling is largely independent of appearance-based classification and proposed model enables simple upgrade of existing systems with information from global and/or local context. Context modeling is based on non-parametric use of appearance-based classification results which is a novel approach compared to previous systems that model context on a limited number of rules expressed with a fixed set of parameters. Model implementation resulted in a system that, in our simulations, showed stable improvement of the appearance-based object classification. DOI: http://dx.doi.org/10.5755/j01.itc.46.1.13610

Occlusion-Aware Real-Time Object Tracking

The online learning methods are popular for visual tracking because of their robust performance for most video sequences. However, the drifting problem caused by noisy updates is still a challenge for most highly adaptive online classifiers. In visual tracking, target object appearance variation, such as deformation and long-term occlusion, easily causes noisy updates. To overcome this problem, a new real-time occlusion-aware visual tracking algorithm is introduced. First, we learn a novel two-stage classifier with circulant structure with kernel, named integrated circulant structure kernels (ICSK). The first stage is applied for transition estimation and the second is used for scale estimation. The circulant structure makes our algorithm realize fast learning and detection. Then, the ICSK is used to detect the target without occlusion and build a classifier pool to save these classifiers with noisy updates. When the target is in heavy occlusion or after long-term occlusion, we redetect it using an optimal classifier selected from the classifier-pool according to an entropy minimization criterion. Extensive experimental results on the full benchmark demonstrate our real-time algorithm achieves better performance than state-of-the-art methods.

Robust Visual Tracking Using Exemplar-Based Detectors

Tracking by detection has become an attractive tracking technique, which treats tracking as an object detection problem and trains a detector to separate the target object from the background in each frame. While this strategy is effective to some extent, we argue that the task in tracking should be searching for a specific object instance instead of an object category. Based on this viewpoint, a novel framework based on object exemplar detectors is proposed for visual tracking. To build a specific and discriminative model to separate the object instance from the background, the proposed method trains an exemplar-based linear discriminant analysis (ELDA) classifier for the object exemplar, using the current tracked instance as the positive sample and massive negative samples obtained both offline and online. To improve the trackers' adaptivity, we use an ensemble of the above ELDA detectors and update them during the tracking to cover the variation in object appearance. Extensive experimental results on a large benchmark data set show that the proposed method outperforms many state-of-the-art trackers, demonstrating the effectiveness and robustness of the ELDA tracker.

Scale adaptive compressive tracking.

Recently, the compressive tracking (CT) method (Zhang et al. in Proceedings of European conference on computer vision, pp 864–877, 2012) has attracted much attention due to its high efficiency, but it cannot well deal with the scale changing objects due to its constant tracking box. To address this issue, in this paper we propose a scale adaptive CT approach, which adaptively adjusts the scale of tracking box with the size variation of the objects. Our method significantly improves CT in three aspects: Firstly, the scale of tracking box is adaptively adjusted according to the size of the objects. Secondly, in the CT method, all the compressive features are supposed independent and equal contribution to the classifier. Actually, different compressive features have different confidence coefficients. In our proposed method, the confidence coefficients of features are computed and used to achieve different contribution to the classifier. Finally, in the CT method, the learning parameter λ is constant, which will result in large tracking drift on the occasion of object occlusion or large scale appearance variation. In our proposed method, a variable learning parameter λ is adopted, which can be adjusted according to the object appearance variation rate. Extensive experiments on the CVPR2013 tracking benchmark demonstrate the superior performance of the proposed method compared to state-of-the-art tracking algorithms.

Multi-period visual tracking via online DeepBoost learning

In this paper, we propose a novel accurate and robust boosting-style tracking-by-detection method. The proposed algorithm adopts a flexible and capacity-conscious object appearance model, which combines the strengths of both local and global visual representations. We firstly propose a joint local–global visual representation, in which main local and global spatial structure information of the target is flexibly embedded in the candidate classifier set with members from multiple complexity families. In addition, to avoid over-fitting our tracker adopts an effective online DeepBoost learning method (ODB). The key capacity-conscious ability of ODB helps to avoid over-fitting and generate a more adaptive and robust tracker. Furthermore, we propose a multi-period tracking framework (MPTF) to enhance the tracker׳s recovery ability for tracking failures. The proposed Multi-period DeepBoost-Tracker (MPDBT) can well encode the object spatial structures and excellently handle object appearance variations, and it can also recover from tracking failures with the help of the proposed MPTF. The experimental results demonstrate that our tracker outperforms the state-of-the-art trackers.