Object Appearance Information Research Articles

DB-Tracker: Multi-Object Tracking for Drone Aerial Video Based on Box-MeMBer and MB-OSNet

Drone aerial videos offer a promising future in modern digital media and remote sensing applications, but effectively tracking several objects in these recordings is difficult. Drone aerial footage typically includes complicated sceneries with moving objects, such as people, vehicles, and animals. Complicated scenarios such as large-scale viewing angle shifts and object crossings may occur simultaneously. Random finite sets are mixed in a detection-based tracking framework, taking the object’s location and appearance into account. It maintains the detection box information of the detected object and constructs the Box-MeMBer object position prediction framework based on the MeMBer random finite set point object tracking. We develop a hierarchical connection structure in the OSNet network, build MB-OSNet to get the object appearance information, and connect feature maps of different levels through the hierarchy such that the network may obtain rich semantic information at different sizes. Similarity measurements are computed and collected for all detections and trajectories in a cost matrix that estimates the likelihood of all possible matches. The cost matrix entries compare the similarity of tracks and detections in terms of position and appearance. The DB-Tracker algorithm performs excellently in multi-target tracking of drone aerial videos, achieving MOTA of 37.4% and 46.2% on the VisDrone and UAVDT data sets, respectively. DB-Tracker achieves high robustness by comprehensively considering the object position and appearance information, especially in handling complex scenes and target occlusion. This makes DB-Tracker a powerful tool in challenging applications such as drone aerial videos.

Class-aware tiny object recognition over large-scale 3D point clouds

Although tremendous strides have been made in object recognition over large-scale 3D point clouds, one of the remaining open challenges is detecting tiny objects. The signal and appearance information of tiny objects are generally sparse and insufficient due to their tiny size, which makes the tiny object difficult to be detected with existing studies both in tiny 2D object recognition and general 3D object recognition since 3D topology relations are lost due to feature projection in tiny 2D object recognition and massive background may dominate the feature learner in general 3D object recognition. To the best of our knowledge, we explore tiny object recognition for the first time over large-scale 3D point clouds by designing deep neural network. To cope with this problem, a novel two-stage approach is proposed in this paper, which can effectively learn an informative and discriminative feature representation and efficiently process large-scale point clouds. Specifically, a class-aware filtering strategy is designed to dispense with the redundant background information. After that, we introduce a novel aggregating and sampling scheme in a supervised manner to progressively increase the receptive field for each 3D point, which can adaptively capture useful signals from tiny objects and learn complex geometric structures. Last, we propose a biased loss function that is inversely correlated to the number of tiny object points, thereby effectively handling the sparsity of tiny objects. Extensive experimental results over two real-world data sets validate the effectiveness and efficiency of the proposed approach.

Bidirectional Posture-Appearance Interaction Network for Driver Behavior Recognition

Driver behavior recognition has become one of the most important tasks for intelligent vehicles. This task, however, is very challenging since the background contents in real-world driving scenarios are often very complex. More critically, the difference between driving behaviors is often very minor, making it extremely difficult to distinguish them. Existing methods often rely only on RGB frames (or skeleton data), which may fail to capture the minor differences between behaviors and appearance information of objects simultaneously and thus fail to achieve promising performance. To address the above issues, in this paper, we propose a bidirectional posture-appearance interaction network (BPAI-Net), which simultaneously considers RGB frames and skeleton ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e.</i> , posture) data for driver behavior recognition. Specifically, we propose a posture-guided convolutional neural network (PG-CNN) and an appearance-guided graph convolutional network (AG-GCN) to extract appearance and posture features, respectively. To exploit the complementary information between appearance and posture, we use the appearance features from PG-CNN for guiding AG-GCN to exploit the contextual information ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">e.g.</i> , nearby objects) to enhance posture features. Then, we use the enhanced posture features from AG-GCN to help PG-CNN focus on critical local areas of video frames that are related to driver behaviors. In this sense, we are able to use the interaction between two modalities to extract more discriminative features and thus improve the recognition accuracy. Experimental results on Drive&Act dataset show that our method outperforms state-of-the-art methods by a large margin (67.83% vs. 63.64%). Furthermore, we collect a bus driver behavior recognition dataset and yield consistent performance gain against baseline methods, demonstrating the effectiveness of our method in real-world applications. The source code and trained models are available at github.com/SCUT-AILab/BPAI-Net/.

Detecting and Tracking Small and Dense Moving Objects in Satellite Videos: A Benchmark

Satellite video cameras can provide continuous observation for a large-scale area, which is important for many remote sensing applications. However, achieving moving object detection and tracking in satellite videos remains challenging due to the insufficient appearance information of objects and lack of high-quality datasets. In this article, we first build a large-scale satellite video dataset with rich annotations for the task of moving object detection and tracking. This dataset is collected by the Jilin-1 satellite constellation and composed of 47 high-quality videos with 1 646 038 instances of interest for object detection and 3711 trajectories for object tracking. We then introduce a motion modeling baseline to improve the detection rate and reduce false alarms based on accumulative multiframe differencing and robust matrix completion. Finally, we establish the first public benchmark for moving object detection and tracking in satellite videos and extensively evaluate the performance of several representative approaches on our dataset. Comprehensive experimental analyses and insightful conclusions are also provided. The dataset is available at <uri>https://github.com/QingyongHu/VISO</uri>.

A Robust Quadruplet and Faster Region-Based CNN for UAV Video-Based Multiple Object Tracking in Crowded Environment

Multiple object tracking (MOT) from unmanned aerial vehicle (UAV) videos has faced several challenges such as motion capture and appearance, clustering, object variation, high altitudes, and abrupt motion. Consequently, the volume of objects captured by the UAV is usually quite small, and the target object appearance information is not always reliable. To solve these issues, a new technique is presented to track objects based on a deep learning technique that attains state-of-the-art performance on standard datasets, such as Stanford Drone and Unmanned Aerial Vehicle Benchmark: Object Detection and Tracking (UAVDT) datasets. The proposed faster RCNN (region-based convolutional neural network) framework was enhanced by integrating a series of activities, including the proper calibration of key parameters, multi-scale training, hard negative mining, and feature collection to improve the region-based CNN baseline. Furthermore, a deep quadruplet network (DQN) was applied to track the movement of the captured objects from the crowded environment, and it was modelled to utilize new quadruplet loss function in order to study the feature space. A deep 6 Rectified linear units (ReLU) convolution was used in the faster RCNN to mine spatial–spectral features. The experimental results on the standard datasets demonstrated a high performance accuracy. Thus, the proposed method can be used to detect multiple objects and track their trajectories with a high accuracy.

A multi-context representation approach with multi-task learning for object counting

Object counting is a fundamental while challenging computer vision task, as it requires the object appearance information as well as semantic understanding of the object. In this paper, we propose an end-to-end multi-context embedding deep network for object counting(MCENet), which observes the object counting task from the three different perspectives to count the number of vehicles in the traffic video frame, or to estimate the number of the pedestrian in the largely congested scene. The first sub-network of MCENet extracts the potential features for the appearance context and the semantic context from different-level layers. The two different-level features from the first sub-network are transferred into the two parallel and complementary sub-networks, which are used to model the appearance context and semantic context for final counting. And thus the multiple contexts are represented and embedded to assist the counting task. Extensive experimental evaluations are reported in this paper, using up to three different object counting benchmarks, which show the proposed approach achieves a competitive performance in all these heterogeneous scenarios.

Conditional GAN based individual and global motion fusion for multiple object tracking in UAV videos

Multiple Object Tracking (MOT) meets great challenges in videos captured by Unmanned Aerial Vehicles (UAVs). Different from traditional videos, due to high altitude and abrupt motion changes of UAVs, the sizes of target objects in UAVs videos are usually very small and the appearance information of target objects is unreliable. The motion analysis is meaningful to associate multiple objects in UAV videos. However, the traditional motion analysis models inevitably suffer from the autonomous motion of UAVs. In this paper, we proposed a Conditional Generative Adversarial Networks (GAN) based model to predict complex motions in UAV videos. We regard the objects motions and the UAV movement as the individual motions and global motions respectively. They are complementary with each other and are employed jointly to facilitate accurate motion prediction. Specifically, a social Long Short Term Memory network is exploited to estimate the individual motion of objects, and a Siamese network is constructed to generate the global motion to reflect the view changes from UAVs, and a conditional GAN is developed to generate the final motion affinity. Extensive experimental results are conducted on public UAV datasets contained various types of objects and 4 different kinds of object detection inputs. Robust motion prediction and improved MOT performance are achieved compared with state-of-the-art methods.

Deep sensorimotor learning for RGB-D object recognition

Research findings in cognitive neuroscience establish that humans, early on, develop their understanding of real-world objects by observing others interact with them or by performing active exploration and physical interactions with them. This fact has motivated the so-called “sensorimotor” learning approach, where the object appearance information (sensory) is combined with the object affordances (motor), i.e. the types of actions a human can perform with the object. In this work, the aforementioned paradigm is adopted, and a neuro-biologically inspired two-stream model for RGB-D object recognition is investigated. Both streams are realized as state-of-the-art deep neural networks that process and fuse appearance and affordance information in multiple ways. In particular, three model variants are developed to efficiently encode the spatio-temporal nature of the hand–object interaction, while an attention mechanism that relies on the appearance stream confidence is also investigated. Additionally, a suitable auxiliary loss is proposed for model training, utilized to further optimize both information streams. Experiments on the challenging SOR3D dataset, which consists of 14 object types and 13 object affordances, demonstrate the efficacy of the proposed model in RGB-D object recognition. Overall, the best performing developed model achieves 90.70% classification accuracy, which is further increased to 91.98% when trained using the auxiliary loss. The latter corresponds to 46% relative error reduction compared to the appearance-only classifier performance. Finally, a cross-view analysis on the SOR3D dataset provides valuable feedback for the viewpoint impact on the affordance information.

Neuromorphic Vision Datasets for Pedestrian Detection, Action Recognition, and Fall Detection.

Neuromorphic Vision Datasets for Pedestrian Detection, Action Recognition, and Fall Detection.

Occluded-Object 3D Reconstruction Using Camera Array Synthetic Aperture Imaging.

With the three-dimensional (3D) coordinates of objects captured by a sequence of images taken in different views, object reconstruction is a technique which aims to recover the shape and appearance information of objects. Although great progress in object reconstruction has been made over the past few years, object reconstruction in occlusion situations remains a challenging problem. In this paper, we propose a novel method to reconstruct occluded objects based on synthetic aperture imaging. Unlike most existing methods, which either assume that there is no occlusion in the scene or remove the occlusion from the reconstructed result, our method uses the characteristics of synthetic aperture imaging that can effectively reduce the influence of occlusion to reconstruct the scene with occlusion. The proposed method labels occlusion pixels according to variance and reconstructs the 3D point cloud based on synthetic aperture imaging. Accuracies of the point cloud are tested by calculating the spatial difference between occlusion and non-occlusion conditions. The experiment results show that the proposed method can handle the occluded situation well and demonstrates a promising performance.

Long-term object tracking combined offline with online learning

We propose a simple yet effective method for long-term object tracking. Different from the traditional visual tracking method, which mainly depends on frame-to-frame correspondence, we combine high-level semantic information with low-level correspondences. Our framework is formulated in a confidence selection framework, which allows our system to recover from drift and partly deal with occlusion. To summarize, our algorithm can be roughly decomposed into an initialization stage and a tracking stage. In the initialization stage, an offline detector is trained to get the object appearance information at the category level, which is used for detecting the potential target and initializing the tracking stage. The tracking stage consists of three modules: the online tracking module, detection module, and decision module. A pretrained detector is used for maintaining drift of the online tracker, while the online tracker is used for filtering out false positive detections. A confidence selection mechanism is proposed to optimize the object location based on the online tracker and detection. If the target is lost, the pretrained detector is utilized to reinitialize the whole algorithm when the target is relocated. During experiments, we evaluate our method on several challenging video sequences, and it demonstrates huge improvement compared with detection and online tracking only.

An Analysis of Object Appearance Information and Context Based Classification

The purpose of categorizing objects is to identify and locate within an image, instances of an object category. It is a difficult task to identify objects in images when such images contain poor quality, occlusion, background clutter, or noise. It is more difficult when there are too many objects in same scene (background). Most models for categorizing object exploit context information and appearance information from objects in order to enhance recognition accuracy. In the scene, interaction among objects (context information) can successfully help in removing ambiguity of appearance inputs during recognition task. Appearance information can help identify object classes successfully up to some level. The issue of integrating several kinds of contextual information in order to have a robust categorization of object is address in this paper. Various methods of exploiting contextual information for categorizing object are review. An isolated object. The object’s appearance is not sufficient enough to reveal the accurate information about the identity of the object.

Food image classification using local appearance and global structural information

This paper proposes food image classification methods exploiting both local appearance and global structural information of food objects. The contribution of the paper is threefold. First, non-redundant local binary pattern (NRLBP) is used to describe the local appearance information of food objects. Second, the structural information of food objects is represented by the spatial relationship between interest points and encoded using a shape context descriptor formed from those interest points. Third, we propose two methods of integrating appearance and structural information for the description and classification of food images. We evaluated the proposed methods on two datasets. Experimental results verified that the combination of local appearance and structural features can improve classification performance.

Parsing radiographs by integrating landmark set detection and multi-object active appearance models.

This work addresses the challenging problem of parsing 2D radiographs into salient anatomical regions such as the left and right lungs and the heart. We propose the integration of an automatic detection of a constellation of landmarks via rejection cascade classifiers and a learned geometric constellation subset detector model with a multi-object active appearance model (MO-AAM) initialized by the detected landmark constellation subset. Our main contribution is twofold. First, we propose a recovery method for false positive and negative landmarks which allows to handle extreme ranges of anatomical and pathological variability. Specifically we (1) recover false negative (missing) landmarks through the consensus of inferences from subsets of the detected landmarks, and (2) choose one from multiple false positives for the same landmark by learning Gaussian distributions for the relative location of each landmark. Second, we train a MO-AAM using the true landmarks for the detectors and during test, initialize the model using the detected landmarks. Our model fitting allows simultaneous localization of multiple regions by encoding the shape and appearance information of multiple objects in a single model. The integration of landmark detection method and MO-AAM reduces mean distance error of the detected landmarks from 20.0mm to 12.6mm. We assess our method using a database of scout CT scans from 80 subjects with widely varying pathology.

Exploring Tiny Images: The Roles of Appearance and Contextual Information for Machine and Human Object Recognition

Typically, object recognition is performed based solely on the appearance of the object. However, relevant information also exists in the scene surrounding the object. In this paper, we explore the roles that appearance and contextual information play in object recognition. Through machine experiments and human studies, we show that the importance of contextual information varies with the quality of the appearance information, such as an image's resolution. Our machine experiments explicitly model context between object categories through the use of relative location and relative scale, in addition to co-occurrence. With the use of our context model, our algorithm achieves state-of-the-art performance on the MSRC and Corel data sets. We perform recognition tests for machines and human subjects on low and high resolution images, which vary significantly in the amount of appearance information present, using just the object appearance information, the combination of appearance and context, as well as just context without object appearance information (blind recognition). We also explore the impact of the different sources of context (co-occurrence, relative-location, and relative-scale). We find that the importance of different types of contextual information varies significantly across data sets such as MSRC and PASCAL.

Multiple kernel relevance vector machine for geospatial objects detection in high-resolution remote sensing images

Geospatial objects detection within complex environment is a challenging problem in remote sensing area. In this paper, we derive an extension of the Relevance Vector Machine (RVM) technique to multiple kernel version. The proposed method learns an optimal kernel combination and the associated classifier simultaneously. Two feature types are extracted from images, forming basis kernels. Then these basis kernels are weighted combined and resulted the composite kernel exploits interesting points and appearance information of objects simultaneously. Weights and the detection model are finally learnt by a new algorithm. Experimental results show that the proposed method improve detection accuracy to above 88%, yields good interpretation for the selected subset of features and appears sparser than traditional single-kernel RVMs.

Object appearance and picture-specific viewpoint are not integrated in long-term memory

Previous work has demonstrated that visual long-term memory (VLTM) stores detailed information about object appearance. The current experiments investigate whether object appearance information in VLTM is integrated within representations that contain picture-specific viewpoint information. In three experiments using both incidental and intentional encoding instructions, participants were unable to perform above chance on recognition tests that required recognizing the conjunction of object appearance and viewpoint information (Experiments 1a, 1b, 2, and 3). However, performance was better when object appearance information (Experiments 1a, 1b, and 2) or picture-specific viewpoint information (Experiment 3) alone was sufficient to succeed on the memory test. These results replicate previous work demonstrating good memory for object appearance and viewpoint. However the current results suggest that object appearance and viewpoint are not episodically integrated in VLTM.

Robust Tracking Method to Intersections of Objects with Similar Patterns

This paper proposes an approach of robust tracking for crossing of similar pattern objects. It is based on the particle filter. As far as only appearance information of a target object is used for the particle filter, it fails in tracking the target object when it intersects other objects with similar patterns. The proposed method uses velocity information of the target object and distance information between the target object and other moving objects in addition to appearance information. The situation judges where the target object exists and how to calculate the likelihood of each particle is changed according to the situation. The proposed method can track targets efficiently and accurately by using the result of background subtraction. Moreover, the method is improved so that it can track objects even when they are hidden wholly by a background object. Results are demonstrated by experiments using real video sequences.

Multi-object trajectory tracking

The majority of existing tracking algorithms are based on the maximum a posteriori solution of a probabilistic framework using a Hidden Markov Model, where the distribution of the object state at the current time instance is estimated based on current and previous observations. However, this approach is prone to errors caused by distractions such as occlusions, background clutters and multi-object confusions. In this paper, we propose a multiple object tracking algorithm that seeks the optimal state sequence that maximizes the joint multi-object state-observation probability. We call this algorithm trajectory tracking since it estimates the state sequence or “trajectory” instead of the current state. The algorithm is capable of tracking unknown time-varying number of multiple objects. We also introduce a novel observation model which is composed of the original image, the foreground mask given by background subtraction and the object detection map generated by an object detector. The image provides the object appearance information. The foreground mask enables the likelihood computation to consider the multi-object configuration in its entirety. The detection map consists of pixel-wise object detection scores, which drives the tracking algorithm to perform joint inference on both the number of objects and their configurations efficiently. The proposed algorithm has been implemented and tested extensively in a complete CCTV video surveillance system to monitor entries and detect tailgating and piggy-backing violations at access points for over six months. The system achieved 98.3% precision in event classification. The violation detection rate is 90.4% and the detection precision is 85.2%. The results clearly demonstrate the advantages of the proposed detection based trajectory tracking framework.