Extract Appearance Features Research Articles

A motion-appearance-aware network for object change detection

Object change detection (OCD), which aims to segment moving objects from an input frame, has attracted growing attention. Most OCD algorithms rely on scene diversity or ignore the interframe spatiotemporal structural dependence, which limits their applicability. In this paper, we propose a motion-appearance-aware network (MAAN) for learning robust feature representations. Specifically, a module for mining information at multiple time scales, which can adaptively adjust information elements, is designed to refine the motion feature. Meanwhile, salient object knowledge is obtained with the help of the extracted appearance features. To enhance the semantic consistency and trim redundant connections, we construct a fusion module called multi-view feature evolution, which effectively fuses motion and appearance information by global communication and local guidance, respectively. Moreover, we develop two strategies for obtaining uniform and consistent moving objects during information propagation. One is to feed the predicted mask of the previous frame into the decoder, and the other is to match different levels of motion cues at multiple time scales to the decoder. Finally, extensive experiments on four public datasets (i.e., LASIESTA, CDnet2014, INO, and AICD) indicate that the proposed approach outperforms other methods.

Bidirectional Posture-Appearance Interaction Network for Driver Behavior Recognition

Driver behavior recognition has become one of the most important tasks for intelligent vehicles. This task, however, is very challenging since the background contents in real-world driving scenarios are often very complex. More critically, the difference between driving behaviors is often very minor, making it extremely difficult to distinguish them. Existing methods often rely only on RGB frames (or skeleton data), which may fail to capture the minor differences between behaviors and appearance information of objects simultaneously and thus fail to achieve promising performance. To address the above issues, in this paper, we propose a bidirectional posture-appearance interaction network (BPAI-Net), which simultaneously considers RGB frames and skeleton ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e.</i> , posture) data for driver behavior recognition. Specifically, we propose a posture-guided convolutional neural network (PG-CNN) and an appearance-guided graph convolutional network (AG-GCN) to extract appearance and posture features, respectively. To exploit the complementary information between appearance and posture, we use the appearance features from PG-CNN for guiding AG-GCN to exploit the contextual information ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">e.g.</i> , nearby objects) to enhance posture features. Then, we use the enhanced posture features from AG-GCN to help PG-CNN focus on critical local areas of video frames that are related to driver behaviors. In this sense, we are able to use the interaction between two modalities to extract more discriminative features and thus improve the recognition accuracy. Experimental results on Drive&Act dataset show that our method outperforms state-of-the-art methods by a large margin (67.83% vs. 63.64%). Furthermore, we collect a bus driver behavior recognition dataset and yield consistent performance gain against baseline methods, demonstrating the effectiveness of our method in real-world applications. The source code and trained models are available at github.com/SCUT-AILab/BPAI-Net/.

Fused deep learning based Facial Expression Recognition of students in online learning mode

SummaryIn this research work, Facial Expression Recognition (FER) is used in the analysis of facial expressions during the online learning sessions in the prevailing pandemic situation. An integrated geometric and appearance feature extraction is presented for the FER of the students participating in the online classes. The integrated features provided a low‐dimensional significant feature area for better facial data representation. Feasible Weighted Squirrel Search Optimization (FW‐SSO) algorithm is applied for selecting the optimal features due to its efficient exploration of the search space and enhancement of the dynamic search. The output of the FW‐SSO algorithm is used for tuning the autoencoder. Autoencoder is used for combining the G&amp;A features, for feature optimization process. Classification is done by using Long Short‐Term Memory (LSTM) network with Attention Mechanism (ALSTM), as it is highly efficient in capturing the long‐term dependency of the facial landmarks in the image/video sequences. The proposed fused deep learning method focuses on the fusion of the G&amp;A features for high discrimination. Experimental analysis using FER‐2013 and LIRIS datasets demonstrated that the proposed method achieved maximum accuracy of 85.96% than the existing architectures and maximum accuracy of 88.24% than the VGGNet‐CNN architecture.

Lightweight and Deep Appearance Embedding for Multiple Object Tracking

The main challenge of Multiple Object Tracking (MOT) is that there is great uncertainty in data association when using the tracked predicted values and tracked trajectories. Meanwhile, the MOT is complex and time-consuming. When equipment resources are limited or real-time requirement is high, its application is very limited. Therefore, we propose a Lightweight Deep Appearance Embedding (LDAE) to assist the association of trajectories. Firstly, in addition to motion information in data association, we also introduce more discriminative appearance features to participate in the affinity measure to effectively distinguish similar targets. Secondly, according to the idea of feature mapping, we design a lightweight deep appearance embedding module. It can help extract appearance features with less computation. Finally, we propose a simulated occlusion strategy for the training of the LDAE, which helps improve the ability to recognise different targets in dense scenes. The LDAE dramatically reduces the computational cost and improves the accuracy of data association. Extensive experiments are conducted on the MOT datasets (MOT16, MOT17 and MOT20), which prove that the LDAE outperforms several state-of-the-art trackers in the tracking accuracy and anti-occlusion performance. Furthermore, we apply the LDAE to escalators, which can achieve fast and stable tracking effect.

Compositional action recognition with multi-view feature fusion.

Most action recognition tasks now treat the activity as a single event in a video clip. Recently, the benefits of representing activities as a combination of verbs and nouns for action recognition have shown to be effective in improving action understanding, allowing us to capture such representations. However, there is still a lack of research on representational learning using cross-view or cross-modality information. To exploit the complementary information between multiple views, we propose a feature fusion framework, and our framework is divided into two steps: extraction of appearance features and fusion of multi-view features. We validate our approach on two action recognition datasets, IKEA ASM and LEMMA. We demonstrate that multi-view fusion can effectively generalize across appearances and identify previously unseen actions of interacting objects, surpassing current state-of-the-art methods. In particular, on the IKEA ASM dataset, the performance of the multi-view fusion approach improves 18.1% over the performance of the single-view approach on top-1.

ESAR, An Expert Shoplifting Activity Recognition System

Abstract Shoplifting is a troubling and pervasive aspect of consumers, causing great losses to retailers. It is the theft of goods from the stores/shops, usually by hiding the store item either in the pocket or in carrier bag and leaving without any payment. Revenue loss is the most direct financial effect of shoplifting. Therefore, this article introduces an Expert Shoplifting Activity Recognition (ESAR) system to reduce shoplifting incidents in stores/shops. The system being proposed seamlessly examines each frame in video footage and alerts security personnel when shoplifting occurs. It uses dual-stream convolutional neural network to extract appearance and salient motion features in the video sequences. Here, optical flow and gradient components are used to extract salient motion features related to shoplifting movement in the video sequence. Long Short Term Memory (LSTM) based deep learner is modeled to learn the extracted features in the time domain for distinguishing person actions (i.e., normal and shoplifting). Analyzing the model behavior for diverse modeling environments is an added contribution of this paper. A synthesized shoplifting dataset is used here for experimentations. The experimental outcomes show that the proposed approach attains better consequences up to 90.26% detection accuracy compared to the other prevalent approaches.

STCNet: spatiotemporal cross network for industrial smoke detection

Industrial smoke emissions present a serious threat to natural ecosystems and human health. Prior works have shown that using computer vision techniques to identify smoke is a low-cost and convenient method. However, translucent smoke detection is a challenging task because of the irregular contours and complex motion state. To overcome these problems, we propose a novel spatiotemporal cross network (STCNet) to recognize industrial smoke emissions. The proposed STCNet involves a spatial pathway to extract appearance features and a temporal pathway to capture smoke motion information. Our STCNet is more targeted and goal oriented for dealing with translucent, nonrigid smoke objects. The spatial path can easily recognize obvious nonsmoking objects such as trees and buildings, and the temporal path can highlight the obscure traces of motion smoke. Our STCNet achieves the mutual guidance of multilevel spatiotemporal information by bidirectional feature fusion on multilevel feature maps. Extensive experiments on public datasets show that our STCNet achieves clear improvements against the best competitors by 6.2%. We also perform in-depth ablation studies on STCNet to explore the impacts of different feature fusion methods for the entire model. The code will be available at https://github.com/Caoyichao/STCNet.

Deep Shape-Aware Person Re-Identification for Overcoming Moderate Clothing Changes

Although person re-identification (person re-id) has advanced substantially in recent years, most methods are based on the assumption that the identities would not change clothes. This assumption might not hold in practice considering criminals who intentionally change clothes. In this work, we attempt to solve person re-id under moderate clothing change. Since the human body shape is considered as relatively more invariant under moderate clothing changes, we propose to learn a reliable shape-aware feature representation by mutually learning both colorful images and contour images. Instead of directly extracting shape features from contour images, we utilize contour feature learning as regularization and excavate more effective shape-aware feature representations from colorful images. We propose a multi-scale appearance and contour deep infomax (MAC-DIM) to maximize mutual information between colorful appearance features and contour shape features, and in this way, the extracted appearance features are constrained to be shape-aware in terms of both low-level visual properties and high-level semantics. To better model the long-range human body shape and explicitly capture contour segment relations, we introduce hierarchical graph modeling as aggregation headers, propagating structural context through graph convolutional networks (GCNs). The extensive results on benchmarks under clothing changes demonstrate the effectiveness of our shape-aware feature learning scheme.

CSA6D: Channel-Spatial Attention Networks for 6D Object Pose Estimation

6D object pose estimation plays a crucial role in robotic manipulation and grasping tasks. The aim to estimate the 6D object pose from RGB or RGB-D images is to detect objects and estimate their orientations and translations relative to the given canonical models. RGB-D cameras provide two sensory modalities: RGB and depth images, which could benefit the estimation accuracy. But the exploitation of two different modality sources remains a challenging issue. In this paper, inspired by recent works on attention networks that could focus on important regions and ignore unnecessary information, we propose a novel network: Channel-Spatial Attention Network (CSA6D) to estimate the 6D object pose from RGB-D camera. The proposed CSA6D includes a pre-trained 2D network to segment the interested objects from RGB image. Then it uses two separate networks to extract appearance and geometrical features from RGB and depth images for each segmented object. Two feature vectors for each pixel are stacked together as a fusion vector which is refined by an attention module to generate a aggregated feature vector. The attention module includes a channel attention block and a spatial attention block which can effectively leverage the concatenated embeddings into accurate 6D pose prediction on known objects. We evaluate proposed network on two benchmark datasets YCB-Video dataset and LineMod dataset and the results show it can outperform previous state-of-the-art methods under ADD and ADD-S metrics. Also, the attention map demonstrates our proposed network searches for the unique geometry information as the most likely features for pose estimation. From experiments, we conclude that the proposed network can accurately estimate the object pose by effectively leveraging multi-modality features.

Online multi-object tracking using multi-function integration and tracking simulation training

Recently, with the development of deep-learning, the performance of multi-object tracking algorithms based on deep neural networks has been greatly improved. However, most methods separate different functional modules into multiple networks and train them independently on specific tasks. When these network modules are used directly, they are not compatible with each other effectively, nor can they be better adapted to the multi-object tracking task, which leads to a poor tracking effect. Therefore, a network structure is designed to aggregate the regression of objects between frames and the extraction of appearance features into one model to improve the harmony between various functional modules of multi-object tracking. To improve the support for the multi-object tracking task, an end-to-end training method is also proposed to simulate the multi-object tracking process during the training and expand the training data by using the historical position of the target combined with the prediction of the motion model. A metric loss that can take advantage of the historical appearance features of the target is also used to train the extraction module of appearance features to improve the temporal correlation of extracted appearance features. Evaluation results on the MOTChallenge benchmark datasets show that the proposed approach achieves state-of-the-art performance.

Deep Human-Interaction and Association by Graph-Based Learning for Multiple Object Tracking in the Wild

Multiple Object Tracking (MOT) in the wild has a wide range of applications in surveillance retrieval and autonomous driving. Tracking-by-Detection has become a mainstream solution in MOT, which is composed of feature extraction and data association. Most of the existing methods focus on extracting targets’ individual features and optimizing the association by hand-crafted algorithms. In this paper, we specially consider the interrelation cue between targets and we propose Human-Interaction Model (HIM) to extract interaction features between the tracked target and its surrounding. The interaction model has more discriminative features to distinguish objects, especially in crowded (dense) scene. Meanwhile we propose an efficient end-to-end model, Deep Association Network (DAN), to optimize the association with graph-based learning mechanism. Both HIM and DAN are constructed by three kinds of deep networks, which include Convolutional Neural Network (CNN), Recurrent Neural Network (RNN) and Graph Neural Network (GNN). The CNNs extract appearance features from bounding box images, the RNNs encoder motion features from historical positions of trajectory. And then the GNNs aim to extract interaction features and optimize graph structure to associate the objects in different frames. In addition, we present a novel end-to-end training strategy for Deep Association Network and Human-Interaction Model. Our experimental results demonstrate performance of our method reaches the state-of-the-art on MOT15, MOT16 and DukeMTMCT datasets.

Rethinking Motion Representation: Residual Frames With 3D ConvNets.

Recently, 3D convolutional networks yield good performance in action recognition. However, an optical flow stream is still needed for motion representation to ensure better performance, whose cost is very high. In this paper, we propose a cheap but effective way to extract motion features from videos utilizing residual frames as the input data in 3D ConvNets. By replacing traditional stacked RGB frames with residual ones, 35.6% and 26.6% points improvements over top-1 accuracy can be achieved on the UCF101 and HMDB51 datasets when trained from scratch using ResNet-18-3D. We deeply analyze the effectiveness of this modality compared to normal RGB video clips, and find that better motion features can be extracted using residual frames with 3D ConvNets. Considering that residual frames contain little information of object appearance, we further use a 2D convolutional network to extract appearance features and combine them together to form a two-path solution. In this way, we can achieve better performance than some methods which even used an additional optical flow stream. Moreover, the proposed residual-input path can outperform RGB counterpart on unseen datasets when we apply trained models to video retrieval tasks. Huge improvements can also be obtained when the residual inputs are applied to video-based self-supervised learning methods, revealing better motion representation and generalization ability of our proposal.

Integrated Multiscale Appearance Features and Motion Information Prediction Network for Anomaly Detection.

The rise of video-prediction algorithms has largely promoted the development of anomaly detection in video surveillance for smart cities and public security. However, most current methods relied on single-scale information to extract appearance (spatial) features and lacked motion (temporal) continuity between video frames. This can cause a loss of partial spatiotemporal information that has great potential to predict future frames, affecting the accuracy of abnormality detection. Thus, we propose a novel prediction network to improve the performance of anomaly detection. Due to the objects of various scales in each video, we use different receptive fields to extract detailed appearance features by the hybrid dilated convolution (HDC) module. Meanwhile, the deeper bidirectional convolutional long short-term memory (DB-ConvLSTM) module can remember the motion information between consecutive frames. Furthermore, we use RGB difference loss to replace optical flow loss as temporal constraint, which greatly reduces the time for optical flow extraction. Compared with the state-of-the-art methods in the anomaly-detection task, experiments prove that our method can more accurately detect abnormalities in various video surveillance scenes.

Discriminative feature extraction for video person re-identification via multi-task network

The goal of video-based person re-identification is to match different pedestrians in various image sequences across non-overlapping cameras. A critical issue of this task is how to exploit the useful information provided by videos. To solve this problem, we propose a novel feature learning framework for video-based person re-identification. The proposed method aims at capturing the most significant information in the spatial and temporal domains and then building a discriminative and robust feature representation for each sequence. More specifically, to learn more effective frame-wise features, we apply several attributes to the video-based task and build a multi-task network for the identity and attribute classifications. In the training phase, we present a multi-loss function to minimize intra-class variances and maximize inter-class differences. After that, the feature aggregation network is employed to aggregate frame-wise features and extract the temporal information from the video. Furthermore, considering that adjacent frames typically have similar appearance features, we propose the concept of “non-redundant appearance feature extraction” to obtain the sequence-level appearance descriptors of pedestrians. Based on the complementarity between the temporal feature and the non-redundant appearance feature, we combine them in the distance learning phase by assigning them different distance-weighted coefficients. Extensive experiments are conducted on three video-based datasets and the results demonstrate the superiority and effectiveness of our method.

A new method of abnormal behavior detection using LSTM network with temporal attention mechanism

In this paper, we propose an accurate and effective method for detecting abnormal behavior. We consider the video as a series of frame sequences; in the training phase, our deep learning framework is used to extract appearance features and learn the relationship between historical features and current features in the normal video. In the testing phase, the predicted features that differ from the actual features are considered as abnormal. Our model is designed as a feature prediction framework with a new temporal attention mechanism. In the feature extraction stage, we transform a pre-trained Vgg16 network into a fully convolutional neural network and used the third pooling layer output as the appearance feature extraction to effectively capture static appearance features. Then, a new temporal attention mechanism is introduced to learn the contribution of different historical appearance features at the same position to the current features, thereby solving the problem of representing dynamic motion features. Finally, the LSTM network is used to decode the historical feature sequences with temporal attention to predict the features at the current moment. Those actual features that differ from the predicted features are considered as abnormal features. Using upsampling for the abnormal features locates abnormal behavior on the original frames. Experiments on two benchmark datasets demonstrate the competitive performance of our method with the state-of-the-art methods.

Online Multiple Object Tracking with Reid Feature Extraction Network and Similarity Matrix Network

In multiple object tracking(MOT), data association is a crucial part. By constructing similarity loss matrix for trajectories and detections, they can be matched correspondingly by using Hungarian matching algorithm. However, the similarity loss is often obtained by calculating the Euclidean distance or other handcraft distance metrics of features extracted between objects, which may not be robust enough, resulting in matching inaccuracy. In this paper, we propose a novel MOT method with applying deep learning to feature extraction and data association. We firstly design the appearance feature extraction network(AFN) to learn effective features by training it on a large-scale person re-identification dataset(reid). Then, we propose the similarity matrix estimation network (SMN) to obtain reliable similarity by training it on the public MOT Challenge dataset, MOT17. Additionally, the similarity matrix output of SMN includes the dummy objects, which are used to deal with the association problems of object missing and object appearance between frames. In the end, our proposed MOT method is evaluated on MOT15, MOT17 and ablation study is carried out.

Accurate Segmentation of Cerebrovasculature From TOF-MRA Images Using Appearance Descriptors

Analyzing cerebrovascular changes can significantly lead to not only detecting the presence of serious diseases e.g., hypertension and dementia, but also tracking their progress. Such analysis could be better performed using Time-of-Flight Magnetic Resonance Angiography (ToF–MRA) images, but this requires accurate segmentation of the cerebral vasculature from the surroundings. To achieve this goal, we propose a fully automated cerebral vasculature segmentation approach based on extracting both prior and current appearance features that have the ability to capture the appearance of macro and micro-vessels in ToF–MRA. The appearance prior is modeled with a novel translation and rotation invariant Markov-Gibbs Random Field (MGRF) of voxel intensities with pairwise interaction analytically identified from a set of training data sets. The appearance of the cerebral vasculature is also represented with a marginal probability distribution of voxel intensities by using a Linear Combination of Discrete Gaussians (LCDG) that its parameters are estimated by using a modified Expectation-Maximization (EM) algorithm. The extracted appearance features are separable and can be classified by any classifier, as demonstrated by our segmentation results. To validate the accuracy of our algorithm, we tested the proposed approach on in-vivo data using 270 data sets, which were qualitatively validated by a neuroradiology expert. The results were quantitatively validated using the three commonly used metrics for segmentation evaluation: the Dice coefficient, the modified Hausdorff distance, and the absolute volume difference. The proposed approach showed a higher accuracy compared to two of the existing segmentation approaches.

Generating Visually Aligned Sound from Videos.

We focus on the task of generating sound from natural videos, and the sound should be both temporally and content-wise aligned with visual signals. This task is extremely challenging because some sounds generated outside a camera can not be inferred from video content. The model may be forced to learn an incorrect mapping between visual content and these irrelevant sounds. To address this challenge, we propose a framework named REGNET. In this framework, we first extract appearance and motion features from video frames to better distinguish the object that emits sound from complex background information. We then introduce an innovative audio forwarding regularizer that directly considers the real sound as input and outputs bottlenecked sound features. Using both visual and bottlenecked sound features for sound prediction during training provides stronger supervision for the sound prediction. The audio forwarding regularizer can control the irrelevant sound component and thus prevent the model from learning an incorrect mapping between video frames and sound emitted by the object that is out of the screen. During testing, the audio forwarding regularizer is removed to ensure that REGNET can produce purely aligned sound only from visual features. Extensive evaluations based on Amazon Mechanical Turk demonstrate that our method significantly improves both temporal and contentwise alignment. Remarkably, our generated sound can fool the human with a 68.12% success rate. Code and pre-trained models are publicly available at https://github.com/PeihaoChen/regnet.

MonoFENet: Monocular 3D Object Detection with Feature Enhancement Networks.

Monocular 3D object detection has the merit of low cost and can be served as an auxiliary module for autonomous driving system, becoming a growing concern in recent years. In this paper, we present a monocular 3D object detection method with feature enhancement networks, which we call MonoFENet. Specifically, with the estimated disparity from the input monocular image, the features of both the 2D and 3D streams can be enhanced and utilized for accurate 3D localization. For the 2D stream, the input image is used to generate 2D region proposals as well as to extract appearance features. For the 3D stream, the estimated disparity is transformed into 3D dense point cloud, which is then enhanced by the associated front view maps. With the RoI Mean Pooling layer, 3D geometric features of RoI point clouds are further enhanced by the proposed point feature enhancement (PointFE) network. The region-wise features of image and point cloud are fused for the final 2D and 3D bounding boxes regression. The experimental results on the KITTI benchmark reveal that our method can achieve state-of-the-art performance for monocular 3D object detection.

Multi-Level Haar Wavelet based Facial Expression Recognition using Logistic Regression

Background/Objective: Facial expressions play an equally important role as verbal communication and tonal expressions. Recognition of facial expression is important in industrial automation, security, medical and many other fields. Methods/ Statistical Analysis: In this paper, we propose multilevel haar wavelet-based approach, which extracts appearance features from prominent face regions at two different scales. The approach first segments most informative geometric components such as eye, mouth, eyebrows etc. using the Viola-Jones cascade object detector. Haar features of segmented components are extracted. One vs. All logistic regression model is used for the classification. Haar features are simple to compute and they can effectively represent the signal in low dimension, yet preserves the energy of the signal. Findings: The performance of the proposed approach is tested on well-known CK, JAFFE and TFEID facial expression datasets, and it achieves 90.48%, 88.57% and 96.84% accuracy for the respective dataset.