Feature Disentanglement Research Articles

Multi‐view facial action unit detection via deep feature enhancement

Multi-view facial action unit (AU) analysis has been a challenging research topic due to multiple disturbing variables, including subject identity biases, variational facial action unit intensities, facial occlusions and non-frontal head-poses. A deep feature enhancement (DFE) framework is presented to tackle some of these coupled complex disturbing variables for multi-view facial action unit detection. The authors' DFE framework is a novel end-to-end three-stage feature learning model with taking subject identity biases, dynamic facial changes and head-pose into consideration. It contains three feature enhancement modules, including coarse-grained local and holistic spatial feature learning (LHSF), spatio-temporal feature learning (STF) and head-pose feature disentanglement (FD). Experimental results show that the proposed method achieved state-of-the-art recognition performance on the FERA2017 dataset. The code is released at http://aip.seu.edu.cn/cgtang/.

Context-Aware Wireless-Based Cross-Domain Gesture Recognition

Recently, significant efforts have been made to enable WiFi-based gesture recognition. However, models trained with data collected from specific domain suffer from significant performance degradation when applied in a new domain. In practice, various WiFi sensing techniques have provided us with a full knowledge of domain information including discrete variables, i.e., environment and subject, as well as continuous variables, i.e., location and orientation. Previous works haven't fully explored these domain information or need to integrate substantial links' information to use them. Intuitively, we can boost gesture recognition accuracy by accounting for all these domain information with different properties. We propose a new framework not being restricted to link number which combines an adversarial learning scheme with feature disentanglement modules. They together conduct two-stage alignment between each of the source domains and the target domain to eliminate all gesture irrespective information. We also present an attention scheme based on discriminative information of each source and target domain to promote positive transfer from source to target domain. Our model is evaluated on the Widar 3.0 data set and achieves an improvement of 3%-12.7% in cross-domain average accuracy, demonstrating the superiority.

Deep learning-based multi-modal computing with feature disentanglement for MRI image synthesis.

Different Magnetic resonance imaging (MRI) modalities of the same anatomical structure are required to present different pathological information from the physical level for diagnostic needs. However, it is often difficult to obtain full-sequence MRI images of patients owing to limitations such as time consumption and high cost. The purpose of this work is to develop an algorithm for target MRI sequences prediction with high accuracy, and provide more information for clinical diagnosis. We propose a deep learning-based multi-modal computing model for MRI synthesis with feature disentanglement strategy. To take full advantage of the complementary information provided by different modalities, multi-modal MRI sequences are utilized as input. Notably, the proposed approach decomposes each input modality into modality-invariant space with shared information and modality-specific space with specific information, so that features are extracted separately to effectively process the input data. Subsequently, both of them are fused through the adaptive instance normalization (AdaIN) layer in the decoder. In addition, to address the lack of specific information of the target modality in the test phase, a local adaptive fusion (LAF) module is adopted to generate a modality-like pseudo-target with specific information similar to the ground truth. To evaluate the synthesis performance, we verify our method on the BRATS2015 dataset of 164 subjects. The experimental results demonstrate our approach significantly outperforms the benchmark method and other state-of-the-art medical image synthesis methods in both quantitative and qualitative measures. Compared with the pix2pixGANs method, the PSNR improves from 23.68 to 24.8. Moreover the ablation studies have also verified the effectiveness of important components of the proposed method. The proposed method could be effective in prediction of target MRI sequences, and useful for clinical diagnosis and treatment.

Informative Feature Disentanglement for Unsupervised Domain Adaptation

Unsupervised Domain Adaptation (UDA) aims at learning a classifier for an unlabeled target domain by transferring knowledge from a labeled source domain with a related but different distribution. The strategy of aligning the two domains in latent feature space via metric discrepancy or adversarial learning has achieved considerable progress. However, these existing approaches mainly focus on adapting the entire image and ignore the bottleneck that occurs when forced adaptation of uninformative domain-specific variations undermines the effectiveness of learned features. To address this problem, we propose a novel component called Informative Feature Disentanglement (IFD), which is equipped with the adversarial network or the metric discrepancy model, respectively. Accordingly, the new network architectures, named IFDAN and IFDMN, enable informative feature refinement before the adaptation. The proposed IFD is designed to disentangle informative features from the uninformative domain-specific variations, which are produced by a Variational Autoencoder (VAE) with lateral connections from the encoder to the decoder. We cooperatively apply the IFD to conduct supervised disentanglement for the source domain and unsupervised disentanglement for the target domain. In this way, informative features are disentangled from the domain-specific details before the adaptation. Extensive experimental results on three gold-standard domain adaptation datasets, e.g., Office31, Office-Home and VisDA-C, demonstrate the effectiveness of the proposed IFDAN and IFDMN models for UDA.

Unsupervised Domain Adaptation for Small Bowel Segmentation using Disentangled Representation.

We present a novel unsupervised domain adaptation method for small bowel segmentation based on feature disentanglement. To make the domain adaptation more controllable, we disentangle intensity and non-intensity features within a unique two-stream auto-encoding architecture, and selectively adapt the non-intensity features that are believed to be more transferable across domains. The segmentation prediction is performed by aggregating the disentangled features. We evaluated our method using intravenous contrast-enhanced abdominal CT scans with and without oral contrast, which are used as source and target domains, respectively. The proposed method showed clear improvements in terms of three different metrics compared to other domain adaptation methods that are without the feature disentanglement. The method brings small bowel segmentation closer to clinical application.

Joint Feature Disentanglement and Hallucination for Few-Shot Image Classification.

Few-shot learning (FSL) refers to the learning task that generalizes from base to novel concepts with only few examples observed during training. One intuitive FSL approach is to hallucinate additional training samples for novel categories. While this is typically done by learning from a disjoint set of base categories with sufficient amount of training data, most existing works did not fully exploit the intra-class information from base categories, and thus there is no guarantee that the hallucinated data would represent the class of interest accordingly. In this paper, we propose Feature Disentanglement and Hallucination Network (FDH-Net), which jointly performs feature disentanglement and hallucination for FSL purposes. More specifically, our FDH-Net is able to disentangle input visual data into class-specific and appearance-specific features. With both data recovery and classification constraints, hallucination of image features for novel categories using appearance information extracted from base categories can be achieved. We perform extensive experiments on two fine-grained datasets (CUB and FLO) and two coarse-grained ones (mini-ImageNet and CIFAR-100). The results confirm that our framework performs favorably against state-of-the-art metric-learning and hallucination-based FSL models.

CGMVAE: Coupling GMM Prior and GMM Estimator for Unsupervised Clustering and Disentanglement

Impressive progress has been recently witnessed on deep unsupervised clustering and feature disentanglement. In this paper, we propose a novel method on top of one recent architecture with a novel explanation of Gaussian mixture model (GMM) membership, accompanied by a GMM loss to enhance the clustering. The GMM loss is optimized with the explicitly computed parameters under our coupled GMM inspired framework. Specifically, our model takes the advantage of implicitly learning a GMM in latent space by neural networks (GMM prior as the first GMM), and explicitly clustering via the other GMM framework (GMM estimator as the second GMM). We further introduce a Dirichlet conjugate loss as a regularization term to prevent the GMM estimator from degenerating to few Gaussians. Eventually, we further propose an application of apparel generation based on the proposed method which requires only three selection steps. Extensive experiments on publicly available datasets demonstrate the effectiveness of our method, in terms of clustering and disentanglement performance.

EBIT: Weakly-supervised image translation with edge and boundary enhancement

Abstract In this paper, targeting image translation between the thermal and visible domains, we propose a novel framework to enhance the edge and boundary features during translation. We tackle the unsupervised training task where sample image pairs from two domains are randomly chosen so the image content does not match, but we also take advantage of paired feature maps during the feature disentanglement process. This can be considered to be weakly-supervisedtraining. First, since thermal images usually have vague edges, we propose to apply a Canny operator to strengthen the edge features of the thermal images. Next, to define the correct object boundaries, we extract the boundary features from silhouette masks which are paired with the visible domain images. Then we disentangle the features of both domains into a domain-shared latent space and a domain-exclusive latent space. In the domain-shared latent space, the edge and boundary features extracted from the thermal and visible domains respectively act as domain-shared information which is used to render the edges and define the boundaries of the translated images. In the domain-exclusive latent space, domain-exclusive information such as colour is used to render the colour of objects of the translated images. In addition, we propose a pixel-wise adversarial loss rather than more traditional ones. The experimental results show that the proposed method has the ability to render realistic edge and colour features within the correct object boundaries and outperforms several state-of-the-art methods.

GMFAD: Towards Generalized Visual Recognition via Multilayer Feature Alignment and Disentanglement.

The deep learning based approaches which have been repeatedly proven to bring benefits to visual recognition tasks usually make a strong assumption that the training and test data are drawn from similar feature spaces and distributions. However, such an assumption may not always hold in various practical application scenarios on visual recognition tasks. Inspired by the hierarchical organization of deep feature representation that progressively leads to more abstract features at higher layers of representations, we propose to tackle this problem with a novel feature learning framework, which is called GMFAD, with better generalization capability in a multilayer perceptron manner. We first learn feature representations at the shallow layer where shareable underlying factors among domains (e.g., a subset of which could be relevant for each particular domain) can be explored. In particular, we propose to align the domain divergence between domain pair(s) by considering both inter-dimension and inter-sample correlations, which have been largely ignored by many cross-domain visual recognition methods. Subsequently, to learn more abstract information which could further benefit transferability, we propose to conduct feature disentanglement at the deep feature layer. Extensive experiments based on different visual recognition tasks demonstrate that our proposed framework can learn better transferable feature representation compared with state-of-the-art baselines.

On Learning Disentangled Representations for Gait Recognition.

Gait, the walking pattern of individuals, is one of the important biometrics modalities. Most of the existing gait recognition methods take silhouettes or articulated body models as gait features. These methods suffer from degraded recognition performance when handling confounding variables, such as clothing, carrying and viewing angle. To remedy this issue, we propose a novel AutoEncoder framework, GaitNet, to explicitly disentangle appearance, canonical and pose features from RGB imagery. The LSTM integrates pose features over time as a dynamic gait feature while canonical features are averaged as a static gait feature. Both of them are utilized as classification features. In addition, we collect a Frontal-View Gait (FVG) dataset to focus on gait recognition from frontal-view walking, which is a challenging problem since it contains minimal gait cues compared to other views. FVG also includes other important variations, e.g., walking speed, carrying, and clothing. With extensive experiments on CASIA-B, USF, and FVG datasets, our method demonstrates superior performance to the SOTA quantitatively, the ability of feature disentanglement qualitatively, and promising computational efficiency. We further compare our GaitNet with state-of-the-art face recognition to demonstrate the advantages of gait biometrics identification under certain scenarios, e.g., long-distance/lower resolutions, cross viewing angles. Source code is available at http://cvlab.cse.msu.edu/project-gaitnet.html.

View Transfer on Human Skeleton Pose: Automatically Disentangle the View-Variant and View-Invariant Information for Pose Representation Learning

Learning a good pose representation is significant for many applications, such as human pose estimation and action recognition. However, the representations learned by most approaches are not intrinsic and their transferability in different datasets and different tasks is limited. In this paper, we introduce a method to learn a versatile representation, which is capable of recovering unseen corrupted skeletons, being applied to the human action recognition, and transferring pose from one view to another view without knowing the relationships of cameras. To this end, a sequential bidirectional recursive network (SeBiReNet) is proposed for modeling kinematic dependency between skeleton joints. Utilizing the SeBiReNet as the core module, a denoising autoencoder is designed to learn intrinsic pose features through the task of recovering corrupted skeletons. Instead of only extracting the view-invariant feature as many other methods, we disentangle the view-invariant feature from the view-variant feature in the latent space and use them together as a representation of the human pose. For a better feature disentanglement, an adversarial augmentation strategy is proposed and applied to the denoising autoencoder. Disentanglement of view-variant and view-invariant features enables us to realize view transfer on 3D poses. Extensive experiments on different datasets and different tasks verify the effectiveness and versatility of the learned representation.

Appearance-invariant place recognition by adversarially learning disentangled representation

Place recognition is an essential component to address the problem of visual navigation and SLAM. The long-term place recognition is challenging as the environment exhibits significant variations across different times of the days, months, and seasons. In this paper, we view appearance changes as multiple domains and propose a Feature Disentanglement Network (FDNet) based on a convolutional auto-encoder and adversarial learning to extract two independent deep features — content and appearance. In our network, the content feature is learned which only retains the content information of images through the competition with the discriminators and content encoder. Besides, we utilize the triplets loss to make the appearance feature encode the appearance information. The generated content features are directly used to measure the similarity of images without dimensionality reduction operations. We use datasets that contain extreme appearance changes to carry out experiments, which show how meaningful recall at 100% precision can be achieved by our proposed method where existing state-of-art approaches often get worse performance.

Prediction Consistency Guided Convolutional Neural Networks for Cross-Domain Bearing Fault Diagnosis

An underlying assumption in bearing fault diagnosis is that the training data from a source domain and the test data from a target domain obey the same distribution. But this assumption can be easily violated in practical industrial environments due to domain shift, which leads to significant performance degradation. To overcome this issue, we propose a novel convolutional neural network model to identify cross-domain bearing fault types based on 1-D vibration signals. Different from current single-network-based approaches, our model comprises a student network and a teacher network that simultaneously conduct data distribution matching and discriminative feature learning. Moreover, the two networks promote each other with the label prediction consistency constraint, so that the discriminative knowledge is able to transfer between the domains. Our model bridges the semantic information of the source vibration signals and the distribution information of the target vibration signals by jointly performing cross-domain feature disentanglement and adaptation. The proposed method is evaluated extensively on the Case Western Reserve University bearing fault dataset in two scenarios: varying working loads and different sensor locations. Experimental results show the superior performance of our method compared with existing shallow and deep learning methods in the literature.