Human Parsing Research Articles

Data augmentation in human-centric vision

AbstractThis survey presents a comprehensive analysis of data augmentation techniques in human-centric vision tasks, a first of its kind in the field. It delves into a wide range of research areas including person ReID, human parsing, human pose estimation, and pedestrian detection, addressing the significant challenges posed by overfitting and limited training data in these domains. Our work categorizes data augmentation methods into two main types: data generation and data perturbation. Data generation covers techniques like graphic engine-based generation, generative model-based generation, and data recombination, while data perturbation is divided into image-level and human-level perturbations. Each method is tailored to the unique requirements of human-centric tasks, with some applicable across multiple areas. Our contributions include an extensive literature review, providing deep insights into the influence of these augmentation techniques in human-centric vision and highlighting the nuances of each method. We also discuss open issues and future directions, such as the integration of advanced generative models like Latent Diffusion Models, for creating more realistic and diverse training data. This survey not only encapsulates the current state of data augmentation in human-centric vision but also charts a course for future research, aiming to develop more robust, accurate, and efficient human-centric vision systems.

Knowledge enhanced multi-task learning for simultaneous optimization of human parsing and pose estimation

Human parsing and pose estimation are both closely related to the human body structure, and it is advantageous to deal with both together in order to assist each other’s learning. Nevertheless, the two tasks have their own unique characteristics, and it is therefore difficult to efficiently and simultaneously obtain a good performance for both tasks in one single multi-task learning (MTL) network. This paper proposes a new MTL network (named “KDNet”) to enhance the communication between human parsing and pose estimation through knowledge distillation so as to improve the simultaneous learning of both tasks. In the proposed KDNet, a consistent representation module is used to enhance the related information sharing between the tasks, and a single-task model-based loss weighting method is developed to balance the loss levels. The proposed KDNet has been verified across three benchmark datasets, and in comparison to other state-of-the-art methods it achieves substantial performance gains without increasing model parameters during inference. The codes are shared at https://github.com/Yhdian/KDNet.git.

Learning differentiable categorical regions with Gumbel-Softmax for person re-identification

Locating diverse body parts and perceiving part visibility are essential to person re-identification (re-ID). Most existing methods employ an extra model, e.g., pose estimation or human parsing, to locate parts, or generate pseudo labels to train the part locator incorporated with the re-ID model. In this paper, we aim at learning diverse horizontal stripes with foreground refinement to pursue pixel-level part alignment via only using person identity labels. Specifically, we proposed a Gumbel-Softmax based Differential Categorical Region (DCR) learning method and make two contributions. (1) A stripe-wise regularization. Given an image, the part locator produce part probability maps. The continuous values in the probability maps are discretized into zero or argmax value in the horizontal stripes by the Gumbel-Softmax. Gumbel-Softmax allows us to use the argmax discrete value for part diversity regularization in the forward pass, but can still estimate gradients in the backward pass. (2) A self-refinement method to suppress the background noise in the stripes. We employ a lightweight foreground perception head to produce foreground probability map with only person identity labels supervision. Benefits from discretization of the categorical stripes, we can conveniently obtain the part pseudo label by element-wise multiplying the categorical stripes with foreground probability map. Finally, DCR can locate the body parts at pixel-level and extract part-aligned representation. Experimental results on both holistic and occluded re-ID datasets confirm that our approach significantly improves the learned representation and the achieved performance is on par with the state-of-the-art methods. The code is available at https://github.com/deepalchemist/differentiable-categorical-region

FCGNet: Foreground and Class Guided Network for human parsing

Understanding the inherent hierarchical human structure is key to human parsing. To capture the human-specific characteristic, it is necessary to focus on the spatial and class information corresponding to the foreground (i.e., human) in an image. Inspired by these insights, we introduce two supervision signals, spatial foreground information and existent class information in the image. By utilizing foreground information as guidance, the network is guided to generate a human-focused feature map and capture the pixel-wise hierarchical characteristics by computing correlations between pixels. Furthermore, we guide the network to consider class information in the image at the feature level and capture the class-wise relationship by calculating correlations between channels. Moreover, during the training phase, we prevent the network from misclassifying pixels into confusing classes by providing the existent class information in the image to the network at the prediction level. Our model achieves state-of-the-art performance with significantly reduced parameters and Multiply-Accumulate Operations (MACs) in three public benchmarks.

Explore human parsing modality for action recognition

AbstractMultimodal‐based action recognition methods have achieved high success using pose and RGB modality. However, skeletons sequences lack appearance depiction and RGB images suffer irrelevant noise due to modality limitations. To address this, the authors introduce human parsing feature map as a novel modality, since it can selectively retain effective semantic features of the body parts while filtering out most irrelevant noise. The authors propose a new dual‐branch framework called ensemble human parsing and pose network (EPP‐Net), which is the first to leverage both skeletons and human parsing modalities for action recognition. The first human pose branch feeds robust skeletons in the graph convolutional network to model pose features, while the second human parsing branch also leverages depictive parsing feature maps to model parsing features via convolutional backbones. The two high‐level features will be effectively combined through a late fusion strategy for better action recognition. Extensive experiments on NTU RGB + D and NTU RGB + D 120 benchmarks consistently verify the effectiveness of our proposed EPP‐Net, which outperforms the existing action recognition methods. Our code is available at https://github.com/liujf69/EPP‐Net‐Action.

From Simple to Complex Scenes: Learning Robust Feature Representations for Accurate Human Parsing.

Human parsing has attracted considerable research interest due to its broad potential applications in the computer vision community. In this paper, we explore several useful properties, including high-resolution representation, auxiliary guidance, and model robustness, which collectively contribute to a novel method for accurate human parsing in both simple and complex scenes. Starting from simple scenes: we propose the boundary-aware hybrid resolution network (BHRN), an advanced human parsing network. BHRN utilizes deconvolutional layers and multi-scale supervision to generate rich high-resolution representations. Additionally, it includes an edge perceiving branch designed to enhance the fineness of part boundaries. Building on BHRN, we construct a dual-task mutual learning (DTML) framework. It not only provides implicit guidance to assist the parser by incorporating boundary features, but also explicitly maintains the high-order consistency between the parsing prediction and the ground truth. Toward complex scenes: we develop a domain transform method to enhance the model robustness. By transforming the input space from the spatial domain to the polar harmonic Fourier moment domain, the mapping relationship to the output semantic space is highly stable. This transformation yields robust representations for both clean and corrupted data. When evaluated on standard benchmark datasets, our method achieves superior performance compared to state-of-the-art human parsing methods. Furthermore, our domain transform strategy significantly improves the robustness of DTML dramatically in most complex scenes.

WNet: A dual‐encoded multi‐human parsing network

AbstractIn recent years, multi‐human parsing has become a focal point in research, yet prevailing methods often rely on intermediate stages and lacking pixel‐level analysis. Moreover, their high computational demands limit real‐world efficiency. To address these challenges and enable real‐time performance, low‐latency end‐to‐end network is proposed. This approach leverages vision transformer and convolutional neural network in a dual‐encoded network, featuring a lightweight Transformer‐based vision encoder) and a convolution encoder based on Darknet. This combination adeptly captures long‐range dependencies and spatial relationships. Incorporating a fuse block enables the seamless merging of features from the encoders. Residual connections in the decoder design amplify information flow. Experimental validation on crowd instance‐level human parsing and look into person datasets showcases the WNet's effectiveness, achieving high‐speed multi‐human parsing at 26.7 frames per second. Ablation studies further underscore WNet's capabilities, emphasizing its efficiency and accuracy in complex multi‐human parsing tasks.

Modality adaptation via feature difference learning for depth human parsing

In the field of human parsing, depth data offers unique advantages over RGB data due to its illumination invariance and geometric detail, which motivates us to explore human parsing with only depth input. However, depth data is challenging to collect at scale due to the specialized equipment required. In contrast, RGB data is readily available in large quantities, presenting an opportunity to enhance depth-only parsing models with semantic knowledge learned from RGB data. However, fully finetuning the RGB-pretrained encoder leads to high training costs and inflexible domain generalization, while keeping the encoder frozen suffers from a large RGB-depth modality gap and restricts the parsing performance. To alleviate the limitations of these naive approaches, we introduce a Modality Adaptation pipeline via Feature Difference Learning (MAFDL) which leverages the RGB knowledge to facilitate depth human parsing. A Difference-Guided Depth Adapter (DGDA) is proposed within MAFDL to learn the feature differences between RGB and depth modalities, adapting depth features into RGB feature space to bridge the modality gap. Furthermore, we also design a Feature Alignment Constraint (FAC) to impose explicit alignment supervision at pixel and batch levels, making the modality adaptation more comprehensive. Extensive experiments on the NTURGBD-Parsing-4K dataset show that our method surpasses previous state-of-the-art approaches.

A multitask tensor-based relation network for cloth-changing person re-identification

The Cloth-changing person re-identification (CC-ReID) is more challenging than person re-identification (ReID) because of the unreliability of cloth-relevant features. Existing CC-ReID methods output the unique latent vector feature for the same pedestrian image. However, on the one hand, the latent vector feature generated by the pooling layer will lose more spatial information. On the other hand, the features that they focus on are the same for the same pedestrian image under different comparison pedestrian images, which does not consider the pedestrian pair relation information. For this, we propose a Multitask Tensor-based Relation Network (MTTRN) for CC-ReID. In MTTRN, we utilize the augmented cloth-changing identity images, human parsing images and head images to guide the model learning more fine-grained cloth-irrelevant feature cues. We propose a novel channel subspace generator and use the tensor mode-n product to generate the subspace tensor features instead of the vector feature generated by the pooling layer, which can retain more spatial feature information. Furthermore, we assume that the model will focus on different features for the same pedestrian image under different comparison pedestrian images in the same feature subspace or the same pedestrian image pair in different feature subspaces, in which the tensor feature pair relation is considered fully to mine more robust features. Extensive experiments show that our method achieves the state-of-the-art or competitive performance on three CC-ReID benchmark datasets and demonstrate the robustness of our model.

Semantically enhanced attention map‐driven occluded person re‐identification

AbstractOccluded person re‐identification (Re‐ID) is to identify a particular person when the person's body parts are occluded. However, challenges remain in enhancing effective information representation and suppressing background clutter when considering occlusion scenes. This paper proposes a novel attention map‐driven network (AMD‐Net) for occluded person Re‐ID. In AMD‐Net, human parsing labels are introduced to supervise the generation of partial attention maps, while a spatial‐frequency interaction module is suggested to complement the higher‐order semantic information from the frequency domain. Furthermore, a Taylor‐inspired feature filter for mitigating background disturbance and extracting fine‐grained features is proposed. Moreover, a part‐soft triplet loss, which is robust to non‐discriminative body partial features is also designed. Experimental results on Occluded‐Duke, Occluded‐Reid, Market‐1501, and Duke‐MTMC datasets show that this method outperforms existing state‐of‐the‐art methods. The code is available at: https://github.com/ISCLab‐Bistu/SA‐ReID.

CycleVTON: A Cycle Mapping Framework for Parser-Free Virtual Try-On

Image-based virtual try-on aims to transfer a target clothing onto a specific person. A significant challenge is arbitrarily matched clothing and person lack corresponding ground truth to supervised learning. A recent pioneering work leveraged an improved cycleGAN to enable one network to generate the desired image for another network during training. However, there is no difference in the result distribution before and after the clothing changes. Therefore, using two different networks is unnecessary and may even increase the difficulty of convergence. Furthermore, the introduced human parsing used to provide body structure information in the input also have a negative impact on the try-on result. How to employ a single network for supervised learning while eliminating human parsing? To tackle these issues, we present a Cycle mapping Virtual Try-On Network (CycleVTON), which can produce photo-realistic try-on results by using a cycle mapping framework without the parser. In particular, we introduce a flow constraint loss to achieve supervised learning of arbitrarily matched clothing and person as inputs to the deformer, thus naturally mimicking the interaction between clothing and the human body. Additionally, we design a skin generation strategy that can adapt to the shape of the target clothing by dynamically adjusting the skin region, i.e., by first removing and then filling skin areas. Extensive experiments conducted on challenging benchmarks demonstrate that our proposed method exhibits superior performance compared to state-of-the-art methods.

Deep Learning Technique for Human Parsing: A Survey and Outlook

Deep Learning Technique for Human Parsing: A Survey and Outlook

Prior based Pyramid Residual Clique Network for human body image super-resolution

Recent research in the analysis of human images, such as human parsing and pose estimation, usually requires input images to have a sufficiently high resolution. However, small images of people are commonly encountered in our daily lives, particularly in surveillance applications. This paper aims to ultra-resolve a tiny person image to its high-resolution counterpart by learning effective feature representations and exploiting useful human body prior knowledge. First, we propose the Residual Clique Block (RCB) to fully exploit compact feature representations for image Super-Resolution (SR). Second, a series of RCBs are cascaded in a coarse-to-fine manner to construct the Pyramid Residual Clique Network (PRCN), which simultaneously reconstructs multiple SR results (e.g. 2×, 4×, and 8×) in one feed-forward pass. Third, we utilize the human parsing map as the shape prior, and the high-frequency sub-bands of Uniform Discrete Curvelet Transform (UDCT) as the texture prior to enhance the details of reconstructed human body image. Experimental results demonstrate that our proposed method achieves state-of-the-art performance with superior visual quality and PSNR/SSIM scores. Moreover, we show that our results can considerably enhance the performance of human parsing and pose estimation tasks.

Causality and signalling of garden-path sentences.

Sheaves are mathematical objects that describe the globally compatible data associated with open sets of a topological space. Original examples of sheaves were continuous functions; later they also became powerful tools in algebraic geometry, as well as logic and set theory. More recently, sheaves have been applied to the theory of contextuality in quantum mechanics. Whenever the local data are not necessarily compatible, sheaves are replaced by the simpler setting of presheaves. In previous work, we used presheaves to model lexically ambiguous phrases in natural language and identified the order of their disambiguation. In the work presented here, we model syntactic ambiguities and study a phenomenon in human parsing called garden-pathing. It has been shown that the information-theoretic quantity known as 'surprisal' correlates with human reading times in natural language but fails to do so in garden-path sentences. We compute the degree of signalling in our presheaves using probabilities from the large language model BERT and evaluate predictions on two psycholinguistic datasets. Our degree of signalling outperforms surprisal in two ways: (i) it distinguishes between hard and easy garden-path sentences (with a [Formula: see text]-value [Formula: see text]), whereas existing work could not, (ii) its garden-path effect is larger in one of the datasets (32 ms versus 8.75 ms per word), leading to better prediction accuracies. This article is part of the theme issue 'Quantum contextuality, causality and freedom of choice'.

Reducing vulnerable internal feature correlations to enhance efficient topological structure parsing

Most cropping-and-segmenting pattern parsers typically establish a single metric/scheme to reason diverse inner correlations, resulting in over-general and redundant representations. To make pattern parsing procedure more streamlined and concise, a part-relation scissor network (PRSN) with multi-constrained attention shifters (MCASs) and multi-head attention expectation-maximum routing agreement (MhAEMRA) is proposed on the basic of matrix-based capsule network (CapsNet). MCASs detect and prune fragile part-to-whole correlations from the perspectives of inter-part diversity and intra-object cohesion. They stipulate that only those primary entities fulfilling the criteria of inter-part diversity and intra-object cohesiveness can update senior entities. MhAEMRA is defined to shield the redundant capsule voting signals. PRSN gradually parses objective semantic patterns by clustering highly associated secondary entities in a bottom-up “part backtracking” manner. Quantitative and ablation experiments surrounding face and human parsing tasks demonstrate the superiority of PRSN over the state-of-the-arts, especially for the definition of fine-grained semantic boundaries.

Toward Accurate Human Parsing Through Edge Guided Diffusion.

Existing human parsing frameworks commonly employ joint learning of semantic edge detection and human parsing to facilitate the localization around boundary regions. Nevertheless, the parsing prediction within the interior of the part contour may still exhibit inconsistencies due to the inherent ambiguity of fine-grained semantics. In contrast, binary edge detection does not suffer from such fine-grained semantic ambiguity, leading to a typical failure case where misclassification occurs inner the part contour while the semantic edge is accurately detected. To address these challenges, we develop a novel diffusion scheme that incorporates guidance from the detected semantic edge to mitigate this problem by propagating corrected classified semantics into the misclassified regions. Building upon this diffusion scheme, we present an Edge Guided Diffusion Network (EGDNet) for human parsing, which can progressively refine the parsing predictions to enhance the accuracy and coherence of human parsing results. Moreover, we design a horizontal-vertical aggregation to exploit inherent correlations among body parts along both the horizontal and vertical axes, which aims at enhancing the initial parsing results. Extensive experimental evaluations on various challenging datasets demonstrate the effectiveness of the proposed EGDNet. Remarkably, our EGDNet shows impressive performances on six benchmark datasets, including four human body parsing datasets (LIP, CIHP, ATR, and PASCAL-Person-Part), and two human face parsing datasets (CelebAMask-HQ and LaPa).

CPI-Parser: Integrating Causal Properties Into Multiple Human Parsing.

Existing methods of multiple human parsing (MHP) apply deep models to learn instance-level representations for segmenting each person into non-overlapped body parts. However, learned representations often contain many spurious correlations that degrade model generalization, leading learned models to be vulnerable to visually contextual variations in images (e.g., unseen image styles/external interventions). To tackle this, we present a causal property integrated parsing model termed CPI-Parser, which is driven by fundamental causal principles involving two causal properties for human parsing (i.e., the causal diversity and the causal invariance). Specifically, we assume that an image is constructed by a mix of causal factors (the characteristics of body parts) and non-causal factors (external contexts), where only the former ones decide the essence of human parsing. Since causal/non-causal factors are unobservable, the proposed CPI-Parser is required to separate key factors that satisfy the causal properties from an image. In this way, the parser is able to rely on causal factors w.r.t relevant evidence rather than non-causal factors w.r.t spurious correlations, thus alleviating model degradation and yielding improved parsing ability. Notably, the CPI-Parser is designed in a flexible way and can be integrated into any existing MHP frameworks. Extensive experiments conducted on three widely used benchmarks demonstrate the effectiveness and generalizability of our method. Code and models are released (https://github.com/HAG-uestc/CPI-Parser) for research purpose.

Dynamic Interaction Dilation for Interactive Human Parsing

Interactive segmentation pursues generating high-quality pixel-level predictions with a few user-provided clicks, which is gaining attention for its convenience in segmentation data annotation. Users are allowed to iteratively refine the prediction by adding clicks until the result is satisfactory. Existing interactive methods usually transform the clicks into a set of localization maps by Euclidian distance computation or RGB texture extraction to guide the segmentation, which makes the click transformation a core module in interactive segmentation networks. However, when adopted in human images where large poses, occlusions, and bad illuminations are prevailing, prior transformation methods tend to cause uncorrectable overlapping across localization maps, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e.</i> , one click corresponds to multiple transformed values at the same position in different localization map channels, which are difficult to form a good match among human parts and limit the interaction efficiency. Furthermore, the inappropriately transformed information is hard to be refined with the static transformation manner, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e.</i> , based on the fixed formulas / RGB textures, which is out of tune with the dynamically refined interaction process. Hence, we design a dynamic transformation scheme for interactive human parsing (IHP) named Dynamic Interaction Dilation Net ( <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">DID-Net</b> ), which serves as an initial attempt to break the limitations of static transformation while capturing long-range dependencies of clicks within each human part. Specifically, we construct a Dynamic Dilation Module ( <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">DD-Module</b> ) to dilate clicks radially in several directions assisted by human body edge detection. The continually refined edges guide to improve the dilation quality in each interaction iteration, thereby better fitting user intention. Furthermore, we propose an Adaptive Interaction Excitation Block ( <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">AIE-Block</b> ) to exploit potential semantic clues buried in the dilated clicks and emphasize semantic expression for each human part by feature recalibration. Our DID-Net achieves state-of-the-art performance on 3 public human parsing benchmarks.

ReSParser: Fully Convolutional Multiple Human Parsing With Representative Sets

Multiple human parsing (MHP) is typically treated as two sub-tasks, i.e., instance separation and body part segmentation. Existing methods usually tackle the sub-tasks by adopting a two-stage strategy, which regards MHP as an ROI-based (i.e., detect-then-segment) or grouping-based (i.e., segment-then-grouping) paradigm. However, the strong dependence between the two sub-tasks limits the potential of an MHP method, since it often requires qualified prior predictions. Besides, isolated models responsible for the two sub-tasks bring a significant computational burden. Unlike existing methods, we regard MHP as a hierarchical set prediction problem and handle two sub-tasks using several landmarks of body parts. Motivated by this, we propose a novel multiple human parser with representative sets, termed ReSParser. In ReSParser, several landmarks of body parts are hierarchically estimated, resulting in coarse-to-fine representative sets. After that, each representative set is adaptively responsible for segmenting pixels into semantically consistent regions belonging to the corresponding person. In such a manner, the ReSParser simultaneously addresses two sub-tasks in a fully convolutional fashion, thus eliminating the dependence between two sub-tasks and significantly alleviating computational complexity. Extensive experiments on two challenging benchmarks demonstrate that our proposed ReSParser is an efficient framework with a superior parsing performance, which significantly outperforms that of other ROI-free yet grouping-free methods. Besides, it achieves competitive results to that of the best two-stage methods such as RP-RCNN, but requires a much lower inference time, showing a good precision-speed trade-off. Code and models are publicly available <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/JosonChan1998/RepParser</uri> . We hope the ReSParser serves as a new baseline for multiple human parsing research in the future.

Prior-structure Driven Weakly-supervised Learning for Fine-grained Human Parsing

Prior-structure Driven Weakly-supervised Learning for Fine-grained Human Parsing