Skeleton Representation Research Articles

Prediction of thrombosis in atrial fibrillation with compact atrial shape representation

Abstract Funding Acknowledgements Type of funding sources: Public grant(s) – EU funding. Main funding source(s): IHU LIRYC ANR-10-IAHU-04 ERACoSysMed PARIS Project Background The impact of LA shape on clot formation in AF is yet to be understood Purpose To analyze the impact of LA, PVs and LAA orientations on clot formation Methods Pts with history of AF and an indication for LAA closure underwent pre-operative cardiac CT. Those with either a prior history of embolism or a clot detected on arterial and venous-enhanced CT images were categorized as clot positive. On CT images, the LA was segmented and meshed. Automated labelling of PVs and LAA was employed to build a compact skeleton representation of the LA (by registering labels from a template to all cases, and by connecting the center of mass of each labels). LA skeletons were used to train a novel Neural Network model for joint classification of heterogeneous data, such as images taken at different phases of the cardiac cycle. Such model consists of one encoder per dataset followed by a common classifier, enforcing a joint representation of all datasets in the same latent space. This is made possible by writing the task as a variational problem to optimise, we call this model a multi-channel variational classifier. The model was trained to predict clot positive patients and its generalizability was assessed on a test population. Results 237 pts were included (age 74±8, 70% males, CHADS-VASc 4.3±1.2). CTs were acquired during systole in 117 and diastole in 120. 100 (42%) patients were considered clot positive. Segmentation, labeling and skeletization was successfully achieved in all pts. The model was trained using a 10-Fold cross validation. We compared the results after training on diastole and systole cases independently and by jointly training on both datasets. At testing, our model reached a 0.72 overall accuracy for the prediction of LAA clots (0.83 for systole and 0.61 for diastole), while training separately on systole and diastole was prone to strong overfitting and mode collapse with 0.52 and 0.48 accuracy, respectively. Conclusion While LA shape, LA size, PV and LAA orientations alone may not be sufficient for robust clot prediction, we introduce a compressed representation of the global anatomy that closely relates to LAA thrombosis, possibly identifying global features related to adverse hemodynamics. The method is reproducible and introduces a novel approach to accommodate heterogeneous diastole and systole datasets.

PO-706-04 COMPACT REPRESENTATION OF ATRIAL SHAPE FOR THROMBOSIS RISK PREDICTION IN AF

The impact of LA shape on clot formation in AF is yet to be understood To analyze the impact of LA, PVs and LAA orientations on clot formation Pts with history of AF and an indication for LAA closure underwent pre-operative cardiac CT. Those with either a prior history of embolism or a clot detected on arterial and venous-enhanced CT images were categorized as clot positive. On CT images, the LA was segmented and meshed. Automated labelling of PVs and LAA was employed to build a compact skeleton representation of the LA (by registering labels from a template to all cases, and by connecting the center of mass of each labels). LA skeletons were used to train a novel Neural Network model for joint classification of heterogeneous data, such as images taken at different phases of the cardiac cycle. Such model consists of one encoder per dataset followed by a common classifier, enforcing a joint representation of all datasets in the same latent space. This is made possible by writing the task as a variational problem to optimise, we call this model a multi-channel variational classifier. The model was trained to predict clot positive patients and its generalizability was assessed on a test population. 237 pts were included (age 74±8, 70% males, CHA₂DS₂-VASc 4.3±1.2). CTs were acquired during systole in 117 and diastole in 120. 100 (42%) patients were considered clot positive. Segmentation, labeling and skeletization was successfully achieved in all pts. The model was trained using a 10-Fold cross validation. We compared the results after training on diastole and systole cases independently andby jointly training on both datasets. At testing, our model reached a 0.72 overall accuracy for the prediction of LAA clots (0.83 for systole and 0.61 for diastole), while training separately on systole anddiastole was prone to strong overfitting and mode collapse with 0.52 and 0.48 accuracy, respectively. While LA shape, LA size, PV and LAA orientations alone may not be sufficient for robust clot prediction, we introduce a compressed representation of the global anatomy that closely relates to LAA thrombosis, possibly identifying global features related to adverse hemodynamics. The method is reproducible andintroduces a novel approach to accommodate heterogeneous diastole and systole datasets.

Graph2Net: Perceptually-Enriched Graph Learning for Skeleton-Based Action Recognition

Skeleton representation has attracted a great deal of attention recently as an extremely robust feature for human action recognition. However, its non-Euclidean structural characteristics raise new challenges for conventional solutions. Recent studies have shown that there is a native superiority in modeling spatiotemporal skeleton information with a Graph Convolutional Network (GCN). Nevertheless, the skeleton graph modeling normally focuses on the physical adjacency of the elements of the human skeleton sequence, which contrasts with the requirement to provide a perceptually meaningful representation. To address this problem, in this paper, we propose a perceptually-enriched graph learning method by introducing innovative features to spatial and temporal skeleton graph modeling. For the spatial information modeling, we incorporate a Local-Global Graph Convolutional Network (LG-GCN) that builds a multifaceted spatial perceptual representation. This helps to overcome the limitations caused by over-reliance on the spatial adjacency relationships in the skeleton. For temporal modeling, we present a Region-Aware Graph Convolutional Network (RA-GCN), which directly embeds the regional relationships conveyed by a skeleton sequence into a temporal graph model. This innovation mitigates the deficiency of the original skeleton graph models. In addition, we strengthened the ability of the proposed channel modeling methods to extract multi-scale representations. These innovations result in a lightweight graph convolutional model, referred to as Graph2Net, that simultaneously extends the spatial and temporal perceptual fields, and thus enhances the capacity of the graph model to represent skeleton sequences. We conduct extensive experiments on NTU-RGB+D 60&120, Northwestern-UCLA, and Kinetics-400 datasets to show that our results surpass the performance of several mainstream methods while limiting the model complexity and computational overhead.

Revisiting -Reciprocal Distance Re-Ranking for Skeleton-Based Person Re-Identification

Person re-identification (re-ID) as a retrieval task often utilizes a re-ranking model to improve performance. Existing re-ranking methods are typically designed for conventional person re-ID with RGB images, while skeleton representation re-ranking for skeleton-based person re-ID still remains to be explored. To fill this gap, we revisit the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$k$</tex-math></inline-formula> -reciprocal distance re-ranking model in this letter, and propose a generic re-ranking method that exploits the salient skeleton features to perform <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$k$</tex-math></inline-formula> -reciprocal distance encoding for skeleton-based person re-ID re-ranking. In particular, we devise the skeleton sequence pooling to aggregate the most salient features of skeletons within a sequence, and combine both original Euclidean distance and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$k$</tex-math></inline-formula> -reciprocal distance to re-rank the skeleton sequence representations for person re-ID. Furthermore, we propose the context-based Rank-1 voting that jointly exploits the initial ranking list and re-ranking list to vote for the top candidate to enhance the Rank-1 matching. Extensive experiments on three public benchmarks demonstrate that our approach can effectively re-rank different state-of-the-art skeleton representations and significantly improve their person re-ID performance.

Skeleton Split Strategies for Spatial Temporal Graph Convolution Networks

Action recognition has been recognized as an activity in which individuals’ behaviour can be observed. Assembling profiles of regular activities such as activities of daily living can support identifying trends in the data during critical events. A skeleton representation of the human body has been proven to be effective for this task. The skeletons are presented in graphs form-like. However, the topology of a graph is not structured like Euclidean-based data. Therefore, a new set of methods to perform the convolution operation upon the skeleton graph is proposed. Our proposal is based on the Spatial Temporal-Graph Convolutional Network (ST-GCN) framework. In this study, we proposed an improved set of label mapping methods for the ST-GCN framework. We introduce three split techniques (full distance split, connection split, and index split) as an alternative approach for the convolution operation. The experiments presented in this study have been trained using two benchmark datasets: NTU-RGB + D and Kinetics to evaluate the performance. Our results indicate that our split techniques outperform the previous partition strategies and are more stable during training without using the edge importance weighting additional training parameter. Therefore, our proposal can provide a more realistic solution for real-time applications centred on daily living recognition systems activities for indoor environments.

An Effectively Dynamic Path Optimization Approach for the Tree Skeleton Extraction from Portable Laser Scanning Point Clouds

One key step to the tree structure study is skeleton processing. Although there are lots of extraction approaches, the existing methods have paid less attention to extraction effectiveness, which highly use redundant points to formulate the skeleton and bring difficulties to the subsequent 3D modeling. This work proposes a four-step framework for the purpose of skeleton extraction. Firstly, candidate skeleton points are filtered from input data based on the spatial slice projection and grouped using the Euclidean distance analysis. Secondly, a key dynamic path optimization step is used to formulate a tree skeleton using the candidate point information. Thirdly, the optimized path is filled by interpolating points to achieve complete skeletons. Finally, short skeletons are removed based on the distance between branching points and ending points, and then, the extraction skeletons are smoothed for improving the visual quality. Our main contribution lies in that we find the global minimization cost path from every point to the root using a novel energy function. The formulated objective function contains a data term to constrain the distance between points and paths, and a smoothness term to constrain the direction continuities. Experimental scenes include three different types of trees, and input point clouds are collected by a portable laser scanning system. Skeleton extraction results demonstrate that we achieved completeness and correctness of 81.10% and 99.21%. respectively. Besides, our effectiveness is up to 79.26%, which uses only 5.82% of the input tree points in the skeleton representation, showing a promising effective solution for the tree skeleton and structure study.

Human skeleton representation for 3D action recognition based on complex network coding and LSTM

3D skeleton sequences contain more effective and discriminative information than RGB video and are more suitable for human action recognition. Accurate extraction of human skeleton information is the key to the high accuracy of action recognition. Considering the correlation between joint points, in this work, we first propose a skeleton feature extraction method based on complex network. The relationship between human skeleton points in each frame is coded as a network. The changes of action over time are described by a time series network composed of skeleton points. Network topology attributes are used as feature vectors, complex network coding and LSTM are combined to recognize human actions. The method was verified on the NTU RGB + D60, MSR Action3D and UTKinect-Action3D dataset, and have achieved good performance, respectively. It shows that the method of extracting skeleton features based on complex network can properlyidentify different actions. This method that considers the temporal information and the relationship between skeletons at the same time plays an important role in the accurate recognition of human actions.

An Action Recognition Method for Volleyball Players Using Deep Learning

This paper investigates the extraction of volleyball players’ skeleton information and provides a deep learning-based solution for recognizing the players’ actions. For this purpose, the convolutional neural network-based approach for recognizing volleyball players’ actions is used. The Lie group skeleton has a large data dimension since it is used to represent the features retrieved from the model. The convolutional neural network is used for feature learning and classification in order to process high-dimensional data, minimize the complexity of the recognition process, and speed up the calculation. This paper uses the Lie group skeleton representation model to extract the geometric feature of the skeleton information in the feature extraction stage and the geometric transformation (rotation and translation) between different limbs to represent the volleyball players’ movements in the feature representation stage. The approach is evaluated using the datasets Florence3D actions, MSR action pairs, and UTKinect action. The average recognition rate of our method is 93.00%, which is higher than that of the existing literature with high attention and reflects better accuracy and robustness.

Combining CNN and LSTM for activity of daily living recognition with a 3D matrix skeleton representation

In socially assistive robotics, human activity recognition plays a central role when the adaptation of the robot behavior to the human one is required. In this paper, we present an activity recognition approach for activities of daily living based on deep learning and skeleton data. In the literature, ad hoc features extraction/selection algorithms with supervised classification methods have been deployed, reaching an excellent classification performance. Here, we propose a deep learning approach, combining CNN and LSTM, that exploits both the learning of spatial dependencies correlating the limbs in a skeleton 3D grid representation and the learning of temporal dependencies from instances with a periodic pattern that works on raw data and so without requiring an explicit feature extraction process. These models are proposed for real-time activity recognition, and they are tested on the CAD-60 dataset. Results show that the proposed model behaves better than an LSTM model thanks to the automatic features extraction of the limbs’ correlation. “New Person” results show that the CNN-LSTM model achieves 95.4% of precision and 94.4% of recall, while the “Have Seen” results are 96.1% of precision and 94.7% of recall.

Les occupations gravettiennes de la Grotte Santa Maria di Agnano (Pouilles, Italie) – Zone SMA-Esterno : typo-technologie lithique et archéozoologie

The Gravettian in Apulia constitutes a singular cultural entity in the context of the prehistoric chrono-cultures of Upper Paleolithic in Italy. This situation can particularly be observed in the Gravettian sequence of the Paglicci cave on Mount Gargano, the lithic series of which show an original development. In the very heart of Apulia, until recently, no field for comparison could be found for the time sequence of the Paglicci cave. But the recent discovery of Gravettian occupation in the stratigraphic sequence of the SMA-external area of Santa Maria di Agnano makes it possible to revisit the status of the Paglicci cave, but also to clarify the specific characters of the Gravettian in Apulia. The lithic industry of SU 9-6 of SMA-Esterno is characterized by the presence of many burins on retouched truncation associated with straight backed points, Gravette or Vachons type. This association reminds us of the assemblage of Paglicci layer 21, attributed to Middle Gravettian. C14 datings for both sites match. Finally, at the base of the Gravettian SMA sequence, SU 11 provided large dihedral burins matching those found Paglicci layers 23-22 attributed to ancient Gravettian. The preliminary zooarchaeological study was carried out on a sample of 2007 remains of large mammals. In total, 816 taxinomic and anatomic remains (NRDt) could be identified, i.e. a 40.7% determination rate. The faunic spectrum consists of 10 taxons, among which 4 species of carnivores (leopard, wolf, fox, badger), 5 ungulates (horse, aurochs, ibex, deer and wild boar) and 1 leporid (hare). Ungulates prevail, as far as remains (98.7% of NRDt) or the minimum number of individuals (82.4% of NMI total) are concerned. Taphonomic analyses indicate that bones were not subject to deep post-depositional modifications. Consequences of weathering are visible on 17% of the cortical surfaces. Non-anthropic biological modifications are scarce: 13% of the material were impacted by roots and 0.7% show the results of carnivorous activity. The hare may be the only species, among herbivores, that was accumulated by a carnivore. The analysis of bone fractures and the traces left by the edge of a lithic tool reveal that the bone material was essentially accumulated and modified by human groups. Hunting is clearly aimed at both Horses (Equus caballus) and Aurochs (Bos primigenius), with occasional Red Deer (Cervus elaphus), Wild Boar (Sus s. scrofa) and Ibex (Capra ibex). The individuals represented are mainly sub-adults and mature adults. The human groups living on the site probably got their meat resources from the plain bordering the Adriatic shore and the vast, more or less flat, plateau overlooking the site. According to the skeleton representations in which the stylopod and zeugopod prevailed, after an initial treatment at the slaughter site, the Gravettians carried the most nutritious parts to the camp. The study of human activities indicates extensive processing of carcasses for their meat and extraction of marrow from long bones. Finally, specific research for fat from the cancellous tissue of the epiphyses of both long and short bones is proposed.

Content-Based Management of Human Motion Data: Survey and Challenges

Digitization of human motion using skeleton representations offers exciting possibilities for a large number of applications but, at the same time, requires innovative techniques for their effective and efficient processing. Content-based processing of skeleton data has developed rapidly in recent years, focusing mainly on specialized prototypes with limited consideration of generic data management possibilities. In this survey article, we synthesize and categorize the existing approaches and outline future research challenges brought by the increasing availability of human motion data. In particular, we first discuss the problems of suitable representation and segmentation of continuous skeleton data obtained from various sources. Then, we concentrate on comparison models for assessing the similarity of time-restricted pieces of motions, as required by any content-based management operation. Next, we review the techniques for evaluating similarity queries over collections of motion sequences and filtering query-relevant parts from continuous motion streams. Finally, we summarize the usability of existing techniques in perspective application domains and discuss the new challenges related to current technological and infrastructural developments. We especially assess the existing techniques from the perspective of scalability and propose future research directions for dealing with large and diverse volumes of skeleton data.

Attention-Based Multilevel Co-Occurrence Graph Convolutional LSTM for 3-D Action Recognition

Action recognition is essential for many human-centered applications in the Internet of Things (IoT). Especially, in the Internet of Medical Things (IoMT), action recognition shows great importance in surgical assistance, patient monitoring, etc. Recently, 3-D skeleton sequence-based action recognition draws broad attention. It is a challenging task that needs effective modeling on intraframe skeleton representations and interframe temporal dynamics. Standard long short-term memory (LSTM)-based models are widely used for sequence modeling due to its long-term memory, yet they are unable to fully model the relationship between different body joints or persons to extract crucial co-occurrence features from different levels. To handle this shortcoming, we propose an attention-based multilevel co-occurrence graph convolutional LSTM (AMCGC-LSTM). By integrating graph convolutional networks (GCNs) into LSTM, the proposed model is capable of leveraging body structural information from skeletons and strengthening the multilevel co-occurrence (MC) feature learning. Specifically, we first design the spatial attention module for feature enhancement of key joints from skeleton inputs. Second, we design MC memory units coupled with GCN to automatically model the spatial relationship between joints, and simultaneously capture the co-occurrence features from different joints, persons, and frames. Finally, we construct aggregated features of MCs (AFMCs) from MC memory units to better represent the intraframe action context encoding, and leverage a concurrent LSTM (Co-LSTM) to further model their temporal dynamics for action recognition. Our model significantly outperforms mainstream methods on NTU RGB+D 60/120 data set, mutual action subset of NTU RGB+D 60/120 data set, and Northewestern-UCLA data set.

Pen Trace Reconstruction with Skeleton Representation of a Handwritten Text Image

This paper presents a new approach for the pen trace reconstruction task in handwritten text images. The proposed approach is based on the skeleton representation of a binary image which provides sufficient information about the pen movement trajectory during the writing. As such, the initial problem can be viewed as a problem of constructing a specific walk in the skeleton graph covering all its edges. The proposed approach consists of splitting the graph into structural elements and estimating the pen movement direction in each of them with the resulting trace being composed of the traces of these elements.

A Mixed-Perception Approach for Safe Human-Robot Collaboration in Industrial Automation.

Digital-enabled manufacturing systems require a high level of automation for fast and low-cost production but should also present flexibility and adaptiveness to varying and dynamic conditions in their environment, including the presence of human beings; however, this presence of workers in the shared workspace with robots decreases the productivity, as the robot is not aware about the human position and intention, which leads to concerns about human safety. This issue is addressed in this work by designing a reliable safety monitoring system for collaborative robots (cobots). The main idea here is to significantly enhance safety using a combination of recognition of human actions using visual perception and at the same time interpreting physical human–robot contact by tactile perception. Two datasets containing contact and vision data are collected by using different volunteers. The action recognition system classifies human actions using the skeleton representation of the latter when entering the shared workspace and the contact detection system distinguishes between intentional and incidental interactions if physical contact between human and cobot takes place. Two different deep learning networks are used for human action recognition and contact detection, which in combination, are expected to lead to the enhancement of human safety and an increase in the level of cobot perception about human intentions. The results show a promising path for future AI-driven solutions in safe and productive human–robot collaboration (HRC) in industrial automation.

On the robustness of skeleton detection against adversarial attacks

Human perception of an object’s skeletal structure is particularly robust to diverse perturbations of shape. This skeleton representation possesses substantial advantages for parts-based and invariant shape encoding, which is essential for object recognition. Multiple deep learning-based skeleton detection models have been proposed, while their robustness to adversarial attacks remains unclear. (1) This paper is the first work to study the robustness of deep learning-based skeleton detection against adversarial attacks, which are only slightly unlike the original data but still imperceptible to humans. We systematically analyze the robustness of skeleton detection models through exhaustive adversarial attacking experiments. (2) We propose a novel Frequency attack, which can directly exploit the regular and interpretable perturbations to sharply disrupt skeleton detection models. Frequency attack consists of an excitatory-inhibition waveform with high frequency attribution, which confuses edge-sensitive convolutional filters due to the sudden contrast between crests and troughs. Our comprehensive results verify that skeleton detection models are also vulnerable to adversarial attacks. The meaningful findings will inspire researchers to explore more potential robust models by involving explicit skeleton features.

Virtual reality framework for editing and exploring medial axis representations of nanometric scale neural structures

We present a novel virtual reality (VR) based framework for the exploratory analysis of nanoscale 3D reconstructions of cellular structures acquired from rodent brain samples through serial electron microscopy. The system is specifically targeted on medial axis representations (skeletons) of branched and tubular structures of cellular shapes, and it is designed for providing to domain scientists: i) effective and fast semi-automatic interfaces for tracing skeletons directly on surface-based representations of cells and structures, ii) fast tools for proofreading, i.e., correcting and editing of semi-automatically constructed skeleton representations, and iii) natural methods for interactive exploration, i.e., measuring, comparing, and analyzing geometric features related to cellular structures based on medial axis representations. Neuroscientists currently use the system for performing morphology studies on sparse reconstructions of glial cells and neurons extracted from a sample of the somatosensory cortex of a juvenile rat. The framework runs in a standard PC and has been tested on two different display and interaction setups: PC-tethered stereoscopic head-mounted display (HMD) with 3D controllers and tracking sensors, and a large display wall with a standard gamepad controller. We report on a user study that we carried out for analyzing user performance on different tasks using these two setups.

S3D-CNN: skeleton-based 3D consecutive-low-pooling neural network for fall detection

Most existing deep-learning-based fall detection methods use either 2D neural network without considering movement representation sequences, or whole sequences instead of only those in the fall period. These characteristics result in inaccurate extraction of human action features and failure to detect falls due to background interferences or activity representation beyond the fall period. To alleviate these problems, a skeleton-based 3D consecutive-low-pooling neural network (S3D-CNN) for fall detection is proposed in this paper. In the S3D-CNN, an activity feature clustering selector is designed to extract the skeleton representation in depth videos using pose estimation algorithm and form optimized skeleton sequence of fall period. A 3D consecutive-low-pooling (3D-CLP) neural network is proposed to process these representation sequences by improving network in terms of layer number, pooling kernel size, and single input frame number. The proposed method is evaluated on public and self-collected datasets respectively, outperforming the existing methods.

Seismic inversion by Newtonian machine learning

We present a wave-equation inversion method that inverts skeletonized seismic data for the subsurface velocity model. The skeletonized representation of the seismic traces consists of the low-rank latent-space variables predicted by a well-trained autoencoder neural network. The input to the autoencoder consists of seismic traces, and the implicit function theorem is used to determine the Fréchet derivative, i.e., the perturbation of the skeletonized data with respect to the velocity perturbation. The gradient is computed by migrating the shifted observed traces weighted by the skeletonized data residual, and the final velocity model is the one that best predicts the observed latent-space parameters. We denote this as inversion by Newtonian machine learning (NML) because it inverts for the model parameters by combining the forward and backward modeling of Newtonian wave propagation with the dimensional reduction capability of machine learning. Empirical results suggest that inversion by NML can sometimes mitigate the cycle-skipping problem of conventional full-waveform inversion (FWI). Numerical tests with synthetic and field data demonstrate the success of NML inversion in recovering a low-wavenumber approximation to the subsurface velocity model. The advantage of this method over other skeletonized data methods is that no manual picking of important features is required because the skeletal data are automatically selected by the autoencoder. The disadvantage is that the inverted velocity model has less resolution compared with the FWI result, but it can serve as a good initial model for FWI. Our most significant contribution is that we provide a general framework for using wave-equation inversion to invert skeletal data generated by any type of neural network. In other words, we have combined the deterministic modeling of Newtonian physics and the pattern matching capabilities of machine learning to invert seismic data by NML.

Cross-Domain Complementary Learning Using Pose for Multi-Person Part Segmentation

Supervised deep learning with pixel-wise training labels has great successes on multi-person part segmentation. However, data labeling at pixel-level is very expensive. To solve the problem, people have been exploring to use synthetic data to avoid the data labeling. Although it is easy to generate labels for synthetic data, the results are much worse compared to those using real data and manual labeling. The degradation of the performance is mainly due to the domain gap, i.e., the discrepancy of the pixel value statistics between real and synthetic data. In this paper, we observe that real and synthetic humans both have a skeleton (pose) representation. We found that the skeletons can effectively bridge the synthetic and real domains during the training. Our proposed approach takes advantage of the rich and realistic variations of the real data and the easily obtainable labels of the synthetic data to learn multi-person part segmentation on real images without any human-annotated labels. Through experiments, we show that without any human labeling, our method performs comparably to several state-of-the-art approaches which require human labeling on Pascal-Person-Parts and COCO-DensePose datasets. On the other hand, if part labels are also available in the real-images during training, our method outperforms the supervised state-of-the-art methods by a large margin. We further demonstrate the generalizability of our method on predicting novel keypoints in real images where no real data labels are available for the novel keypoints detection. Code and pre-trained models are available at https://github.com/kevinlin311tw/CDCL-human-part-segmentation

A multimodal approach for human activity recognition based on skeleton and RGB data

Human action recognition plays a fundamental role in the design of smart solution for home environments, particularly in relation to ambient assisted living applications, where the support of an automated system could improve the quality of life of humans trying to interpret and anticipate user needs, recognizing unusual behaviors or preventing dangerous situations (e.g. falls). In this work the potentialities of the Kinect sensor are fully exploited to design a robust approach for activity recognition combining the analysis of skeleton and RGB data streams. The skeleton representation is designed to capture the most representative body postures, while the temporal evolution of actions is better highlighted by the representation obtained from RGB images. The experimental results confirm that the combination of these two data sources allow to capture highly discriminative features resulting in an approach able to achieve state-of-the-art performance on public benchmarks.