Pose Changes Research Articles

Bipartite Graph Reasoning GANs for Person Pose and Facial Image Synthesis

We present a novel bipartite graph reasoning Generative Adversarial Network (BiGraphGAN) for two challenging tasks: person pose and facial image synthesis. The proposed graph generator consists of two novel blocks that aim to model the pose-to-pose and pose-to-image relations, respectively. Specifically, the proposed bipartite graph reasoning (BGR) block aims to reason the long-range cross relations between the source and target pose in a bipartite graph, which mitigates some of the challenges caused by pose deformation. Moreover, we propose a new interaction-and-aggregation (IA) block to effectively update and enhance the feature representation capability of both a person’s shape and appearance in an interactive way. To further capture the change in pose of each part more precisely, we propose a novel part-aware bipartite graph reasoning (PBGR) block to decompose the task of reasoning the global structure transformation with a bipartite graph into learning different local transformations for different semantic body/face parts. Experiments on two challenging generation tasks with three public datasets demonstrate the effectiveness of the proposed methods in terms of objective quantitative scores and subjective visual realness. The source code and trained models are available at https://github.com/Ha0Tang/BiGraphGAN.

MonoEF: Extrinsic Parameter Free Monocular 3D Object Detection

Monocular 3D object detection is an important task in autonomous driving. It can be easily intractable where there exists ego-car pose change w.r.t. ground plane. This is common due to the slight fluctuation of road smoothness and slope. Due to the lack of insight in industrial application, existing methods on open datasets neglect the camera pose information, which inevitably results in the detector being susceptible to camera extrinsic parameters. The perturbation of objects is very popular in most autonomous driving cases for industrial products. To this end, we propose a novel method to capture camera pose to formulate the detector free from extrinsic perturbation. Specifically, the proposed framework predicts camera extrinsic parameters by detecting vanishing point and horizon change. A converter is designed to rectify perturbative features in the latent space. By doing so, our 3D detector works independent of the extrinsic parameter variations and produces accurate results in realistic cases, e.g., potholed and uneven roads, where almost all existing monocular detectors fail to handle. Experiments demonstrate our method yields the best performance compared with the other state-of-the-arts by a large margin on both KITTI 3D and nuScenes datasets.

LiDARTouch: Monocular metric depth estimation with a few-beam LiDAR

Vision-based depth estimation is a key feature in autonomous systems, which often relies on a single camera or several independent ones. In such a monocular setup, dense depth is obtained with either additional input from one or several expensive LiDARs, e.g., with 64 beams, or camera-only methods, which suffer from scale-ambiguity and infinite-depth problems. In this paper, we propose a new alternative of densely estimating metric depth by combining a monocular camera with a light-weight LiDAR, e.g., with 4 beams, typical of today’s automotive-grade mass-produced laser scanners. Inspired by recent self-supervised methods, we introduce a novel framework, called LiDARTouch, to estimate dense depth maps from monocular images with the help of “touches” of LiDAR, i.e., without the need for dense ground-truth depth. In our setup, the minimal LiDAR input contributes on three different levels: as an additional model’s input, in a self-supervised LiDAR reconstruction objective function, and to estimate changes of pose (a key component of self-supervised depth estimation architectures). Our LiDARTouch framework achieves new state of the art in self-supervised depth estimation on the KITTI dataset, thus supporting our choices of integrating the very sparse LiDAR signal with other visual features. Moreover, we show that the use of a few-beam LiDAR alleviates scale ambiguity and infinite-depth issues that camera-only methods suffer from. We also demonstrate that methods from the fully-supervised depth-completion literature can be adapted to a self-supervised regime with a minimal LiDAR signal.

Hue-Preserving Color Transforms for LED Wall Virtual Production Workflows

Virtual production using light-emitting diode (LED) wall display technology is gaining popularity in the entertainment industry to produce motion pictures, episodic television, live broadcast, and esports content. This new paradigm typically uses one or more large LED displays that contain millions of individual LEDs that are used to display a virtual background and/or foreground objects from a virtual scene that is simultaneously captured by the digital photographic camera on set, resulting in the so-called “in-camera visual effects.” In a virtual production workflow, digital cameras capture the actors and objects on the set in the foreground while simultaneously capturing the image shown on the LED wall behind the actors. Realtime video game rendering engines update the image shown on the LED wall to compensate for changes in camera location, camera pose, and lens focal length. Modern digital camera workflows typically include color transforms that were not designed to accurately render the large areas of saturated colors that are possible when capturing the image shown on an LED wall. Some examples of a hue shift that can occur in typical workflows are blue to cyan, red to pink, red to orange, and green to yellow. We found that these hue shifts can occur dynamically while racking focus to and from the LED wall and also statically when the LED wall is kept out of focus, which is a common technique that is used to minimize moiré artifacts. This article explores a simple modification of these common color transforms to preserve the hue of the scene while also creating a similar look to the existing color transforms.

Graph convolutional network with triplet attention learning for person re-identification

Person re-identification (re-ID) is a method that uses several non-overlapping cameras to identify the same individual. Person re-ID has been employed successfully in a diversity of computer vision applications. This task is made more difficult by occlusions, abrupt illumination, pose changes among camera views, cluttered backgrounds, and inaccurate detections. Therefore, we propose a new graph convolutional network with attention modules. This research reveals a new attention network that encompasses the encoder-decoder and the triplet attention module. The proposed attention module employs the self-attention process to achieve potent and discriminatory features by utilizing temporal, spatial, and channel context information. The triplet attention module is utilized to capture cross-dimension dependencies and pedestrian features, and also reduces the impact of the imperfect pedestrian image to remedy the occlusion issue. The encoder-decoder is used to observe the whole-body shape. Experiments on several publicly available datasets reveal that our method has a high degree of generalization and outperforms existing methods. On Market1501, the proposed method outperformed the recent approaches with an accuracy of 92.98% for rank-1. According to the results, our method ameliorates quantitative and qualitative person re-ID methods.

A siamese network-based approach for vehicle pose estimation.

We propose a deep learning-based vehicle pose estimation method based on a monocular camera called FPN PoseEstimateNet. The FPN PoseEstimateNet consists of a feature extractor and a pose calculate network. The feature extractor is based on Siamese network and a feature pyramid network (FPN) is adopted to deal with feature scales. Through the feature extractor, a correlation matrix between the input images is obtained for feature matching. With the time interval as the label, the feature extractor can be trained independently of the pose calculate network. On the basis of the correlation matrix and the standard matrix, the vehicle pose changes can be predicted by the pose calculate network. Results show that the network runs at a speed of 6 FPS, and the parameter size is 101.6 M. In different sequences, the angle error is within 8.26° and the maximum translation error is within 31.55 m.

Keypoint-Based Disentangled Pose Network for Category-Level 6-D Object Pose Tracking.

Category-level 6-D object pose tracking is very challenging in the field of 3-D computer vision. Keypoint-based object pose estimation has demonstrated its effectiveness in dealing with it. However, current approaches first estimate the keypoints through a neural network and further compute the interframe pose change via least-squares optimization. They estimate rotation and translation in the same way, ignoring the differences between them. In this work, we propose a keypoint-based disentangled pose network, which disentangles the 6-D object pose change to 3-D rotation and 3-D translation. Specifically, the translation is directly estimated by the network and the rotation is indirectly calculated by singular value decomposition according to the keypoints. Extensive experiments on the NOCS-REAL275 dataset demonstrate the superiority of our method.

Hand Activity Recognition From Automatic Estimated Egocentric Skeletons Combining Slow Fast and Graphical Neural Networks

In this paper, we present an unified framework for understanding hand action from the first-person video. The proposed framework composes two main components. The first component estimates three-dimensional (3D) hand joints from RGB images. Two network structures derived from the baseline HopeNet network are proposed: convolutional neural networks (CNNs) which are traditional multi-layer CNN and CNN combining with GraphCNN to perform 3D hand pose estimation, without the use of GraphUNet as in baseline HopeNet method. The second component of the framework recognizes hand action from skeleton stream. We first deploy two recent advanced neuronal networks that are PA-ResGCN and Double-feature Double-motion (DDNet). To focus more on the hand pose changes, we improve DDNet with two normalization strategies of the hand joints. Finally, we fuse PA-ResGCN with our improved DDNet to still boost the recognition performance. We evaluate our proposed methods on First-Person Hand Action Benchmark dataset. Experiments show that our model for 3D hand joints estimation achieves the best precision (36.6 mm). Our hand joint normalization strategies improve the original DDNet from 0.71% to 4.05% of accuracy with the ground-truth hand pose while the improvement is significantly larger (from 2.96% to 10.98%) with the estimated hand pose. The late fusion schemes outperform different state-of-the-art methods for the hand action recognition with the highest accuracy of 86.67%. These experimental results show potential and extendable possibilities for developing practical first-person vision applications.

Industrial serial robot calibration considering geometric and deformation errors

The absolute accuracy of the industrial serial robot is affected by the geometric errors from machining and assembling, and the elastic deformation errors from the large payload and flexible joints. The inherent features and correlations of both geometric and deformation errors have not been thoroughly discussed, leading to the unsatisfactory calibration results. In this paper, the geometric and deformation propagations are separately deduced and then combined to form a complete model having both types of errors. Geometric errors, i.e. the joint twist errors and initial transformation errors, are described at the initial pose and remain the same during the task execution of the robot. But the deformation errors are evaluated at the current pose and change values versus the change of poses. Based on the features of both errors, it is summarized that the deflections of joints are independent from the geometric errors and would not affect the geometric error propagation. Then, the redundancy within the geometric errors is proved and the singularity between two types of errors are discussed. A Generalized Cross-Validation method is adopted to solve the ill-conditioning problem caused by the different unit of the identified parameters. The simultaneous identification of both types of errors are implemented. Finally, a step-by-step compensation is proposed for a convenient error correction. A UR3 robot is taken as an example to illustrate and verify the proposed calibration method. The mean positioning absolute accuracy is 0.4672 mm after calibration. Comparisons with the calibration of only geometric errors indicates the proposed calibration method leads to higher absolute accuracy.

Improved Graph Convolutional Neural Network for Dance Tracking and Pose Estimation.

Movement recognition technology is widely used in various practical application scenarios, but there are few researches on dance movement recognition at present. Aiming at the problem of low accuracy of dance movement recognition due to complex pose changes in dance movements, this paper designed an improved graph convolutional neural network algorithm for dance tracking and pose estimation. In this method, the spatial and temporal characteristics of motion are extracted from the skeleton joint diagram of human body. Then, GCN (graph convolutional neural) is used to extract potential spatial information between skeleton nodes. Finally, LSTM (long short-term memory) extracts the time series features before and after human actions as a supplement and performs late fusion of the prediction outputs of the two networks, respectively, to improve the problem of insufficient generalization ability of single network. The experimental results show that this method can effectively improve the accuracy of dance movement recognition in general movement recognition data set and dance pose data set. It has certain application value in dance self-help teaching, professional dancer movement correction, and other application scenarios.

Multitask, Multilabel, and Multidomain Learning With Convolutional Networks for Emotion Recognition.

Automated emotion recognition in the wild from facial images remains a challenging problem. Although recent advances in deep learning have assumed a significant breakthrough in this topic, strong changes in pose, orientation, and point of view severely harm current approaches. In addition, the acquisition of labeled datasets is costly and the current state-of-the-art deep learning algorithms cannot model all the aforementioned difficulties. In this article, we propose applying a multitask learning loss function to share a common feature representation with other related tasks. Particularly, we show that emotion recognition benefits from jointly learning a model with a detector of facial action units (collective muscle movements). The proposed loss function addresses the problem of learning multiple tasks with heterogeneously labeled data, improving previous multitask approaches. We validate the proposal using three datasets acquired in noncontrolled environments, and an application to predict compound facial emotion expressions.

Attention-Mechanism-Based Real-Time Gaze Tracking in Natural Scenes With Residual Blocks

Gaze tracking is widely used in fatigue driving detection, eye disease diagnosis, mental illness diagnosis, website or advertising design, virtual reality, gaze-control devices, and human–computer interaction. However, the influence of light, specular reflection, and occlusion, and the change of head pose, especially the ever-changing human pose in natural scenes, have brought great challenges to the accurate gaze tracking. In this article, gaze tracking in natural scenes is studied, and a method based on the convolutional neural network with residual blocks is proposed, in which the attention mechanism is integrated into the network to improve the accuracy of gaze tracking. Furthermore, it is tested on the GazeFollow database, which contains six kinds of databases. The results show that the performance of the proposed method outperforms that of the other state-of-the-art methods in natural scenes. Moreover, the proposed method has better real-time performance and is more suitable for practical applications.

Research on occlusion block face recognition based on feature point location

AbstractAiming at improving the poor face recognition accuracy under occlusion, a multipose block occlusion face recognition method based on feature point location is proposed. Face segmentation is carried out according to the location results of face feature points and occlusion areas. The local features of each face block are extracted by deep convolution neural network. The dynamic adaptive weighting method is used to give different weights to the face block information, and the occluded face recognition is completed according to the results of face segmentation, which effectively reduces the impact of pose change and occlusion on face recognition. The experiment is analyzed from two aspects: frontal occlusion and multipose face occlusion. The results show that fewer blocks in the frontal occlusion experiment are conducive to maintaining good local integrity and relatively good recognition performance; When the proportion of frontal occlusion reaches 50%, the recognition rate of our algorithm can still be up to 92.68%, which is significantly better than other algorithms.

Automated Smart Attendance System Using Face Recognition

Abstract: In the human body, the face contains many important details, so the face is the most significant factor in identifying a person. There are several main ways to detect the presence of a person. Biometrics for capturing attendance. This is a time consuming process. In this paper, we will use a collection of rules to develop a model that classifies each character's face from the captured images. The HOG algorithm for recording student attendance. HOG (Histogram of Oriented Gradients) is one of the methods used to represent and classify images, is a common and effective method, and was chosen for its robustness against changes in pose and lighting. .. The proposed ASAS (Automated Smart Attendance System) captures images and compares them to the images stored in the databases. The database is updated during student enrollment using an automated process that also includes names and role numbers. Key Terms: HOG, python, ASAS

Self-supervised rigid transformation equivariance for accurate 3D point cloud registration

Transformation equivariance has been widely investigated in 3D point cloud representation learning for more informative descriptors, which formulates the change of the representation with respect to the transformation of the input point clouds explicitly. In this paper, we extend this property to the task of 3D point cloud registration and propose a rigid transformation equivariance (RTE) for accurate 3D point cloud registration. Specifically, RTE formulates the change of the relative pose explicitly with respect to the rigid transformation of the input point clouds. To exploit RTE, we adopt a Siamese structure network with two shared registration branches. One focuses on the input pair of point clouds, and the other one focuses on the new pair achieved by applying two random rigid transformations to the input point clouds respectively. Since the change of the two output relative poses has been predicted according to RTE, a new additional self-supervised loss is obtained to supervise the training. This general network structure can be integrated with most learning-based point cloud registration frameworks easily to improve the performance. Our method adopts the state-of-the-art virtual point-based pipelines as our shared branches, in which we propose a data-driven matching based on learned cost volume (LCV) rather than traditional hand-crafted matching strategies. Experimental evaluations on both synthetic datasets and real datasets validate the effectiveness of our proposed framework. The source code will be made public.

Tracking of rigid head motion during MRI using anEEG system.

PurposeTo demonstrate a novel method for tracking of head movements during MRI using electroencephalography (EEG) hardware for recording signals induced by native imaging gradients.Theory and MethodsGradient switching during simultaneous EEG–fMRI induces distortions in EEG signals, which depend on subject head position and orientation. When EEG electrodes are interconnected with high‐impedance carbon wire loops, the induced voltages are linear combinations of the temporal gradient waveform derivatives. We introduce head tracking based on these signals (CapTrack) involving 3 steps: (1) phantom scanning is used to characterize the target sequence and a fast calibration sequence; (2) a linear relation between changes of induced signals and head pose is established using the calibration sequence; and (3) induced signals recorded during target sequence scanning are used for tracking and retrospective correction of head movement without prolonging the scan time of the target sequence. Performance of CapTrack is compared directly to interleaved navigators.ResultsHead‐pose tracking at 27.5 Hz during echo planar imaging (EPI) was demonstrated with close resemblance to rigid body alignment (mean absolute difference: [0.14 0.38 0.15]‐mm translation, [0.30 0.27 0.22]‐degree rotation). Retrospective correction of 3D gradient‐echo imaging shows an increase of average edge strength of 12%/−0.39% for instructed/uninstructed motion with CapTrack pose estimates, with a tracking interval of 1561 ms and high similarity to interleaved navigator estimates (mean absolute difference: [0.13 0.33 0.12] mm, [0.28 0.15 0.22] degrees).ConclusionMotion can be estimated from recordings of gradient switching with little or no sequence modification, optionally in real time at low computational burden and synchronized to image acquisition, using EEG equipment already found at many research institutions.

FERGCN: facial expression recognition based on graph convolution network.

Due to the problems of occlusion, pose change, illumination change, and image blur in the wild facial expression dataset, it is a challenging computer vision problem to recognize facial expressions in a complex environment. To solve this problem, this paper proposes a deep neural network called facial expression recognition based on graph convolution network (FERGCN), which can effectively extract expression information from the face in a complex environment. The proposed FERGCN includes three essential parts. First, a feature extraction module is designed to obtain the global feature vectors from convolutional neural networks branch with triplet attention and the local feature vectors from key point-guided attention branch. Then, the proposed graph convolutional network uses the correlation between global features and local features to enhance the expression information of the non-occluded part, based on the topology graph of key points. Furthermore, the graph-matching module uses the similarity between images to enhance the network’s ability to distinguish different expressions. Results on public datasets show that our FERGCN can effectively recognize facial expressions in real environment, with RAF-DB of 88.23%, SFEW of 56.15% and AffectNet of 62.03%.

Online Correction of Camera Poses for the Surround-view System: A Sparse Direct Approach

The surround-view module is an indispensable component of a modern advanced driving assistance system. By calibrating the intrinsics and extrinsics of the surround-view cameras accurately, a top-down surround-view can be generated from raw fisheye images. However, poses of these cameras sometimes may change. At present, how to correct poses of cameras in a surround-view system online without re-calibration is still an open issue. To settle this problem, we introduce the sparse direct framework and propose a novel optimization scheme of a cascade structure. This scheme is actually composed of two levels of optimization and two corresponding photometric error based models are proposed. The model for the first-level optimization is called the ground model, as its photometric errors are measured on the ground plane. For the second level of the optimization, it’s based on the so-called ground-camera model, in which photometric errors are computed on the imaging planes. With these models, the pose correction task is formulated as a nonlinear least-squares problem to minimize photometric errors in overlapping regions of adjacent bird’s-eye-view images. With a cascade structure of these two levels of optimization, an appropriate balance between the speed and the accuracy can be achieved. Experiments show that our method can effectively eliminate the misalignment caused by cameras’ moderate pose changes in the surround-view system. Source code and test cases are available online at https://cslinzhang.github.io/CamPoseCorrection/ .

The devil is in the face: Exploiting harmonious representations for facial expression recognition

Despite the recent effort from computer vision community, facial expression recognition (FER) remains a largely unsolved problem. This is because the appearance of people’s face undergoes dramatic changes due to changes in view angle, pose, illumination plus ambiguous facial expressions and low-quality facial images. In this work, we show the advantage of feature representation learning by dynamically graph message propagating subject to FER discriminative learning constraints and minimizing the distance of expression-agnostic transformed instance feature pairs. Specifically, we formulate a novel Harmonious Representation Learning (HRL) model for joint learning of landmark-guided graph message propagation, and spatially invariant feature learning using only generic matching metrics. Extensive comparative evaluations demonstrate the superiority of our proposed approach for FER over a variety of state-of-the-art methods on three major benchmark datasets including SFEW 2.0, RAF-DB, and CK+.

A proposed standard for quantifying 3-D hindlimb joint poses in living and extinct archosaurs.

The last common ancestor of birds and crocodylians plus all of its descendants (clade Archosauria) dominated terrestrial Mesozoic ecosystems, giving rise to disparate body plans, sizes, and modes of locomotion. As in the fields of vertebrate morphology and paleontology more generally, studies of archosaur skeletal structure have come to depend on tools for acquiring, measuring, and exploring three‐dimensional (3‐D) digital models. Such models, in turn, form the basis for many analyses of musculoskeletal function. A set of shared conventions for describing 3‐D pose (joint or limb configuration) and 3‐D kinematics (change in pose through time) is essential for fostering comparison of posture/movement among such varied species, as well as for maximizing communication among scientists. Following researchers in human biomechanics, we propose a standard methodological approach for measuring the relative position and orientation of the major segments of the archosaur pelvis and hindlimb in 3‐D. We describe the construction of anatomical and joint coordinate systems using the extant guineafowl and alligator as examples. Our new standards are then applied to three extinct taxa sampled from the wider range of morphological, postural, and kinematic variation that has arisen across >250 million years of archosaur evolution. These proposed conventions, and the founding principles upon which they are based, can also serve as starting points for measuring poses between elements within a hindlimb segment, for establishing coordinate systems in the forelimb and axial skeleton, or for applying our archosaurian system more broadly to different vertebrate clades.