Target Pose Research Articles

Precise region semantics‐assisted GAN for pose‐guided person image generation

AbstractGenerating a realistic person's image from one source pose conditioned on another different target pose is a promising computer vision task. The previous mainstream methods mainly focus on exploring the transformation relationship between the keypoint‐based source pose and the target pose, but rarely investigate the region‐based human semantic information. Some current methods that adopt the parsing map neither consider the precise local pose‐semantic matching issues nor the correspondence between two different poses. In this study, a Region Semantics‐Assisted Generative Adversarial Network (RSA‐GAN) is proposed for the pose‐guided person image generation task. In particular, a regional pose‐guided semantic fusion module is first developed to solve the imprecise match issue between the semantic parsing map from a certain source image and the corresponding keypoints in the source pose. To well align the style of the human in the source image with the target pose, a pose correspondence guided style injection module is designed to learn the correspondence between the source pose and the target pose. In addition, one gated depth‐wise convolutional cross‐attention based style integration module is proposed to distribute the well‐aligned coarse style information together with the precisely matched pose‐guided semantic information towards the target pose. The experimental results indicate that the proposed RSA‐GAN achieves a 23% reduction in LPIPS compared to the method without using the semantic maps and a 6.9% reduction in FID for the method with semantic maps, respectively, and also shows higher realistic qualitative results.

Semi-automatic Framework for Voxel Human Deformation Modeling.

With the advancement of computer and medical imaging technologies, a number of high-resolution, voxel-based, full-body human anatomical models have been developed for medical education, industrial design, and physics simulation studies. However, these models are limited in many applications because they are often only in an upstanding posture. To quickly develop multi-pose human models for different applications. A semi-automatic framework for voxel deformation is proposed in the study. This paper describes a framework for human pose deformation based on three-dimensional (3D) medical images. The voxel model is first converted into a surface model using a surface reconstruction algorithm. Second, a deformation skeleton based on human bones is defined, and the surface model is bound to the skeleton. The bone Glow algorithm is used to assign weights to the surface vertices. Then, the model is deformed to the target posture by using the Smoothed Rotation Enhanced As-Rigid-As-Possible (SR-ARAP) algorithm. Finally, the volume-filling algorithm is applied to refill the tissues into the deformed surface model. The proposed framework is used to deform two standing human models, and the sitting and running models are developed. The results show that the framework can successfully develop the target pose. When compared to the results of the As-Rigid-As-Possible algorithm, SR-ARAP preserves local tissues better. The study proposes a frame for voxel human model deformation and improves the local tissue integrity during deformation.

CaPTURe: Cartoon Pose Transfer Using Reverse Attention

Most of the previous pose transfer methods require additional input data such as joint keypoints of the target pose extracted by a pre-trained network in a human domain. However, in the fields where cartoon characters are used, such as animation or webtoons, the body proportions and structures of the characters often deviate from those of real-life humans. Therefore, it is not appropriate to utilize a pre-trained network designed for the human domain to extract additional data from these cartoon characters. Even if the network is newly trained in the cartoon domain, expensive data labeling is necessary. As a result, most of the previous pose transfer methods are not suitable for application in the cartoon domain. To address these issues, we propose a cartoon pose transfer network named CaPTURe that can successfully perform pose transfer in the cartoon domain with only target images and no other input data. Here, we incorporate the attention mechanism to accurately generate the desired identity. Additionally, we employ reverse attention to enhance the precision of generating the target pose. This approach eliminates the influence of identity information in the pose feature, enabling our network to focus solely on utilizing pose information. Consequently, through comparative experiments using a cartoon domain dataset, CaPTURe shows significant improvements in quantitative results over the previous state-of-the-art methods. Specifically, it achieves a 4.7% reduction in L1 Distance, a 0.1% improvement in SSIM, and a 0.8% enhancement in LPIPS. Moreover, CaPTURe exhibits superior qualitative performance than the previous state-of-the-art methods. In addition, we demonstrate that CaPTURe is capable of achieving effective pose transfer not just in the cartoon domains, but also in the human domains, as evidenced by our experiments on a human domain dataset.

Compliant robotic behaviors for satellite servicing.

The demands of traditional industrial robotics differ significantly from those of space robotics. While industry requires robots that can perform repetitive tasks with precision and speed, the space environment needs robots to cope with uncertainties, dynamics, and communication delays or interruptions, similar to human astronauts. These demands make a well-suited application for compliant robotics and behavior-based programming. Pose Target Wrench Limiting (PTWL) is a compliant behavior paradigm developed specifically to meet these demands. PTWL controls a robot by moving a virtual attractor to a target pose. The attractor applies virtual forces, based on stiffness and damping presets, to an underlying admittance controller. Guided by virtual forces, the robot will follow the attractor until safety conditions are violated or success criteria are met. We tested PTWL on a variety of quasi-static tasks that may be useful for future space operations. Our results demonstrate that PTWL is an extremely powerful tool. It makes teleoperation easy and safe for a wide range of quasi-static tasks. It also facilitates the creation of semi-autonomous state machines that can reliably complete complex tasks with minimal human intervention.

Motion Capture for Sporting Events Based on Graph Convolutional Neural Networks and Single Target Pose Estimation Algorithms

Human pose estimation refers to accurately estimating the position of the human body from a single RGB image and detecting the location of the body. It serves as the basis for several computer vision tasks, such as human tracking, 3D reconstruction, and autonomous driving. Improving the accuracy of pose estimation has significant implications for the advancement of computer vision. This paper addresses the limitations of single-branch networks in pose estimation. It presents a top-down single-target pose estimation approach based on multi-branch self-calibrating networks combined with graph convolutional neural networks. The study focuses on two aspects: human body detection and human body pose estimation. The human body detection is for athletes appearing in sports competitions, followed by human body pose estimation, which is divided into two methods: coordinate regression-based and heatmap test-based. To improve the accuracy of the heatmap test, the high-resolution feature map output from HRNet is used for deconvolution to improve the accuracy of single-target pose estimation recognition.

End-to-end active non-cooperative target tracking of free-floating space manipulators

Active non-cooperative target tracking is an important ability for free-floating space manipulator (FFSM) systems. Aiming at the difficulty of pose and posture estimation of non-cooperative targets and the complexity of FFSM planning and control, the paper proposed an end-to-end target tracking algorithm using deep reinforcement learning (DRL) algorithm. The algorithm uses raw camera images as input and directly outputs the manipulator joint velocities. To deal with the partially observable Markov decision processes (POMDP) problem caused by single image input, we combine the soft actor–critic algorithm with recurrent neural networks. Therefore, the velocity information of the target is introduced to transform the task into a fully observable Markov decision-making process (MDP). Our approach is completely data-driven. It avoids the position and posture estimation of non-cooperative target and the complex modeling motion planning process of FFSM. We trained and tested the algorithm by building a simulation environment in CoppeliaSim. The results showed that our approach achieved good results in active non-cooperative target tracking of FFSM systems and demonstrated the great potential of DRL algorithms in solving space tasks.

A pose estimation method of space non-cooperative target based on ORBFPFH SLAM

In this paper, to improve the pose measurement accuracy of the Time-of-Flight (ToF) camera for space non-cooperative targets, a pose estimation method based on ORBFPFH Simultaneous Localization and Mapping (SLAM) with effective integration of strength and depth measurement information is proposed. The primary process of the method is as follows: First, a unique ORBFPFH Bag of Words (BoW) model for space targets is trained based on the ToF image dataset of spacecraft; Secondly, the pose of the non-cooperative space target is tracked based on the ORBFPFH feature and optimized using the pose graph; Then, based on the ORBFPFH BoW model, loop closure is detected to reduce the cumulative error. Finally, the proposed method is tested on the ToF image dataset: compared with the advanced ORB-SLAM2 algorithm, the proposed method has a minor translation error and rotation error on the test data, with a mean translation error less than 0.144 m and mean rotation error less than 0.642°. The test results show that the proposed method can improve the pose estimation accuracy of space non-cooperative targets, achieve a better 3D point cloud reconstruction effect, and provide technical support for space applications such as rendezvous and docking of non-cooperative spacecraft. © 2001 Elsevier Science. All rights reserved

Optimization of working position and posture of a 5-DOF hybrid automatic drilling system based on an improved GA-BP neural network

ABSTRACT After compensation for positioning error, such errors become different in multipose space. To reduce the positioning error of a 5-degree-of-freedom automatic drilling system, an optimization algorithm is used to optimize the drilling position and posture. The forwards kinematics of the automatic drilling system were determined by a vector cross product. Combined with structural deformation calculation, the workspace of the drilling system was analysed. The pose errors in multipose space were calculated. A BP neural network was used to establish the mapping between the target pose and the pose error after error compensation, and the improved genetic algorithm was used to optimize the working pose. To verify the robustness of the proposed method, drilling comparison experiments were carried out. After optimization, the maximum position and posture error of the parallel posture alignment mechanism were reduced by 71.11% and 67.57%, respectively. The maximum position and normal error of the automatic drilling system were reduced by 54.37% and 19.64%, respectively. Drilling experiments show that the proposed algorithm can meet the accuracy requirements of aircraft assembly fields. The algorithm can be used for working position and posture optimization of a mechanism equipped with posture alignment and an end effector module.

AROS: Affordance Recognition with One-Shot Human Stances.

We present Affordance Recognition with One-Shot Human Stances (AROS), a one-shot learning approach that uses an explicit representation of interactions between highly articulated human poses and 3D scenes. The approach is one-shot since it does not require iterative training or retraining to add new affordance instances. Furthermore, only one or a small handful of examples of the target pose are needed to describe the interactions. Given a 3D mesh of a previously unseen scene, we can predict affordance locations that support the interactions and generate corresponding articulated 3D human bodies around them. We evaluate the performance of our approach on three public datasets of scanned real environments with varied degrees of noise. Through rigorous statistical analysis of crowdsourced evaluations, our results show that our one-shot approach is preferred up to 80% of the time over data-intensive baselines.

Machine learning-based framework for optimally solving the analytical inverse kinematics for redundant manipulators

Solving the analytical inverse kinematics (IK) of redundant manipulators in real time is a difficult problem in robotics since its solution for a given target pose is not unique. Moreover, choosing the optimal IK solution with respect to application-specific demands helps to improve the robustness and to increase the success rate when driving the manipulator from its current configuration towards a desired pose. This is necessary, especially in high-dynamic tasks like catching objects in mid-flights. To compute a suitable target configuration in the joint space for a given target pose in the trajectory planning context, various factors such as travel time or manipulability must be considered. However, these factors increase the complexity of the overall problem which impedes real-time implementation. In this paper, a real-time framework to compute the analytical inverse kinematics of a redundant robot is presented. To this end, the analytical IK of the redundant manipulator is parameterized by so-called redundancy parameters, which are combined with a target pose to yield a unique IK solution. Most existing works in the literature either try to approximate the direct mapping from the desired pose of the manipulator to the solution of the IK or cluster the entire workspace to find IK solutions. In contrast, the proposed framework directly learns these redundancy parameters by using a neural network (NN) that provides the optimal IK solution with respect to the manipulability and the closeness to the current robot configuration. Monte Carlo simulations show the effectiveness of the proposed approach which is accurate and real-time capable ($\approx$ \SI{32}{\micro\second}) on the KUKA LBR iiwa 14 R820.

Аналіз стану дослідження проблеми визначення параметрів відносного положення об'єктів орбітального сервісу

Recently considerable attention has been paid to the problem of estimating the pose of an on-orbit service object. Determining the pose at a close distance still remains an open line of research, especially for non-cooperative objects (targets) of on-orbit service. The goal of this work is to overview the state of the art in the problem of determining the relative motion parameters of on-orbit service objects with emphasis on close proximity operations with non-cooperative and unknown targets. The method employed is the analysis of publications devoted to this problem over the last decade. The analysis showed the following. Determining the pose of a non-cooperative orbital object using video systems is a classical approach due to the advantages of light weight and low power consumption. Video camera based pose estimation algorithms usually require prior knowledge of the target features. The main methods of pose estimation still involve approaches based on the recognition and correspondence of image features for consecutive frames or with a target model. Another major approach to pose determination is lidar navigation, where the recognition and correspondence of features based on lidar-derived target surface point clouds are also common methods. Recently, a trend has emerged towards the development of non-feature methods for target pose determination, including unknown targets. The three-dimensional nature of lidar point cloud data is favorable for target pose estimation without any target model. As to the applicability of target pose estimation methods to an unknown target, the implementation of the obvious approach based on constructing a three-dimensional model of the target by processing a series of its images prior to estimating its spatial motion takes a lot of time, which is critical in close proximity operations. The trend in target pose estimation is the development of methods for simultaneous estimation of the pose and shape of an unknown object. In general, the case of an unknown object has not yet been fully investigated.

Multi-Body 3D-2D Registration for Robot-Assisted Joint Reduction: Preclinical Evaluation in the Ankle Syndesmosis.

Existing methods to improve the accuracy of tibiofibular joint reduction present workflow challenges, high radiation exposure, and a lack of accuracy and precision, leading to poor surgical outcomes. To address these limitations, we propose a method to perform robot-assisted joint reduction using intraoperative imaging to align the dislocated fibula to a target pose relative to the tibia. The approach (1) localizes the robot via 3D-2D registration of a custom plate adapter attached to its end effector, (2) localizes the tibia and fibula using multi-body 3D-2D registration, and (3) drives the robot to reduce the dislocated fibula according to the target plan. The custom robot adapter was designed to interface directly with the fibular plate while presenting radiographic features to aid registration. Registration accuracy was evaluated on a cadaveric ankle specimen, and the feasibility of robotic guidance was assessed by manipulating a dislocated fibula in a cadaver ankle. Using standard AP and mortise radiographic views registration errors were measured to be less than 1 mm and 1° for the robot adapter and the ankle bones. Experiments in a cadaveric specimen revealed up to 4 mm deviations from the intended path, which was reduced to <2 mm using corrective actions guided by intraoperative imaging and 3D-2D registration. Preclinical studies suggest that significant robot flex and tibial motion occur during fibula manipulation, motivating the use of the proposed method to dynamically correct the robot trajectory. Accurate robot registration was achieved via the use of fiducials embedded within the custom design. Future work will evaluate the approach on a custom radiolucent robot design currently under construction and verify the solution on additional cadaveric specimens.

Six-Dimensional Target Pose Estimation for Robot Autonomous Manipulation: Methodology and Verification

The autonomous and precise grasping operation of robots is considered challenging in situations where there are different objects with different shapes and postures. In this study, we proposed a method of 6-D target pose estimation for robot autonomous manipulation. The proposed method is based on: 1) a fully convolutional neural network for scene semantic segmentation and 2) fast global registration to achieve target pose estimation. To verify the validity of the proposed algorithm, we built a robot grasping operation system and used the point cloud model of the target object and its pose estimation results to generate the robot grasping posture control strategy. Experimental results showed that the proposed method can achieve a six-degree-of-freedom pose estimation for arbitrarily placed target objects and complete the autonomous grasping of the target. Comparative experiments demonstrated that the proposed target pose estimation method achieved a significant improvement in average accuracy and real-time performance compared with traditional methods.

Fully Automatic Visual Servoing Control for Underwater Vehicle Manipulator Systems Based on a Heuristic Inverse Kinematics

The use of underwater vehicle manipulator systems (UVMS) equipped with cameras has gained significant attention due to their capacity to perform underwater tasks autonomously. However, controlling both the manipulator and the remotely operated vehicle (ROV) based on the vision system information is not an easy task, especially in situations where the vehicle cannot be parked/held stationary. Most of the existing approaches work based on complex matrix calculations for the inverse kinematics (IK), which can lead to high computational costs and the need to deal with singularity problems. A problem arises when the amount of time needed to calculate the UVMS configuration can result in reduced frequency of target pose estimation, beyond the point where the target has moved out of the camera field of view. Therefore, this paper proposes an autonomous visual servoing approach for UVMS, including an extension of a heuristic technique named M-FABRIK (Mobile - Forward and Backward Reaching IK) to calculate the UVMS inverse kinematics in a simple and fast way. This approach aims to control both the configuration of the manipulator and ROV position in order to allow underwater intervention in situations where the ROV cannot be parked/held stationary. This solution allows the vehicle to be positioned according to additional criteria, besides avoiding matrix inversion and being robust to singularities. Trials have been performed with a manipulator mounted on a work-class ROV for an autonomous underwater monitoring task and results demonstrate a simple and fast approach, which is able to set the configuration of the manipulator as well as the ROV for visual servoing applications in real-time, such as for monitoring, tracking and intervention tasks underwater.

High-Fidelity Face Reenactment Via Identity-Matched Correspondence Learning

Face reenactment aims to generate an animation of a source face using the poses and expressions from a target face. Although recent methods have made remarkable progress by exploiting generative adversarial networks, they are limited in generating high-fidelity and identity-preserving results due to the inappropriate driving information and insufficiently effective animating strategies. In this work, we propose a novel face reenactment framework that achieves both high-fidelity generation and identity preservation. Instead of sparse face representations (e.g., facial landmarks and keypoints), we utilize the Projected Normalized Coordinate Code (PNCC) to better preserve facial details. We propose to reconstruct the PNCC with the source identity parameters and the target pose and expression parameters estimated by 3D face reconstruction to factor out the target identity. By adopting the reconstructed representation as the driving information, we address the problem of identity mismatch. To effectively utilize the driving information, we establish the correspondence between the reconstructed representation and the source representation based on the features extracted by an encoder network. This identity-matched correspondence is then utilized to animate the source face using a novel feature transformation strategy. The generator network is further enhanced by the proposed geometry-aware skip connection. Once trained, our model can be applied to previously unseen faces without further training or fine-tuning. Through extensive experiments, we demonstrate the effectiveness of our method in face reenactment and show that our model outperforms state-of-the-art approaches both qualitatively and quantitatively. Additionally, the proposed PNCC reconstruction module can be easily inserted into other methods and improve their performance in cross-identity face reenactment.

Input-Constrained Hybrid Control of a Hyper-Redundant Mobile Medical Manipulator.

To reduce the risk of infection in medical personnel working in infectious-disease areas, we proposed a hyper-redundant mobile medical manipulator (HRMMM) to perform contact tasks in place of healthcare workers. A kinematics-based tracking algorithm was designed to obtain highly accurate pose tracking. A kinematic model of the HRMMM was established and its global Jacobian matrix was deduced. An expression of the tracking error based on the Rodrigues rotation formula was designed, and the relationship between tracking errors and gripper velocities was derived to ensure accurate object tracking. Considering the input constraints of the physical system, a joint-constraint model of the HRMMM was established, and the variable-substitution method was used to transform asymmetric constraints to symmetric constraints. All constraints were normalized by dividing by their maximum values. A hybrid controller based on pseudo-inverse (PI) and quadratic programming (QP) was designed to satisfy the real-time motion-control requirements in medical events. The PI method was used when there was no input saturation, and the QP method was used when saturation occurred. A quadratic performance index was designed to ensure smooth switching between PI and QP. The simulation results showed that the HRMMM could approach the target pose with a smooth motion trajectory, while meeting different types of input constraints.

Pose- and Attribute-consistent Person Image Synthesis

Person Image Synthesis aims at transferring the appearance of the source person image into a target pose. Existing methods cannot handle large pose variations and therefore suffer from two critical problems: (1) synthesis distortion due to the entanglement of pose and appearance information among different body components and (2) failure in preserving original semantics (e.g., the same outfit). In this article, we explicitly address these two problems by proposing a Pose- and Attribute-consistent Person Image Synthesis Network (PAC-GAN). To reduce pose and appearance matching ambiguity, we propose a component-wise transferring model consisting of two stages. The former stage focuses only on synthesizing target poses, while the latter renders target appearances by explicitly transferring the appearance information from the source image to the target image in a component-wise manner. In this way, source-target matching ambiguity is eliminated due to the component-wise disentanglement of pose and appearance synthesis. Second, to maintain attribute consistency, we represent the input image as an attribute vector and impose a high-level semantic constraint using this vector to regularize the target synthesis. Extensive experimental results on the DeepFashion dataset demonstrate the superiority of our method over the state of the art, especially for maintaining pose and attribute consistencies under large pose variations.

High-precision pose measurement method based on binocular vision in dark lighting environments

Measuring the pose of non-cooperative targets in space is a critical supporting technology for cleaning up space debris and recovering items. However, most existing methods are simulation experiments conducted in good lighting environments and tend to show poor performance in dark lighting environments. A target pose measurement method based on binocular vision is proposed, which is suitable for dark lighting environments. First, the traditional features from accelerated segment test algorithm are improved to reduce the influence of illumination on the performance of feature point extraction under various postures. The point feature and line feature are combined to extract image features more easily in a dark lighting environment while retaining the high accuracy of the pose measurement algorithm based on point features. Second, the normalized cross-correlation coefficient matching method is combined with the epipolar constraint to narrow the search range of the matching points from the two-dimensional plane to the epipolar line, which substantially improves the matching efficiency and accuracy of the matching algorithm. Finally, post-processing through feature matching is performed to reduce the probability of mismatches. Simulation and physical experiment results show that our method can stably extract features and obtain high-precision target pose information in well-illuminated as well as dark lighting environments, making it suitable for high-precision target pose measurement under insufficient illumination.

A novel Multi-scale architecture driven by decoupled semantic attention transfer for person image generation

Person image generation is a challenging task aimed to transfer the person of the source image from a source pose to a target pose while preserving its style. In this paper, we proposed a Generative Adversarial Network based on Decoupled Semantic Attention Transfer (DSAT-GAN), focusing on that local semantic representations of different image styles and contents cannot be accurately decoupled and transferred. This architecture employs a novel Multi-scale Semantic Mapping Generation Network (Ms-SMGN), driven by two network modules with different semantic attention mechanism, aiming to accurately align and transfer the representations of local semantics at different spatial scales. Then, a channel-separated convolution is applied in the encoding networks instead of the traditional channel fully-connected operation, which reduces computational complexity while realizing channel semantic decoupling. Moreover, a Gram matrix-based global style loss is introduced to further enhance the consistency of high-level semantic between generated and target images. Experiments on Market-1501 and DeepFashion datasets show that DSAT-GAN has superior performance compared with other recent baselines. Additionally, this architecture can be extended to the data enhancement scenes to significantly improve the accuracy of person Re-identification.

Target Recognition in SAR Images by Deep Learning with Training Data Augmentation.

Mass production of high-quality synthetic SAR training imagery is essential for boosting the performance of deep-learning (DL)-based SAR automatic target recognition (ATR) algorithms in an open-world environment. To address this problem, we exploit both the widely used Moving and Stationary Target Acquisition and Recognition (MSTAR) SAR dataset and the Synthetic and Measured Paired Labeled Experiment (SAMPLE) dataset, which consists of selected samples from the MSTAR dataset and their computer-generated synthetic counterparts. A series of data augmentation experiments are carried out. First, the sparsity of the scattering centers of the targets is exploited for new target pose synthesis. Additionally, training data with various clutter backgrounds are synthesized via clutter transfer, so that the neural networks are better prepared to cope with background changes in the test samples. To effectively augment the synthetic SAR imagery in the SAMPLE dataset, a novel contrast-based data augmentation technique is proposed. To improve the robustness of neural networks against out-of-distribution (OOD) samples, the SAR images of ground military vehicles collected by the self-developed MiniSAR system are used as the training data for the adversarial outlier exposure procedure. Simulation results show that the proposed data augmentation methods are effective in improving both the target classification accuracy and the OOD detection performance. The purpose of this work is to establish the foundation for large-scale, open-field implementation of DL-based SAR-ATR systems, which is not only of great value in the sense of theoretical research, but is also potentially meaningful in the aspect of military application.