Camera Viewpoint Research Articles

Image segmentation of tunnel water leakage defects in complex environments using an improved Unet model

Computer vision technology provides an intelligent means for detecting tunnel water leakage areas. However, the accuracy of defect feature extraction and segmentation is limited by factors such as insufficient lighting and environmental interference inside tunnels. To address the problem, this paper proposes a tunnel water leakage area segmentation network model called Customized Side Guided-Unet (CSG-Unet), using Unet as the baseline model. The main contributions are: (1) To improve the accuracy of water leakage area extraction, a customized side guided term is introduced to direct the net’s attention to the changes in light and shade within the image. A parallel attention network module is designed to extract internal information from the guided term. Subsequently, a strengthened channel attention module aggregates the guided term and the original information to achieve accurate segmentation of water leakage areas; (2) To address the scarcity of tunnel water leakage area datasets, a basic dataset is constructed by collecting data from open-source datasets and manually gathered data in tunnels. On this basis, perspective transformation is used to change the camera viewpoint, gaussian noise is randomly added to the images in the dataset to simulate images taken in dimly lit scenes, thereby expanding the dataset and enhancing the network’s generalization. The CSG-Unet network was trained using the constructed training set, achieving a mean Intersection over Union (mi IoU) of 85.54%, a mean Dice coefficient (mi Dice) of 85.26%, and a mean Pixel Accuracy (mi PA) of 90.85%. Compared to its baseline network, U-Net (tiny), these metrics show an improvement of over 3.2% in each indicator. Finally, a visual comparison between the improved network and the baseline network further confirms that the proposed model can effectively adapt to the segmentation of water leakage areas in complex environments.

Using 3D Hand Pose Data in Recognizing Human–Object Interaction and User Identification for Extended Reality Systems

Object detection and action/gesture recognition have become imperative in security and surveillance fields, finding extensive applications in everyday life. Advancement in such technologies will help in furthering cybersecurity and extended reality systems through the accurate identification of users and their interactions, which plays a pivotal role in the security management of an entity and providing an immersive experience. Essentially, it enables the identification of human–object interaction to track actions and behaviors along with user identification. Yet, it is performed by traditional camera-based methods with high difficulties and challenges since occlusion, different camera viewpoints, and background noise lead to significant appearance variation. Deep learning techniques also demand large and labeled datasets and a large amount of computational power. In this paper, a novel approach to the recognition of human–object interactions and the identification of interacting users is proposed, based on three-dimensional hand pose data from an egocentric camera view. A multistage approach that integrates object detection with interaction recognition and user identification using the data from hand joints and vertices is proposed. Our approach uses a statistical attribute-based model for feature extraction and representation. The proposed technique is tested on the HOI4D dataset using the XGBoost classifier, achieving an average F1-score of 81% for human–object interaction and an average F1-score of 80% for user identification, hence proving to be effective. This technique is mostly targeted for extended reality systems, as proper interaction recognition and users identification are the keys to keeping systems secure and personalized. Its relevance extends into cybersecurity, augmented reality, virtual reality, and human–robot interactions, offering a potent solution for security enhancement along with enhancing interactivity in such systems.

Safety, Efficiency, and Mental Workload in Simulated Teledriving of a Vehicle as Functions of Camera Viewpoint.

Teleoperation services are expected to operate on-road and often in urban areas. In current teleoperation applications, teleoperators gain a higher viewpoint of the environment from a camera on the vehicle's roof. However, it is unclear how this viewpoint compares to a conventional viewpoint in terms of safety, efficiency, and mental workload. In the current study, teleoperators (n = 148) performed driving tasks in a simulated urban environment with a conventional viewpoint (i.e., the simulated camera was positioned inside the vehicle at the height of a driver's eyes) and a higher viewpoint (the simulated camera was positioned on the vehicle roof). The tasks required negotiating road geometry and other road users. At the end of the session, participants completed the NASA-TLX questionnaire. Results showed that participants completed most tasks faster with the higher viewpoint and reported lower frustration and mental demand. The camera position did not affect collision rates nor the probability of hard braking and steering events. We conclude that a viewpoint from the vehicle roof may improve teleoperation efficiency without compromising driving safety, while also lowering the teleoperators' mental workload.

420 Automated identification and behavioral estrus detection in freestall-housed dairy cows using computer vision and machine learning: A two-layered algorithm approach

Abstract A critical aspect of dairy management involves maintaining accurate records of individual animals and promptly detecting estrus behavior, which is vital for successful breeding and reproductive outcomes. Recent advancements in precision livestock management, utilizing computer vision and machine learning technologies, have paved the way for automated and precise solutions in these areas. The objective of this study is to investigate the application of computer vision and machine learning for animal identification and behavioral estrus detection in dairy cows housed in freestall settings, employing a two-layered algorithmic approach. The research was conducted at the Joe Bearden Dairy Research Center, where a total of 10 lactating Holstein-Friesian cows were assessed across a 7-d trial period. For synchronization efforts, cows were provided a single injection of Prostaglandin F2α at trial initiation with expected estrus behaviors to be exhibited within the following 56 h. Continuous video was captured in 4k resolution using a five-camera closed caption media recording system strategically placed in the pen corners and over the feed bunk alleyway to capture multiple perspectives. Video data extrapolated into frames and manual annotation was utilized to generate the machine learning algorithm’s training data. Frames were randomly selected from a scattering of 56 h of video across all five camera viewpoints and down sampled to 384 x 640 p for both layers. A two-layer approach was employed utilizing an algorithm for each aspect of the study with the first model detecting the behavior, and the second model using the detected region of interest to identify specific cows. The first layer was trained on 682 frames and validated against 123 manually annotated frames. The second layer was trained on 102 frames and validated against 16 frames manually annotated for individual animals. Results demonstrated that the model used for mounting behavior identification has high precision (0.819) and recall (0.89), achieving a 0.926 mAP at an IoU of 0.5. This model achieved an F1 score of 0.86 at a confidence threshold of 0.268. The results demonstrated that the model used for individual animal identification across different classes achieved high precision (0.861) and recall (0.842), with a mAP of 0.897 at an IoU of 0.5. The F1 score for all classes was 0.89 at a confidence threshold of 0.788. In conclusion, these promising findings underscore the potential of computer vision and machine learning technologies to revolutionize dairy cow management practices. This innovative approach not only has potential to enhance operational efficiency, but also contributes significantly to improved reproductive management, thereby advancing precision and productivity in dairy operations.

A Multi-Scale Graph Attention-Based Transformer for Occluded Person Re-Identification

The objective of person re-identification (ReID) tasks is to match a specific individual across different times, locations, or camera viewpoints. The prevalent issue of occlusion in real-world scenarios affects image information, rendering the affected features unreliable. The difficulty and core challenge lie in how to effectively discern and extract visual features from human images under various complex conditions, including cluttered backgrounds, diverse postures, and the presence of occlusions. Some works have employed pose estimation or human key point detection to construct graph-structured information to counteract the effects of occlusions. However, this approach introduces new noise due to issues such as the invisibility of key points. Our proposed module, in contrast, does not require the use of additional feature extractors. Our module employs multi-scale graph attention for the reweighting of feature importance. This allows features to concentrate on areas genuinely pertinent to the re-identification task, thereby enhancing the model’s robustness against occlusions. To address these problems, a model that employs multi-scale graph attention to reweight the importance of features is proposed in this study, significantly enhancing the model’s robustness against occlusions. Our experimental results demonstrate that, compared to baseline models, the method proposed herein achieves a notable improvement in mAP on occluded datasets, with increases of 0.5%, 31.5%, and 12.3% in mAP scores.

Factorized visual representations in the primate visual system and deep neural networks.

Object classification has been proposed as a principal objective of the primate ventral visual stream and has been used as an optimization target for deep neural network models (DNNs) of the visual system. However, visual brain areas represent many different types of information, and optimizing for classification of object identity alone does not constrain how other information may be encoded in visual representations. Information about different scene parameters may be discarded altogether ('invariance'), represented in non-interfering subspaces of population activity ('factorization') or encoded in an entangled fashion. In this work, we provide evidence that factorization is a normative principle of biological visual representations. In the monkey ventral visual hierarchy, we found that factorization of object pose and background information from object identity increased in higher-level regions and strongly contributed to improving object identity decoding performance. We then conducted a large-scale analysis of factorization of individual scene parameters - lighting, background, camera viewpoint, and object pose - in a diverse library of DNN models of the visual system. Models which best matched neural, fMRI, and behavioral data from both monkeys and humans across 12 datasets tended to be those which factorized scene parameters most strongly. Notably, invariance to these parameters was not as consistently associated with matches to neural and behavioral data, suggesting that maintaining non-class information in factorized activity subspaces is often preferred to dropping it altogether. Thus, we propose that factorization of visual scene information is a widely used strategy in brains and DNN models thereof.

Convolutional MLP orthogonal fusion of multiscale features for visual place recognition

Visual place recognition (VPR) involves obtaining robust image descriptors to cope with differences in camera viewpoints and drastic external environment changes. Utilizing multiscale features improves the robustness of image descriptors; however, existing methods neither exploit the multiscale features generated during feature extraction nor consider the feature redundancy problem when fusing multiscale information when image descriptors are enhanced. We propose a novel encoding strategy—convolutional multilayer perceptron orthogonal fusion of multiscale features (ConvMLP-OFMS)—for VPR. A ConvMLP is used to obtain robust and generalized global image descriptors and the multiscale features generated during feature extraction are used to enhance the global descriptors to cope with changes in the environment and viewpoints. Additionally, an attention mechanism is used to eliminate noise and redundant information. Compared to traditional methods that use tensor splicing for feature fusion, we introduced matrix orthogonal decomposition to eliminate redundant information. Experiments demonstrated that the proposed architecture outperformed NetVLAD, CosPlace, ConvAP, and other methods. On the Pittsburgh and MSLS datasets, which contained significant viewpoint and illumination variations, our method achieved 92.5% and 86.5% Recall@1, respectively. We also achieved good performances—80.6% and 43.2%—on the SPED and NordLand datasets, respectively, which have more extreme illumination and appearance variations.

Visual Reinforcement Learning for Dynamic Object Detection

Abstract. Object detection is a widely studied task in computer vision. Current methods often focus on images captured from appropriate viewpoints. However, there is a large disparity between objects observed from different viewpoints in the real world. Dynamic Object Detection (DOD) method automatically adjusts the camera viewpoint in a visual scene to sequentially find optimal viewpoints. Currently, the DOD tasks are usually modeled as a sequential decision-making problem and solved using reinforcement learning methods. Existing approaches face challenges with sparse rewards and training instability. To tackle these issues, we proposed a single-step reward function and a lightweight network, respectively. The single-step reward function, which provides timely feedback, gives an efficient training process for DOD tasks. The lightweight network with few parameters can ensure the stability of the training process. To evaluate the effectiveness of our method, we developed a simulation dataset based on UE4, which consists of 1800 training images and 450 testing images. The dataset includes five object categories: vans, cars, trailers, box trucks and SUVs. Experiments demonstrate that our method outperforms SOTA object detectors on our simulation dataset. Specifically, the average precisions(APs) are improved from 89.1% to 96.0% when using the YOLOv8 object detector.

Meet JEANIE: A Similarity Measure for 3D Skeleton Sequences via Temporal-Viewpoint Alignment

Video sequences exhibit significant nuisance variations (undesired effects) of speed of actions, temporal locations, and subjects’ poses, leading to temporal-viewpoint misalignment when comparing two sets of frames or evaluating the similarity of two sequences. Thus, we propose Joint tEmporal and cAmera viewpoiNt alIgnmEnt (JEANIE) for sequence pairs. In particular, we focus on 3D skeleton sequences whose camera and subjects’ poses can be easily manipulated in 3D. We evaluate JEANIE on skeletal Few-shot Action Recognition (FSAR), where matching well temporal blocks (temporal chunks that make up a sequence) of support-query sequence pairs (by factoring out nuisance variations) is essential due to limited samples of novel classes. Given a query sequence, we create its several views by simulating several camera locations. For a support sequence, we match it with view-simulated query sequences, as in the popular Dynamic Time Warping (DTW). Specifically, each support temporal block can be matched to the query temporal block with the same or adjacent (next) temporal index, and adjacent camera views to achieve joint local temporal-viewpoint warping. JEANIE selects the smallest distance among matching paths with different temporal-viewpoint warping patterns, an advantage over DTW which only performs temporal alignment. We also propose an unsupervised FSAR akin to clustering of sequences with JEANIE as a distance measure. JEANIE achieves state-of-the-art results on NTU-60, NTU-120, Kinetics-skeleton and UWA3D Multiview Activity II on supervised and unsupervised FSAR, and their meta-learning inspired fusion.

Learning Autonomous Viewpoint Adjustment from Human Demonstrations for Telemanipulation

Teleoperation systems find many applications from earlier search-and-rescue to more recent daily tasks. It is widely acknowledged that using external sensors can decouple the view of the remote scene from the motion of the robot arm during manipulation, facilitating the control task. However, this design requires the coordination of multiple operators or may exhaust a single operator as s/he needs to control both the manipulator arm and the external sensors. To address this challenge, our work introduces a viewpoint prediction model, the first data-driven approach that autonomously adjusts the viewpoint of a dynamic camera to assist in telemanipulation tasks. This model is parameterized by a deep neural network and trained on a set of human demonstrations. We propose a contrastive learning scheme that leverages viewpoints in a camera trajectory as contrastive data for network training. We demonstrated the effectiveness of the proposed viewpoint prediction model by integrating it into a real-world robotic system for telemanipulation. User studies reveal that our model outperforms several camera control methods in terms of control experience and reduces the perceived task load compared to manual camera control. As an assistive module of a telemanipulation system, our method significantly reduces task completion time for users who choose to adopt its recommendation.

DermSynth3D: Synthesis of in-the-wild annotated dermatology images

In recent years, deep learning (DL) has shown great potential in the field of dermatological image analysis. However, existing datasets in this domain have significant limitations, including a small number of image samples, limited disease conditions, insufficient annotations, and non-standardized image acquisitions. To address these shortcomings, we propose a novel framework called DermSynth3D. DermSynth3D blends skin disease patterns onto 3D textured meshes of human subjects using a differentiable renderer and generates 2D images from various camera viewpoints under chosen lighting conditions in diverse background scenes. Our method adheres to top-down rules that constrain the blending and rendering process to create 2D images with skin conditions that mimic in-the-wild acquisitions, ensuring more meaningful results. The framework generates photo-realistic 2D dermatological images and the corresponding dense annotations for semantic segmentation of the skin, skin conditions, body parts, bounding boxes around lesions, depth maps, and other 3D scene parameters, such as camera position and lighting conditions. DermSynth3D allows for the creation of custom datasets for various dermatology tasks. We demonstrate the effectiveness of data generated using DermSynth3D by training DL models on synthetic data and evaluating them on various dermatology tasks using real 2D dermatological images. We make our code publicly available at https://github.com/sfu-mial/DermSynth3D.

PoseGen: Learning to Generate 3D Human Pose Dataset with NeRF

This paper proposes an end-to-end framework for generating 3D human pose datasets using Neural Radiance Fields (NeRF). Public datasets generally have limited diversity in terms of human poses and camera viewpoints, largely due to the resource-intensive nature of collecting 3D human pose data. As a result, pose estimators trained on public datasets significantly underperform when applied to unseen out-of-distribution samples. Previous works proposed augmenting public datasets by generating 2D-3D pose pairs or rendering a large amount of random data. Such approaches either overlook image rendering or result in suboptimal datasets for pre-trained models. Here we propose PoseGen, which learns to generate a dataset (human 3D poses and images) with a feedback loss from a given pre-trained pose estimator. In contrast to prior art, our generated data is optimized to improve the robustness of the pre-trained model. The objective of PoseGen is to learn a distribution of data that maximizes the prediction error of a given pre-trained model. As the learned data distribution contains OOD samples of the pre-trained model, sampling data from such a distribution for further fine-tuning a pre-trained model improves the generalizability of the model. This is the first work that proposes NeRFs for 3D human data generation. NeRFs are data-driven and do not require 3D scans of humans. Therefore, using NeRF for data generation is a new direction for convenient user-specific data generation. Our extensive experiments show that the proposed PoseGen improves two baseline models (SPIN and HybrIK) on four datasets with an average 6% relative improvement.

106 Advancing precision livestock management in the dairy industry: An early exploration of individual cow tracking using YOLOv5-Based computer vision

Abstract Precision livestock management in the dairy industry holds promise for addressing the ability of future livestock producers to accurately track multiple health factors related to animal welfare and milk production. While there are significant challenges with integrating sensors into the dairy, a key challenge lies in accurately tracking individual cows across multiple sensors, allowing data correlation with health and milk records. The emergence of artificial intelligence and other innovative technologies provides a unique opportunity for livestock producers. The objective of this study is to examine the ability of computer-based vision using YOLOv5 to automatically detect and identify individual animals housed in a group setting. The study was conducted at the Joe Bearden Dairy Research Center, where lactating cows (seven Holstein-Friesian, three Jersey) were housed within a single pen of a freestall with sand bedding. Animals were managed in accordance with the standard operating procedure of the dairy. Video was captured in 4k resolution via a four camera closed caption media recording system. Cameras were placed at an angle in each corner of the pen providing multiple perspectives of each animal throughout the study. Initial training video data for the algorithm were generated through manual annotation from a single viewpoint for proof of concept. The video capture was segmented into a total of 3,241 frames, with a total of 84 frames manually annotated using open-source software and utilized for training data spanning 75 epochs. Eight images were used for validation. Transfer learning was applied using pretrained weights for the YOLOv5s model. The trained model achieved high precision (0.97) and recall (0.96) during validation. Notably, the F1-Confidence curve showcases a good F1 score of 0.96 at a confidence interval of 0.69. In conclusion, this preliminary work sets the stage for effective sensing and individual cow tracking, with far-reaching implications for dairy farming precision and productivity. While the trained model exhibited strong performance within the confines of its training data, challenges arise when presented with previously unseen angles of the animals, leading to potential identification errors. Moreover, the presence of sand beds occasionally obstructs clear views of the animals, introducing additional uncertainties. To address these challenges and enhance the robustness of the model, future work should incorporate images captured from additional camera viewpoints, ensuring a more comprehensive and reliable identification system for precision animal management in the dairy industry.

VERI-D: A new dataset and method for multi-camera vehicle re-identification of damaged cars under varying lighting conditions

Vehicle re-identification (V-ReID) is a critical task that aims to match the same vehicle across images from different camera viewpoints. The previous studies have leveraged attribute clues, such as color, model, and license plate, to enhance the V-ReID performance. However, these methods often lack effective interaction between the global–local features and the final V-ReID objective. Moreover, they do not address the challenging issues in real-world scenarios, such as high viewpoint variations, extreme illumination conditions, and car appearance changes (e.g., due to damage or wrong driving). We propose a novel framework to tackle these problems and advance the research in V-ReID, which can handle various types of car appearance changes and achieve robust V-ReID under varying lighting conditions. Our main contributions are as follows: (i) we propose a new Re-ID architecture named global–local self-attention network, which integrates local information into the feature learning process and enhances the feature representation for V-ReID and (ii) we introduce a novel damaged vehicle Re-ID dataset called VERI-D, which is the first publicly available dataset that focuses on this challenging yet practical scenario. The dataset contains both natural and synthetic images of damaged vehicles captured from multiple camera viewpoints and under different lighting conditions. (iii) We conduct extensive experiments on the VERI-D dataset and demonstrate the effectiveness of our approach in addressing the challenges associated with damaged vehicle re-identification. We also compare our method to several state-of-the-art V-ReID methods and show its superiority.

GRASPA-fying the Panda: Easily Deployable, Fully Reproducible Benchmarking of Grasp Planning Algorithms

Although robotic grasp planning has been extensively studied in the literature, comparing the performance of different approaches still proves challenging due to the lack of standardization in hardware setup and benchmarking protocols. This work addresses the issue with a threefold contribution. First, it provides a standardized hardware platform and a software framework integrating a benchmarking protocol for grasp planning algorithms (GRASPA). Second, it uses such a framework to benchmark three state-of-the-art algorithms in a reproducible way. Third, it employs the framework to investigate the effect of camera pose variance in visual-based grasp planning. We show how the proposed benchmarking setup can be used to provide insight into the results, not only to compare different vision-based grasp planners but also to evaluate different parameter configurations within the same grasp planner, for instance, camera viewpoint with respect to the scene. To ease the reproducibility of our results and usability of the platform, we provide extensive information for replicating the experimental setup and installing our software in the supplementary material (available at <ext-link ext-link-type="uri" xlink:href="https://doi.org/10.1109/MRA.2023.3256272" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://doi.org/10.1109/MRA.2023.3256272</ext-link> ). All the software used in this article is freely available online in the form of Docker images.

Leveraging Self-Attention Mechanism for Deep Learning in Hand-Gesture Recognition System

This research addresses the complex challenge of recognizing hand gestures irrespective of the user's body posture, a crucial issue in medical treatment for people with speech impairments and human-machine interfaces where precise gesture interpretation is vital. The aim is to engineer an advanced hand gesture recognition system, effective across various body positions and camera viewpoints. A novel flexible camera arrangement was employed, integrating a CNN-Transformer hybrid model, leveraging the strengths of Convolutional Neural Networks and the self-attention mechanism of Transformers. Developed using Python and the PyTorch deep learning framework, the system focuses on sophisticated image processing techniques. A thorough literature review on gesture recognition systems and multi-view analysis was conducted to inform the development. The system demonstrated exceptional accuracy in recognizing hand gestures in diverse body postures and from multiple camera perspectives, significantly outperforming existing methods. It marked a significant advancement in decoding complex gestures, a key aspect for medical applications and intricate human-machine is primarily interactions. This beneficial for people with speech impairments, rehabilitation, and in human-machine interfaces, poised to revolutionize patient care and enhance interaction with advanced machinery and computer systems.

Attribute Descent: Simulating Object-Centric Datasets on the Content Level and Beyond.

This article aims to use graphic engines to simulate a large number of training data that have free annotations and possibly strongly resemble to real-world data. Between synthetic and real, a two-level domain gap exists, involving content level and appearance level. While the latter is concerned with appearance style, the former problem arises from a different mechanism, i.e., content mismatch in attributes such as camera viewpoint, object placement and lighting conditions. In contrast to the widely-studied appearance-level gap, the content-level discrepancy has not been broadly studied. To address the content-level misalignment, we propose an attribute descent approach that automatically optimizes engine attributes to enable synthetic data to approximate real-world data. We verify our method on object-centric tasks, wherein an object takes up a major portion of an image. In these tasks, the search space is relatively small, and the optimization of each attribute yields sufficiently obvious supervision signals. We collect a new synthetic asset VehicleX, and reformat and reuse existing the synthetic assets ObjectX and PersonX. Extensive experiments on image classification and object re-identification confirm that adapted synthetic data can be effectively used in three scenarios: training with synthetic data only, training data augmentation and numerically understanding dataset content.

AvatarStudio: Text-Driven Editing of 3D Dynamic Human Head Avatars

Capturing and editing full-head performances enables the creation of virtual characters with various applications such as extended reality and media production. The past few years witnessed a steep rise in the photorealism of human head avatars. Such avatars can be controlled through different input data modalities, including RGB, audio, depth, IMUs, and others. While these data modalities provide effective means of control, they mostly focus on editing the head movements such as the facial expressions, head pose, and/or camera viewpoint. In this paper, we propose AvatarStudio, a text-based method for editing the appearance of a dynamic full head avatar. Our approach builds on existing work to capture dynamic performances of human heads using Neural Radiance Field (NeRF) and edits this representation with a text-to-image diffusion model. Specifically, we introduce an optimization strategy for incorporating multiple keyframes representing different camera viewpoints and time stamps of a video performance into a single diffusion model. Using this personalized diffusion model, we edit the dynamic NeRF by introducing view-and-time-aware Score Distillation Sampling (VT-SDS) following a model-based guidance approach. Our method edits the full head in a canonical space and then propagates these edits to the remaining time steps via a pre-trained deformation network. We evaluate our method visually and numerically via a user study, and results show that our method outperforms existing approaches. Our experiments validate the design choices of our method and highlight that our edits are genuine, personalized, as well as 3D- and time-consistent.

Interaction-Driven Active 3D Reconstruction with Object Interiors

We introduce an active 3D reconstruction method which integrates visual perception, robot-object interaction , and 3D scanning to recover both the exterior and interior , i.e., unexposed, geometries of a target 3D object. Unlike other works in active vision which focus on optimizing camera viewpoints to better investigate the environment, the primary feature of our reconstruction is an analysis of the interactability of various parts of the target object and the ensuing part manipulation by a robot to enable scanning of occluded regions. As a result, an understanding of part articulations of the target object is obtained on top of complete geometry acquisition. Our method operates fully automatically by a Fetch robot with built-in RGBD sensors. It iterates between interaction analysis and interaction-driven reconstruction, scanning and reconstructing detected moveable parts one at a time, where both the articulated part detection and mesh reconstruction are carried out by neural networks. In the final step, all the remaining, non-articulated parts, including all the interior structures that had been exposed by prior part manipulations and subsequently scanned, are reconstructed to complete the acquisition. We demonstrate the performance of our method via qualitative and quantitative evaluation, ablation studies, comparisons to alternatives, as well as experiments in a real environment.

Face and body-shape integration model for cloth-changing person re-identification

Among the existing deep learning-based person re-identification (ReID) methods, human parsing based on semantic segmentation is the most promising solution for ReID because such models can learn to identify fine-grained details of different body parts or apparel of a target semantically. However, intra-class variations such as illumination changes, multi-pose angles, and cloth-changing (CC) across different non-overlapping camera viewpoints present a crucial challenge for this approach. Among these challenges, a person CC is the most distinctive problem for ReID models, which often fail to associate the target in new cloth against the learned feature semantics of the previous cloth worn in a different timeline. In this paper, we propose a face and body-shape integration (FBI) network as a tactical solution to address the long-term person CC-ReID problem. The FBI comprises hierarchically stacked parsing and edge prediction (PEP) CNN blocks that generate fine-grained human-parsing output at the initial stage. We then aligned the PEP to our proposed model agnostic plug-in feature overlay module (FOM) to mask cloth-relevant body attributes except the facial features pooled from the input sample. Thus, our human parsing PEP and FOM modules are attuned to discriminatively learn cloth-irrelevant features of the target pedestrian(s) to optimize the effectiveness of person ReID in solitary or minimally crowded areas. In our extensive person CC-ReID experiments, our FBI model achieves 83.4/61.8 in R1 and 91.7/65.8 in mAP evaluation results on the PRCC and LTCC datasets, respectively; thereby significantly out-competing several previous state-of-the-art ReID methods, and validating the effectiveness of the FBI.