Raw Depth Research Articles

465 Comparative analysis of semantic segmentation and deep regression models with supervised pre-training for accurate prediction of pig body weight from video data: Insights from industry-scale datasets

Abstract Accurate pig body weight (BW) measurement is essential for producers as it is related to pig growth, health, and marketing, yet conventional manual weighing methods are time-consuming and may cause potential stress to the animals. Although there is a growing trend towards the adoption of three-dimensional cameras coupled with computer vision techniques for pig BW estimation, their validation using industry-scale data are still limited. Among the prevailing methodologies, semantic segmentation and supervised pre-training regression are prominent models currently used. Therefore, the objectives of this study were: 1) to estimate pig BW from repeatedly measured video data obtained from a commercial setting, 2) to compare the performance of the two image analysis methods: thresholding segmentation and deep regression approaches, and 3) to evaluate the predictive ability of BW estimation models. An Intel RealSense D435 camera was installed in the commercial farm to collect top-view videos of 540 pigs biweekly at five different time points over three months. At the same time, manually measured BW records were collected using a digital weighing system. We used an automated video conversion pipeline and fine-tuned YOLOv8 to pre-process the raw depth videos. Subsequently, we acquired a total of 151,756 depth images and depth map files. Adaptive thresholding was applied to segment the pig body from the background. Four image-derived biometric features, including dorsal length, abdominal width, height, and volume, were estimated from the segmented images and fitted using ordinary least squares and random forest models. We applied transfer learning by initializing the weights of five deep learning models, including ResNet50, Xception, EfficientNetV2S, ConvNeXtBase, and Vision Transformer, with pre-trained weights from ImageNet. We then fine-tuned these models on the pig depth images. The last layer of each model was adapted to linear regression, enabling direct estimation of BW for each image without the need for additional image pre-processing steps. We employed random repeated subsampling cross-validation, dividing 80% of the pigs for training and 20% for testing, to evaluate prediction performance at each time point. The best prediction coefficients of determination and mean absolute percentage error for each time point were 0.76, 0.86, 0.90, 0.83, 0.90, and 4.57%, 3.76%, 3.01%, 3.41%, 4.84%, respectively. On average, the Xception model resulted in the best prediction coefficient of determination and mean absolute percentage error of 0.90 and 3.01%. Our results suggest that deep learning-based supervised learning models improve the prediction performance of pig BW from industry-scale depth video data.

High-Throughput and Accurate 3D Scanning of Cattle Using Time-of-Flight Sensors and Deep Learning.

We introduce a high-throughput 3D scanning system designed to accurately measure cattle phenotypes. This scanner employs an array of depth sensors, i.e., time-of-flight (ToF) sensors, each controlled by dedicated embedded devices. The sensors generate high-fidelity 3D point clouds, which are automatically stitched using a point could segmentation approach through deep learning. The deep learner combines raw RGB and depth data to identify correspondences between the multiple 3D point clouds, thus creating a single and accurate mesh that reconstructs the cattle geometry on the fly. In order to evaluate the performance of our system, we implemented a two-fold validation process. Initially, we quantitatively tested the scanner for its ability to determine accurate volume and surface area measurements in a controlled environment featuring known objects. Next, we explored the impact and need for multi-device synchronization when scanning moving targets (cattle). Finally, we performed qualitative and quantitative measurements on cattle. The experimental results demonstrate that the proposed system is capable of producing high-quality meshes of untamed cattle with accurate volume and surface area measurements for livestock studies.

NGHIÊN CỨU XÂY DỰNG MÔ HÌNH NHẬN DIỆN VẬT THỂ TRONG SUỐT DỰA TRÊN THỊ GIÁC LẬP THỂ VÀ TRÍ TUỆ NHÂN TẠO

The article focuses on researching the construction of a transparent object (glass) recognition model based on the application of computer vision techniques and artificial intelligence models. Stereo Matching image processing techniques have been used to build a raw depth image from a Stereo Camera. The goal is to reconstruct the depth image, recover in-depth information, and generate a complete depth image to effectively identify the position of transparent objects in reality. Additionally, the research involves designing a software interface for observing depth images, point clouds, and controlling the robotic arm for object grasping in threedimensional space. The following results were obtained: The quality of the depth image reconstruction model is improved compared to the ClearGrasp model when evaluated on ClearGrasp datasets; Determine orientations on how to improve models and algorithms for reconstructing depth images in a more quantitative manner. The success rate of picking up a glass cup is over 90% in cases of objects on the floor; This rate reaches over 70% when objects are placed at different heights. The software interface displays detailed information and facilitates communication, controlling depth images, point clouds, and position graphs (x, y, z). At the same time, it is easy to interact and convenient during the experiment

G2-MonoDepth: A General Framework of Generalized Depth Inference From Monocular RGB+X Data.

Monocular depth inference is a fundamental problem for scene perception of robots. Specific robots may be equipped with a camera plus an optional depth sensor of any type and located in various scenes of different scales, whereas recent advances derived multiple individual sub-tasks. It leads to additional burdens to fine-tune models for specific robots and thereby high-cost customization in large-scale industrialization. This article investigates a unified task of monocular depth inference, which infers high-quality depth maps from all kinds of input raw data from various robots in unseen scenes. A basic benchmark G2-MonoDepth is developed for this task, which comprises four components: (a) a unified data representation RGB+X to accommodate RGB plus raw depth with diverse scene scale/semantics, depth sparsity ([0%, 100%]) and errors (holes/noises/blurs), (b) a novel unified loss to adapt to diverse depth sparsity/errors of input raw data and diverse scales of output scenes, (c) an improved network to well propagate diverse scene scales from input to output, and (d) a data augmentation pipeline to simulate all types of real artifacts in raw depth maps for training. G2-MonoDepth is applied in three sub-tasks including depth estimation, depth completion with different sparsity, and depth enhancement in unseen scenes, and it always outperforms SOTA baselines on both real-world data and synthetic data.

RDFC-GAN: RGB-Depth Fusion CycleGAN for Indoor Depth Completion.

Raw depth images captured in indoor scenarios frequently exhibit extensive missing values due to the inherent limitations of the sensors and environments. For example, transparent materials frequently elude detection by depth sensors; surfaces may introduce measurement inaccuracies due to their polished textures, extended distances, and oblique incidence angles from the sensor. The presence of incomplete depth maps imposes significant challenges for subsequent vision applications, prompting the development of numerous depth completion techniques to mitigate this problem. Numerous methods excel at reconstructing dense depth maps from sparse samples, but they often falter when faced with extensive contiguous regions of missing depth values, a prevalent and critical challenge in indoor environments. To overcome these challenges, we design a novel two-branch end-to-end fusion network named RDFC-GAN, which takes a pair of RGB and incomplete depth images as input to predict a dense and completed depth map. The first branch employs an encoder-decoder structure, by adhering to the Manhattan world assumption and utilizing normal maps from RGB-D information as guidance, to regress the local dense depth values from the raw depth map. The other branch applies an RGB-depth fusion CycleGAN, adept at translating RGB imagery into detailed, textured depth maps while ensuring high fidelity through cycle consistency. We fuse the two branches via adaptive fusion modules named W-AdaIN and train the model with the help of pseudo depth maps. Comprehensive evaluations on NYU-Depth V2 and SUN RGB-D datasets show that our method significantly enhances depth completion performance particularly in realistic indoor settings.

Neural-ABC: Neural Parametric Models for Articulated Body With Clothes.

In this paper, we introduce Neural-ABC, a novel parametric model based on neural implicit functions that can represent clothed human bodies with disentangled latent spaces for identity, clothing, shape, and pose. Traditional mesh-based representations struggle to represent articulated bodies with clothes due to the diversity of human body shapes and clothing styles, as well as the complexity of poses. Our proposed model provides a unified framework for parametric modeling, which can represent the identity, clothing, shape and pose of the clothed human body. Our proposed approach utilizes the power of neural implicit functions as the underlying representation and integrates well-designed structures to meet the necessary requirements. Specifically, we represent the underlying body as a signed distance function and clothing as an unsigned distance function, and they can be uniformly represented as unsigned distance fields. Different types of clothing do not require predefined topological structures or classifications, and can follow changes in the underlying body to fit the body. Additionally, we construct poses using a controllable articulated structure. The model is trained on both open and newly constructed datasets, and our decoupling strategy is carefully designed to ensure optimal performance. Our model excels at disentangling clothing and identity in different shape and poses while preserving the style of the clothing. We demonstrate that Neural-ABC fits new observations of different types of clothing. Compared to other state-of-the-art parametric models, Neural-ABC demonstrates powerful advantages in the reconstruction of clothed human bodies, as evidenced by fitting raw scans, depth maps and images. We show that the attributes of the fitted results can be further edited by adjusting their identities, clothing, shape and pose codes. The dataset and trained parametric model will be available at https://ustc3dv.github.io/NeuralABC/.

Attention-guided Multi-modality Interaction Network for RGB-D Salient Object Detection

The past decade has witnessed great progress in RGB-D salient object detection (SOD). However, there are two bottlenecks that limit its further development. The first one is low-quality depth maps. Most existing methods directly use raw depth maps to perform detection, but low-quality depth images can bring negative impacts to the detection performance. Hence, it is not desirable to utilize depth maps indiscriminately. The other one is how to effectively predict salient maps with clear boundary and complete salient region. To address these problems, an Attention-Guided Multi-Modality Interaction Network (AMINet) is proposed. First, we propose a new quality enhancement strategy for unreliable depth images, named D epth E nhancement M odule ( DEM ). With respect to the second issue, we propose C ross- M odality A ttention M odule ( CMAM ) to rapidly locate salient region. The B oundary- A ware M odule ( BAM ) is designed to utilize high-level feature to guide the low-level feature generation in a top-down way to make up for the dilution of the boundary. To further improve the accuracy, we propose A trous R efined B lock ( ARB ) to adaptively compensate for the shortcoming of atrous convolution. By integrating these interactive modules, features from depth and RGB streams can be refined efficiently, which consequently boosts the detection performance. Experimental results demonstrate the proposed AMINet exceeds state-of-the-art (SOTA) methods on several public RGB-D datasets.

Autonomous Robotic Bin Picking Platform Generated From Human Demonstration and YOLOv5

Abstract Vision-based robots have been utilized for pick-and-place operations by their ability to find object poses. As they progress into handling a variety of objects with cluttered state, more flexible and lightweight operations have been presented. In this paper, an autonomous robotic bin-picking platform is proposed. It combines human demonstration with a collaborative robot for the flexibility of the objects and YOLOv5 neural network model for faster object localization without prior computer-aided design models or dataset in the training. After a simple human demonstration of which target object to pick and place, the raw color and depth images were refined, and the one on top of the bin was utilized to create synthetic images and annotations for the YOLOv5 model. To pick up the target object, the point cloud was lifted using the depth data corresponding to the result of the trained YOLOv5 model, and the object pose was estimated by matching them with Iterative Closest Points (ICP) algorithm. After picking up the target object, the robot placed it where the user defined it in the previous human demonstration stage. From the result of experiments with four types of objects and four human demonstrations, it took a total of 0.5 s to recognize the target object and estimate the object pose. The success rate of object detection was 95.6%, and the pick-and-place motion of all the found objects was successful.

A Depth-Based Hybrid Approach for Safe Flight Corridor Generation in Memoryless Planning.

This paper presents a depth-based hybrid method to generate safe flight corridors for a memoryless local navigation planner. It is first proposed to use raw depth images as inputs in the learning-based object-detection engine with no requirement for map fusion. We then employ an object-detection network to directly predict the base of polyhedral safe corridors in a new raw depth image. Furthermore, we apply a verification procedure to eliminate any false predictions so that the resulting collision-free corridors are guaranteed. More importantly, the proposed mechanism helps produce separate safe corridors with minimal overlap that are suitable to be used as space boundaries for path planning. The average intersection of union (IoU) of corridors obtained by the proposed algorithm is less than 2%. To evaluate the effectiveness of our method, we incorporated it into a memoryless planner with a straight-line path-planning algorithm. We then tested the entire system in both synthetic and real-world obstacle-dense environments. The obtained results with very high success rates demonstrate that the proposed approach is highly capable of producing safe corridors for memoryless local planning.

Detecting variation of clonal hematopoiesis using machine-learning in liquid biopsy.

e15027 Background: Tumor‐derived DNA released to the plasma of cancer patients which could be captured and detected by ultra-deep sequencing. NGS provides the technical probability of somatic mutations detection from circulating cell‐free DNA (cfDNA) in plasma samples, while clonal hematopoiesis (CH) mutations affect the accuracy of liquid biopsy for clinical diagnosis or choice of drug and tracing of minimal residual disease (MRD). CH mutation screening of cfDNA samples still an unsolved issue, herein we propose a method that based on multi-variable tree model which screen the CH mutation effectively. Methods: We sequenced 1992 cfDNA (raw depth 35000x) and matched peripheral blood lymphocyte (PBL, raw depth 2000x) samples by 1123 genes target capture sequencing from cohorts which include 962 non-small cell lung cancers (NSCLC), 631 gastric cancers (GC), 379 other non-hematological cancers and 40 healthy normal control. First, we compared the coherence patterns of cfDNA-PBL between CH genes (ASXL1, DNMT3A, TET2) hotspot mutation and tumor associated mutation (EGFR L858R T790M, KRAS G12D G13D, TP53 R43H R141H). Next, we labeled canonical mutations by: point mutation and indels; top frequent tumor genes (from cbioportal collected TCGA cancers); published frequent CH genes; known hotspots and unknown significance genes. Furthermore, hotspot tumor mutation and hotspot CH mutations and other mutations were labeled. kNN classify all mutation to labeled mutations groups, and decided the next steps tree model’s node thresholds build a classify tree model screen the confident CH and somatic mutations. Finally, we validated 6 TP53 CH mutation from healthy normal control by ddPCR. Results: Previous published imbalance test model: 'fisher test p.value > 0.5 & 0.5 < OR < 1.5' only covered 23.2%, 26.9%, 17.5% top 10 hotspot CH mutations (ASXL1, DNMT3A, TET2) correspondingly, while 0.75 quantiles both g_VAF and t_VAF less than 1% and alter reads count less than 20. In terms of tumor associated mutations only 3/155,6/123,7/108,7/71,323/998,16/189 exceed the imbalance test models threshold, especially L858R with median t_VAF 3.4% and t_alter_count 60. Population frequency distribution of CH and tumor genes mutation were different significantly (median 0.776% and 0.569%; p.value < 1.44e-05, F-test). After assign hotspot and CH mutations classify by kNN, The tree model screened 42.9% and 83.6% CH mutations in cancer and healthy normal samples respectively. TP53 were common both in tumor and CH mutation ambiguously, tree model screened 6 TP53 CH mutation (c.12G > A, R43H, G113D, R70C, c.16C > T, A80T) from normal plasma sample all validated by ddPCR. Conclusions: CH mutations were ubiquitous in plasma sample and abundance pattern different from common tumor mutations significantly. CH mutation could be screened by optimal condition assembled decision trees effectively.

Optimal Transport for Unsupervised Denoising Learning.

Recently, much progress has been made in unsupervised denoising learning. However, existing methods more or less rely on some assumptions on the signal and/or degradation model, which limits their practical performance. How to construct an optimal criterion for unsupervised denoising learning without any prior knowledge on the degradation model is still an open question. Toward answering this question, this work proposes a criterion for unsupervised denoising learning based on the optimal transport theory. This criterion has favorable properties, e.g., approximately maximal preservation of the information of the signal, whilst achieving perceptual reconstruction. Furthermore, though a relaxed unconstrained formulation is used in practical implementation, we prove that the relaxed formulation in theory has the same solution as the original constrained formulation. Experiments on synthetic and real-world data, including realistic photographic, microscopy, depth, and raw depth images, demonstrate that the proposed method even compares favorably with supervised methods, e.g., approaching the PSNR of supervised methods while having better perceptual quality. Particularly, for spatially correlated noise and realistic microscopy images, the proposed method not only achieves better perceptual quality but also has higher PSNR than supervised methods. Besides, it shows remarkable superiority in harsh practical conditions with complex noise, e.g., raw depth images. Code is available at https://github.com/wangweiSJTU/OTUR.

Surface normal and Gaussian weight constraints for indoor depth structure completion

Raw depth maps captured by depth sensors generally contain missing contents due to glossy, transparent, and sparsity problems. Recent methods well completed flat regions of raw depth maps; however, ignored the accuracy of depth structures. In this paper, an effective depth structure completion method is developed to infer missing depth structures. First, a raw depth map is divided into flat regions and depth structures based on a structure prediction network. Second, two local features including surface normals and Gaussian weights are extracted from a reference RGB image to impose constraints on flat regions and depth structures, separately. Third, a kernel least-square module is adopted to handle the texture-copy artifacts problem. Finally, an iterative optimization model is developed by embedding the two constraints into a Markov random field. The cost function of the model comprises three terms, which limit data fidelity between completed depth map and raw depth map, smoothness of flat regions, and accuracy of depth structures, respectively. The proposed method is evaluated on four indoor datasets including Matterport3D, RealSense, ScanNet, and NYUv2, and compared with eight recent baselines. Quantitative results demonstrate that RMSE and MAE of completed depth maps are considerably reduced by 22.0% and 45.3%, respectively. Visual results show the superiority in completing depth structures and suppressing texture-copy artifacts. Generalization test verify the effectiveness on unseen datasets.

The Human Kaboom by Adam Rubin

Reviewed by: The Human Kaboom by Adam Rubin Wesley Jacques Rubin, Adam The Human Kaboom; illus. by Marta Altés, Daniel Gray-Barnett, Rodolfo Montalvo, Daniel Salmieri, Adam de Souza, and Gracey Zhang. Putnam, 2023 [416p] Trade ed. ISBN 9780593462393 $18.99 E-book ed. ISBN 9780593462416 $10.99 Reviewed from digital galleys R* Gr. 4-8 “The Human Kaboom” acts as the prompt—and all of the titles—for the six stories in this explosively eclectic collection of illustrated stories, which author Rubin hopes will inspire creativity in readers. These particular kabooms deliver that motivation with aplomb, from a swanky hotel’s not-quite murder mystery to a near-future where siblings outsmart some thick-headed rock monsters to a blind tight-rope walker’s final performance. While shared humor and a propensity for the outlandish tie the stories together on some stylistic levels, the varying approaches to the prompt also share an unexpectedly raw depth; several characters lose loved ones tragically, love is bittersweet, and even tremendous superpowers are shown in a dark, violent, isolating light. A unique popcorn recipe follows Rubin call’s for kaboom-y story submissions to end the book, giving readers options for their explosive creativity. Copyright © 2022 The Board of Trustees of the University of Illinois

A Bio-Inspired Simultaneous Surface and Underwater Risk Assessment Method Based on Stereo Vision for USVs in Nearshore Clean Waters

Ensuring safety of unmanned surface vehicles (USVs) during nearshore field operations is challenging as they face unknown risks of collision with surface and underwater obstacles. To address this problem, we propose a bio-inspired collision risk-assessment method for ensuring safe USVs operation in nearshore clean waters. Collision risks are typically unknown due to the imprecision of underwater measurements for the water-air boundary light refraction, and disturbances caused by wave-induced USV motion. To tackle these two causes of imprecision, our collision risk-assessment method employs one single stereo camera and an IMU. We use visual information from the stereo camera to model the light rays at the water-air interface, rectify the underwater measurements, and monitor for surface obstacles, and IMU data to model wave disturbances via time-frequency analysis. We verify the performance of this new method in both the indoor and outdoor experiments, which demonstrates that it achieves a 65.44% improvement in the depth accuracy at the water-air interface compared with raw depth measurements, and provides a range of safety levels for guiding USV operations. This new collision risk-assessment method significantly expands the USVs working areas while ensuring safety during field exploration. Additionally, the simplicity of the sensor setup and thus minimal cost of this method underscore its utility for supporting USV-based various explorations in clear waters.

Interactive nonlocal joint learning network for red, green, blue plus depth salient object detection

Research into red, green, blue plus depth salient object detection (SOD) has identified the challenging problem of how to exploit raw depth features and fuse cross-modal (CM) information. To solve this problem, we propose an interactive nonlocal joint learning (INL-JL) network for quality RGB-D SOD. INL-JL benefits from three key components. First, we carry out joint learning to extract common features from RGB and depth images. Second, we adopt simple yet effective CM fusion blocks in lower levels while leveraging the proposed INL blocks in higher levels, aiming to purify the depth features and to make CM fusion more efficient. Third, we utilize a dense multiscale transfer strategy to infer saliency maps. INL-JL advances the state-of-the-art methods on five public datasets, demonstrating its power to promote the quality of RGB-D SOD.

LBENet: Lightweight boundary enhancement network for detecting salient objects in RGB-D images

ObjectiveRecent methods for RGB-D salient object detection are capable of achieving desirable performance by leveraging depth features extracted using convolutional neural networks (CNNs). However, the lack of pretrained networks currently available for extracting representative features from depth data, ambiguity in the fusion of depth and RGB features, and the contamination effect originating from unreliable depth data still hinder the further improvement of these methods. Recently, a boundary cue was embedded in a deep model to highlight the boundary region of a salient object and improve the performance of SOD. Furthermore, other methods use Backbone to extract depth map features, which increases the complexity of the model. MethodsBased on this, we propose a lightweight boundary enhancement network (LBENet) for RGB-D salient object detection. The proposed LBENet comprises two main modules: a normalized max-min filter (NMMF) and a boundary weight module (BWM). First, instead of directly using CNNs to extract features from the depth map, which may be suboptimal owing to the use of unreliable depth data, we used the NMMF to extract boundary information from the raw depth data that contain less noisy features and rich boundary information and used the BWM to adaptatively enhance the RGB boundary region features. A boundary loss function allows for the learning of precise boundaries of salient objects ResultThe proposed end-to-end method achieves state-of-the-art performance on two widely used benchmarks, while improving the number of FLOPs (44.271 G), number of parameters (15.96 M), model size (65.4 MB), and inference speed (29.41 frame/s).

Real-time human action recognition using raw depth video-based recurrent neural networks

This work proposes and compare two different approaches for real-time human action recognition (HAR) from raw depth video sequences. Both proposals are based on the convolutional long short-term memory unit, namely ConvLSTM, with differences in the architecture and the long-term learning. The former uses a video-length adaptive input data generator (stateless) whereas the latter explores the stateful ability of general recurrent neural networks but is applied in the particular case of HAR. This stateful property allows the model to accumulate discriminative patterns from previous frames without compromising computer memory. Furthermore, since the proposal uses only depth information, HAR is carried out preserving the privacy of people in the scene, since their identities can not be recognized. Both neural networks have been trained and tested using the large-scale NTU RGB+D dataset. Experimental results show that the proposed models achieve competitive recognition accuracies with lower computational cost compared with state-of-the-art methods and prove that, in the particular case of videos, the rarely-used stateful mode of recurrent neural networks significantly improves the accuracy obtained with the standard mode. The recognition accuracies obtained are 75.26% (CS) and 75.45% (CV) for the stateless model, with an average time consumption per video of 0.21 s, and 80.43% (CS) and 79.91%(CV) with 0.89 s for the stateful one.

Abstract 1238: Comprehensive ctDNA monitoring provides early signal of clinical benefit with a novel personalized neoantigen directed immunotherapy for late-stage cancer patients

Abstract Neoantigen directed immunotherapy holds promise to increase the likelihood of patients with solid tumor devoid of immune infiltration benefiting from immune checkpoint immunotherapy (CPI). A heterologous prime-boost vaccination approach consisting of Chimpanzee Adenovirus (ChAd) prime and multiple self-amplifying mRNA (SAM) boosts, delivering 20 neoantigens, has been evaluated in a Phase 1/2 clinical trial in late-stage solid tumor patients in combination with nivolumab and ipilimumab (NCT03639714). Neoantigen dynamics, tumor burden and genomic correlates of response were studied over time in 20 patients (8 GEA, 2 NSCLC, 10 MSS-CRC). Exome sequences from archival (sample used for neoantigen selection), baseline (start of immunization) and on-treatment biopsies were analyzed for 20, 16 and 10 patients respectively. Paired pre- and post-vaccine tumor transcriptomes were analyzed for 6 patients. Personalized capture baits were designed for all non-synonymous mutations detected in archival biopsies (mean 146; range: 67-402) for ctDNA monitoring. Longitudinal ctDNA samples were collected monthly on treatment (mean 7; range: 1-18). ctDNA duplex UMI libraries were captured and sequenced to a target mean raw depth >80,000x and reduced to 3x per strand consensus duplex reads. The majority of vaccine neoantigens were detected in ctDNA (87%; range 45%-100%) and mean neoantigen variant allele frequency (VAF) strongly correlated with all monitored mutations VAF (R2 = 0.90, p < 0.0001) through treatment. The percentage of vaccine neoantigens detected was higher than that for all monitored mutations in the same samples with a median of 80% (21%-98%) in ctDNA and 70% (44%-100%) in biopsies. Five of 9 MSS-CRC patients with measurable baseline ctDNA achieved molecular responses (MR, >50% reduction in ctDNA from baseline) that correlated with OS and PFS, and in some patients, was accompanied by radiologic tumor shrinkage. One MSS-CRC patient with MR for >7 months acquired novel biallelic loss-of-function mutations in TAP1 following 1 year of study treatment. Differential gene expression analysis from paired pre- and post-vaccine biopsies (including 2 MSS-CRC pairs with MR) demonstrated significant upregulation in gene signatures associated with immune-inflamed tumor microenvironments including interferon alpha and gamma responses. We demonstrate that tumor-informed neoantigen selection and vaccine manufacturing while patients receive chemotherapy is feasible, since the majority of neoantigens are retained in the tumor post-chemotherapy. Further, neoantigen-directed immunotherapy appears to drive clinical benefit in patients with advanced MSS-CRC tumors, where CPI alone has provided minimal benefit. Comprehensive ctDNA longitudinal monitoring enables real time assessment of clinical response and acquired resistance. Citation Format: Desiree Schenck, Rita Zhou, Alexis Mantilla, Oliver Spiro, Taylor Patch, Adrienne Johnson, Daniel Navarro Gomez, Brian S. Henick, Chih-Yi Liao, Sameek Roychowdhury, Steve Maron, Benjamin Solomon, Alexander I. Spira, Daniel V. Catenacci, Andrew R. Fergusson, Raphael F. Rousseau, Karin Jooss, Matthew J. Davis. Comprehensive ctDNA monitoring provides early signal of clinical benefit with a novel personalized neoantigen directed immunotherapy for late-stage cancer patients [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2022; 2022 Apr 8-13. Philadelphia (PA): AACR; Cancer Res 2022;82(12_Suppl):Abstract nr 1238.

3D human pose estimation with cross-modality training and multi-scale local refinement

This paper aims to study two major research problems for 3D human pose estimation using depth data. First we seek an effective way for applying the RGB pre-trained 2D CNN model to 3D pose field, so as to transfer large-scale RGB annotation information to depth domain. In particular, we proposed a cross-modality CNN training strategy, where the key idea is to set a partial Batch Normalization (BN) layer within the RGB pre-trained 2D CNN model to weaken the distribution divergence between the RGB and depth data during training. To involve richer 3D descriptive cues, the raw depth data is appended with the normal vector map. Albeit coarse-to-fine human pose estimation with local refinement is helpful to enhance performance. While the way for setting the optimal local observation scale is not well addressed. Towards this crucial problem, we propose to fuse the multi-scale local information jointly. A multi-scale local refinement network is proposed accordingly, where the small local region focuses on capturing the fine information. On the other hand, the large local region contains richer semantic contextual information. The experiments on two 3D human pose estimation datasets with depth data verify the effectiveness and real-time running capacity of our proposition.

3DFCNN: real-time action recognition using 3D deep neural networks with raw depth information

This work describes an end-to-end approach for real-time human action recognition from raw depth image-sequences. The proposal is based on a 3D fully convolutional neural network, named 3DFCNN, which automatically encodes spatio-temporal patterns from raw depth sequences. The described 3D-CNN allows actions classification from the spatial and temporal encoded information of depth sequences. The use of depth data ensures that action recognition is carried out protecting people’s privacy, since their identities can not be recognized from these data. The proposed 3DFCNN has been optimized to reach a good performance in terms of accuracy while working in real-time. Then, it has been evaluated and compared with other state-of-the-art systems in three widely used public datasets with different characteristics, demonstrating that 3DFCNN outperforms all the non-DNN-based state-of-the-art methods with a maximum accuracy of 83.6% and obtains results that are comparable to the DNN-based approaches, while maintaining a much lower computational cost of 1.09 seconds, what significantly increases its applicability in real-world environments.