3D Annotations Research Articles

CAD-based Autonomous Vision Inspection Systems

Automated industrial Visual Inspection Systems (VIS) are typically customized for specific applications, limiting their flexibility. They are characterized by a demanding setup, high capital investments, and significant knowledge barriers. In this paper, we propose an alternative architecture for the visual inspection of 3D printed parts or complex assemblies using a robotic arm equipped with hand-eye sensors and controllable lighting system. The core of the proposed Flexible Vision Inspection System (FVIS) is the self-extraction of 3D text annotations from STandard for the Exchange of Product model (STEP) AP242 files. The system self-selects and parametrizes the most suitable inspection algorithm, including lighting settings. Additionally, it autonomously performs self-localization, self-referencing of physical products, and self-planning of robot inspection path based on CAD information. This framework, characterized by self-X, cost-effective, non-invasive, and plug-and-play architecture has the potential to disrupt the business model of vision inspection, enabling an as-a-service solution aligned with the next generation of flexible manufacturing.

MA-ST3D: Motion Associated Self-Training for Unsupervised Domain Adaptation on 3D Object Detection.

Recently, unsupervised domain adaptation (UDA) for 3D object detectors has increasingly garnered attention as a method to eliminate the prohibitive costs associated with generating extensive 3D annotations, which are crucial for effective model training. Self-training (ST) has emerged as a simple and effective technique for UDA. The major issue involved in ST-UDA for 3D object detection is refining the imprecise predictions caused by domain shift and generating accurate pseudo labels as supervisory signals. This study presents a novel ST-UDA framework to generate high-quality pseudo labels by associating predictions of 3D point cloud sequences during ego-motion according to spatial and temporal consistency, named motion-associated self-training for 3D object detection (MA-ST3D). MA-ST3D maintains a global-local pathway (GLP) architecture to generate high-quality pseudo-labels by leveraging both intra-frame and inter-frame consistencies along the spatial dimension of the LiDAR's ego-motion. It also equips two memory modules for both global and local pathways, called global memory and local memory, to suppress the temporal fluctuation of pseudo-labels during self-training iterations. In addition, a motion-aware loss is introduced to impose discriminated regulations on pseudo labels with different motion statuses, which mitigates the harmful spread of false positive pseudo labels. Finally, our method is evaluated on three representative domain adaptation tasks on authoritative 3D benchmark datasets (i.e. Waymo, Kitti, and nuScenes). MA-ST3D achieved SOTA performance on all evaluated UDA settings and even surpassed the weakly supervised DA methods on the Kitti and NuScenes object detection benchmark.

SceneDreamer: Unbounded 3D Scene Generation from 2D Image Collections.

In this work, we present, an unconditional generative model for unbounded 3D scenes, which synthesizes large-scale 3D landscapes from random noise. Our framework is learned from in-the-wild 2D image collections only, without any 3D annotations. At the core of is a principled learning paradigm comprising 1) an efficient yet expressive 3D scene representation, 2) a generative scene parameterization, and 3) an effective renderer that can leverage the knowledge from 2D images. Our approach begins with an efficient bird's-eye-view (BEV) representation generated from simplex noise, which includes a height field for surface elevation and a semantic field for detailed scene semantics. This BEV scene representation enables 1) representing a 3D scene with quadratic complexity, 2) disentangled geometry and semantics, and 3) efficient training. Moreover, we propose a novel generative neural hash grid to parameterize the latent space based on 3D positions and scene semantics, aiming to encode generalizable features across various scenes. Lastly, a neural volumetric renderer, learned from 2D image collections through adversarial training, is employed to produce photorealistic images. Extensive experiments demonstrate the effectiveness of and superiority over state-of-the-art methods in generating vivid yet diverse unbounded 3D worlds. Project Page is available at https://scene-dreamer.github.io/. Code is available at https://github.com/FrozenBurning/SceneDreamer.

Real-time telementoring with 3D drawing annotation in robotic surgery.

In telementoring, differences in teaching methods affect local surgeons' comprehension. Because the object to be operated on is a three-dimensional (3D) structure, voice or 2D annotation may not be sufficient to convey the instructor's intention. In this study, we examined the usefulness of telementoring using 3D drawing annotations in robotic surgery. Kyushu University and Beppu Hospital are located 140km apart, and the study was conducted using a Saroa™ surgical robot by RIVERFIELD Inc. using a commercial guarantee network on optical fiber. Twenty medical students performed vertical mattress suturing using a swine intestinal tract under surgical guidance at the Center for Advanced Medical Innovation Kyushu University. Surgical guidance was provided by Beppu Hospital using voice, 2D, and 3D drawing annotations. All robot operations were performed using 3D images, and only the annotations were independently switched between voice and 2D and 3D images. The operation time, needle movement, and performance were also evaluated. The 3D annotation group tended to have a shorter working time than the control group (25.6 ± 63.2 vs. -36.7 ± 65.4min, P = 0.06). The 3D annotation group had fewer retries than the control group (1.3 ± 1.7 vs. -1.1 ± 0.7, P = 0.006), and there was a tendency for fewer needle drops (0.4 ± 0.7 vs. -0.5 ± 0.9, P = 0.06). The 3D annotation group scored significantly higher than the control group on the Global Evaluate Assessment of Robot Skills (16.8 ± 2.0 vs. 22.8 ± 2.4, P = 0.04). The 3D annotation group also scored higher than the voice (13.4 ± 1.2) and 2D annotation (16.2 ± 1.8) groups (3D vs. voice: P = 0.03, 3D vs. 2D: P = 0.03). Telementoring using 3D drawing annotation was shown to provide good comprehension and a smooth operation for local surgeons.

Interactive 3D Annotation of Objects in Moving Videos from Sparse Multi-view Frames

Segmenting and determining the 3D bounding boxes of objects of interest in RGB videos is an important task for a variety of applications such as augmented reality, navigation, and robotics. Supervised machine learning techniques are commonly used for this, but they need training datasets: sets of images with associated 3D bounding boxes manually defined by human annotators using a labelling tool. However, precisely placing 3D bounding boxes can be difficult using conventional 3D manipulation tools on a 2D interface. To alleviate that burden, we propose a novel technique with which 3D bounding boxes can be created by simply drawing 2D bounding rectangles on multiple frames of a video sequence showing the object from different angles. The method uses reconstructed dense 3D point clouds from the video and computes tightly fitting 3D bounding boxes of desired objects selected by back-projecting the 2D rectangles. We show concrete application scenarios of our interface, including training dataset creation and editing 3D spaces and videos. An evaluation comparing our technique with a conventional 3D annotation tool shows that our method results in higher accuracy. We also confirm that the bounding boxes created with our interface have a lower variance, likely yielding more consistent labels and datasets.

HHCDB: a database of human heterochromatin regions.

Heterochromatin plays essential roles in eukaryotic genomes, such as regulating genes, maintaining genome integrityand silencing repetitive DNA elements. Identifying genome-wide heterochromatin regions is crucial for studying transcriptional regulation. We propose the Human Heterochromatin Chromatin Database (HHCDB) for archiving heterochromatin regions defined by specific or combined histone modifications (H3K27me3, H3K9me2, H3K9me3) according to a unified pipeline. 42839743 heterochromatin regions were identified from 578 samples derived from 241 cell-types/cell lines and 92 tissue types. Genomic information is provided in HHCDB, including chromatin location, gene structure, transcripts, distance from transcription start site, neighboring genes, CpG islands, transposable elements, 3D genomic structureand functional annotations. Furthermore, transcriptome data from 73 single cells were analyzed and integrated to explore cell type-specific heterochromatin-related genes. HHCDB affords rich visualization through the UCSC Genome Browser and our self-developed tools. We have also developed a specialized online analysis platform to mine differential heterochromatin regions in cancers. We performed several analyses to explore the function of cancer-specific heterochromatin-related genes, including clinical feature analysis, immune cell infiltration analysis and the construction of drug-target networks. HHCDB is a valuable resource for studying epigenetic regulation, 3D genomicsand heterochromatin regulation in development and disease. HHCDB is freely accessible at http://hhcdb.edbc.org/.

CENAGIS-ALS BENCHMARK - NEW PROPOSAL FOR DENSE ALS BENCHMARK BASED ON THE REVIEW OF DATASETS AND BENCHMARKS FOR 3D POINT CLOUD SEGMENTATION

Abstract. Benchmarking is an essential tool for scientific and technological progress. This article reviews the benchmarks for 3D point cloud segmentation and classification. Based on the analysis of the articles and the knowledge gathered, it can be concluded that there has been an increase in the number of benchmarks, allowing to compare research results against specific performance metrics independently. However, benchmarks vary regarding the number of classes, spatial size, nomenclature, and class division. In this article, we introduce a new annotated 3D dataset - CENAGIS-ALS Benchmark. We propose a benchmark of highly dense lidar point clouds acquired by Leica CityMapper-2 for the Centre of Warsaw, Poland. The area covers 2 km2, and the data has a density of 275 pts/m2. The dataset consists of a number of classes that are distinguishable for this type of data. In addition to the basic classes, more specialized classes, important from the perspective of urban space, are also distinguished. Moreover, the division of classes consists of three levels of detail from coarse (e.g., a building) to refined elements (e.g., roofs, chimneys, and other rooftop objects). This benchmark can contribute to geospatial societies, considering the large spatial size of the study area with unified data quality and the higher number of classes with the hierarchical division compared to other benchmarking data.

A full-set tooth segmentation model based on improved PointNet++

Segmentation of a complete set of teeth from three-dimensional (3D) intra-oral scanner images is a crucial step in tooth identification procedures. In large-scale disasters with many victims, teeth are often the preferred and reliable source for victim identification due to their hard and non-deformable characteristics. In this paper we present a study on the automatic segmentation of a complete set of teeth from intra-oral scanner images. We propose a tooth segmentation method based on an improved PointNet++ architecture. To address the problem of inadequate segmentation capability of the teeth-gingival boundary of PointNet++, we introduce a single-point preliminary feature extraction (SPFE) module to better preserve the subtle details that may be overlooked by the original PointNet++ model. In addition, a weighted-sum local feature aggregation (WSLFA) mechanism is proposed to replace the max pooling in PointNet++ to better perform feature aggregation. The experimental results on 52 testing datasets using the network trained on 160 annotated 3D intra-oral scanner images demonstrate that our improved PointNet++ method achieves a segmentation accuracy of 97.68%, and performs well under different dental conditions.

Evaluating visual encoding quality of a mixed reality user interface for human–machine co-assembly in complex operational terrain

During human–machine collaboration in manufacturing activities, it is important to provide real-time annotations in the three-dimensional workspace for local workers who may lack relevant experience and knowledge. For example, in MR assembly, workers need to be alerted to avoid entering hazardous areas when manually replacing components. Recently, many researchers have explored various visual cues for expressing physical task progress information in the MR interface of intelligent systems. However, the relationship between the implantation of visual cues and the balance of interface cognition has not been well revealed, especially in tasks that require annotating hazardous areas in complex operational terrains. In this study, we developed a novel MR interface for an intelligent assembly system that supports local scene sharing based on dynamic 3D reconstruction, remote expert behavior intention recognition based on deep learning, and local personnel operational behavior visual feedback based on external bounding box. We compared the encoding results of the proposed MR interface with 3D annotations combined with 3D sketch cues (3DS), which combines 3D spatial cues (3DSC) and 3DS combined with adaptive cues (AVC), through a case study. We found that for physical tasks that require specific area annotations, 3D annotations with context (3DAC) can better improve the quality of manual work and regulate the cognitive load distribution of the MR interface more reasonably.

GAL: Graph-Induced Adaptive Learning for Weakly Supervised 3D Object Detection

Weakly Supervised 3D Object Detection (WS3DOD) aims to perform 3D object detection with little reliance on 3D labels, which greatly reduces the cost of 3D annotations. In recent literature, the pseudo-label-based approach brings impressive performance, which generates 3D pseudo-labels from 2D bounding boxes. Despite their success, two key issues remain unresolved that reduce the quality of 3D pseudo-labels: 1) the existing local object locating algorithm can not capture complete clusters of points globally, and 2) the existing algorithm can not capture sparse points caused by the unevenly distributed points obtained by LiDAR cameras. Hence, we propose GAL, a Graph-induced Adaptive Learning algorithm, to generate 3D pseudo-labels. First, we propose the Cluster Locating algorithm based on the Minimum Spanning Tree (MST) to globally locate the objects, which can leverage the characteristic that points inside an object are compact while points between objects are discrete. Second, we propose a density-guided adaptive learning algorithm to optimise the Cluster Locating algorithm, named Cuboid Drift. Cuboid Drift considers the inhomogeneous distribution of reflected points on different reflective surfaces of LiDAR imaging. Finally, 3D pseudo-labels generated by GAL are leveraged to train 3D detectors. Extensive experiments on the challenging <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">KITTI </i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">DAIR-V2X-V</i> dataset demonstrate that GAL without 3D labels can be comparable with strongly supervised approaches and outperforms the previous state-of-the-art WS3DOD methods. Moreover, our method saves 88% of the time spent on pseudo-label generation.

MHCanonNet: Multi-Hypothesis Canonical lifting Network for self-supervised 3D human pose estimation in the wild video

Recent advancements in 3D Human Pose Estimation using fully-supervised learning approach have shown impressive results; however, these methods heavily rely on large amounts of annotated 3D data, which are challenging to obtain outside controlled laboratory environments. Therefore, in this study, we propose a new self-supervised training method designed to train a 3D human pose estimation network using unlabeled multi-view images. The model trains relative depths between joints without any 3D annotation by satisfying multi-view consistency constraints from unlabeled multi-view videos without camera calibration, while simultaneously learning representations of multiple plausible pose hypotheses. For this reason, we call our proposed network a Multi-Hypothesis Canonical Lifting Network (MHCanonNet). By enriching the diversity of extracted features and keeping various possibilities open, our network accurately estimates the final 3D pose. The key idea lies in the design of a novel and unbiased reconstruction objective function that combines multiple hypotheses from different viewpoints. The proposed approach demonstrates state-of-the-art results not only on two popular benchmark datasets, Human3.6M and MPI-INF-3DHP but also on an in-the-wild dataset, Ski-Pose, surpassing existing self-supervised training methods.

Efficient Annotation and Learning for 3D Hand Pose Estimation: A Survey

AbstractIn this survey, we present a systematic review of 3D hand pose estimation from the perspective of efficient annotation and learning. 3D hand pose estimation has been an important research area owing to its potential to enable various applications, such as video understanding, AR/VR, and robotics. However, the performance of models is tied to the quality and quantity of annotated 3D hand poses. Under the status quo, acquiring such annotated 3D hand poses is challenging, e.g., due to the difficulty of 3D annotation and the presence of occlusion. To reveal this problem, we review the pros and cons of existing annotation methods classified as manual, synthetic-model-based, hand-sensor-based, and computational approaches. Additionally, we examine methods for learning 3D hand poses when annotated data are scarce, including self-supervised pretraining, semi-supervised learning, and domain adaptation. Based on the study of efficient annotation and learning, we further discuss limitations and possible future directions in this field.

Consistent 3D Hand Reconstruction in Video via Self-Supervised Learning.

We present a method for reconstructing accurate and consistent 3D hands from a monocular video. We observe that the detected 2D hand keypoints and the image texture provide important cues about the geometry and texture of the 3D hand, which can reduce or even eliminate the requirement on 3D hand annotation. Accordingly, in this work, we propose S2HAND, a self-supervised 3D hand reconstruction model, that can jointly estimate pose, shape, texture, and the camera viewpoint from a single RGB input through the supervision of easily accessible 2D detected keypoints. We leverage the continuous hand motion information contained in the unlabeled video data and explore S2HAND(V), which uses a set of weights shared S2HAND to process each frame and exploits additional motion, texture, and shape consistency constrains to obtain more accurate hand poses, and more consistent shapes and textures. Experiments on benchmark datasets demonstrate that our self-supervised method produces comparable hand reconstruction performance compared with the recent full-supervised methods in single-frame as input setup, and notably improves the reconstruction accuracy and consistency when using the video training data.

A comprehensive survey of long-range tertiary interactions and motifs in non-coding RNA structures.

Understanding the 3D structure of RNA is key to understanding RNA function. RNA 3D structure is modular and can be seen as a composition of building blocks of various sizes called tertiary motifs. Currently, long-range motifs formed between distant loops and helical regions are largely less studied than the local motifs determined by the RNA secondary structure. We surveyed long-range tertiary interactions and motifs in a non-redundant set of non-coding RNA 3D structures. A new dataset of annotated LOng-RAnge RNA 3D modules (LORA) was built using an approach that does not rely on the automatic annotations of non-canonical interactions. An original algorithm, ARTEM, was developed for annotation-, sequence- and topology-independent superposition of two arbitrary RNA 3D modules. The proposed methods allowed us to identify and describe the most common long-range RNA tertiary motifs. Along with the prevalent canonical A-minor interactions, a large numberof previously undescribed staple interactions were observed. The most frequent long-range motifs were found to belong to three main motif families: planar staples, tilted staples, and helical packing motifs.

Aïoli: A reality-based 3D annotation cloud platform for the collaborative documentation of cultural heritage artefacts

Archaeologists, architects, engineers, materials specialists, teachers, curators and restorers of cultural property, contribute to the daily knowledge and conservation of heritage artefacts. For many years, the development of digital technologies has produced important results in the collection, visualisation and indexing of digital resources. Whilst these advances have made it possible to introduce new tools that are making documentation practices evolve within the cultural heritage community, the management of multi-dimensional and multi-format data introduces new problems and challenges, in particular the development of relevant analysis and interpretation methods, the sharing and correlation of heterogeneous data among several actors and contexts, and the centralised archiving of documentation results for long-term preservation purposes. Despite their different approaches and tools for observation, description and analysis, the actors of cultural heritage documentation all have a common interest and central focus: the heritage object, the physical one, whether it is a site, a building, a sculpture, a painting, a work of art, or an archaeological fragment. This is the starting point for the development of “Aïoli”, a reality-based 3D annotation platform, which allows a multidisciplinary community to build semantically-enriched 3D descriptions of heritage artefacts from simple images and spatialised annotations coupled with additional resources. Resulting from more than 10 years of methodological and technological contributions concerning different steps of the workflow starting from reality-based 3D reconstruction to the analysis of cultural heritage data, this platform introduces an innovative framework for the comprehensive, large-scale collaborative documentation of cultural heritage by integrating state-of-the-art technological components within a cloud infrastructure accessible via web interfaces from PCs, tablets and smartphones online and onsite. This paper aims to provide a synthesis of the work done to build it, to describe its main features, and to discuss the limitations and opportunities it raises for cultural heritage studies.

Frozen Section Timeout: Pilot Study to Reconcile Margins Using 3D Resected Specimen and Defect Scans.

Opportunities exist to improve intraoperative communication and documentation of resection margin details. We instituted a "frozen section timeout" that centers around visualization of the paired resection specimen and surgical defect-facilitating effective, bidirectional exchange of information. We designed an interactive form for use during the "frozen section timeout" including annotated 3D virtual models of the resected specimen and surgical defect, plus a "line-item" table for primary and supplemental margin results. The "timeout" was conducted over a Zoom call between the operating room and frozen section laboratory. The form was simultaneously projected and discussed while all members of the surgical care team stopped activities. Nurses, co-surgeons, and all other members of the surgical team were encouraged to take part in this process. Twenty-six frozen section timeouts were conducted during head and neck surgeries in the Department of Otolaryngology at Mount Sinai West Hospital. These timeouts were facilitated by the lead surgeon, and all other activities were halted to ensure that critical information was shared, documented, and agreed upon. During the timeout, the annotated specimen and defect scans were displayed, clearly demonstrating the at-risk margins and the corresponding location and breadth of supplemental margins harvested. Incorporating a frozen section timeout can improve intraoperative communication, increase transparency, and potentially eliminate uncertainty regarding margin status and tumor clearance. Visualization of at-risk margins and the corresponding location and breadth of supplemental margins promises an unprecedented level of documentation and understanding. This novel technique can establish a new and improved standard of care. NA Laryngoscope, 134:725-731, 2024.

3D pose estimation and localization of construction equipment from single camera images by virtual model integration

Computer vision technologies are receiving much attention for automatic site monitoring. However, previous approaches mainly focused on two-dimensional (2D) pixel information, and there was a limit to providing detailed three-dimensional (3D) pose information. Some studies have built a training database only using virtual models, but there were still limitations in representing the actual equipment. To address these limitations, the authors propose a method for estimating 3D poses of equipment from single camera images by integrating virtual models. The proposed method consists of three main processes: (1) 2D – 3D annotation using a virtual model; (2) preprocessing for pose estimation in two ways (i.e., image matching or key points detection); and (3) 3D pose localization. As a result, the root mean square error for the 3D pose of construction equipment was 1.49 m (using image matching) and 1.56 m (using key points detection). The results imply that the proposed method successfully estimated the 3D pose of the equipment with a single camera. The findings of this study can contribute to detailed productivity or safety analysis based on more diverse 3D information than 2D pixel-based analysis.

Weakly Supervised 3D Multi-Person Pose Estimation for Large-Scale Scenes Based on Monocular Camera and Single LiDAR

Depth estimation is usually ill-posed and ambiguous for monocular camera-based 3D multi-person pose estimation. Since LiDAR can capture accurate depth information in long-range scenes, it can benefit both the global localization of individuals and the 3D pose estimation by providing rich geometry features. Motivated by this, we propose a monocular camera and single LiDAR-based method for 3D multi-person pose estimation in large-scale scenes, which is easy to deploy and insensitive to light. Specifically, we design an effective fusion strategy to take advantage of multi-modal input data, including images and point cloud, and make full use of temporal information to guide the network to learn natural and coherent human motions. Without relying on any 3D pose annotations, our method exploits the inherent geometry constraints of point cloud for self-supervision and utilizes 2D keypoints on images for weak supervision. Extensive experiments on public datasets and our newly collected dataset demonstrate the superiority and generalization capability of our proposed method. Project homepage is at \url{https://github.com/4DVLab/FusionPose.git}.

Context-Aware Transformer for 3D Point Cloud Automatic Annotation

3D automatic annotation has received increased attention since manually annotating 3D point clouds is laborious. However, existing methods are usually complicated, e.g., pipelined training for 3D foreground/background segmentation, cylindrical object proposals, and point completion. Furthermore, they often overlook the inter-object feature correlation that is particularly informative to hard samples for 3D annotation. To this end, we propose a simple yet effective end-to-end Context-Aware Transformer (CAT) as an automated 3D-box labeler to generate precise 3D box annotations from 2D boxes, trained with a small number of human annotations. We adopt the general encoder-decoder architecture, where the CAT encoder consists of an intra-object encoder (local) and an inter-object encoder (global), performing self-attention along the sequence and batch dimensions, respectively. The former models intra-object interactions among points and the latter extracts feature relations among different objects, thus boosting scene-level understanding. Via local and global encoders, CAT can generate high-quality 3D box annotations with a streamlined workflow, allowing it to outperform existing state-of-the-arts by up to 1.79% 3D AP on the hard task of the KITTI test set.

Semantic segmentation of mobile mapping point clouds via multi-view label transfer

We study how to learn semantic segmentation of 3D point clouds from small training sets. The problem arises because annotating 3D point clouds is a lot more time-consuming and error-prone than annotating 2D images. On the one hand this means that one cannot afford to create a large enough training dataset for each new project. On the other hand it also means that there is not nearly as much public data available as there is for images, which one could use to pretrain a generic feature extractor that could then, with only little dedicated training data, be adapted (“fine-tuned”) to the task at hand. To address this bottleneck we explore the possibility to transfer knowledge from the 2D image domain to 3D point clouds. That strategy is of particular interest for mobile mapping systems that capture both point clouds and images, in a fully calibrated setting that makes it easy to connect the two domains. We find that, as expected, naively segmenting in image space and mapping the resulting labels onto the point cloud is not sufficient, as visual ambiguities, residual calibration errors, etc. affect the result. Instead, we propose a system that learns to merge image evidence from a varying number viewpoint, and 3D geometry information, into a common representation that encodes point-wise 3D semantics. To validate our approach we make use of a new mobile mapping dataset with 88M annotated 3D points and 2205 oriented multi-view images. In a series of experiments, we show how much label noise is caused by simplistic label transfer, and how well existing semantic segmentation architectures can correct it. Finally, we demonstrate that adding our learned 2D-to-3D multi-view label transfer significantly improves the performance of different segmentation backbones.