3D Object Localization Research Articles

MSSA: Multi-Representation Semantics-Augmented Set Abstraction for 3D Object Detection

Accurate recognition and localization of 3D objects is a fundamental research problem in 3D computer vision. Benefiting from transformation-free point cloud processing and flexible receptive fields, point-based methods have become accurate in 3D point cloud modeling, but still fall behind voxel-based competitors in 3D detection. We observe that the set abstraction module, commonly utilized by point-based methods for downsampling points, tends to retain excessive irrelevant background information, thus hindering the effective learning of features for object detection tasks. To address this issue, we propose MSSA, a Multi-representation Semantics-augmented Set Abstraction for 3D object detection. Specifically, we first design a backbone network to encode different representation features of point clouds, which extracts point-wise features through PointNet to preserve fine-grained geometric structure features, and adopts VoxelNet to extract voxel features and BEV features to enhance the semantic features of key points. Second, to efficiently fuse different representation features of keypoints, we propose a Point feature-guided Voxel feature and BEV feature fusion (PVB-Fusion) module to adaptively fuse multi-representation features and remove noise. At last, a novel Multi-representation Semantic-guided Farthest Point Sampling (MS-FPS) algorithm is designed to help set abstraction modules progressively downsample point clouds, thereby improving instance recall and detection performance with more important foreground points. We evaluate MSSA on the widely used KITTI dataset and the more challenging nuScenes dataset. Experimental results show that compared to PointRCNN, our method improves the AP of “moderate” level for three classes of objects by 7.02%, 6.76%, and 5.44%, respectively. Compared to the advanced point-voxel-based method PV-RCNN, our method improves the AP of “moderate” level by 1.23%, 2.84%, and 0.55% for the three classes, respectively.

LiDAR-Based Intensity-Aware Outdoor 3D Object Detection.

LiDAR-based 3D object detection and localization are crucial components of autonomous navigation systems, including autonomous vehicles and mobile robots. Most existing LiDAR-based 3D object detection and localization approaches primarily use geometric or structural feature abstractions from LiDAR point clouds. However, these approaches can be susceptible to environmental noise due to adverse weather conditions or the presence of highly scattering media. In this work, we propose an intensity-aware voxel encoder for robust 3D object detection. The proposed voxel encoder generates an intensity histogram that describes the distribution of point intensities within a voxel and is used to enhance the voxel feature set. We integrate this intensity-aware encoder into an efficient single-stage voxel-based detector for 3D object detection. Experimental results obtained using the KITTI dataset show that our method achieves comparable results with respect to the state-of-the-art method for car objects in 3D detection and from a bird's-eye view and superior results for pedestrian and cyclic objects. Furthermore, our model can achieve a detection rate of 40.7 FPS during inference time, which is higher than that of the state-of-the-art methods and incurs a lower computational cost.

Vision-Based Object Manipulation for Activities of Daily Living Assistance Using Assistive Robot

The increasing prevalence of upper and lower extremity (ULE) functional deficiencies presents a significant challenge, as it restricts individuals’ ability to perform daily tasks independently. Robotic devices are emerging as assistive devices to assist individuals with limited ULE functionalities in activities of daily living (ADLs). While assistive manipulators are available, manual control through traditional methods like joysticks can be cumbersome, particularly for individuals with severe hand impairments and vision limitations. Therefore, autonomous/semi-autonomous control of a robotic assistive device to perform any ADL task is open to research. This study addresses the necessity of fostering independence in ADLs by proposing a creative approach. We present a vision-based control system for a six-degrees-of-freedom (DoF) robotic manipulator designed for semi-autonomous “pick-and-place” tasks, one of the most common activities among ADLs. Our approach involves selecting and training a deep-learning-based object detection model with a dataset of 47 ADL objects, forming the base for a 3D ADL object localization algorithm. The proposed vision-based control system integrates this localization technique to identify and manipulate ADL objects (e.g., apples, oranges, capsicums, and cups) in real time, returning them to specific locations to complete the “pick-and-place” task. Experimental validation involving an xArm6 (six DoF) robot from UFACTORY in diverse settings demonstrates the system’s adaptability and effectiveness, achieving an overall 72.9% success rate in detecting, localizing, and executing ADL tasks. This research contributes to the growing field of autonomous assistive devices, enhancing independence for individuals with functional impairments.

MMoNet3D: Enhancing monocular 3D object localization with modern advancements

MMoNet3D, an innovative project, addresses monocular multi-object detection and 3D localization in real-time, surpassing the limitations of the existing MoNet3D system. Utilizing a custom deep learning model inspired by Centernet, it excels in oriented car detection across static images, videos, and live webcam feeds. The system, powered by the KITTI dataset, showcases its potential impact on real-time monocular multi-object detection. Its unique features, including 3D bounding box visualization and a user-friendly GUI, set MMoNet3D apart, making it suitable for embedded advanced driving-assistance systems. This advancement promises heightened accuracy and efficiency, contributing to enhanced vehicular safety and operational efficacy. MMoNet3D stands as a beacon in computer vision, paving the way for intelligent transportation systems.

Mono3DVG: 3D Visual Grounding in Monocular Images

We introduce a novel task of 3D visual grounding in monocular RGB images using language descriptions with both appearance and geometry information. Specifically, we build a large-scale dataset, Mono3DRefer, which contains 3D object targets with their corresponding geometric text descriptions, generated by ChatGPT and refined manually. To foster this task, we propose Mono3DVG-TR, an end-to-end transformer-based network, which takes advantage of both the appearance and geometry information in text embeddings for multi-modal learning and 3D object localization. Depth predictor is designed to explicitly learn geometry features. The dual text-guided adapter is proposed to refine multiscale visual and geometry features of the referred object. Based on depth-text-visual stacking attention, the decoder fuses object-level geometric cues and visual appearance into a learnable query. Comprehensive benchmarks and some insightful analyses are provided for Mono3DVG. Extensive comparisons and ablation studies show that our method significantly outperforms all baselines. The dataset and code will be released.

MonoAux: Fully Exploiting Auxiliary Information and Uncertainty for Monocular 3D Object Detection.

Monocular 3D object detection plays a pivotal role in autonomous driving, presenting a formidable challenge by requiring the precise localization of 3D objects within a single image, devoid of depth information. Most existing methods in this domain fall short of harnessing the limited information available in monocular 3D detection tasks. They typically provide only a single detection outcome, omitting essential uncertainty analysis and result post-processing during model inference, thus limiting overall model performance. In this paper, we propose a comprehensive framework that maximizes information extraction from monocular images while encompassing diverse depth estimation and incorporating uncertainty analysis. Specifically, we mine additional information intrinsic to the monocular 3D detection task to augment supervision, thereby addressing the information scarcity challenge. Moreover, our framework handles depth estimation by recovering multiple sets of depth values from calculated visual heights. The final depth estimate and 3D confidence are determined through an uncertainty fusion process, effectively reducing inference errors. Furthermore, to address task weight allocation in multi-task training, we present a versatile training strategy tailored to monocular 3D detection. This approach leverages measurement indicators to monitor task progress, adaptively adjusting loss weights for different tasks. Experimental results on the KITTI and Waymo dataset confirm the effectiveness of our approach. The proposed method consistently provides enhanced performance across various difficulty levels compared to the original framework while maintaining real-time efficiency.

Vision-Based Object Localization and Classification for Electric Vehicle Driving Assistance

The continuous advances in intelligent systems and cutting-edge technology have greatly influenced the development of intelligent vehicles. Recently, integrating multiple sensors in cars has improved and spread the advanced drive-assistance systems (ADAS) solutions for achieving the goal of total autonomy. Despite current self-driving approaches and systems, autonomous driving is still an open research issue that must guarantee the safety and reliability of drivers. This work employs images from two cameras and Global Positioning System (GPS) data to propose a 3D vision-based object localization and classification method for assisting a car during driving. The experimental platform is a prototype of a two-sitter electric vehicle designed and assembled for navigating the campus under controlled mobility conditions. Simultaneously, color and depth images from the primary camera are combined to extract 2D features, which are reprojected into 3D space. Road detection and depth features isolate point clouds representing the objects to construct the occupancy map of the environment. A convolutional neural network was trained to classify typical urban objects in the color images. Experimental tests validate car and object pose in the occupancy map for different scenarios, reinforcing the car position visually estimated with GPS measurements.

Cyber Mobility Mirror: A Deep Learning-Based Real-World Object Perception Platform Using Roadside LiDAR

Object perception plays a fundamental role in Cooperative Driving Automation (CDA) which is regarded as a revolutionary promoter for next-generation transportation systems. However, the vehicle-based perception may suffer from the limited sensing range and occlusion as well as low penetration rates in connectivity. In this paper, we propose <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Cyber Mobility Mirror</i> ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">CMM</i> ), a next-generation real-world object perception system for 3D object detection, tracking, localization, and reconstruction, to explore the potential of roadside sensors for enabling CDA in the real world. The CMM system consists of six main components: i) the data pre-processor to retrieve and preprocess the raw data; ii) the roadside 3D object detector to generate 3D detection results; iii) the multi-object tracker to identify detected objects; iv) the global locator to generate geo-localization information; v) the mobile-edge-cloud-based communicator to transmit perception information to equipped vehicles, and vi) the onboard advisor to reconstruct and display the real-time traffic conditions. An automatic perception evaluation approach is proposed to support the assessment of data-driven models without human-labeling requirements and a CMM field-operational system is deployed at a real-world intersection to assess the performance of the CMM. Results from field tests demonstrate that our CMM prototype system can achieve 96.99% precision and 83.62% recall for detection and 73.55% ID-recall for tracking. High-fidelity real-time traffic conditions (at the object level) can be geo-localized with a root-mean-square error (RMSE) of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$0.69m$</tex-math> </inline-formula> and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$0.33m$</tex-math> </inline-formula> for lateral and longitudinal direction, respectively, and displayed on the GUI of the equipped vehicle with a frequency of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$3-4 Hz$</tex-math> </inline-formula> .

3D integral imaging depth estimation of partially occluded objects using mutual information and Bayesian optimization.

Integral imaging (InIm) is useful for passive ranging and 3D visualization of partially-occluded objects. We consider 3D object localization within a scene and in occlusions. 2D localization can be achieved using machine learning and non-machine learning-based techniques. These techniques aim to provide a 2D bounding box around each one of the objects of interest. A recent study uses InIm for the 3D reconstruction of the scene with occlusions and utilizes mutual information (MI) between the bounding box in this 3D reconstructed scene and the corresponding bounding box in the central elemental image to achieve passive depth estimation of partially occluded objects. Here, we improve upon this InIm method by using Bayesian optimization to minimize the number of required 3D scene reconstructions. We evaluate the performance of the proposed approach by analyzing different kernel functions, acquisition functions, and parameter estimation algorithms for Bayesian optimization-based inference for simultaneous depth estimation of objects and occlusion. In our optical experiments, mutual-information-based depth estimation with Bayesian optimization achieves depth estimation with a handful of 3D reconstructions. To the best of our knowledge, this is the first report to use Bayesian optimization for mutual information-based InIm depth estimation.

3D Object Detection via 2D Segmentation-Based Computational Integral Imaging Applied to a Real Video.

This study aims to achieve accurate three-dimensional (3D) localization of multiple objects in a complicated scene using passive imaging. It is challenging, as it requires accurate localization of the objects in all three dimensions given recorded 2D images. An integral imaging system captures the scene from multiple angles and is able to computationally produce blur-based depth information about the objects in the scene. We propose a method to detect and segment objects in a 3D space using integral-imaging data obtained by a video camera array. Using objects' two-dimensional regions detected via deep learning, we employ local computational integral imaging in detected objects' depth tubes to estimate the depth positions of the objects along the viewing axis. This method analyzes object-based blurring characteristics in the 3D environment efficiently. Our camera array produces an array of multiple-view videos of the scene, called elemental videos. Thus, the proposed 3D object detection applied to the video frames allows for 3D tracking of the objects with knowledge of their depth positions along the video. Results show successful 3D object detection with depth localization in a real-life scene based on passive integral imaging. Such outcomes have not been obtained in previous studies using integral imaging; mainly, the proposed method outperforms them in its ability to detect the depth locations of objects that are in close proximity to each other, regardless of the object size. This study may contribute when robust 3D object localization is desired with passive imaging, but it requires a camera or lens array imaging apparatus.

Monocular 3D Object Detection Based on Pseudo Multimodal Information Extraction and Keypoint Estimation

Three-dimensional object detection is an essential and fundamental task in the field of computer vision which can be widely used in various scenarios such as autonomous driving and visual navigation. In view of the current insufficient utilization of image information in current monocular camera-based 3D object detection algorithms, we propose a monocular 3D object detection algorithm based on pseudo-multimodal information extraction and keypoint estimation. We utilize the original image to generate pseudo-lidar and a bird’s-eye view, and then feed the fused data of the original image and pseudo-lidar to the keypoint-based network for an initial 3D box estimation, finally using the bird’s-eye view to refine the initial 3D box. The experimental performance of our method exceeds state-of-the-art algorithms under the evaluation criteria of 3D object detection and localization on the KITTI dataset, achieving the best experimental performance so far.

Object Recognition and Localization for Pick-and-Place Task using Difference-based Dynamic Movement Primitives

In this paper, a framework is developed for three dimensional (3D) object recognition, localization, and anthropomorphic manipulation using YOLOv5, two low-cost webcams, and a Franka Emika (FE) Panda manipulator. Autonomous pick-and-place tasks are executed in a human-like manner using impedance control and novel difference-based Dynamic Movement Primitives (DMP) trajectory planning. The high accuracy vision system is developed with two webcams by using triangular relation to determine the object's position. The difference-based DMP algorithm and task execution plan are used to learn unique trajectories for different objects based on human demonstration and the object's coordinates. By integrating these components together, experimental studies on pick-and-place tasks are successfully performed for two different objects.

Monopulse ladar: super-resolved 3D localization with Si-photonic serpentine optical phased arrays

We present an optical ranging and super-resolution object localization method, monopulse ladar, used to determine the angle of a point target in two dimensions to a few percent of an optical beam width from differential measurements of four just-resolved waveform-encoded beams while simultaneously providing target range via either coherent or incoherent coded waveform correlation. A common optical carrier is shifted by four GHz-scale tones, each modulated with distinct ranging waveforms, which when transmitted from a Si-photonic 2D wavelength-steered serpentine optical phased array (SOPA) aperture form an encoded rectangular beam cluster that propagates to and scatters from a distant point target. Superposed backscattered target returns from each beam are decoded by correlation with reference waveforms at the receiver. The angular position of the target along the two orthogonal axes is calculated from pairwise ratios of beam amplitudes, while target range is determined from the round-trip time delay of each beam as measured with a wideband correlation peak. The analysis of coherent and incoherent monopulse ladar architectures presented herein indicates that a 50-fold increase in angular resolution-to the tens of arcseconds level-of a point target located within a wide field of regard is achievable while maintaining cm-scale resolution-limited ranging using a single SOPA tile transmitter, with further improvement in angular resolution possible through arrayed tiling of SOPAs. Implementation of monopulse ladar with a SOPA aperture enables non-mechanically steered high-resolution 3D object localization in a compact, low-control complexity form factor.

Object Localization Assistive System Based on CV and Vibrotactile Encoding.

Intelligent assistive systems can navigate blind people, but most of them could only give non-intuitive cues or inefficient guidance. Based on computer vision and vibrotactile encoding, this paper presents an interactive system that provides blind people with intuitive spatial cognition. Different from the traditional auditory feedback strategy based on speech cues, this paper firstly introduces a vibration-encoded feedback method that leverages the haptic neural pathway and enables the users to interact with objects other than manipulating an assistance device. Based on this strategy, a wearable visual module based on an RGB-D camera is adopted for 3D spatial object localization, which contributes to accurate perception and quick object localization in the real environment. The experimental results on target blind individuals indicate that vibrotactile feedback reduces the task completion time by over 25% compared with the mainstream voice prompt feedback scheme. The proposed object localization system provides a more intuitive spatial navigation and comfortable wearability for blindness assistance.

An efficient regression method for 3D object localization in machine vision systems

&lt;p&gt;Machine vision or robot vision plays is playing an important role in many industrial systems and has a lot of potential applications in the future of automation tasks such as in-house robot managing, swarm robotics controlling, product line observing, and robot grasping. One of the most common yet challenging tasks in machine vision is 3D object localization. Although several works have been introduced and achieved good results for object localization, there is still room to further improve the object location determination. In this paper, we introduce a novel 3D object localization algorithm in which a checkerboard pattern-based method is used to initialize the object location and followed by a regression model to regularize the object location. The proposed object localization is employed in a low-cost robot grasping system where only one simple 2D camera is used. Experimental results showed that the proposed algorithm significantly improves the accuracy of the object localization when compared to the relevant works.&lt;/p&gt;

End-to-End Optimization of LiDAR Beam Configuration for 3D Object Detection and Localization

Existing learning methods for LiDAR-based applications use 3D points scanned under a pre-determined beam configuration, e.g., the elevation angles of beams are often evenly distributed. Those fixed configurations are task-agnostic, so simply using them can lead to sub-optimal performance. In this work, we take a new route to learn to optimize the LiDAR beam configuration for a given application. Specifically, we propose a reinforcement learning-based learning-to-optimize (RL-L2O) framework to automatically optimize the beam configuration in an end-to-end manner for different LiDAR-based applications. The optimization is guided by the final performance of the target task and thus our method can be integrated easily with any LiDAR-based application as a simple drop-in module. The method is especially useful when a low-resolution (low-cost) LiDAR is needed, for instance, for system deployment at a massive scale. We use our method to search for the beam configuration of a low-resolution LiDAR for two important tasks: 3D object detection and localization. Experiments show that the proposed RL-L2O method improves the performance in both tasks significantly compared to the baseline methods. We believe that a combination of our method with the recent advances of programmable LiDARs can start a new research direction for LiDAR-based active perception. The code is publicly available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">github.com/vniclas/lidar_beam_selection</uri> .

Improved Indoor 3D Localization using LoRa Wireless Communication

New communication technologies are being developed for the Internet of Things (IoT), such as the wireless long distance and low power networks (LPWAN). The LoRa (Long Range) protocol, a wireless communication technology for long-range applications, is a type of LPWAN that has been recently widely used. The 3D localization and tracking of objects using LPWAN is a hotspot in IoT applications. In spite of many methods for improving the accuracy of 3D localization using traditional wireless communication such as Wi-Fi, ZigBee and Bluetooth, a few of them are available for long-range applications. In this way, this paper presents an improved 3D localization system based on LoRa communication and altimeter for indoor applications. The method uses the traditional trilateration technique with the signal strength indicator (RSSI) together with experimental update of parameters for improving the localization accuracy. Experimental tests were done in two indoor scenarios shown a significant improvement of up to 80%, with a maximum confident interval (95%) of 0,121m, in the 3D localization accuracy, in addition to better results than related papers in the literature.

3D Object Recognition and Localization with a Dense LiDAR Scanner

Dense scanning is an effective solution for refined geometrical modeling applications. The previous studies in dense environment modeling mostly focused on data acquisition techniques without emphasizing autonomous target recognition and accurate 3D localization. Therefore, they lacked the capability to output semantic information in the scenes. This article aims to make complementation in this aspect. The critical problems we solved are mainly in two aspects: (1) system calibration to ensure detail-fidelity for the 3D objects with fine structures, (2) fast outlier exclusion to improve 3D boxing accuracy. A lightweight fuzzy neural network is proposed to remove most background outliers, which was proven in experiments to be effective for various objects in different situations. With precise and clean data ensured by the two abovementioned techniques, our system can extract target objects from the original point clouds, and more importantly, accurately estimate their center locations and orientations.

Kimera: From SLAM to spatial perception with 3D dynamic scene graphs

Humans are able to form a complex mental model of the environment they move in. This mental model captures geometric and semantic aspects of the scene, describes the environment at multiple levels of abstractions (e.g., objects, rooms, buildings), includes static and dynamic entities and their relations (e.g., a person is in a room at a given time). In contrast, current robots’ internal representations still provide a partial and fragmented understanding of the environment, either in the form of a sparse or dense set of geometric primitives (e.g., points, lines, planes, and voxels), or as a collection of objects. This article attempts to reduce the gap between robot and human perception by introducing a novel representation, a 3D dynamic scene graph (DSG), that seamlessly captures metric and semantic aspects of a dynamic environment. A DSG is a layered graph where nodes represent spatial concepts at different levels of abstraction, and edges represent spatiotemporal relations among nodes. Our second contribution is Kimera, the first fully automatic method to build a DSG from visual–inertial data. Kimera includes accurate algorithms for visual–inertial simultaneous localization and mapping (SLAM), metric–semantic 3D reconstruction, object localization, human pose and shape estimation, and scene parsing. Our third contribution is a comprehensive evaluation of Kimera in real-life datasets and photo-realistic simulations, including a newly released dataset, uHumans2, which simulates a collection of crowded indoor and outdoor scenes. Our evaluation shows that Kimera achieves competitive performance in visual–inertial SLAM, estimates an accurate 3D metric–semantic mesh model in real-time, and builds a DSG of a complex indoor environment with tens of objects and humans in minutes. Our final contribution is to showcase how to use a DSG for real-time hierarchical semantic path-planning. The core modules in Kimera have been released open source.

A Fast and Robust Rotation Search and Point Cloud Registration Method for 2D Stitching and 3D Object Localization

Rotation search and point cloud registration are two fundamental problems in robotics, geometric vision, and remote sensing, which aim to estimate the rotation and transformation between the 3D vector sets and point clouds, respectively. Due to the presence of outliers (probably in very large numbers) among the putative vector or point correspondences in real-world applications, robust estimation is of great importance. In this paper, we present Inlier searching using COmpatible Structures (ICOS), a novel, efficient, and highly robust solver for both the correspondence-based rotation search and point cloud registration problems. Specifically, we (i) propose and construct a series of compatible structures for the two problems, based on which various invariants can be established, and (ii) design time-efficient frameworks to filter out outliers and seek inliers from the invariant-constrained random sampling based on the compatible structures proposed. In this manner, even with extreme outlier ratios, inliers can be effectively sifted out and collected for solving the optimal rotation and transformation, leading to our robust solver ICOS. Through plentiful experiments over standard datasets, we demonstrated that: (i) our solver ICOS is fast, accurate, and robust against over 95% outliers with nearly a 100% recall ratio of inliers for rotation search and both known-scale and unknown-scale registration, outperforming other state-of-the-art methods, and (ii) ICOS is practical for use in real-world application problems including 2D image stitching and 3D object localization.