Multi-view Object Research Articles

Toward Real-World Multi-View Object Classification: Dataset, Benchmark, and Analysis

Toward Real-World Multi-View Object Classification: Dataset, Benchmark, and Analysis

Simultaneous Multi-View Object Recognition and Grasping in Open-Ended Domains

To aid humans in everyday tasks, robots need to know which objects exist in the scene, where they are, and how to grasp and manipulate them in different situations. Therefore, object recognition and grasping are two key functionalities for autonomous robots. Most state-of-the-art approaches treat object recognition and grasping as two separate problems, even though both use visual input. Furthermore, the knowledge of the robot is fixed after the training phase. In such cases, if the robot encounters new object categories, it must be retrained to incorporate new information without catastrophic forgetting. To resolve this problem, we propose a deep learning architecture with an augmented memory capacity to handle open-ended object recognition and grasping simultaneously. In particular, our approach takes multi-views of an object as input and jointly estimates pixel-wise grasp configuration as well as a deep scale- and rotation-invariant representation as output. The obtained representation is then used for open-ended object recognition through a meta-active learning technique. We demonstrate the ability of our approach to grasp never-seen-before objects and to rapidly learn new object categories using very few examples on-site in both simulation and real-world settings. Our approach empowers a robot to acquire knowledge about new object categories using, on average, less than five instances per category and achieve 95%\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$95\\%$$\\end{document} object recognition accuracy and above 91%\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$91\\%$$\\end{document} grasp success rate on (highly) cluttered scenarios in both simulation and real-robot experiments. A video of these experiments is available online at: https://youtu.be/n9SMpuEkOgk

Recurrent Volume-Based 3-D Feature Fusion for Real-Time Multiview Object Pose Estimation

Recurrent Volume-Based 3-D Feature Fusion for Real-Time Multiview Object Pose Estimation

Few-shot multi-view object classification via dual augmentation network

Existing multi-view object classification algorithms usually rely on sufficient labeled multi-view objects, which substantially restricts their scalability to novel classes with few annotated training samples in real-world applications. Aiming to go beyond these limitations, we explore a novel yet challenging task, few-shot multi-view object classification (FS-MVOC), which expects the network to build its classification ability efficiently based on limited labeled multi-view objects. To this end, we design a dual augmentation network (DANet) to provide excellent performance for the under-explored FS-MVOC task. On the one hand, we employ an attention-guided multi-view representation augmentation (AMRA) strategy to help the model focus on salient features and suppress unnecessary ones on multiple views of multi-view objects, resulting in more discriminative multi-view representations. On the other hand, during the meta-training stage, we adopt the category prototype augmentation (CPA) strategy to improve the class-representativeness of each prototype and increase the inter-prototype difference by injecting Gaussian noise in the deep feature space. Extensive experiments on the benchmark datasets (Meta-ModelNet and Meta-ShapeNet) indicate the effectiveness and robustness of DANet.

MORE: simultaneous multi-view 3D object recognition and pose estimation

Simultaneous object recognition and pose estimation are two key functionalities for robots to safely interact with humans as well as environments. Although both object recognition and pose estimation use visual input, most state of the art tackles them as two separate problems since the former needs a view-invariant representation, while object pose estimation necessitates a view-dependent description. Nowadays, multi-view convolutional neural network (MVCNN) approaches show state-of-the-art classification performance. Although MVCNN object recognition has been widely explored, there has been very little research on multi-view object pose estimation methods, and even less on addressing these two problems simultaneously. The pose of virtual cameras in MVCNN methods is often pre-defined in advance, leading to bound the application of such approaches. In this paper, we propose an approach capable of handling object recognition and pose estimation simultaneously. In particular, we develop a deep object-agnostic entropy estimation model, capable of predicting the best viewpoints of a given 3D object. The obtained views of the object are then fed to the network to simultaneously predict the pose and category label of the target object. Experimental results showed that the views obtained from such positions are descriptive enough to achieve a good accuracy score. Furthermore, we designed a real-life serve drink scenario to demonstrate how well the proposed approach worked in real robot tasks. Code is available online at: https://github.com/SubhadityaMukherjee/more_mvcnn.

MOL: Towards accurate weakly supervised remote sensing object detection via Multi-view nOisy Learning

Weakly supervised object detection (WSOD) has become a growing trend in remote sensing images (RSIs) analysis, which aims to train an object detector given only image-level annotations (i.e., categories of existing objects in the given image), reducing the dependency on expensive object-level annotations. However, an object detector trained on such weak annotations is susceptible to the issues of background interference and missing detection due to the complex backgrounds and densely arranged objects of RSIs. In this paper, we develop a novel Multi-view nOisy Learning framework, named MOL, to tackle the abovementioned problems, which consists of two sequential stages: reliable object discovery and progressive object mining. (i) In reliable object discovery, we formulate WSOD as a multiple instance learning problem to discover potential objects in the given image, which process inevitably suffers background interference due to a lack of accurate supervision signal. To this end, we take inspiration from noisy learning theory and propose a temporal consistency-based instance selection strategy for discovering reliable foreground objects, reducing the risk of background interference. (ii) In progressive object mining, we serve the observed reliable objects as the initial pseudo-labels for building an object detector and propose a novel multi-view object mining strategy to progressively mine neglected objects from multiple distinct views, alleviating the missing detection issue. In this way, a well-trained object detector is obtained, which can achieve satisfactory performance in RSIs. Experimental results on two public benchmarks demonstrate that our method outperforms previous state-of-the-art methods by a large margin of 13.97% mAP and 1.69% mAP on NWPU VHR-10.v2 and DIOR, respectively. The code is available at: https://github.com/GC-WSL/MOL.

3D Reconstruction for Multi-view Objects

Deep learning-based 3D reconstruction neural networks have achieved good performance on generating 3D features from 2D features. However, they often lead to feature loss in reconstruction. In this paper we propose a multi-view object 3D reconstruction neural network, named P2VNet. The depth estimation module of the front and back layers of P2VNet realizes the smooth transformation from 2D features to 3D features, which improves the performance of single view reconstruction. A multi-scale fusion sensing module in multi-view fusion is also proposed, where more receptive fields are added to generate richer context-aware features. We also introduce 3DFocal Loss to replace binary cross-entropy to address the problems of unbalanced space occupation of the voxel grid and complex division of partial grid occupation. Our experimental results have demonstrated that P2VNet has achieved higher accuracy than existing works.

Task-Oriented Communication for Multidevice Cooperative Edge Inference

This paper investigates task-oriented communication for multi-device cooperative edge inference, where a group of distributed low-end edge devices transmit the extracted features of local samples to a powerful edge server for inference. While cooperative edge inference can overcome the limited sensing capability of a single device, it substantially increases the communication overhead and may incur excessive latency. To enable low-latency cooperative inference, we propose a learning-based communication scheme that optimizes local feature extraction and distributed feature encoding in a task-oriented manner, i.e., to remove data redundancy and transmit information that is essential for the downstream inference task rather than reconstructing the data samples at the edge server. Specifically, we leverage Tishby’s information bottleneck (IB) principle (Tishby et al., 2000) to extract the task-relevant feature at each edge device, and adopt the distributed information bottleneck (DIB) framework of Aguerri and Zaidi, 2021, to formalize a single-letter characterization of the optimal rate-relevance tradeoff for distributed feature encoding. To admit flexible control of the communication overhead, we extend the DIB framework to a distributed deterministic information bottleneck (DDIB) objective that explicitly incorporates the representational costs of the encoded features. As the IB-based objectives are computationally prohibitive for high-dimensional data, we adopt variational approximations to make the optimization problems tractable. To compensate for the potential performance loss due to the variational approximations, we also develop a selective retransmission (SR) mechanism to identify the redundancy in the encoded features among multiple edge devices to attain additional communication overhead reduction. Extensive experiments on multi-view image classification and multi-view object recognition tasks evidence that the proposed task-oriented communication scheme achieves a better rate-relevance tradeoff than existing methods.

Hierarchical multi-view metric learning with HSIC regularization

As the information era develops rapidly, it’s common to utilize multiple features from different sources to represent one object. Measuring the similarity between multi-view objects is the fundamental task in multi-view learning. To effectively measure the similarity between multi-view samples, multi-view metric learning has gained extensive attention recently. Nevertheless, most existing methods merely focus on the closeness of similar pairs and the separability of dissimilar ones inside each view, so that rich consensus properties existing in multi-views data might be ignored to some extent. To mitigate this issue, we come up with a novel method entitled Hierarchical Multi-view Metric learning with HSIC regularization (HM2H). HM2H aims to simultaneously maintain the closeness of similar points and the separability of dissimilar ones in intra-view and inter-view. Since multiple views depict different perspectives of the same object, the shared metric is introduced to capture the consensus information among those views. Moreover, we take advantage of the Hilbert–Schmidt Independence Criterion to seek the maximum distribution agreement of the multi-view dataset. Correspondingly, an algorithm based on Alternating Direction Method is provided to solve the proposed HM2H. Finally, various experimental results on five visual recognition datasets confirm the effectiveness and feasibility of our proposed method.

On the robustness of self-supervised representations for multi-view object classification

• Analyse self-supervised and supervised embeddings regarding viewpoint robustness in both controlled and real-world settings. • Controlled setting created by approximating view using a homography on multiple common benchmark datasets. • Performance in-the-wild quantified by analysing the representations on multiple real-world, multi-view datasets. • Demonstrate that self-supervised representations are more robust to both controlled and real-world settings. • Show that self-supervised representations require less context than supervised ones in recognising objects in novel views. It is known that representations from self-supervised pre-training can perform on par, and often better, on various downstream tasks than representations from fully-supervised pre-training. This has been shown in a host of settings such as generic object classification and detection, semantic segmentation, and image retrieval . However, some issues have recently come to the fore that demonstrate some of the failure modes of self-supervised representations, such as performance on non-ImageNet-like data, or complex scenes. In this paper, we show that self-supervised representations based on the instance discrimination objective lead to better representations of objects that are more robust to changes in the viewpoint and perspective of the object. We perform experiments of modern self-supervised methods against multiple supervised baselines to demonstrate this, including approximating object viewpoint variation through homographies , and real-world tests based on several multi-view datasets. We find that self-supervised representations are more robust to object viewpoint and appear to encode more pertinent information about objects that facilitate the recognition of objects from novel views.

Autonomous Robotic Manipulation: Real-Time, Deep-Learning Approach for Grasping of Unknown Objects

Recent advancement in vision-based robotics and deep-learning techniques has enabled the use of intelligent systems in a wider range of applications requiring object manipulation. Finding a robust solution for object grasping and autonomous manipulation became the focus of many engineers and is still one of the most demanding problems in modern robotics. This paper presents a full grasping pipeline proposing a real-time data-driven deep-learning approach for robotic grasping of unknown objects using MATLAB and convolutional neural networks. The proposed approach employs RGB-D image data acquired from an eye-in-hand camera centering the object of interest in the field of view using visual servoing. Our approach aims at reducing propagation errors and eliminating the need for complex hand tracking algorithm, image segmentation, or 3D reconstruction. The proposed approach is able to efficiently generate reliable multi-view object grasps regardless of the geometric complexity and physical properties of the object in question. The proposed system architecture enables simple and effective path generation and a real-time tracking control. In addition, our system is modular, reliable, and accurate in both end effector path generation and control. We experimentally justify the efficacy and effectiveness of our overall system on the Barrett Whole Arm Manipulator.

PSNet: Perspective-sensitive convolutional network for object detection

Multi-view object detection is challenging due to the influence of the different view-angles on intra-class similarity. The uniformed feature representation of traditional detectors couples the object’s perspective attribute and semantic feature, and the variances of perspective will cause intra-class differences. In this paper, a robust perspective-sensitive network (PSNet) is proposed to overcome the above problem. The uniformed feature is replaced by the perspective-specific structural feature, which makes the network perspective sensitive. Its essence is to learn multiple perspective spaces. In each perspective space, the semantic feature is decoupled from the perspective attribute and is robust to perspective variances. Perspective-sensitive RoI pooling and loss function are proposed for perspective-sensitive learning. Experiments on Pascal3D + and SpaceNet MOVI show the effectiveness and superiority of the PSNet.

2D-to-3D Projection for Monocular and Multi-View 3D Object Detection in Outdoor Scenes

In this article, we introduce the task of multi-view RGB-based 3D object detection as an end-to-end optimization problem. In a multi-view formulation of the 3D object detection problem, several images of a static scene are used to detect objects in the scene. To address the 3D object detection problem in a multi-view formulation, we propose a novel 3D object detection method named ImVoxelNet. ImVoxelNet is based on a fully convolutional neural network. Unlike existing 3D object detection methods, ImVoxelNet works directly with 3D representations and does not mediate 3D object detection through 2D object detection. The proposed method accepts multi-view inputs. The number of monocular images in each multi-view input can vary during training and inference; actually, this number might be unique for each multi-view input. Moreover, we propose to treat a single RGB image as a special case of a multi-view input. Accordingly, the proposed method can also accept monocular inputs with no modifications. Through extensive evaluation, we demonstrate that the proposed method successfully handles a variety of outdoor scenes. Specifically, it achieves state-of-the-art results in car detection on KITTI (monocular) and nuScenes (multi-view) benchmarks among all methods that accept RGB images. The proposed method operates in real-time, which makes it possible to integrate it into the navigation systems of autonomous devices. The results of this study can be used to address tasks of navigation, path planning, and semantic scene mapping.

Modality adaptation in multimodal data

Recently, multimodal data has received much attention. In classical machine learning, it is assumed that all data comes from one modality while in multimodal machine learning, the information comes from different modalities. In multimodal machine learning, transiting, or fusing knowledge from different modalities is an important step. Hence, in these steps, the different marginal distributions between different modalities should be taken into account. However, in recent years, modality adaptation has not gotten enough attention. The motivation of this work is to consider modality adaptation to effectively encode the shared common or complementary knowledge in multimodal data. To reduce the modality shift, we present a new perspective on the modality adaptation algorithm. In multimodal data, by applying the existing domain adaptation techniques to reduce the modality shift, a problem arises because of the insufficient capability of those techniques in preserving complementary knowledge. Our proposed modality adaptation is designed such that it simultaneously considers both the shared and complementary knowledge of each modality while preserving the discriminative ability of each modality in the label space. To evaluate the proposed approach, we have applied it to two different multimodal applications: multi-view object detection and RGBD image semantic segmentation. Our results show that the proposed modality adaptation technique is successful in transferring and fusing knowledge.

Fine-Grained Categorization From RGB-D Images

In the field of computer vision, fine-grained visual categorization has attracted a lot of attention and made great progress due to convolutional neural networks and a large number of publicly available datasets. With next-generation sensing technology, RGB-D cameras can provide high-quality synchronized RGB and depth images for solving many computer vision problems. Although RGB-D cameras have been used in the context of multi-view object category detection and scene understanding, they have not been widely used in fine-grained classification. In this paper, we introduce a multiview RGB-D dataset RGBD-FG for fine-grained categorization. Currently, the dataset contains 93 051 RGB-D images covering 19 super-categories and 50 sub-categories of common vegetables and fruit, and is organized in a hierarchical manner. We provide extensive experimental results to establish state-of-the-art benchmarks for our dataset, illustrating its diversity and scope for improvement through future work. We also propose a novel modality-specific multimodal network called FS-Multimodal network, which can solve two limitations of multimodal networks trained based on fine-tuning techniques: over-fitting and lack of effective depth-specific features. We hope that our study lays the foundations for fine-grained categorization of RGB-D data.

Learning the Global Descriptor for 3-D Object Recognition Based on Multiple Views Decomposition

The key point of view based strategies for the analysis of 3D object is to obtain a global descriptor from a collection of its rendered views on 2D images. The views are always redundantly sampled as to ensure the completeness of the information. In this paper, we bring new insight into the study of multi-view object recognition, which models an object as a View Mixture Model (VMM). We argue that each object represented by the multiple views can be decomposed into just a few latent views. Based on the VMM, we introduce a decomposition module to mine the representations of these latent views for the construction of a compact and comprehensive descriptor. After that, we further propose a view alignment module to ensure the descriptor is robust to the variation of view permutation. We evaluate our method on the ModelNet-40, ModelNet-10 and ShapeNetCore55 datasets. The experimental results show that our method can learn efficient and comprehensive representation for 3D objects, and achieves state-of-the-art performance on both the 3D object classification and retrieval tasks. Lastly, experiments are conducted for benchmarking various popular CNN backbones on the 3D object recognition task, with a view to achieving fair comparisons and promoting the future research in this area. Codes for our paper are released: &#x201C;<uri>https://github.com/hjjpku/multi_view_sort</uri> &#x201D;.

Orthogonal multi-view three-dimensional object representations in memory revealed by serial reproduction

Orthogonal multi-view three-dimensional object representations in memory revealed by serial reproduction

Active multiview recognition with hidden Markov temporal support

Our paper deals with active multiview object recognition focusing on the directional support of sequential multiple shots. Since inertial sensors are easily available nowadays, we propose the use of them to estimate the orientation change of the camera and thus to estimate the probability of relative poses. With the help of relative orientation change, we can compute transition probabilities between possible poses and can use a hidden Markov model to evaluate state (pose) sequences and can thus increase the recognition rate. Furthermore, we can plan our next viewing position to minimize the risk of misclassification, resulting in higher overall recognition rates. Besides giving the theoretical details, we use two datasets to illustrate the performance of our model through several tests including occlusion, blur, Gaussian noise, and to compare to a solution with a long short-term memory network.

One-shot learning technique regression of reconfigurable learning network for generative modeling in interconnected imaging infrastructure

Learning visual models of object classes conventionally require hundreds or a large number of training samples. Conventional gradient-based approaches for target recognition require lot of data to be trained on and require exhaustive training with high computational expense. Hence, when a new condition or untrained data is encountered, such systems inadequately misconfigure newly learned feature sets in the trained model. This misconfigures the structure of re-learned features and is then carried out in subsequent recognition stages. Thus, a development in this scenario with low training time will allow us to fend of this disadvantage. This study presents a new automatic target recognition framework that gives the enhanced performance of target-recognition system when several imaging sensors are connected with one another. This is in contrast with traditional automatic target recognition frameworks, which utilizes one-on-one computational model over synthetic-aperture radar image-processing systems. The work comprises of a learning-based classifications strategy when dealing with sharing of learned parameters over the network to discern critical changes in target-recognition performance by utilizing a novel one-shot learning-based reconfigurable learning network for image processing platform. This upgrades the networked connected CCTV and multiview synthetic-aperture radar image object identification and recognition process.

Multi-view object topography measurement with optical sectioning structured illumination microscopy

Various optical instruments have been developed for three-dimensional (3D) surface topography, including the white light interference, reflectance confocal microscopes, and digital holographic microscopes, etc. However, the steep local slope of objects may cause the light to be reflected in a way that it will not be captured by the objective lens because of the finite collection angle of the objective. To solve this "shadow problem," we report a method to enlarge the collection angle range of optical sectioning structured illumination microscopy by capturing sectioned images of the objects from multiple angle of views. We develop a multi-view image fusion algorithm to reconstruct a single 3D image. Using this method, we detect previously invisible details whose slopes are beyond the collection angle of the objective. The proposed approach is useful for height map measurement and quantitative analyses in a variety of fields, such as biology, materials science, microelectronics, etc.