Framework For Object Recognition Research Articles

A deep learning-based framework for object recognition in ecological environments with dense focal loss and occlusion

A deep learning-based framework for object recognition in ecological environments with dense focal loss and occlusion

Online Active Continual Learning for Robotic Lifelong Object Recognition.

In real-world applications, robotic systems collect vast amounts of new data from ever-changing environments over time. They need to continually interact and learn new knowledge from the external world to adapt to the environment. Particularly, lifelong object recognition in an online and interactive manner is a crucial and fundamental capability for robotic systems. To meet this realistic demand, in this article, we propose an online active continual learning (OACL) framework for robotic lifelong object recognition, in the scenario of both classes and domains changing with dynamic environments. First, to reduce the labeling cost as much as possible while maximizing the performance, a new online active learning (OAL) strategy is designed by taking both the uncertainty and diversity of samples into account to protect the information volume and distribution of data. In addition, to prevent catastrophic forgetting and reduce memory costs, a novel online continual learning (OCL) algorithm is proposed based on the deep feature semantic augmentation and a new loss-based deep model and replay buffer update, which can mitigate the class imbalance between the old and new classes and alleviate confusion between two similar classes. Moreover, the mistake bound of the proposed method is analyzed in theory. OACL allows robots to select the most representative new samples to query labels and continually learn new objects and new variants of previously learned objects from a nonindependent and identically distributed (i.i.d.) data stream without catastrophic forgetting. Extensive experiments conducted on real lifelong robotic vision datasets demonstrate that our algorithm, even trained with fewer labeled samples and replay exemplars, can achieve state-of-the-art performance on OCL tasks.

Implicit Sparse Shape Representation: A Unified Framework for Object Segmentation and Recognition

Given a classified probabilistic shape dictionary, and an image with a shape similar to some of the elements in the dictionary, this letter introduces a sparse representation based framework with a twofold goal. First, to select a sparse shape combination from the dictionary that best represents the shape, and second, to accurately segment the image taking into account both the sparse shape combination and the image information. A new energy function that combines the region-based segmentation with sparse representation is introduced to accomplish both goals simultaneously. The experimental results show the superior segmentation and recognition capabilities of the proposed model.

SpiReco: Fast and Efficient Recognition of High-Speed Moving Objects with Spike Cameras

Benefited from the high temporal resolution and high dynamic range, spike cameras have shown great potential in recognizing high-speed moving objects. However, the computer vision community has not explored this task due to the lack of spike data and annotations of high-speed moving objects. This paper contributes a novel dataset, named <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">SpiReco</i> ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Spi</i> king datasets for <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Reco</i> gnition), by recording high-speed moving objects using a spike camera. To annotate the dataset, image labels from established datasets such as MNIST, CIFAR10, and CALTECH101 are utilized. Based on this new dataset, this paper proposes the first spike-based object recognition framework. The proposed framework includes a denoise module, which is designed to suppress spike noise by learning spatio-temporal correlation from neighbouring pixels. Additionally, a motion enhancement module is introduced to address high-speed and random motions. Afterward, binarized neural networks are adopted to save computation costs. These efforts result in a fast and efficient processing framework for spiking data. Experimental results demonstrate the effectiveness of the proposed methods. For example, the proposed spike-based recognition framework achieves 80.2% accuracy in recognizing 101 classes of high-speed moving objects using only 2.2ms of spike streams. The SpiReco is available at https://github.com/Evin-X/SpiReco.

Augmented Reality Frameworks for Object Recognition in Learning Application Domain: A Systematic Review

Object recognition is a technology based on computer vision techniques to identify the object in images or video. A previous study revealed that object recognition is useful to improve the learning outcome in augmented reality applications. However, there is yet to be a comprehensive augmented reality (AR) framework on object recognition in the learning application domain setting. This study aims to review the existing AR frameworks in object recognition and highlight the important components needed in AR framework. This Systematic Literature Review (SLR) was developed based on the PRISMA method. A preliminary search resulted in a total of 70 articles. After the removal of duplicates and a more rigorous screening based on the titles and abstracts, the number of articles was reduced to 26 articles. Then, for final review, only 9 articles were chosen. Based on the SLR, the components needed in the AR framework for object recognition are tracking, image classification, object localization, object detection, object segmentation, interface and interaction. Some of the limitations identified from the existing AR framework for object recognition include the lower-resolution camera that uses compressed images and due to the low detection accuracy and the limited of the tools. The proposed framework suggests the initial for AR framework object recognition.

SOLARNet: A single stage regression based framework for efficient and robust object recognition in aerial images

Object recognition and localization play a crucial role in aerial images and their applications. The aerial images are challenging due to the large aspect ratio, arbitrary orientation, variation in scales, and non-uniform and cluttered object distribution. To address these challenges, we propose an efficient and robust model called the Simultaneous Object Localization and Recognition Network (SOLARNet), which is a fusion network that integrates two different sub-networks: PixelAttentionDetector (PD) and RotationDetector (RD). The PD considers features from different scales and cluttered objects, while RD handles rotation invariance, giving horizontal and oriented object detection results. The state-of-the-art model fails to improve accuracy when images are adversarially attacked. SOLARNet is not only efficient in terms of accuracy but also robust concerning to Fast Gradient Sign Method (FGSM), Basic Iterative Method (BIM), Jacobian Based Saliency Map (JSM) adversarial attacks, which is the crucial factor for any mission-critical system. We have performed experiments and achieved the accuracy on the publicly available DOTA dataset (75.00% mAP, 66.40% mAP) and DIOR dataset (88.60% mAP, 81.50% mAP) for horizontal and oriented object recognition tasks respectively while having high inference speed. Qualitative and quantitative results reported in this paper substantiate the superiority of SOLARNet over other state-of-the-art methodologies.

Object Recognition Using Mechanical Impact, Viscoelasticity, and Surface Friction During Interaction.

Current robotic haptic object recognition relies on statistical measures derived from movement dependent interaction signals such as force, vibration or position. Mechanical properties, which can be estimated from these signals, are intrinsic object properties that may yield a more robust object representation. Therefore, this paper proposes an object recognition framework using multiple representative mechanical properties: stiffness, viscosity and friction coefficient as well as the coefficient of restitution, which has been rarely used to recognise objects. These properties are estimated in real-time using a dual Kalman filter (without tangential force measurements) and then are used for object classification and clustering. The proposed framework was tested on a robot identifying 20 objects through haptic exploration. The results demonstrate the technique's effectiveness and efficiency, and that all four mechanical properties are required for the best recognition rate of 98.18 ± 0.424%. For object clustering, the use of these mechanical properties also results in superior performance when compared to methods based on statistical parameters.

Detection and classification of white blood cells with an improved deep learning-based approach

The analysis of white blood cells, which defend the body against deadly infections and disease-causing substances, is an important issue in the medical world. The concentrations of these cells in the blood, examined in 5 classes, i.e. monocytes, eosinophils, basophils, lymphocytes, and neutrophils, vary according to the types of diseases in the body. The peripheral blood smear is widely used to analyze blood cells. Manual evaluation of this method is laborious and time-consuming. At the same time, many environmental and humanistic parameters affect the method's performance. Therefore, in the presented study, a real-time detection process is realized. Firstly, YOLOv5s, YOLOv5x, and Detectron 2 R50-FPN pretrained models in the object recognition framework are used. Next, two original contributions are made to the study to improve the model's performance. The first contribution includes optimizing the activation function, an essential criterion in training the model, and an arrangement provided in the architecture. With this proposed approach, an improvement of 0.006 is achieved in the recognition rates of all classes. The second contribution is the combined use of the YOLO and Detectron2 frameworks, which have two different object evaluation processes. The success rate achieved with this hybrid structure provided an improvement between 3.44% and 14.7% compared to the outputs obtained from the YOLO and Detectron2 pretrained models. In addition, the maximum accuracy rate of this hybrid structure on the test dataset for detection and classification of white blood cells is obtained as 98%.

Deep Active Cross-Modal Visuo-Tactile Transfer Learning for Robotic Object Recognition

We propose for the first time, a novel deep active visuo-tactile cross-modal full-fledged framework for object recognition by autonomous robotic systems. Our proposed network <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">xAVTNet</i> is actively trained with labelled point clouds from a vision sensor with one robot and tested with an active tactile perception strategy to recognise objects never touched before using another robot. We propose a novel visuo-tactile loss ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">VTLoss</i> ) to minimise the discrepancy between the visual and tactile domains for unsupervised domain adaptation. Our framework leverages the strengths of deep neural networks for cross-modal recognition along with active perception and active learning strategies for increased efficiency by minimising redundant data collection. Our method is extensively evaluated on a real robotic system and compared against baselines and other state-of-art approaches. We demonstrate clear outperformance in recognition accuracy compared to the state-of-art visuo-tactile cross-modal recognition method.

Rotation Prediction Based Representative View Locating Framework for 3D Object Recognition

Finding representative views of 3D objects is a key problem in the field of 3D object analysis. We can obtain most of the crucial information of 3D objects from their representative views. In this paper, we propose a framework for learning the features of multi-view rendered images extracted from 3D objects in order to locate representative views of 3D objects. The learning method includes a reinforcement learning based rotation direction prediction (RDP) method and a deep learning based rotation angle prediction (RAP) method. The RDP uses a deep deterministic policy gradient (DDPG) algorithm to learn rotation policies. We improved DDPG to make RDP more applicable for learning 3D object rotation action. RAP uses a convolutional neural network to predict the rotation angles of representative views. We also propose a 3D object classification network. The network reconstructs the rendered images using an encoder–decoder based rendered image reconstruction method and trains the images composed of the original and reconstructed images. Finally, a series of experiments are conducted to validate the feasibility of the proposed methods. Experimental results show the competitive performance of the proposed framework. • We propose a classification reliability discrimination and a reward mechanism for learning 3D object rotation policies. • We improve the classification performance and demonstrate the optimum angles are oblique angles using RDP and RAP. • With the EDIR method, we reconstruct new rendered images without tiny unique details and improve the generalization ability.

Partial Visual-Tactile Fused Learning for Robotic Object Recognition

Currently, visual-tactile fusion learning for robotic object recognition has achieved appealing performance, due to the fact that visual and tactile data can offer complementary information. However: 1) the distinct gap between vision and touch makes it difficult to fully explore the complementary information, which would further lead to performance degradation and 2) most of the existing visual-tactile fused learning methods assume that visual and tactile data are complete, which is often difficult to be satisfied in many real-world applications. In this article, we propose a partial visual-tactile fused (PVTF) framework for robotic object recognition to address these challenges. Specifically, we first employ two modality-specific (MS) encoders to encode partial visual-tactile data into two incomplete subspaces (i.e., visual subspace and tactile subspace). Then, a modality gap mitigated (MGM) network is adopted to discover modality-invariant high-level label information, which is utilized to generate gap loss and further help updating the MS encoders for relatively consistent visual and tactile subspaces generation. In this way, the huge gap between vision and touch is mitigated, which would further contribute to mine the complementary visual-tactile information. Finally, to achieve data completeness and complementary visual-tactile information exploration simultaneously, a cycle subspace leaning technique is proposed to project the incomplete subspaces into a complete subspace by fully exploiting all the obtainable samples, where complete latent representations with maximum complementary information can be learned. A lot of comparative experiments conducted on three visual-tactile datasets validate the advantage of the proposed PVTF framework, by comparing with state-of-the-art baselines.

HCCG: Efficient high compatibility correspondence grouping for 3D object recognition and 6D pose estimation in cluttered scenes

Recognizing and localizing queried objects in point clouds is a critical technique for robotic manipulation and bin picking tasks. Even though it has been steadily studied, it is still a challenging task for scenes with heavy occlusion and clutter. To copy with this intractable problem, this paper presents a fast and robust 3D object recognition framework, especially an efficient high compatibility correspondence grouping (HCCG) technique achieved by the correspondence ranking, clustering and expanding operations. Utilizing the HCCG technique, we first cluster the compatible correspondences into several high compatibility correspondence groups. Then, for each group, a 6DoF pose hypothesis is generated by using a point pair feature constraints (PPFCs)-based outlier removal module and a local reference frame (LRF)-based pose estimation algorithm. Finally, a robust pose verification operator is carried out to reject the false positive pose hypotheses and pick up the correct target object pose. Our approach is not only suitable for the multi-object recognition task, but also inherently yields the capability of detecting multiple instances thanks to the efficient HCCG technique. Extensive experiments on four challenging datasets show that our approach yields efficient and timely solutions and its advantages are further verified by comparing with the latest methods.

A cooperative image object recognition framework and task offloading optimization in edge computing

The development of AIoT (Artificial Intelligence of Things) and communication technologies has recently brought with it many promising applications. Intelligent surveillance is one of the new models of applications that synthesizes both technologies of IoTs and the object recognition method of artificial intelligence. However, this system suffers from the heavy computation load problem, but the processing devices are generally lightweight with only limited memory and computation power. In this paper, we propose a cooperative image object recognition framework to integrate the process of AIoT devices into the edge computing environment to overcome this problem. The proposed scheme offloads some captured surveillance video frames to nearby GPU equipped edge servers and then integrates the returned results to enhance the overall recognition accuracy of the surveillance system. We also formulate the AIoT Task Offloading scheduling to a mathematical programming Problem (named ATOP) and prove it to be an NP-complete problem. A heuristic Particle Swarm Optimization algorithm is also proposed to determine near optimal offloading scheduling for the ATOP such that the overall object recognition precision is maximized. Although the object recognition precision is enhanced, there still remains a gap to the ground truth, and the cumulative gap might cause significant errors as times goes by. In this research, we also develop a mechanism to help the designer to automatically diagnose when the cumulative errors become significant. Simulation results demonstrate that the proposed framework is feasible and the proposed Particle Swarm Optimization algorithm for the ATOP outperforms the greedy approach and the genetic algorithm. The simulation results also show that the abnormal data diagnosis mechanism can figure out when the cumulative errors happen using a real-world human-flow counting data set.

Variable Stiffness Object Recognition with a CNN-Bayes Classifier on a Soft Gripper.

Soft grippers significantly widen the palpation capabilities of robots, ranging from soft to hard materials without the assistance of cameras. From a medical perspective, the detection of size and shape of hard inclusions concealed within soft three-dimensional (3D) objects is meaningful for the early detection of cancer through palpation. This article proposes a framework for variable-stiffness object recognition using tactile information collected by force sensitive resistors on a three-finger soft gripper. A 15 × 50 spatiotemporal tactile image is generated for each 3D palpation process and then fed into a convolutional neural network (CNN) for object identification. The training set consists of tactile images generated from different grasping orientations. We developed our own CNN architecture, named SoftTactNet, and compared its performance with several state-of-the-art CNNs on the image dataset produced by our experiments. The results show that our proposed method excels in distinguishing 3D shapes and sizes of objects enclosed by a thick soft foam. The average recognition rate is significantly improved using a Naive Bayes classifier, reaching a 97% recognition accuracy. The detection of shapes and sizes of hard objects underneath soft tissues is extremely important for breast and testicular cancer early detection, a field where Soft Robots can shine with inexpensive and ubiquitous devices.

Joint object recognition and pose estimation using multiple-anchor triplet learning of canonical plane

• Our method performs object recognition and pose estimation simultaneously. • Our network is trained with only supervised object class labels. • We propose multiple-anchor triplet learning to avoid the problem of local minima. • Our method is superior to various existing methods. Accurate object recognition and pose estimation models are essential for practical applications of robot arms, such as picking products on a shelf. Training such a model often requires a large-scale dataset with qualified labels for both object classes and pose parameters, and collecting accurate pose labels is particularly costly. A recent paper Ueno et al.(2019)[28] proposed a triplet learning framework for joint object recognition and pose estimation without explicit pose labels by learning a spatial transformer network to estimate the pose difference of an input image from an anchor image depicting the same object in a reference pose . However, our analysis suggests that the pose estimation accuracy is severely degraded for input images with large pose differences. To address this problem, we propose a new learning approach called multiple-anchor triplet learning. The basic idea is to give dense reference poses by preparing multiple anchors so that there is at least one anchor image having a small pose difference to the input image. Our multiple-anchor triplet learning is an extension of the standard single-anchor triplet learning to the multiple-anchor case. Inspired by the idea of multiple instance learning , we introduce a selection layer that automatically chooses the best anchor for each input image and allows the network to be trained end-to-end to minimize triplet-based losses. Experiments with three benchmark datasets in product picking scenarios demonstrate that our method significantly outperforms existing methods in both object recognition and pose estimation accuracy.

DeepRecog: Threefold underwater image deblurring and object recognition framework for AUV vision systems

DeepRecog: Threefold underwater image deblurring and object recognition framework for AUV vision systems

Multi-view dual attention network for 3D object recognition

The existing view-based 3D object classification and recognition methods ignore the inherent hierarchical correlation and distinguishability of views, making it difficult to further improve the classification accuracy. In order to solve this problem, this paper proposes an end-to-end multi-view dual attention network framework for high-precision recognition of 3D objects. On one hand, we obtain three feature layers of query, key, and value through the convolution layer. The spatial attention matrix is generated by the key-value pairs of query and key, and each feature in the value of the original feature space branch is assigned different importance, which clearly captures the prominent detail features in the view, generates the view space shape descriptor, and focuses on the detail part of the view with the feature of category discrimination. On the other hand, a channel attention vector is obtained by compressing the channel information in different views, and the attention weight of each view feature is scaled to find the correlation between the target views and focus on the view with important features in all views. Integrating the two feature descriptors together to generate global shape descriptors of the 3D model, which has a stronger response to the distinguishing features of the object model and can be used for high-precision 3D object recognition. The proposed method achieves an overall accuracy of 96.6% and an average accuracy of 95.5% on the open-source ModelNet40 dataset, compiled by Princeton University when using Resnet50 as the basic CNN model. Compared with the existing deep learning methods, the experimental results demonstrate that the proposed method achieves state-of-the-art performance in the 3D object classification accuracy.

FastGCN + ARSRGemb: a novel framework for object recognition

In recent years research has been producing an important effort to encode the digital image content. Most of the adopted paradigms only focus on local features and lack in information about location and relationships between them. To fill this gap, we propose a framework built on three cornerstones. First, ARSRG (Attributed Relational SIFT (Scale-Invariant Feature Transform) regions graph), for image representation, is adopted. Second, a graph embedding model, with purpose to work in a simplified vector space, is applied. Finally, Fast Graph Convolutional Networks perform classification phase on a graph based dataset representation. The framework is evaluated on state of art object recognition datasets through a wide experimental phase and is compared with well-known competitors.

Novel methods for noisy 3D point cloud based object recognition

3D point cloud based object recognition becomes increasingly important in the last few years, as the widely use of point cloud over the low-cost 3D sensors have developed rapidly. However, the obtained 3D point cloud is inevitably contaminated with noise due to physical and environmental factors, which has a negative impact on recognition task. To address this problem, a complete object recognition framework for 3D noisy point cloud is presented into which a pre-processing step of filtering is integrated for the first time. In the filtering phase, our two proposed approaches, named Guided 3D Point Cloud Filter (G3DF) and Iterative Guidance Normal Filter (IGNF), are taken into account to produce high-quality point cloud model. Then, on the basis of advantages of local-based and global-based descriptors, a new type of feature descriptor, called Local-to-Global Histogram (LGH), is proposed, which contains Local Viewpoint Feature Histogram (LVFH) and Local Ensemble of Shape Function (LESF). Experimental results show that the comprehensive classification performance yielded by using proposed filters and descriptors is competitive compared to other state-of-the-art combinations. In particularly, the composition of G3DF and LVFH is more suited for real-time applications.

Tactile Sensing Using Machine Learning-Driven Electrical Impedance Tomography

Electrical Impedance Tomography (EIT) tactile sensors have limited success in equipping robots with tactile sensing capabilities due to the low spatial resolution of the resulting tactile images and the presence of image artifacts. To address these limitations, we propose a modular framework for invariant recognition of objects, within the context of an EIT artificial skin sensor. Three interconnected problems, i.e., EIT image reconstruction, segmentation and object recognition, are tackled in this work with the aid of machine learning. A novel conductivity surface decomposition approach, based on low order bivariate polynomials and RBF networks is introduced for the efficient solution of the EIT inverse problem. Next, segmentation of the reconstructed images is performed using a convolutional neural network and transfer learning. Finally, a subspace KNN ensemble classifier is trained on the set of object descriptors extracted from the segmented inhomogeneities to classify the objects. The proposed framework provides an accuracy of 97.5% on unseen data.