Object Bounding Box Research Articles

Fast and Robust Visual Tracking with Few-Iteration Meta-Learning

Visual object tracking has been a major research topic in the field of computer vision for many years. Object tracking aims to identify and localize objects of interest in subsequent frames, given the bounding box of the first frame. In addition, the object-tracking algorithms are also required to have robustness and real-time performance. These requirements create some unique challenges, which can easily become overfitting if given a very small training dataset of objects during offline training. On the other hand, if there are too many iterations in the model-optimization process during offline training or in the model-update process during online tracking, it will cause the problem of poor real-time performance. We address these problems by introducing a meta-learning method based on fast optimization. Our proposed tracking architecture mainly contains two parts, one is the base learner and the other is the meta learner. The base learner is primarily a target and background classifier, in addition, there is an object bounding box prediction regression network. The primary goal of a meta learner based on the transformer is to learn the representations used by the classifier. The accuracy of our proposed algorithm on OTB2015 and LaSOT is 0.930 and 0.688, respectively. Moreover, it performs well on VOT2018 and GOT-10k datasets. Combined with the comparative experiments on real-time performance, our algorithm is fast and robust.

Pillar-Based Cooperative Perception from Point Clouds for 6G-Enabled Cooperative Autonomous Vehicles

3D object detection is a significant aspect of the perception module in autonomous driving; however, with current technology, data sharing between vehicles and cloud servers for cooperative 3D object detection under the strict latency requirement is limited by the communication bandwidth. The sixth-generation (6G) networks have accelerated the transmission rate of the sensor data significantly with extreme low-latency and high-speed data transmission. However, which sensor data format and when to transmit it are still challenging. To address these issues, this study proposes a cooperative perception framework combined with a pillar-based encoder and Octomap-based compression at edges for connected autonomous vehicles to reduce the amount of missing detection in blind spots and further distances. This approach satisfies the constraints on the accuracy of the task perception and provides drivers or autonomous vehicles with sufficient reaction time by applying fixed encoders to learn a representation of point clouds (LiDAR sensor data). Extensive experiment results show that the proposed approach outperforms the previous cooperative perception schemes running at 30 Hz, and the accuracy of the object bounding box results in further distances (greater than 12 m). Furthermore, this approach achieves a lower total delay for the procession of the fusion data and the transmission of the cooperative perception message. To the best of our knowledge, this study is the first to introduce a pillar-based encoder and Octomap-based compression framework for cooperative perception between vehicles and edges in connected autonomous driving.

Adaptive Fusion CNN Features for RGBT Object Tracking

Thermal sensors play an important role in intelligent transportation system. This paper studies the problem of RGB and thermal (RGBT) tracking in challenging situations by leveraging multimodal data. A RGBT object tracking method is proposed in correlation filter tracking framework based on short term historical information. Given the initial object bounding box, hierarchical convolutional neural network (CNN) is employed to extract features. The target is tracked for RGB and thermal modalities separately. Then the backward tracking is implemented in the two modalities. The difference between each pair is computed, which is an indicator of the tracking quality in each modality. Considering the temporal continuity of sequence frames, we also incorporate the history data into the weights computation to achieve a robust fusion of different source data. Experiments on three RGBT datasets show the proposed method achieves comparable results to state-of-the-art methods.

DEEP LEARNING-BASED TRACKING OF MULTIPLE OBJECTS IN THE CONTEXT OF FARM ANIMAL ETHOLOGY

Abstract. Automatic detection and tracking of individual animals is important to enhance their welfare and to improve our understanding of their behaviour. Due to methodological difficulties, especially in the context of poultry tracking, it is a challenging task to automatically recognise and track individual animals. Those difficulties can be, for example, the similarity of animals of the same species which makes distinguishing between them harder, or sudden changes in their body shape which may happen due to putting on or spreading out the wings in a very short period of time. In this paper, an automatic poultry tracking algorithm is proposed. This algorithm is based on the well-known tracktor approach and tackles multi-object tracking by exploiting the regression head of the Faster R-CNN model to perform temporal realignment of object bounding boxes. Additionally, we use a multi-scale re-identification model to improve the re-association of the detected animals. For evaluating the performance of the proposed method in this study, a novel dataset consisting of seven image sequences that show chicks in an average pen farm in different stages of growth is used.

Multi-expert human action recognition with hierarchical super-class learning

In still image human action recognition, existing studies have mainly leveraged extra bounding box information along with class labels to mitigate the lack of temporal information in still images. However, preparing additional annotations such as human and object bounding boxes is time-consuming and also prone to human errors because these annotations are prepared manually. In this paper, we propose a two-phase multi-expert classification method for human action recognition by means of super-class learning and without any extra information. Specifically, a coarse-grained phase selects the most relevant fine-grained experts. Then, the fine-grained experts encode the intricate details within each super-class so that the inter-class variation increases. In the proposed approach, to choose the best configuration for each super-class and characterize inter-class dependency between different action classes, we propose a novel Graph-Based Class Selection (GCS) algorithm. Moreover, the proposed method copes with long-tailed distribution, which the existing studies have not addressed in action recognition. Extensive experimental evaluations are conducted on various public human action recognition datasets, including Stanford40, Pascal VOC 2012 Action, BU101+, and IHAR datasets. The experimental results demonstrate that the proposed method yields promising improvements. To be more specific, in IHAR, Sanford40, Pascal VOC 2012 Action, and BU101+ benchmarks, the proposed approach outperforms the state-of-the-art studies by 8.92%, 0.41%, 0.66%, and 2.11% with much less computational cost and without any auxiliary annotation information. Besides, it is proven that in addressing action recognition with long-tailed distribution, the proposed method outperforms its counterparts by a significant margin.

2차원 영상에서의 깊이 추정을 위한 객체 인식 제어 방법

In this paper, we propose a method for controlling a manipulator using a single 2D camera and an experimental system. Typically, to control the manipulator, the camera is constantly moved using a motor to navigate or detect the reference object in a fixed frame. The proposed control method does not require conventional high-performance depth cameras and can derive depth values from images acquired using a low-cost 2D camera. A real-time image frame detects objects that are the criteria and configures the learning model and communication node to draw an object-bounding box accordingly. In this work, we convert the change in size of this bounding box into a depth value. When the depth value is thus derived from the 2D image, the target position value of the manipulator can be determined. Finally, the manipulator system is configured with detection control.

Adaptive depth-aware visual relationship detection

Visual relationship detection aims at detecting the interaction between objects from flat image, where visual appearance and spatial relationship between different objects are two key factors for detection. However, most existing methods usually extract 2D information of object from flat images, which lacks depth information compared to actual 3D space. To obtain and utilize depth information for visual relationship detection, we construct Depth VRDs dataset as an extension of the VRD dataset and propose adaptive depth-aware visual relationship detection network(ADVRD). In terms of visual appearance, we propose depth-aware visual fusion module to use additional depth visual information to guide RGB visual information where needs to be strengthened. In terms of spatial relationship, to generate a more accurate depth representation when locating object depth spatial position, we propose adaptive depth spatial location method which uses regional information variance to measure information relevance in each small region in object bounding box. Experiment results show that depth information can significantly improve the performance of our network on visual relationship detection tasks, especially for zero shots.

Autonomous Recognition of Multiple Surgical Instruments Tips Based on Arrow OBB-YOLO Network

In surgery, the robot can automatically adjust the endoscope to ensure the stability of the surgical image and reduce the burden on the doctor. It is becoming a research hotspot of current surgical assistance. The identification and localization of endoscopic instruments tips is the key to achieve efficient “doctor–robot” collaboration. However, traditional methods have the problems of poor real-time performance and insufficient information expression. This article proposes an autonomous recognition of multiple surgical instrument tips based on arrow object bounding box (OBB)-YOLO network prediction, the proposed method not only realizes the recognition and localization of key parts of the instrument, but also uses arrows to enhance the expression of critical information such as the tip angle under the premise of meeting the real-time requirements for medical scenes. First, based on the modified YOLOv3 model, the Arrow OBB-YOLO network used to predict Arrow OBB is built, and the arrows are constrained inside the bounding box (BB) through coordinate transformation. Second, the optimization expression of arrow errors can enhance the prediction effect of the model, and the designed step-by-step training optimization method significantly accelerates the convergence speed of the model. Then, based on the Arrow OBB model, a method for generating the tracking areas is proposed. Finally, the experimental results show that the proposed method can accurately identify and locate the tips of the instrument with 36.5 fps and can improve the recognition effect of the BB when adding new information expression, which lays the foundation for “doctor–robot” collaborative surgery.

A Feature Decomposition-Based Method for Automatic Ship Detection Crossing Different Satellite SAR Images

In the face of Synthetic Aperture Radar (SAR) image object detection with different distributions of training and test data, traditional supervised learning methods cannot achieve good detection performance. Domain adaptation (DA) method has been shown to have the ability to solve this problem, but existing DA object detection algorithms all use adversarial DA theory for the detection task, which is ineffective in solving object regression localization in the detection task. In this article, to better solve the above problem, an automatic SAR image ship detection method based on feature decomposition crossing different satellites is proposed. The feature extraction layer of backbone network is divided into low level and high level, where domain-invariant feature extractors are designed for the local features extracted from the low level and the global features extracted from the high level, respectively. We argue that the local and global features extracted from source domain and target domain contain domain-specific features (DSF) for adversarial DA and domain-invariant features (DIF) that contribute to object regression localization. Then, we decompose the local features and global features into DSF and DIF via vector decomposition method. For DSF counterpart, we introduce adversarial DA attention for feature alignment. DIF from the local features are fused into the backbone network for high-level global feature extraction. Finally, by using region proposal network and adversarial domain classifier, we can get the accurate bounding box and object class of SAR image objects. Extensive experiments prove that the proposed method outperforms state-of-the-art methods in terms of detection performance.

Object-Agnostic Transformers for Video Referring Segmentation.

Video referring segmentation focuses on segmenting out the object in a video based on the corresponding textual description. Previous works have primarily tackled this task by devising two crucial parts, an intra-modal module for context modeling and an inter-modal module for heterogeneous alignment. However, there are two essential drawbacks of this approach: (1) it lacks joint learning of context modeling and heterogeneous alignment, leading to insufficient interactions among input elements; (2) both modules require task-specific expert knowledge to design, which severely limits the flexibility and generality of prior methods. To address these problems, we here propose a novel Object-Agnostic Transformer-based Network, called OATNet, that simultaneously conducts intra-modal and inter-modal learning for video referring segmentation, without the aid of object detection or category-specific pixel labeling. More specifically, we first directly feed the sequence of textual tokens and visual tokens (pixels rather than detected object bounding boxes) into a multi-modal encoder, where context and alignment are simultaneously and effectively explored. We then design a novel cascade segmentation network to decouple our task into coarse-grained segmentation and fine-grained refinement. Moreover, considering the difficulty of samples, a more balanced metric is provided to better diagnose the performance of the proposed method. Extensive experiments on two popular datasets, A2D Sentences and J-HMDB Sentences, demonstrate that our proposed approach noticeably outperforms state-of-the-art methods.

Multivehicle Object Tracking in Satellite Video Enhanced by Slow Features and Motion Features

With the development of video satellites, multimoving object tracking in satellite video is possible and has become a new challenging task. The difficulties are mainly caused by the characteristics of satellite videos: 1) small objects; 2) low contrast between objects and background; and 3) background in a state of continuous motion. These characteristics make it difficult for the advanced multiobject tracking algorithms in the natural video to give full play to their advantages, resulting in vast false alarms, missed objects, ID switches, and low-confidence bounding boxes. To tackle these problems, a novel multimoving object tracking method considering slow features (SFs) and motion features has been proposed in this research, named SF and motion feature-guided multiobject tracking (SFMFMOT), which realizes the continuous tracking of moving vehicles in satellite videos. A nonmaximum suppression (NMS) module guided by bounding box proposals based on SFs is designed to assist the object detection part by utilizing the sensitivity of SF analysis to the changed pixels. While removing a large number of static false alarms and supplementing missed objects, it improves the recall rate by increasing the confidence score of the correctly detected object bounding boxes. In order to improve the tracking performance, a set of optimization strategies based on motion features and time accumulation information are proposed to smooth the trajectory, remove static false alarms, and duplicate bounding boxes. The proposed method is evaluated in three satellite videos and its superiority is demonstrated.

Robotic Objects Detection and Grasping in Clutter Based on Cascaded Deep Convolutional Neural Network

The complex and changeable robotic operating environment will often cause the low success rate or failure of the robot grasping. This article proposes a grasp pose detection method based on the cascaded convolutional neural network, which can be applied to grasp unknown irregular objects under unstructured environment. The grasping feature and the grasp position candidate bounding-boxes of objects are extracted by Mask-RCNN. To guarantee the generalization and improve the detection rate, grasp angle estimation network Y-Net is proposed to accurately obtain the grasp angle. To solve the problem of insufficient accuracy of grasping position, grasping feasibility evaluation network Q-Net is proposed for acquiring the grasp quality distribution. Finally, the optimal grasp posture is obtained for the robotic object grasping task in cluttered scenes. Experiments are validated in Cornell datasets, Jacquard datasets, and real environments, respectively. The experimental results show that the proposed method can quickly calculate the robot posture for irregular objects with random poses and different shapes. Compared to the previous methods, it has considerable improvement in grasp accuracy and speed. The method can be applied to object grasping scenarios in cluttered scenes and has strong stability and robustness.

Object-wise Secure Image Display Method for Screen Capture Protection

In this study, we propose an image copyright protection technology to prevent analog hole attacks using a deep object detection network. Instead of applying a virtual fence to the whole image, only a region of copyright is protected with fences. The region of copyright interest is called the protection zone and is generated by merging bounding boxes found with a YOLO network. The protection zone is adjusted to satisfy the fence generation rules, and then the virtual fence is finally applied. The main goal of this study is to attenuate the image degradation caused by virtual fences while preserving copyright protection capability. Since the virtual fence method uses the afterimage effect of the human visual system, it is necessary to generate images perceived by a human eye at every moment in order to calculate the objective image quality compared to the original unprotected image. Perceived images are generated at every 0.001 seconds using the mathematical model of the afterimage effect. The protection zone is found using object bounding boxes detected using a YOLO deep neural network. We applied the proposed method to four animation images, and the simulation results show that it achieves image quality that is about 3dB better than that of other methods.

AHDet: A dynamic coarse-to-fine gaze strategy for active object detection

In the work setting of deep learning, most of the neural networks employed for visual object detection in recent years are based on bounding box regression. The performance of active detectors through multi-step decision-making is limited by the rough model design. However, from the perspective of cognitive science, the recognition in the human visual system is a decision process from coarse to fine. Based on the theory of “see the forest first, then the trees”, this paper proposes a dynamic coarse-to-fine gaze strategy for active object detection, named AHDet, which takes the key points as the realization carrier of the coarse-to-fine concept. The detection process is divided into two steps, AIM and HIT. In the step of AIM, the positioning and prior bounding boxes for objects are given by detecting the center points, referring to the first glance. In the step of HIT, bounding boxes are dynamically adjusted to obtain compact bounding boxes with the help of the corner points, referring to the careful observation. With the design of the two-step coarse-to-fine gaze process, AHDet outperforms traditional approaches. A series of experiments performed on MS-COCO and PASCAL VOC dataset demonstrate the advantages of AHDet.

Occlusion-robust online multi-object visual tracking using a GM-PHD filter with CNN-based re-identification

We propose a novel online multi-object visual tracker using a Gaussian mixture Probability Hypothesis Density (GM-PHD) filter and deep appearance learning. The GM-PHD filter has a linear complexity with the number of objects and observations while estimating the states and cardinality of time-varying number of objects, however, it is susceptible to miss-detections and does not include the identity of objects. We use visual-spatio-temporal information obtained from object bounding boxes and deeply learned appearance representations to perform estimates-to-tracks data association for target labeling as well as formulate an augmented likelihood and then integrate into the update step of the GM-PHD filter. We also employ additional unassigned tracks prediction after the data association step to overcome the susceptibility of the GM-PHD filter towards miss-detections caused by occlusion. Extensive evaluations on MOT16, MOT17 and HiEve benchmark data sets show that our tracker significantly outperforms several state-of-the-art trackers in terms of tracking accuracy and identification.

АЛГОРИТМ КОМПЛЕКСИРОВАНИЯ НЕСКОЛЬКИХ ИСТОЧНИКОВ ДАННЫХ В ОБЩУЮ КАРТУ ЗАНЯТОСТИ

The paper deals with the problem of constructing a passability model of environment witha large number of dynamic objects based on data from several different sensors. The aim of thework is to improve the algorithm for constructing the occupancy map by adding data from bothexisting algorithms for moving obstacles detection and from millimeter-wave automotive radar.The study solves the problem of combining data on static environment and dynamic objects intoone general passability model for further trajectory planning. The modification of the algorithmpresented in the article is able to combine data from both occupancy maps based on a threedimensional point cloud from any sensor such as lidar or radar, and arrays of bounding boxes ofobjects with known coordinates, sizes, and orientation. Data aggregation occurs at the level ofbuilding occupancy maps and does not impose requirements on the source of information aboutdynamic obstacles. The algorithm is able to refine the data on the position and size of the dynamicobject by speed from the radar, which allows to plan the trajectory taking into account the movementof dynamic objects. The parallel use of the classical approach allows to detect obstacles inthe event of an error in the output of the dynamic obstacle detection algorithm. The developedalgorithm works in real time on the Jetson AGX Xavier module, and is tested in real conditions ona mobile robotic platform in autonomous mode. Promising directions for further research to improvethe presented approach are formulated.

MOHE-NET: MONOCULAR OBJECT HEIGHT ESTIMATION NETWORK USING DEEP LEARNING AND SCENE GEOMETRY

Abstract. Estimating the heights of objects in the field of view has applications in many tasks such as robotics, autonomous platforms and video surveillance. Object height is a concrete and indispensable characteristic people or machine could learn and capture. Many actions such as vehicle avoiding obstacles will be taken based on it. Traditionally, object height can be estimated using laser ranging, radar or stereo camera. Depending on the application, cost of these techniques may inhibit their use, especially in autonomous platforms. Use of available sensors with lower cost would make the adoption of such techniques at higher rates. Our approach to height estimation requires only a single 2D image. To solve this problem we introduce the Monocular Object Height Estimation Network (MOHE-Net) that includes a cascade of two networks. The first network performs the object detection task. This network detects the bounding box of objects of interest. This information is then input to a second network to estimate the object height and is a linear Multi-layer Perceptron (MLP). The linear MLP model models the camera-scene geometry and does not require training or contain activation function as normal MLP did. The developed approach works for static camera set up as well as moving platform. The proposed approach performs state-of-the-art and can be deployed for obstacle avoidance on autonomous platforms. Our code is available at https://github.com/OSUPCVLab/Ford2019/tree/master/Moving%20Object%20Height% 20Estimation%20Network

Visual Relation Detection using Hybrid Analogical Learning

Visual Relation Detection is currently one of the most popular problems for visual understanding. Many deep-learning models are designed for relation detection on images and have achieved impressive results. However, deep-learning models have several serious problems, including poor training-efficiency and lack of understandability. Psychologists have ample evidence that analogy is central in human learning and reasoning, including visual reasoning. This paper introduces a new hybrid system for visual relation detection combining deep-learning models and analogical generalization. Object bounding boxes and masks are detected using deep-learning models and analogical generalization over qualitative representations is used for visual relation detection between object pairs. Experiments on the Visual Relation Detection dataset indicates that our hybrid system gets comparable results on the task and is more training-efficient and explainable than pure deep-learning models.

Co-Training for Deep Object Detection: Comparing Single-Modal and Multi-Modal Approaches.

Top-performing computer vision models are powered by convolutional neural networks (CNNs). Training an accurate CNN highly depends on both the raw sensor data and their associated ground truth (GT). Collecting such GT is usually done through human labeling, which is time-consuming and does not scale as we wish. This data-labeling bottleneck may be intensified due to domain shifts among image sensors, which could force per-sensor data labeling. In this paper, we focus on the use of co-training, a semi-supervised learning (SSL) method, for obtaining self-labeled object bounding boxes (BBs), i.e., the GT to train deep object detectors. In particular, we assess the goodness of multi-modal co-training by relying on two different views of an image, namely, appearance (RGB) and estimated depth (D). Moreover, we compare appearance-based single-modal co-training with multi-modal. Our results suggest that in a standard SSL setting (no domain shift, a few human-labeled data) and under virtual-to-real domain shift (many virtual-world labeled data, no human-labeled data) multi-modal co-training outperforms single-modal. In the latter case, by performing GAN-based domain translation both co-training modalities are on par, at least when using an off-the-shelf depth estimation model not specifically trained on the translated images.

Augmented Reality Appling with Consistency of Behavior using Oriented Bounding Box Algorithm

Augmented reality can be defined as the process of merging digital information in real videos, it plays big role in many application such as education, medicine, and media industries. The merging of virtual moving object into real videos consists of two main stages: the detection of moving objects into real videos and the collision detection between real and virtual objects. In this study, an algorithm for merging virtual moving object into real video has been proposed, it is based on the three frame differencing technique followed by two pre-processing steps to detect real moving object. While the collision detection stage is based on the intersection between moving object path and the bounding box of real object. The experiment results show the good accuracy of the proposed algorithm in extracting moving object and merging virtual object into video frames.