COCO Benchmark Research Articles

An Object Detection Method Combining Multi-Level Feature Fusion and Region Channel Attention

In deep convolutional neural networks, the introduction of attention mechanism can effectively improve the detection accuracy of multi-scale objects, such as SENet and CBAM. But they use the first-order pooling of feature map to compute the channel weights, which leads to the unreasonable weights allocation. Due to the first-order information can not represent the feature distribution, the differences between channels unable to be distinguished in detail. In this work, we propose a novel object detection method combining multi-level feature fusion and region channel attention (ODMC). First, the positive and negative phase information of multi-level features are fused based on CReLU to enhance the semantic of low-level features and location information of high-level ones. Second, as for the channels after feature fusion, the regional information of feature maps is used to optimize the weight allocation, which can help accurately focus on important channels and suppress irrelevant channels. Finally, the objects are classified and located according to the enhanced features. Extensive experiments on PASCAL VOC and MS COCO benchmark show that ODMC achieves significant improvements over the comparable state-of-the-art detection models with high efficiency.

Learning Category- and Instance-Aware Pixel Embedding for Fast Panoptic Segmentation.

Panoptic segmentation (PS) is a complex scene understanding task that requires providing high-quality segmentation for both thing objects and stuff regions. Previous methods handle these two classes with semantic and instance segmentation modules separately, following with heuristic fusion or additional modules to resolve the conflicts between the two outputs. This work simplifies this pipeline of PS by consistently modeling the two classes with a novel PS framework, which extends a detection model with an extra module to predict category- and instance-aware pixel embedding (CIAE). CIAE is a novel pixel-wise embedding feature that encodes both semantic-classification and instance-distinction information. At the inference process, PS results are simply derived by assigning each pixel to a detected instance or a stuff class according to the learned embedding. Our method not only demonstrates fast inference speed but also the first one-stage method to achieve comparable performance to two-stage methods on the challenging COCO benchmark.

Weakly Supervised Object Detection Using Proposal- and Semantic-Level Relationships.

In recent years, weakly supervised object detection has attracted great attention in the computer vision community. Although numerous deep learning-based approaches have been proposed in the past few years, such an ill-posed problem is still challenging and the learning performance is still behind the expectation. In fact, most of the existing approaches only consider the visual appearance of each proposal region but ignore to make use of the helpful context information. To this end, this paper introduces two levels of context into the weakly supervised learning framework. The first one is the proposal-level context, i.e., the relationship of the spatially adjacent proposals. The second one is the semantic-level context, i.e., the relationship of the co-occurring object categories. Therefore, the proposed weakly supervised learning framework contains not only the cognition process on the visual appearance but also the reasoning process on the proposal- and semantic-level relationships, which leads to the novel deep multiple instance reasoning framework. Specifically, built upon a conventional CNN-based network architecture, the proposed framework is equipped with two additional graph convolutional network-based reasoning models to implement object location reasoning and multi-label reasoning within an end-to-end network training procedure. Comprehensive experiments on the widely used PASCAL VOC and MS COCO benchmarks have been implemented, which demonstrate the superior capacity of the proposed approach when compared with other state-of-the-art methods and baseline models.

A Novel Multi-Scale Feature Fusion Method for Region Proposal Network in Fast Object Detection

Neural network models have been widely used in the field of object detecting. The region proposal methods are widely used in the current object detection networks and have achieved well performance. The common region proposal methods hunt the objects by generating thousands of the candidate boxes. Compared to other region proposal methods, the region proposal network (RPN) method improves the accuracy and detection speed with several hundred candidate boxes. However, since the feature maps contains insufficient information, the ability of RPN to detect and locate small-sized objects is poor. A novel multi-scale feature fusion method for region proposal network to solve the above problems is proposed in this article. The proposed method is called multi-scale region proposal network (MS-RPN) which can generate suitable feature maps for the region proposal network. In MS-RPN, the selected feature maps at multiple scales are fine turned respectively and compressed into a uniform space. The generated fusion feature maps are called refined fusion features (RFFs). RFFs incorporate abundant detail information and context information. And RFFs are sent to RPN to generate better region proposals. The proposed approach is evaluated on PASCAL VOC 2007 and MS COCO benchmark tasks. MS-RPN obtains significant improvements over the comparable state-of-the-art detection models.

Cycle Fusion Network for Multi-Person Pose Estimation

Multi-person pose estimation has been largely improved with the development of convolutional neural network, which however remains a challenging task in computer vision. The challenge is particularly pronounced in high flexibility of occlusion, camera angles, imaging-caused body limbs. Existing methods are mostly based on networks with deep architectures, which are capable of inferring the semantic labels with high accuracy while suffering from positioning accuracy degeneration caused by the loss of the spatial information. To address these problems, we present a novel network called cycle fusion network to improve the pose estimation accuracy in multi-person environments. The cycle fusion network has a two-stage structure. In the first stage, a cycle-path feature fusion is used to increase the semantic information for low level layers and the spatial information for high level layers respectively. In the second stage, a cross-scale feature fusion is used for refining the positions of keypoints. Equipped together with RFB module and multi-level supervision, the proposed method has shown the state-of-the-art performance on the large-scale MS COCO benchmark with extensive experiments.

Linear Context Transform Block

Squeeze-and-Excitation (SE) block presents a channel attention mechanism for modeling global context via explicitly capturing dependencies across channels. However, we are still far from understanding how the SE block works. In this work, we first revisit the SE block, and then present a detailed empirical study of the relationship between global context and attention distribution, based on which we propose a simple yet effective module, called Linear Context Transform (LCT) block. We divide all channels into different groups and normalize the globally aggregated context features within each channel group, reducing the disturbance from irrelevant channels. Through linear transform of the normalized context features, we model global context for each channel independently. The LCT block is extremely lightweight and easy to be plugged into different backbone models while with negligible parameters and computational burden increase. Extensive experiments show that the LCT block outperforms the SE block in image classification task on the ImageNet and object detection/segmentation on the COCO dataset with different backbone models. Moreover, LCT yields consistent performance gains over existing state-of-the-art detection architectures, e.g., 1.5∼1.7% APbbox and 1.0%∼1.2% APmask improvements on the COCO benchmark, irrespective of different baseline models of varied capacities. We hope our simple yet effective approach will shed some light on future research of attention-based models.

Robust one-stage object detection with location-aware classifiers

Recent progress on one-stage detectors focuses on improving the quality of bounding boxes, while they pay less attention to the classification head. In this work, we focus on investigating the influence of the classification head. To understand the behavior of the classifier in one-stage detectors, we resort to the methods of the Explainable deep learning area. We visualize its learned representations via activation maps and analyze its robustness to image scene context. Based on the analysis, we observe that the classifier limits the performance of the detector due to its limited receptive field and the lack of object locations. Then, we design a simple but efficient location-aware multi-dilation module (LAMD) to enhance the weak classifier. We conduct extensive experiments on the COCO benchmark to validate the effectiveness of LAMD. The results suggest that our LAMD can achieve consistent improvements and leads to robust detection across various one-stage detectors with different backbones.

Multi-task Generative Adversarial Network for Detecting Small Objects in the Wild

Object detection results have been rapidly improved over a short period of time with the development of deep convolutional neural networks. Although impressive results have been achieved on large/medium sized objects, the performance on small objects is far from satisfactory and one of remaining open challenges is detecting small object in unconstrained conditions (e.g. COCO and WIDER FACE benchmarks). The reason is that small objects usually lack sufficient detailed appearance information, which can distinguish them from the backgrounds or similar objects. To deal with the small object detection problem, in this paper, we propose an end-to-end multi-task generative adversarial network (MTGAN), which is a general framework. In the MTGAN, the generator is a super-resolution network, which can up-sample small blurred images into fine-scale ones and recover detailed information for more accurate detection. The discriminator is a multi-task network, which describes each inputted image patch with a real/fake score, object category scores, and bounding box regression offsets. Furthermore, to make the generator recover more details for easier detection, the classification and regression losses in the discriminator are back-propagated into the generator during training process. Extensive experiments on the challenging COCO and WIDER FACE datasets demonstrate the effectiveness of the proposed method in restoring a clear super-resolved image from a blurred small one, and show that the detection performance, especially for small sized objects, improves over state-of-the-art methods by a large margin.

Multilabel Image Classification via Feature/Label Co-Projection

This article presents a simple and intuitive solution for multilabel image classification, which achieves the competitive performance on the popular COCO and PASCAL VOC benchmarks. The main idea is to capture how humans perform this task: we recognize both labels (i.e., objects and attributes) and the correlation of labels at the same time. Here, label recognition is performed by a standard ConvNet pipeline, whereas label correlation modeling is done by projecting both labels and image features extracted by the ConvNet to a common latent vector space. Specifically, we carefully design the loss function to ensure that: 1) labels and features that co-appear frequently are close to each other in the latent space and 2) conversely, labels/features that do not appear together are far apart. This information is then combined with the original ConvNet outputs to form the final prediction. The whole model is trained end-to-end, with no additional supervised information other than the image-level supervised information. Experiments show that the proposed method consistently outperforms previous approaches on COCO and PASCAL VOC in terms of mAP, macro/micro precision, recall, and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$F$ </tex-math></inline-formula> -measure. Further, our model is highly efficient at test time, with only a small number of additional weights compared to the base model for direct label recognition.

Location-Aware Box Reasoning for Anchor-Based Single-Shot Object Detection

In the majority of object detection frameworks, the confidence of instance classification is used as the quality criterion of predicted bounding boxes, like the confidence-based ranking in non-maximum suppression (NMS). However, the quality of bounding boxes, indicating the spatial relations, is not only correlated with the classification scores. Compared with the region proposal network (RPN) based detectors, single-shot object detectors suffer the box quality as there is a lack of pre-selection of box proposals. In this paper, we aim at single-shot object detectors and propose a location-aware anchor-based reasoning (LAAR) for the bounding boxes. LAAR takes both the location and classification confidences into consideration for the quality evaluation of bounding boxes. We introduce a novel network block to learn the relative location between the anchors and the ground truths, denoted as a localization score, which acts as a location reference during the inference stage. The proposed localization score leads to an independent regression branch and calibrates the bounding box quality by scoring the predicted localization score so that the best-qualified bounding boxes can be picked up in NMS. Experiments on MS COCO and PASCAL VOC benchmarks demonstrate that the proposed location-aware framework enhances the performances of current anchor-based single-shot object detection frameworks and yields consistent and robust detection results.

A Rich Feature Fusion Single-Stage Object Detector

Single-stage object detectors are quick and highly accurate. Based on the way training model is developed, single-stage object detectors either adopt a training model based on a pre-trained backbone network model, or a model trained from the scratch. The pre-trained backbone network model is associated with the propagation sensitivity both in classification and detection. This leads to deviations in learning goals, and results in an architecture which is limited by the classification network, hence not easy to modify. Training from the scratch is not as efficient as using a pre-trained network, mainly due to the limitations of the predefined network system. In this paper, we combine these two approaches to overcome the above-mentioned shortcomings. In our proposed method a top-down concatenated feature pyramid is built upon a basic FSSD network. The experiments in this paper are conducted on MS COCO and PASCAL VOC data sets. Moreover, we apply VGG16 as the backbone network to further indicate the effectiveness of our proposed method which reaches 33.1 AP on MS COCO benchmark.

Adaptive Convolution for Object Detection

It is quite challenging to detect objects, especially, small objects, in complex scenes. To solve this problem, we propose a novel module named as adaptive convolution block (ACB), which adaptively adjusts the parameters of convolutional filters according to the current feature maps, and then, filter these feature maps with the obtained adaptive convolutional filters to generate enhanced features. Due to such adaptive convolution, the enhanced features can pay more attention to the concerned objects, suppress the interference information caused by irrelevant surroundings, and efficiently improve the detection accuracy. The proposed ACB is light weight and fast. By directly embedding the ACB into the single shot detection framework, we construct a novel real-time adaptive convolutional detector (ACD). Experiments on PASCAL VOC and MS COCO benchmarks confirm that our ACD outperforms the existing state-of-the-art single-stage detection models, and achieves a better tradeoff between accuracy and speed.

High-Level Semantic Networks for Multi-Scale Object Detection

To better solve scale variance problem, deep multi-scale methods usually detect objects of different scales by different in-network layers. However, the semantic levels of features from different layers are usually inconsistent. In this paper, we propose a multi-branch and high-level semantic network by gradually splitting a base network into multiple different branches. As a result, the different branches have same depth and the output features of different branches have similarly high-level semantics. Due to the difference of receptive fields, the different branches are suitable to detect objects of different scales. Meanwhile, the multi-branch network does not introduce additional parameters by sharing the convolutional weights of different branches. To further improve detection performance, skip-layer connections are used to add context to the branch of relatively small receptive field, and dilated convolution is incorporated to enlarge the resolutions of output feature maps. When they are embedded into Faster RCNN architecture, the weighted scores of proposal generation network and proposal classification network are further proposed. Experiments on three pedestrian datasets (i.e., the KITTI dataset, the Caltech dataset, and the Citypersons dataset), one face dataset (i.e., the WIDER FACE dataset), and two general object datasets (i.e., the COCO benchmark and the PASCAL VOC dataset) demonstrate the effectiveness and generality of proposed method. On these datasets, our method achieves state-of-the-art performance.

An over-regression suppression method to discriminate occluded objects of same category

Occlusion is a key challenge in object detection. It is hard to discriminate objects accurately when they gather together and occlude each other, especially when they belong to same category which easily leads to the problem that multiple objects are regressed into the same bounding box. To address this problem, an over-regression suppression (ORS) method is proposed to take full advantage of supervised information. Firstly, annotated information is utilized to compute the overlaps between different ground truth boxes. Then, the regression loss function is redesigned by adding a penalty term which is associated with the aforementioned overlaps to prevent Over-regression. Finally, the validity of the algorithm is proved by making some changes in Faster R-CNN, in which a k-means ++ clustering algorithm is used to automatically generate various size anchors by learning the shape regularities of objects from dataset, and the Soft-NMS, a nearly cost-free method, is introduced to replace the traditional NMS. Extensive evaluations on the challenging PASCAL VOC and MS COCO benchmarks demonstrate the superiority of ORS in handling intra-class occlusion. Its performance increases when dataset contains more large objects and hard samples, as demonstrated by the results on the MS COCO dataset.

Dual-Resolution Dual-Path Convolutional Neural Networks for Fast Object Detection.

Downsampling input images is a simple trick to speed up visual object-detection algorithms, especially on robotic vision and applied mobile vision systems. However, this trick comes with a significant decline in accuracy. In this paper, dual-resolution dual-path Convolutional Neural Networks (CNNs), named DualNets, are proposed to bump up the accuracy of those detection applications. In contrast to previous methods that simply downsample the input images, DualNets explicitly take dual inputs in different resolutions and extract complementary visual features from these using dual CNN paths. The two paths in a DualNet are a backbone path and an auxiliary path that accepts larger inputs and then rapidly downsamples them to relatively small feature maps. With the help of the carefully designed auxiliary CNN paths in DualNets, auxiliary features are extracted from the larger input with controllable computation. Auxiliary features are then fused with the backbone features using a proposed progressive residual fusion strategy to enrich feature representation.This architecture, as the feature extractor, is further integrated with the Single Shot Detector (SSD) to accomplish latency-sensitive visual object-detection tasks. We evaluate the resulting detection pipeline on Pascal VOC and MS COCO benchmarks. Results show that the proposed DualNets can raise the accuracy of those CNN detection applications that are sensitive to computation payloads.

Beyond Weakly Supervised: Pseudo Ground Truths Mining for Missing Bounding-Boxes Object Detection

Due to the shortcomings of the weakly supervised and fully supervised object detection (i.e., unsatisfactory performance and expensive annotations, respectively), leveraging partially labeled images in a cost-effective way to train an object detector has attracted much attention. In this paper, we formulate this challenging task as a missing bounding-boxes’ object detection problem. Specifically, we develop a pseudo ground truth mining procedure to automatically find the missing bounding boxes for the unlabeled instances, called pseudo ground truths here, in the training data, and then combine the mined pseudo ground truths and the labeled annotations to train a fully supervised object detector. Furthermore, we propose an incremental learning framework to gradually incorporate the results of the trained fully supervised detector to improve the performance of the missing bounding-boxes’ object detection. More importantly, we find an effective way to label the massive images with limited labors and funds, which is crucial when building a large-scale weakly/webly labeled dataset for object detection. The extensive experiments on the PASCAL VOC and COCO benchmarks demonstrate that our proposed method can narrow the gap between the fully supervised and weakly supervised object detectors, and outperform the previous state-of-the-art weakly supervised detectors by a large margin (more than 3% mAP absolutely) when the missing rate equals 0.9. Moreover, our proposed method with 30% missing bounding-box annotations can achieve comparable performance to some fully supervised detectors.

Pay Attention to Them: Deep Reinforcement Learning-Based Cascade Object Detection.

This paper proposes a novel and effective approach, namely pay attention to them (PAT), to general object detection, which integrates the bottom-up single-shot convolutional neural networks (CNNs) and a top-down operating strategy. PAT starts by routinely applying a CNN regression detector to the entire input image. It then conducts refinement, which locates a sub-region that probably contains relevant objects through an intelligent agent built with an attentional mechanism and zooms it in to launch the detector again. This refining step is repeated in a cascaded way, where all the bounding boxes produced are scaled according to the original resolution and the sub-marginal and overlapping parts are wiped out to generate the final output. Due to such progressive processing, PAT improves the detection accuracy, especially for the objects of small sizes. Extensive experiments are conducted on the Pascal VOC and MS COCO benchmarks, and the results show that PAT is able to improve the representative baseline detectors, i.e., single shot multibox detector, YOLOv2, and Faster regions with CNN features, with remarkable accuracy gains [about 2%-5% mean Average Precision (mAP)], which demonstrates its competency.

Gaussian Process Surrogate Models for the CMA Evolution Strategy

This article deals with Gaussian process surrogate models for the Covariance Matrix Adaptation Evolutionary Strategy (CMA-ES)-several already existing and two by the authors recently proposed models are presented. The work discusses different variants of surrogate model exploitation and focuses on the benefits of employing the Gaussian process uncertainty prediction, especially during the selection of points for the evaluation with a surrogate model. The experimental part of the article thoroughly compares and evaluates the five presented Gaussian process surrogate and six other state-of-the-art optimizers on the COCO benchmarks. The algorithm presented in most detail, DTS-CMA-ES, which combines cheap surrogate-model predictions with the objective function evaluations in every iteration, is shown to approach the function optimum at least comparably fast and often faster than the state-of-the-art black-box optimizers for budgets of roughly 25-100 function evaluations per dimension, in 10- and less-dimensional spaces even for 25-250 evaluations per dimension.

Image Caption Description of Traffic Scene Based on Deep Learning

It is a hard issue to describe the complex traffic scene accurately in computer vision. The traffic scene is changeable, which causes image captioning easily interfered by light changes and object occlusion. To solve this problem, we propose an image caption generation model based on attention mechanism. Combining convolutional neural network (CNN) and recurrent neural network (RNN) to generate an end-to-end description for traffic images. To generate a semantic description with distinct degree of discrimination, the attention mechanism is applied to language model. Using Flickr8K、Flickr30K and MS COCO benchmark datasets to validate the effectiveness of our method. The accuracy is promoted maximally by 8.6%, 12.4%, 19.3% and 21.5% in different evaluation metrics. Experiments show that our algorithm has good robustness in four different complex traffic scenarios, such as light change, abnormal weather environment, road marked target and various kinds of transportation tools.

Fast image captioning using LSTM

Computer vision and natural language processing have been some of the long-standing challenges in artificial intelligence. In this paper, we explore a generative automatic image annotation model, which utilizes recent advances on both fronts. Our approach makes use of a deep-convolutional neural network to detect image regions, which later will be fed to recurrent neural network that is trained to maximize the likely-hood of the target sentence description of the given image. During our experimentation we found that better accuracy and training was achieved when the image representation from our detection model is coupled with the input word embedding, we also found out most of the information from the last layer of detection model vanishes when it is fed as thought vector for our LSTM decoder. This is mainly because the information within the last fully connected layer of the YOLO model represents the class probabilities for the detected objects and their bounding box and this information is not rich enough. We trained our model on coco benchmark for 60 h on 64,000 training and 12,800-validation dataset achieving 23% accuracy. We also realized a significant training speed drop when we changed the number of hidden units in the LSTM layer from 1470 to 4096.