Microsoft COCO Dataset Research Articles

Boost image captioning with knowledge reasoning

Automatically generating a human-like description for a given image is a potential research in artificial intelligence, which has attracted a great of attention recently. Most of the existing attention methods explore the mapping relationships between words in sentence and regions in image, such unpredictable matching manner sometimes causes inharmonious alignments that may reduce the quality of generated captions. In this paper, we make our efforts to reason about more accurate and meaningful captions. We first propose word attention to improve the correctness of visual attention when generating sequential descriptions word-by-word. The special word attention emphasizes on word importance when focusing on different regions of the input image, and makes full use of the internal annotation knowledge to assist the calculation of visual attention. Then, in order to reveal those incomprehensible intentions that cannot be expressed straightforwardly by machines, we introduce a new strategy to inject external knowledge extracted from knowledge graph into the encoder-decoder framework to facilitate meaningful captioning. Finally, we validate our model on two freely available captioning benchmarks: Microsoft COCO dataset and Flickr30k dataset. The results demonstrate that our approach achieves state-of-the-art performance and outperforms many of the existing approaches.

Design of an ergonomic App for entire rapid body assessment based on Mask RCNN

The rapid entire body assessment (REBA) is a rapid and semi-quantitative ergonomic assessment method for those who engaged in manual handling and/or standing work for a long time. In order to provide a self-assessment tool for operators, an App software based on Mask RCNN is proposed in this paper. The software is developed by adopting the architecture of a mobile terminal App combined with an imaging processing server. The main functions of the App are video capturing of work process, the operation of the process of REBA, human-machine interaction, etc., while the server is work for processing the video imaging transmitted from the App, key-points extraction of worker’s body from the images for work posture identification. One thousand working scene photos marked by VIA are used for training and testing based on the Microsoft COCO dataset to obtain a reliable target detection model. Experiment of container handling scene shows that the App evaluation software has achieved higher evaluation efficiency and accuracy. The validation of this method has been proved compared with manual evaluation based on REBA. And other work posture evaluation methods will be developed in the future to form an ergonomic evaluation software system.

Transformer with sparse self‐attention mechanism for image captioning

Recently, transformer has been applied to the image caption model, in which the convolutional neural network and the transformer encoder act as the image encoder of the model, and the transformer decoder acts as the decoder of the model. However, transformer may suffer from the interference of non-critical objects of a scene and meet with difficulty to fully capture image information due to its self-attention mechanism's dense characteristics. In this Letter, in order to address this issue, the authors propose a novel transformer model with decreasing attention gates and attention fusion module. Specifically, they firstly use attention gate to force transformer to overcome the interference of non-critical objects and capture objects information more efficiently via truncating all the attention weights that smaller than gate threshold. Secondly, through inheriting attentional matrix from the previous layer of each network layer, the attention fusion module enables each network layer to consider other objects without losing the most critical ones. Their method is evaluated using the benchmark Microsoft COCO dataset and achieves better performance compared to the state-of-the-art methods.

Object Recognition with Hybrid Deep Learning Methods and Testing on Embedded Systems

Object recognition applications can be made with deep neural networks. However, this process may require intensive processing load. For this purpose, hybrid object recognition algorithms that can be created for the recognition of an object in the image and the comparison of the working time of these algorithms on various embedded systems are emphasized. While Haar Cascade, Local Binary Pattern (LBP) and Histogram Oriented Gradients (HOG) algorithms are used for object detection, Convolutional Neural Network (CNN) and Deep Neural Network (DNN) algorithms are used for classification. As a result, six hybrid structures such as Haar Cascade+CNN, LBP+CNN, HOG+CNN and Haar Cascade+DNN, LBP+DNN, HOG+DNN are developed. In this study, these 6 hybrid algorithms were analyzed in terms of success percentage and time, then compared with each other. Microsoft COCO dataset was used to train and test all these hybrid algorithms. Object recognition success of CNN was 76.33%. Object recognition success of Haar Cascade+CNN, one of the hybrid methods we recommend, with a success rate of 78.6% is higher than CNN and other hybrid methods. LBP+CNN method recognized objects in 0.487 seconds which is faster than any other hybrid methods. In our study, Nvidia Jetson TX2, Asus TinkerBoard, Raspbbery Pi 3 B+ were used as embedded systems. As a result of these tests, Haar Cascade+CNN method on Nvidia Jetson TX2 was detected in 0.1303 seconds, LBP+DNN and Haar Cascade+DNN methods on Asus Tinker Board were detected in 0.2459 seconds, and HOG+DNN method on Raspberry Pi 3 B+ was detected in 0.7153 seconds.

Fast Vehicle and Pedestrian Detection Using Improved Mask R-CNN

This study presents a simple and effective Mask R-CNN algorithm for more rapid detection of vehicles and pedestrians. The method is of practical value for anticollision warning systems in intelligent driving. Deep neural networks with more layers have greater capacity but also have to perform more complicated calculations. To overcome this disadvantage, this study adopts a Resnet-86 network as a backbone that differs from the backbone structure of Resnet-101 in the Mask R-CNN algorithm within practical conditions. The results show that the Resnet-86 network can reduce the operation time and greatly improve accuracy. The detected vehicles and pedestrians are also screened out based on the Microsoft COCO dataset. The new dataset is formed by screening and supplementing COCO dataset, which makes the training of the algorithm more efficient. Perhaps, the most important part of our research is that we propose a new algorithm, Side Fusion FPN. The parameters in the algorithm have not increased, the amount of calculation has increased by less than 0.000001, and the mean average precision (mAP) has increased by 2.00 points. The results show that, compared with the algorithm of Mask R-CNN, our algorithm decreased the weight memory size by 9.43%, improved the training speed by 26.98%, improved the testing speed by 7.94%, decreased the value of loss by 0.26, and increased the value of mAP by 17.53 points.

A Comprehensive Pipeline for Complex Text-to-Image Synthesis

Synthesizing a complex scene image with multiple objects and background according to text description is a challenging problem. It needs to solve several difficult tasks across the fields of natural language processing and computer vision. We model it as a combination of semantic entity recognition, object retrieval and recombination, and objects’ status optimization. To reach a satisfactory result, we propose a comprehensive pipeline to convert the input text to its visual counterpart. The pipeline includes text processing, foreground objects and background scene retrieval, image synthesis using constrained MCMC, and post-processing. Firstly, we roughly divide the objects parsed from the input text into foreground objects and background scenes. Secondly, we retrieve the required foreground objects from the foreground object dataset segmented from Microsoft COCO dataset, and retrieve an appropriate background scene image from the background image dataset extracted from the Internet. Thirdly, in order to ensure the rationality of foreground objects’ positions and sizes in the image synthesis step, we design a cost function and use the Markov Chain Monte Carlo (MCMC) method as the optimizer to solve this constrained layout problem. Finally, to make the image look natural and harmonious, we further use Poisson-based and relighting-based methods to blend foreground objects and background scene image in the post-processing step. The synthesized results and comparison results based on Microsoft COCO dataset prove that our method outperforms some of the state-of-the-art methods based on generative adversarial networks (GANs) in visual quality of generated scene images.

Image-Text Joint Learning for Social Images with Spatial Relation Model

The rapid developments in sensor technology and mobile devices bring a flourish of social images, and large-scale social images have attracted increasing attention to researchers. Existing approaches generally rely on recognizing object instances individually with geo-tags, visual patterns, etc. However, the social image represents a web of interconnected relations; these relations between entities carry semantic meaning and help a viewer differentiate between instances of a substance. This article forms the perspective of the spatial relationship to exploring the joint learning of social images. Precisely, the model consists of three parts: (a) a module for deep semantic understanding of images based on residual network (ResNet); (b) a deep semantic analysis module of text beyond traditional word bag methods; (c) a joint reasoning module from which the text weights obtained using image features on self-attention and a novel tree-based clustering algorithm. The experimental results demonstrate the effectiveness of using Flickr30k and Microsoft COCO datasets. Meanwhile, our method considers spatial relations while matching.

Tell and guess: cooperative learning for natural image caption generation with hierarchical refined attention

Automatically generating a natural language description of an image is one of the most fundamental and challenging problems in Multimedia Intelligence because it translates information between two different modalities, while such translation requires the ability to understand both modalities. The existing image captioning models have already achieved remarkable performance. However, they heavily rely on the Encoder-Decoder framework is a directional translation which is hard to be further improved. In this paper, we designed the “Tell and Guess” Cooperative Learning model with a Hierarchical Refined Attention mechanism (CL-HRA) that bidirectionally improves the performance to generate more informative captions. The Cooperative Learning (CL) method combines an image caption module (ICM) with an image retrieval module (IRM) - the ICM is responsible for the “Tell” function, which generates informative and natural language descriptions for a given image. While the IRM will “Guess” and try to select that image from a lineup of images based on the given description. Such cooperation mutually improves the learning of two modules. On the other hand, the Hierarchical Refined Attention (HRA) learns to selectively attend the high-level attributes and the low-level visual features, then incorporate them into CL to fulfill the objective gaps from image to caption. The HRA can pay different attention at the different semantic levels to refine the visual representation, while the CL with the human-like mindset is more interpretable to generate a more related caption for the corresponding image. The experimental results on Microsoft COCO dataset show the effectiveness of CL-HRA in terms of several popular image caption generation metrics.

Deep Regionlets: Blended Representation and Deep Learning for Generic Object Detection.

In this article, we propose a novel object detection algorithm named "Deep Regionlets" by integrating deep neural networks and a conventional detection schema for accurate generic object detection. Motivated by the effectiveness of regionlets for modeling object deformations and multiple aspect ratios, we incorporate regionlets into an end-to-end trainable deep learning framework. The deep regionlets framework consists of a region selection network and a deep regionlet learning module. Specifically, given a detection bounding box proposal, the region selection network provides guidance on where to select sub-regions from which features can be learned from. An object proposal typically contains three - 16 sub-regions. The regionlet learning module focuses on local feature selection and transformations to alleviate the effects of appearance variations. To this end, we first realize non-rectangular region selection within the detection framework to accommodate variations in object appearance. Moreover, we design a "gating network" within the regionlet leaning module to enable instance dependent soft feature selection and pooling. The Deep Regionlets framework is trained end-to-end without additional efforts. We present ablation studies and extensive experiments on the PASCAL VOC dataset and the Microsoft COCO dataset. The proposed method yields competitive performance over state-of-the-art algorithms, such as RetinaNet and Mask R-CNN, even without additional segmentation labels.

Solder Joint Recognition Using Mask R-CNN Method

This article proposes a novel solder joint recognition method based on the state-of-the-art Mask Region-convolutional neural network (R-CNN) deep learning method. Traditional classification methods, such as neural networks and statistical methods, can only classify defect type, and the template-matching method can only match the position of the object. Based on Mask R-CNN, our proposed approach can classify, position, and segment the solder joint defect at the same time. To train our Mask R-CNN-based detection method, the transfer learning method uses the ResNet-101, which is initialized and trained on the Microsoft COCO data set. Through experimentation, our proposed method obtained better results than the traditional classification method in solder joint recognition, and it can achieve very high classification accuracy with more than 95% mean of average precision (mAP) for segmentation. The proposed method can classify and identify the position and segment of the solder joint defect simultaneously with very high recognition accuracy.

Boosted Transformer for Image Captioning

Image captioning attempts to generate a description given an image, usually taking Convolutional Neural Network as the encoder to extract the visual features and a sequence model, among which the self-attention mechanism has achieved advanced progress recently, as the decoder to generate descriptions. However, this predominant encoder-decoder architecture has some problems to be solved. On the encoder side, without the semantic concepts, the extracted visual features do not make full use of the image information. On the decoder side, the sequence self-attention only relies on word representations, lacking the guidance of visual information and easily influenced by the language prior. In this paper, we propose a novel boosted transformer model with two attention modules for the above-mentioned problems, i.e., “Concept-Guided Attention” (CGA) and “Vision-Guided Attention” (VGA). Our model utilizes CGA in the encoder, to obtain the boosted visual features by integrating the instance-level concepts into the visual features. In the decoder, we stack VGA, which uses the visual information as a bridge to model internal relationships among the sequences and can be an auxiliary module of sequence self-attention. Quantitative and qualitative results on the Microsoft COCO dataset demonstrate the better performance of our model than the state-of-the-art approaches.

Learning Transparent Object Matting

This paper addresses the problem of image matting for transparent objects. Existing approaches often require tedious capturing procedures and long processing time, which limit their practical use. In this paper, we formulate transparent object matting as a refractive flow estimation problem, and propose a deep learning framework, called TOM-Net, for learning the refractive flow. Our framework comprises two parts, namely a multi-scale encoder-decoder network for producing a coarse prediction, and a residual network for refinement. At test time, TOM-Net takes a single image as input, and outputs a matte (consisting of an object mask, an attenuation mask and a refractive flow field) in a fast feed-forward pass. As no off-the-shelf dataset is available for transparent object matting, we create a large-scale synthetic dataset consisting of 178 K images of transparent objects rendered in front of images sampled from the Microsoft COCO dataset. We also capture a real dataset consisting of 876 samples using 14 transparent objects and 60 background images. Besides, we show that our method can be easily extended to handle the cases where a trimap or a background image is available. Promising experimental results have been achieved on both synthetic and real data, which clearly demonstrate the effectiveness of our approach.

Dynamic length colour palettes

Colour palettes are finite sets of representative colours for image colours. Similar to any clustering problem, the palette's optimal number of colours is an ill-posed problem. Existing techniques assume the number of colours is given by the user. In this work, the authors introduce an approach to generate dynamic length colour palettes. The authors' approach is based on two-stage clustering. At the first stage, they use k-means-based clustering in order to reduce the number of data points in the given image. Then, the second stage is applied using a density-based clustering. The qualitative results show the effectiveness of the proposed approach. At the end of this Letter, they suggest the best range of colours per palette for natural images based on a study performed on 118,060 images from Microsoft COCO dataset.

Matching Image and Sentence with Multi-faceted Representations

In this paper, we propose a novel multimodal matching model for the image and sentence based on their multiple representations. Each representation of the image or sentence undergoes an independent neural network, consisting of multiple layers of nonlinear mappings to yield the corresponding embedding. Besides exploiting the image and sentence relationship based on their embeddings, we propose one novel loss to further exploit the relationship within each single modality, namely, image and sentence based on the yielded multiple embeddings, which is used to train the neural networks simultaneously. The experimental results demonstrate that multiple representations can help to capture the image contents and the sentence semantic meaning more precisely, thus making comprehensive exploitations of the complicated image and sentence matching relationship. More concretely, the proposed matching model significantly outperforms the state-of-the-art approaches in bidirectional image-sentence retrieval on the Flickr8K, Flickr30K, and Microsoft COCO datasets.

Evaluation of deep neural networks for traffic sign detection systems

Abstract Traffic sign detection systems constitute a key component in trending real-world applications, such as autonomous driving, and driver safety and assistance. This paper analyses the state-of-the-art of several object-detection systems (Faster R-CNN, R-FCN, SSD, and YOLO V2) combined with various feature extractors (Resnet V1 50, Resnet V1 101, Inception V2, Inception Resnet V2, Mobilenet V1, and Darknet-19) previously developed by their corresponding authors. We aim to explore the properties of these object-detection models which are modified and specifically adapted to the traffic sign detection problem domain by means of transfer learning. In particular, various publicly available object-detection models that were pre-trained on the Microsoft COCO dataset are fine-tuned on the German Traffic Sign Detection Benchmark dataset. The evaluation and comparison of these models include key metrics, such as the mean average precision (mAP), memory allocation, running time, number of floating point operations, number of parameters of the model, and the effect of traffic sign image sizes. Our findings show that Faster R-CNN Inception Resnet V2 obtains the best mAP, while R-FCN Resnet 101 strikes the best trade-off between accuracy and execution time. YOLO V2 and SSD Mobilenet merit a special mention, in that the former achieves competitive accuracy results and is the second fastest detector, while the latter, is the fastest and the lightest model in terms of memory consumption, making it an optimal choice for deployment in mobile and embedded devices.

Social Image Captioning: Exploring Visual Attention and User Attention.

Image captioning with a natural language has been an emerging trend. However, the social image, associated with a set of user-contributed tags, has been rarely investigated for a similar task. The user-contributed tags, which could reflect the user attention, have been neglected in conventional image captioning. Most existing image captioning models cannot be applied directly to social image captioning. In this work, a dual attention model is proposed for social image captioning by combining the visual attention and user attention simultaneously.Visual attention is used to compress a large mount of salient visual information, while user attention is applied to adjust the description of the social images with user-contributed tags. Experiments conducted on the Microsoft (MS) COCO dataset demonstrate the superiority of the proposed method of dual attention.

Deep feature representation based on privileged knowledge transfer

Feature representation has been an elusive concept until neural networks become popular and exhibit the strong learning capability. However, as supervised labels provide a little meaningful information for feature representation, neural networks suffer from inadequate ability to acquire their representing knowledge by extracting patterns from raw data. To address this issue, in this paper, we propose a novel feature learning model by transferring the spatial relation information to the weights of neural networks. More specially, privileged knowledge stemmed from segmented image is fully utilized in distilling stage to perceive notable objects. The segmented image is regarded as a probability distribution of objects, the notable objects are fetched by maximizing lower bound of mutual information, instead of considering each image as an example of one-hot label. Through this way, spatial relation information is contributed to the feature learning besides class labels. This strategy, denoted as PKT-network, is applied to the image network training. Experimental results show that PKT-network performs excellently for multi-objects representation on Pascal VOC 2012 SegmentationClass (20 object classes) dataset and Microsoft COCO (80 object classes) dataset.

The Lovász Hinge: A Novel Convex Surrogate for Submodular Losses.

Learning with non-modular losses is an important problem when sets of predictions are made simultaneously. The main tools for constructing convex surrogate loss functions for set prediction are margin rescaling and slack rescaling. In this work, we show that these strategies lead to tight convex surrogates iff the underlying loss function is increasing in the number of incorrect predictions. However, gradient or cutting-plane computation for these functions is NP-hard for non-supermodular loss functions. We propose instead a novel surrogate loss function for submodular losses, the Lovász hinge, which leads to O(p logp) complexity with O(p) oracle accesses to the loss function to compute a gradient or cutting-plane. We prove that the Lovász hinge is convex and yields an extension. As a result, we have developed the first tractable convex surrogates in the literature for submodular losses. We demonstrate the utility of this novel convex surrogate through several set prediction tasks, including on the PASCAL VOC and Microsoft COCO datasets.