Microsoft COCO Dataset Research Articles

Novel concept-based image captioning models using LSTM and multi-encoder transformer architecture

Captioning an image involves using a combination of vision and language models to describe the image in an expressive and concise sentence. Successful captioning task requires extracting as much information as possible from the corresponding image. One of these key pieces of information is the topic to which the image belongs. The state-of-the-art methods used topic modeling depending only on caption text in order to extract these topics. The problem with extracting the topics using topic modeling only on caption text is that it lacks the consideration of the image’s semantic information. Instead, concept modeling extracts the concepts directly from the images in addition to considering the corresponding caption text. Concept modeling can be used in image captioning to extremely capture the image contexts and benefit from it to produce more accurate descriptions. In this paper, novel image captioning models are proposed by utilizing the concept modeling technique. The first concept-based model is proposed by utilizing LSTM as a decoder while the second model is proposed in association with new multi-encoder transformer architecture. Standard metrics have been used to evaluate the proposed models using Microsoft COCO and Flickr30K datasets. The proposed models outperformed the related work methods with reduced computational complexity.

Сегментация перекрывающихся изображений деревьев на цифровых снимках лесных массивов

The use of decision support systems based on computer vision and artificial intelligence significantly improves the working conditions for the operators of technological machines in the timber sector, whose work implies high intensity and psycho-emotional overload. By means of computer vision and artificial intelligence the operator can quickly and easily obtain the data on the state of the cutting area and adopt the optimal solution for holding the working operation. This facilitates his work and reduces the time spent searching and analyzing the data on the cutting area. Meanwhile, one of the key elements of such a system is a subsystem for automatic segmentation of objects in the photograph. We have explored the possibility of segmenting overlapping objects in the photographs of forest areas using a convolutional neural network based on the Mask R-CNN architecture. Unlike in most works on similar topics, the objects of this study are color photographs taken by an RGB camera rather than a lidar. This creates the prospect for reducing the cost of hardware and software systems used to support decision-making by the operators of logging machines. The images of the stems and crowns of coniferous and deciduous trees overlapping each other are the segmented objects under consideration. Using the GIMP graphic editor, we have manually marked the color photographs depicting a total of 134 trees of 4 different species: spruce, aspen, birch and pine. Utilizing the developed database, we have carried out an experiment to further train the Mask R-CNN convolutional neural network for segmentation of overlapping parts of the trees in the digital photographs of forest areas. The neural network has been pre-trained using the Microsoft COCO dataset containing more than 200,000 images of 80 different classes of objects such as people, cars, animals and various items. While training the neural network, the images supplied to its input were subjected to a series of various linear and nonlinear geometric transformations, which made it possible to increase the volume of training data by 11 times. As a result, the accuracy of segmentation of the images of the stems and crowns of coniferous and deciduous trees overlapping each other has reached 79 %, which allows the use of neural networks based on a similar architecture in decision support systems for logging machine operators.

Adequate alignment and interaction for cross-modal retrieval

Cross-modal retrieval has attracted widespread attention in many cross-media similarity search applications, especially image-text retrieval in the fields of computer vision and natural language processing. Recently, visual and semantic embedding (VSE) learning has shown promising improvements on image-text retrieval tasks. Most existing VSE models employ two unrelated encoders to extract features, then use complex methods to contextualize and aggregate those features into holistic embeddings. Despite recent advances, existing approaches still suffer from two limitations: 1) without considering intermediate interaction and adequate alignment between different modalities, these models cannot guarantee the discriminative ability of representations; 2) existing feature aggregators are susceptible to certain noisy regions, which may lead to unreasonable pooling coefficients and affect the quality of the final aggregated features. To address these challenges, we propose a novel cross-modal retrieval model containing a well-designed alignment module and a novel multimodal fusion encoder, which aims to learn adequate alignment and interaction on aggregated features for effectively bridging the modality gap. Experiments on Microsoft COCO and Flickr30k datasets demonstrates the superiority of our model over the state-of-the-art methods.

A new YOLO-based method for real-time crowd detection from video and performance analysis of YOLO models.

As seen in the COVID-19 pandemic, one of the most important measures is physical distance in viruses transmitted from person to person. According to the World Health Organization (WHO), it is mandatory to have a limited number of people in indoor spaces. Depending on the size of the indoors, the number of persons that can fit in that area varies. Then, the size of the indoor area should be measured and the maximum number of people should be calculated accordingly. Computers can be used to ensure the correct application of the capacity rule in indoors monitored by cameras. In this study, a method is proposed to measure the size of a prespecified region in the video and count the people there in real time. According to this method: (1) predetermining the borders of a region on the video, (2) identification and counting of people in this specified region, (3) it is aimed to estimate the size of the specified area and to find the maximum number of people it can take. For this purpose, the You Only Look Once (YOLO) object detection model was used. In addition, Microsoft COCO dataset pre-trained weights were used to identify and label persons. YOLO models were tested separately in the proposed method and their performances were analyzed. Mean average precision (mAP), frame per second (fps), and accuracy rate metrics were found for the detection of persons in the specified region. While the YOLO v3 model achieved the highest value in accuracy rate and mAP (both 0.50 and 0.75) metrics, the YOLO v5s model achieved the highest fps rate among non-Tiny models.

Semantic Representations With Attention Networks for Boosting Image Captioning

Image captioning has shown encouraging outcomes with Transformer-based architectures that typically use attention-based methods to establish semantic associations between objects in an image for caption prediction. Nevertheless, when appearance features of objects in an image display low interdependence, attention-based methods have difficulty in capturing the semantic association between them. To tackle this problem, additional knowledge beyond the task-specific dataset is often required to create captions that are more precise and meaningful. In this article, a semantic attention network is proposed to incorporate general-purpose knowledge into a transformer attention block model. This design combines visual and semantic properties of internal image knowledge in one place for fusion, serving as a reference point to aid in the learning of alignments between vision and language and to improve visual attention and semantic association. The proposed framework is validated on the Microsoft COCO dataset, and experimental results demonstrate competitive performance against the current state of the art.

TECMH: Transformer-Based Cross-Modal Hashing For Fine-Grained Image-Text Retrieval

In recent years, cross-modal hash retrieval has become a popular research field because of its advantages of high efficiency and low storage. Cross-modal retrieval technology can be applied to search engines, cross-modal medical processing, etc. The existing main method is to use a multi-label matching paradigm to finish the retrieval tasks. However, such methods do not use fine-grained information in the multi-modal data, which may lead to sub-optimal results. To avoid cross-modal matching turning into label matching, this paper proposes an end-to-end fine-grained cross-modal hash retrieval method, which can focus more on the fine-grained semantic information of multi-modal data. First, the method refines the image features and no longer uses multiple labels to represent text features but uses BERT for processing. Second, this method uses the inference capabilities of the transformer encoder to generate global fine-grained features. Finally, in order to better judge the effect of the fine-grained model, this paper uses the datasets in the image text matching field instead of the traditional label-matching datasets. This article experiment on Microsoft COCO (MS-COCO) and Flickr30K datasets and compare it with the previous classical methods. The experimental results show that this method can obtain more advanced results in the cross-modal hash retrieval field.

Radio-Assisted Human Detection

In this paper, we propose a radio-assisted human detection framework by incorporating radio information into the state-of-the-art detection methods, including anchor-based one-stage detectors and two-stage detectors. We extract the radio localization and identifier information from the radio signals to assist the human detection, due to which the problem of false positives and false negatives can be greatly alleviated. For both detectors, we use the confidence score revision based on the radio localization to improve the detection performance. For two-stage detection methods, we propose to utilize the region proposals generated from radio localization rather than relying on region proposal network (RPN). Moreover, with the radio identifier information, a non-max suppression method with the radio localization constraint has also been proposed to further suppress the false detections and reduce miss detections. Experiments on the simulative Microsoft COCO dataset and Caltech pedestrian datasets show that the mean average precision (mAP) and the miss rate of the state-of-the-art detection methods can be improved with the aid of radio information. Finally, we conduct experiments in real-world scenarios to demonstrate the feasibility of our proposed method in practice.

A non-definitive auto-transfer mechanism for arbitrary style transfers

In recent years, image style transfers have increasingly become a hot research topic in the field of computer vision. The existing CNN-based style transfer methods have been the basis of a substantial amount of research in the content–style fusion area. Artificially controlled style fusions are applied by deterministic computations. Since these methods limit the automatic learning ability of the model, their style transfer effect is unstable. To solve this problem, we propose a non-definitive style auto-transfer module. This module is based on an attention submodule that guides the model in regard to the channels and the space for content–style fusions. Instead of artificially defining the content–style fusion methods, it lets the model learn on its own. We also propose a feature shuffle operation, which reduces the influence of the style image on the content of the result. In addition, to better preserve the high-level and low-level information of the image, our loss function adopts a multi-scale content–style loss and an edge detection loss. All our experiments are conducted on the WikiArt and Microsoft COCO datasets. The experimental results show that our method can achieve more stable and better visual effects than the existing methods.

Structured Semantic Transfer for Multi-Label Recognition with Partial Labels

Multi-label image recognition is a fundamental yet practical task because real-world images inherently possess multiple semantic labels. However, it is difficult to collect large-scale multi-label annotations due to the complexity of both the input images and output label spaces. To reduce the annotation cost, we propose a structured semantic transfer (SST) framework that enables training multi-label recognition models with partial labels, i.e., merely some labels are known while other labels are missing (also called unknown labels) per image. The framework consists of two complementary transfer modules that explore within-image and cross-image semantic correlations to transfer knowledge of known labels to generate pseudo labels for unknown labels. Specifically, an intra-image semantic transfer module learns image-specific label co-occurrence matrix and maps the known labels to complement unknown labels based on this matrix. Meanwhile, a cross-image transfer module learns category-specific feature similarities and helps complement unknown labels with high similarities. Finally, both known and generated labels are used to train the multi-label recognition models. Extensive experiments on the Microsoft COCO, Visual Genome and Pascal VOC datasets show that the proposed SST framework obtains superior performance over current state-of-the-art algorithms. Codes are available at https://github.com/HCPLab-SYSU/HCP-MLR-PL.

Multi-label Image Classification Model via Label Correlation Matrix

Multi-label classification is one of the most essential tasks of computer vision. In the multi-label image classification model, using the correlation between labels is a powerful method for improving the classification effectiveness of the model. However, common methods ignore the interrelationship between the label pairs. On the other hand, introducing a spatial attention mechanism into the model could also improve the classification effectiveness of the model. However, most methods that use the attention mechanism module do not use the correlation information between labels. To solve these issues, we propose a novel multi-label image classification model using the label correlation in the paper. Our model generates label word vectors based on the BERT model that can describe the potential relationship between labels. And then we combine these vectors with static statistics information on labels to construct a new label correlation matrix. Moreover, we introduce label semantic information into the spatial attention mechanism. With the semantic information, the generated spatial attention map could focus on the image feature regions with label correlation, and complete the accurate classification of the model. On the Microsoft COCO data set, this model achieves the best score of 84.3% on mAP, which shows the effectiveness of our model.

Application of YOLO (You Only Look Once) V.4 with Preprocessing Image and Network Experiment

In computer science, specifically in the field of image processing, many reliable algorithms have been found. Previously it was introduced that the YOLO (You Only Look Once) V.3 algorithm. In this case, the application of the YOLO algorithm that we carried out was applied experimentally by utilizing image preprocessing techniques. In this study, image preprocessing was carried out. The image of the Microsoft COCO dataset that was preprocessed in this study used the method of image dimension reduction and image quality improvement. The Microsoft COCO dataset image dimension reduction method used is the Principal Component Analysis (PCA) method and to improve the image quality of the Microsoft COCO dataset using Gaussian Smoothing. Then after the fine-tuning process, there is an increase in the mAP value by an average of 8.99% so that the five models can have an mAP above 80%. The highest mAP value is owned by the model using the schema after the fine-tuning process. From the results of experiments carried out in this study, obtained detection results that have fairly good accuracy in the dataset results. Irregular transformations of position, dimension, composition and, direction can still be captured as the same feature. YOLO's ability in feature engineering is an acknowledgment that has been successfully proven in this research. Although not all the results of using this algorithm are perfect on all data, the results tend to be good. This is related to the services available in the form of a convolutional layer on YOLO reducing downsample or reducing image dimensions by using anchor boxes, this algorithm can also improve accuracy

Integrating Scene Semantic Knowledge into Image Captioning

Most existing image captioning methods use only the visual information of the image to guide the generation of captions, lack the guidance of effective scene semantic information, and the current visual attention mechanism cannot adjust the focus intensity on the image. In this article, we first propose an improved visual attention model. At each timestep, we calculated the focus intensity coefficient of the attention mechanism through the context information of the model, then automatically adjusted the focus intensity of the attention mechanism through the coefficient to extract more accurate visual information. In addition, we represented the scene semantic knowledge of the image through topic words related to the image scene, then added them to the language model. We used the attention mechanism to determine the visual information and scene semantic information that the model pays attention to at each timestep and combined them to enable the model to generate more accurate and scene-specific captions. Finally, we evaluated our model on Microsoft COCO (MSCOCO) and Flickr30k standard datasets. The experimental results show that our approach generates more accurate captions and outperforms many recent advanced models in various evaluation metrics.

A Deep Learning and Transfer Learning Approach for Vehicle Damage Detection

According to the U.S. Department of Transportation, there is an average of six million motor vehicle crashes every year in the United States. For insurance companies, it is very time-consuming and expensive to process claims for detecting and classifying vehicle damages; thus, deep learning techniques have been used to automate this process to reduce the time and the cost. In this paper, Mask R-CNN is used for image segmentation to identify and crop vehicles from images. Then a convolutional neural network (CNN) model is built to classify whether or not the vehicles have damages. In addition, transfer learning is utilized in both image segmentation and classification phases to help build the models for vehicle detection and damage detection, using the pre-trained weights from Microsoft COCO dataset and ImageNet, respectively. 864 images of damaged vehicles collected from public websites, such as Google Images, are used in this research. The experiment on the detection of bumper damages has achieved 87.5% accuracy.

Coconut trees detection and segmentation in aerial imagery using mask region‐based convolution neural network

Food resources face severe damages under extraordinary situations of catastrophes such as earthquakes, cyclones, and tsunamis. Under such scenarios, speedy assessment of food resources from agricultural land is critical as it supports aid activity in the disaster hit areas. In this article, a deep learning approach is presented for the detection and segmentation of coconut tress in aerial imagery provided through the AI competition organized by the World Bank in collaboration with OpenAerialMap and WeRobotics. Maked Region-based Convolutional Neural Network approach was used identification and segmentation of coconut trees. For the segmentation task, Mask R-CNN model with ResNet50 and ResNet1010 based architectures was used. Several experiments with different configuration parameters were performed and the best configuration for the detection of coconut trees with more than 90% confidence factor was reported. For the purpose of evaluation, Microsoft COCO dataset evaluation metric namely mean average precision (mAP) was used. An overall 91% mean average precision for coconut trees detection was achieved.

Mixed-Clipping Quantization for Convolutional Neural Networks

<p indent=0mm>Quantization is the main method to compress convolutional neural networks and accelerate convolutional neural network inference. Most existing quantization methods quantize all layers to the same bit width. Mixed-precision quantization can obtain higher precision under the same compression ratio, but it is difficult to find a mixed-precision quantization strategy. To solve this problem, a mixed-clipping quantization method based on reinforcement learning is proposed. It uses reinforcement learning to search for a mixed-precision quantization strategy, and uses a mixed-clipping method to clip weight data according to the searched quantization strategy before quantization. This method further improves the accuracy of the quantized network. We extensively test this method on a diverse set of models, including ResNet18/50, MobileNet-V2 on ImageNet, as well as YOLOV3 on the Microsoft COCO dataset. The experimental results show that our method can achieve 2.7% and 0.3% higher Top-1 accuracy on MobileNet-V2 (4 bit), as compared to the HAQ and ZeroQ method. And our method can achieve 2.6% higher mAP on YOLOV3 (6 bit), as compared to per-layer quantization method.

ABOships—An Inshore and Offshore Maritime Vessel Detection Dataset with Precise Annotations

Availability of domain-specific datasets is an essential problem in object detection. Datasets of inshore and offshore maritime vessels are no exception, with a limited number of studies addressing maritime vessel detection on such datasets. For that reason, we collected a dataset consisting of images of maritime vessels taking into account different factors: background variation, atmospheric conditions, illumination, visible proportion, occlusion and scale variation. Vessel instances (including nine types of vessels), seamarks and miscellaneous floaters were precisely annotated: we employed a first round of labelling and we subsequently used the CSRT tracker to trace inconsistencies and relabel inadequate label instances. Moreover, we evaluated the out-of-the-box performance of four prevalent object detection algorithms (Faster R-CNN, R-FCN, SSD and EfficientDet). The algorithms were previously trained on the Microsoft COCO dataset. We compared their accuracy based on feature extractor and object size. Our experiments showed that Faster R-CNN with Inception-Resnet v2 outperforms the other algorithms, except in the large object category where EfficientDet surpasses the latter.

Image Description Generation based on Deep Learning

The paper firstly explains the importance and research significance of image description generation and makes a brief research summary on the generation of various algorithms for image description; it further introduces the principle and the algorithm process of the LSTM deep learning algorithm in details, specifically describes establishing steps and the model parameter selection process of the LSTM deep learning algorithm of the Microsoft COCO dataset, implements the deep learning models by employing Python, and finally applies the trained LSTM deep learning model to image description generation by using the Microsoft COCO dataset as an instance.

Cascade Object Detection and Remote Sensing Object Detection Method Based on Trainable Activation Function

Object detection is an important process in surveillance system to locate objects and it is considered as major application in computer vision. The Convolution Neural Network (CNN) based models have been developed by many researchers for object detection to achieve higher performance. However, existing models have some limitations such as overfitting problem and lower efficiency in small object detection. Object detection in remote sensing hasthe limitations of low efficiency in detecting small object and the existing methods have poor localization. Cascade Object Detection methods have been applied to increase the learning process of the detection model. In this research, the Additive Activation Function (AAF) is applied in a Faster Region based CNN (RCNN) for object detection. The proposed AAF-Faster RCNN method has the advantage of better convergence and clear bounding variance. The Fourier Series and Linear Combination of activation function are used to update the loss function. The Microsoft (MS) COCO datasets and Pascal VOC 2007/2012 are used to evaluate the performance of the AAF-Faster RCNN model. The proposed AAF-Faster RCNN is also analyzed for small object detection in the benchmark dataset. The analysis shows that the proposed AAF-Faster RCNN model has higher efficiency than state-of-art Pay Attention to Them (PAT) model in object detection. To evaluate the performance of AAF-Faster RCNN method of object detection in remote sensing, the NWPU VHR-10 remote sensing data set is used to test the proposed method. The AAF-Faster RCNN model has mean Average Precision (mAP) of 83.1% and existing PAT-SSD512 method has the 81.7%mAP in Pascal VOC 2007 dataset.

Extracting Effective Image Attributes with Refined Universal Detection.

Recently, image attributes containing high-level semantic information have been widely used in computer vision tasks, including visual recognition and image captioning. Existing attribute extraction methods map visual concepts to the probabilities of frequently-used words by directly using Convolutional Neural Networks (CNNs). Typically, two main problems exist in those methods. First, words of different parts of speech (POSs) are handled in the same way, but non-nominal words can hardly be mapped to visual regions through CNNs only. Second, synonymous nominal words are treated as independent and different words, in which similarities are ignored. In this paper, a novel Refined Universal Detection (RUDet) method is proposed to solve these two problems. Specifically, a Refinement (RF) module is designed to extract refined attributes of non-nominal words based on the attributes of nominal words and visual features. In addition, a Word Tree (WT) module is constructed to integrate synonymous nouns, which ensures that similar words hold similar and more accurate probabilities. Moreover, a Feature Enhancement (FE) module is adopted to enhance the ability to mine different visual concepts in different scales. Experiments conducted on the large-scale Microsoft (MS) COCO dataset illustrate the effectiveness of our proposed method.

Cross-Modal Attention With Semantic Consistence for Image-Text Matching.

The task of image-text matching refers to measuring the visual-semantic similarity between an image and a sentence. Recently, the fine-grained matching methods that explore the local alignment between the image regions and the sentence words have shown advance in inferring the image-text correspondence by aggregating pairwise region-word similarity. However, the local alignment is hard to achieve as some important image regions may be inaccurately detected or even missing. Meanwhile, some words with high-level semantics cannot be strictly corresponding to a single-image region. To tackle these problems, we address the importance of exploiting the global semantic consistence between image regions and sentence words as complementary for the local alignment. In this article, we propose a novel hybrid matching approach named Cross-modal Attention with Semantic Consistency (CASC) for image-text matching. The proposed CASC is a joint framework that performs cross-modal attention for local alignment and multilabel prediction for global semantic consistence. It directly extracts semantic labels from available sentence corpus without additional labor cost, which further provides a global similarity constraint for the aggregated region-word similarity obtained by the local alignment. Extensive experiments on Flickr30k and Microsoft COCO (MSCOCO) data sets demonstrate the effectiveness of the proposed CASC on preserving global semantic consistence along with the local alignment and further show its superior image-text matching performance compared with more than 15 state-of-the-art methods.