MSCOCO Dataset Research Articles

Enhancement-fusion feature pyramid network for object detection

Scale variation is one of the challenges of object detection. Most state-of-the-art object detectors depend on feature pyramid networks (FPN) for multiscale learning to deal with this problem, in which feature fusion is an essential operation. However, feature fusion does not sufficiently address the difficulty of the detection task. This paper presents an enhancement-fusion feature pyramid network (EFPN) to obtain reliable object representations for object detectors. Specifically, it contains a feature enhancement module (FEM) and a bottom-up path module (BPM). The FEM is used to eliminate the negative impact of the uneven distribution of object scales on the model performance. Then, a BPM is proposed to address the fusion inconsistency in the FPN. Additionally, an attention module (Ac) is added to eliminate the information loss in the bottom-up aggregation process. EFPN is evaluated by combining it with state-of-the-art detection methods. Extensive experimental results on two datasets MS-COCO and VOC2007 demonstrate the effectiveness of the proposed method.

Context-aware and co-attention network based image captioning model

ABSTRACT Previous captioning methods only rely on semantic-level information considering the similarity of features between image regions in visual space and ignoring the linguistic context incorporated in the decoder for caption generation. In this paper, a transformer-based co-attention network is proposed which uses linguistic information to capture the pairwise visual relationships among objects and significant visual features. We infer the entity words from the visual content of objects, during the caption generation process. Also, we infer the interactive words by focusing on the relationship between entity words, based on the relational context between words generated in the course of caption decoding. We use linguistic contextual information as a guiding force to discover the relationships between objects efficiently. Further, we capture both intra-modal and inter-modal interactions using the multilevel co-attention network. Our model attains 44.1/33.6 BLEU@4, 30.8/25.1 METEOR, 61.9/55.1 ROUGE, 132.1/69.8 CIDEr, and 24.1/17.8 SPICE scores on MSCOCO and Flickr30k datasets, respectively.

Medical image annotation and classification employing pyramidal feature specific lightweight deep convolution neural network

ABSTRACT Medical image annotation has significant potential to detect multiple tags. To detect specific tags and labels, most of the conventional learning algorithms took difficulty in matching the tag with corresponding region in the medical images. Hence, most of the medical image annotations fail to reproduce discrimination between various classes. In this research work to tackle medical image discrimination, we proposed a lightweight pyramidal feature-specific learning deep learning network. The proposed pyramidal feature-specific Lightweight Convolutional Neural Network (LDCNN) architecture for classifying local visual regions to annotate the classified region. By employing pyramidal learning, the proposed LDCNN align each medical image annotation with its region. The colour conversion makes low computation complexity, since the medical image exhibits degeneration by colour absorption. Hence, the interpretability and classification effectiveness increase. To evaluate the effectiveness and classification accuracy, we compare the proposed LDCNN with AlexNet and EfficientNet on benchmark datasets like MS-COCO, LC25000 and multiclass Kather datasets. Empirical experimental performance index obtained by the proposed LDCNN outperforms baseline convolutional neural network architecture. The proposed LDCNN achieves 99.6% accuracy, 98.4% sensitivity, 97.9% specificity and 99.1% F1 score. Hence, there is a steady improvement in medical annotation efforts and classification by our proposed lightweight feature-specific learning network.

ThumbDet: One thumbnail image is enough for object detection

Computer vision fields have witnessed great success thanks to deep convolutional neural networks (CNNs). However, state-of-the-art methods often benefit from large models and datasets, which introduce heavy parameters and computational requirements. Deploying such large models in real-world applications is very difficult because of the limited computing resources. Although many researchers focus on designing efficient block structures to compress model parameters, they ignore that the role of large-scale input images is also an important factor for algorithm efficiency. Reducing input resolution is a useful method to boost runtime efficiency, however, traditional interpolation methods assume a fixed degradation criterion that greatly hurts performance. To solve the above problems, in this paper, we propose a novel framework named ThumbDet for reducing model computation while maintaining detection accuracy. In our framework, we first design an image down-sampling module to learn a small-scale image that looks realistic and contains discriminative properties. Furthermore, we propose a distillation-boost supervision strategy to maintain the detection performance of small-scaled images as the original-size inputs. Extensive experiments conducted on a standard object detection dataset MS COCO demonstrate the effectiveness of the proposed method when using very low-resolution images (i.e. 4× down-sampling) as inputs. In particular, ThumbDet achieves satisfactory detection performance (i.e. 32.3% in mAP) while drastically reducing computation and memory requirements (i.e. speed up of 1.26×), outperforming the traditional interpolation methods (e.g. bicubic) by +3.2% absolutely in terms of mAP.

An Improved YOLOv5 Model for Detecting Laser Welding Defects of Lithium Battery Pole

Focus on the requirement for detecting laser welding defects of lithium battery pole, a new model based on the improved YOLOv5 algorithm was proposed in this paper. First, all the 3 × 3 convolutional kernels in the backbone network were replaced by 6 × 6 convolutional kernels to improve the model’s detection capability of a small defect; second, the last layer of the backbone network was replaced by our designed SPPSE module to enhance the detection accuracy of the model; then the improved RepVGG module was introduced in the head network, which can help to improve the inference speed of the model and enhance the feature extraction capability of the network; finally, SIOU was used as the bounding box regression loss function to improve the accuracy and training speed of the model. The experimental results show that our improved YOLOv5 model achieved 97% mAP and 270 fps on our dataset. Compared with conventional methods, ours had the best results. The ablation experiments were conducted on the publicly available datasets PASCAL VOC and MS COCO, and their mAP@0.5 was improved by 2.4% and 3%, respectively. Additionally, our model improved the average detection rate for small targets on the MS COCO dataset by 2.4%, showing that it can effectively detect small target defects.

Image caption generation using Visual Attention Prediction and Contextual Spatial Relation Extraction

Automatic caption generation with attention mechanisms aims at generating more descriptive captions containing coarser to finer semantic contents in the image. In this work, we use an encoder-decoder framework employing Wavelet transform based Convolutional Neural Network (WCNN) with two level discrete wavelet decomposition for extracting the visual feature maps highlighting the spatial, spectral and semantic details from the image. The Visual Attention Prediction Network (VAPN) computes both channel and spatial attention for obtaining visually attentive features. In addition to these, local features are also taken into account by considering the contextual spatial relationship between the different objects. The probability of the appropriate word prediction is achieved by combining the aforementioned architecture with Long Short Term Memory (LSTM) decoder network. Experiments are conducted on three benchmark datasets—Flickr8K, Flickr30K and MSCOCO datasets and the evaluation results prove the improved performance of the proposed model with CIDEr score of 124.2.

Cascaded feature fusion with multi-level self-attention mechanism for object detection

Object detection has been a challenging task due to the complexity and diversity of objects. The emergence of self-attention mechanism provides a new clue for feature fusion in object detection task. Most existing self-attention mechanisms focus on extracting the correlation between global and local information in space or among channels, however it remains problematic issues of how to effectively fuse all those features. To address the above problems, we propose a Pooling and Global feature Fusion Self-attention Mechanism (PGFSM) to capture multi-level correlations among a variety of features, so as to perform cascaded aggregations upon them. PGFSM consists of three parts: Spatial Self-attention Pooling Fusion Module (SSPFM), Channel Self-attention Pooling Fusion Module (CSPFM), and Spatial and Channel Global Self-attention Fusion Module (SCGSFM). SSPFM and CSPFM respectively carried out in space and channel, extract the global maximum pooling and global average pooling self-attention features; SCGSFM extracts the spatial and channel fused characteristic relationship in the global. Finally, the three fused feature relations are added on the original feature to achieve an enhanced trait representation. In test, our PGFSM is embedded into YOLOv4, YOLOv5, and EfficientDet network respectively, and evaluated in PASCAL VOC and MS COCO datasets. The experiment results show that the feature fusion self-attention mechanism improves the performance of object detection compared to each original framework and also the state-of-the-art modules, which proves the effectiveness of our method.

On Distinctive Image Captioning via Comparing and Reweighting.

Recent image captioning models are achieving impressive results based on popular metrics, i.e., BLEU, CIDEr, and SPICE. However, focusing on the most popular metrics that only consider the overlap between the generated captions and human annotation could result in using common words and phrases, which lacks distinctiveness, i.e., many similar images have the same caption. In this paper, we aim to improve the distinctiveness of image captions via comparing and reweighting with a set of similar images. First, we propose a distinctiveness metric-between-set CIDEr (CIDErBtw) to evaluate the distinctiveness of a caption with respect to those of similar images. Our metric reveals that the human annotations of each image in the MSCOCO dataset are not equivalent based on distinctiveness; however, previous works normally treat the human annotations equally during training, which could be a reason for generating less distinctive captions. In contrast, we reweight each ground-truth caption according to its distinctiveness during training. We further integrate a long-tailed weight strategy to highlight the rare words that contain more information, and captions from the similar image set are sampled as negative examples to encourage the generated sentence to be unique. Finally, extensive experiments are conducted, showing that our proposed approach significantly improves both distinctiveness (as measured by CIDErBtw and retrieval metrics) and accuracy (e.g., as measured by CIDEr) for a wide variety of image captioning baselines. These results are further confirmed through a user study.

TFormer: A Transmission-Friendly ViT Model for IoT Devices

Deploying high-performance vision transformer (ViT) models on ubiquitous Internet of Things (IoT) devices to provide high-quality vision services will revolutionize the way we live, work, and interact with the world. Due to the contradiction between the limited resources of IoT devices and resource-intensive ViT models, the use of cloud servers to assist ViT model training has become mainstream. However, due to the larger number of parameters and floating-point operations (FLOPs) of the existing ViT models, the model parameters transmitted by cloud servers are large and difficult to run on resource-constrained IoT devices. To this end, this article proposes a transmission-friendly ViT model, TFormer, for deployment on resource-constrained IoT devices with the assistance of a cloud server. The high performance and small number of model parameters and FLOPs of TFormer are attributed to the proposed hybrid layer and the proposed partially connected feed-forward network (PCS-FFN). The hybrid layer consists of nonlearnable modules and a pointwise convolution, which can obtain multitype and multiscale features with only a few parameters and FLOPs to improve the TFormer performance. The PCS-FFN adopts group convolution to reduce the number of parameters. The key idea of this article is to propose TFormer with few model parameters and FLOPs to facilitate applications running on resource-constrained IoT devices to benefit from the high performance of the ViT models. Experimental results on the ImageNet-1K, MS COCO, and ADE20K datasets for image classification, object detection, and semantic segmentation tasks demonstrate that the proposed model outperforms other state-of-the-art models. Specifically, TFormer-S achieves 5% higher accuracy on ImageNet-1K than ResNet18 with 1.4× fewer parameters and FLOPs.

Deep Supervised Dual Cycle Adversarial Network for Cross-Modal Retrieval

Cross-modal retrieval tasks, which are more natural and challenging than traditional retrieval tasks, have attracted increasing interest from researchers in recent years. Although different modalities with the same semantics have some potential relevance, the feature space heterogeneity still seriously weakens the performance of cross-modal retrieval models. To solve this problem, common space-based methods in which multimodal data is projected into a learned common space for similarity measurement have become the mainstream approach for cross-modal retrieval tasks. However, current methods entangle the modality style and semantic content in the common space and neglect to fully explore the semantic and discriminative representation/reconstruction of the semantic content. This often results in an unsatisfactory retrieval performance. To solve these issues, this paper proposes a new Deep Supervised Dual Cycle Adversarial Network (DSDCAN) model based on common space learning. It is composed of two cross-modal cycle GANs, one for the image and one for the text. The proposed cycle GAN model disentangles the semantic content and modality style features by making the data of one modality well reconstructed from the extracted modal style feature and the content feature of the other modality. Then, a discriminative semantic and label loss is proposed by fully considering the category, sample contrast, and label supervision to enhance the semantic discrimination of the common space representation. Besides this, to make the data distribution between two modalities similar, a second-order similarity is presented as a distance measurement of the cross-modal representation in the common space. Extensive experiments have been conducted on the Wikipedia, Pascal Sentence, NUS-WIDE-10k, PKU XMedia, MSCOCO, NUS-WIDE, Flickr30k and MIRFlickr datasets. The results demonstrate that the proposed method can achieve a higher performance than the state-of-the-art methods.

AFI-GAN: Improving feature interpolation of feature pyramid networks via adversarial training for object detection

Recent convolutional detectors learn strong semantic features by generating and combining multi-scale features via feature interpolation. However, simple interpolation incurs often noisy and blurred features. To resolve this, we propose a novel adversarially-trained interpolator which can substitute for the traditional interpolation effortlessly. In specific, we design AFI-GAN consisting of an AF interpolator and a feature patch discriminator. In addition, we present a progressive adversarial learning and AFI-GAN losses to generate multi-scale features for downstream detection tasks. However, we can also finetune the proposed AFI-GAN with the recent multi-scale detectors without the adversarial learning once a pre-trained AF interpolator is provided. We prove the effectiveness and flexibility of our AF interpolator, and achieve the better box and mask APs by 2.2% and 1.6% on average compared to using other interpolation. Moreover, we achieve an impressive detection score of 57.3% mAP on the MSCOCO dataset. Code is available at https://github.com/inhavl-shlee/AFI-GAN.

Relational-Convergent Transformer for image captioning

Image captioning describes the visual content of a given image by using natural language sentences, and plays a key role in the fusion and utilization of the image features. However, in the existing image captioning models, the decoder sometimes fails to efficiently capture the relationships between image features because of their lack of sequential dependencies. In this paper, we propose a Relational-Convergent Transformer (RCT) network to obtain complex intramodality representations in image captioning. In RCT, a Relational Fusion Module (RFM) is designed for capturing the local and global information of an image by a recursive fusion. Then, a Relational-Convergent Attention (RCA) is proposed, which is composed of a self-attention and a hierarchical fusion module for aggregating global relational information to extract a more comprehensive intramodal contextual representation. To validate the effectiveness of the proposed model, extensive experiments are conducted on the MSCOCO dataset. The experimental results show that the proposed method outperforms some of the state-of-the-art methods.

EYOLOX: An Efficient One-Stage Object Detection Network Based on YOLOX

Object detection has drawn the attention of many researchers due to its wide application in computer vision-related applications. In this paper, a novel model is proposed for object detection. Firstly, a new neck is designed for the proposed detection model, including an efficient SPPNet (Spatial Pyramid Pooling Network), a modified NLNet (Non Local Network) and a lightweight adaptive feature fusion module. Secondly, the detection head with double residual branch structure is presented to reduce the delay of a decoupled head and improve the detection ability. Finally, these improvements are embedded in YOLOX as plug-and-play modules for forming a high-performance detector, EYOLOX (EfficientYOLOX). Extensive experiments demonstrate that the EYOLOX achieves significant improvements, which increases YOLOX-s from 40.5% to 42.2% AP on the MS COCO dataset with a single GPU. Moreover, the performance of the detection of EYOLOX also outperforms YOLOv6 and some SOTA methods with the same number of parameters and GFLOPs. In particular, EYOLOX has only been trained on the COCO-2017 dataset without using any other datasets, and only the pre-training weights of the backbone part are loaded.

Analysis of the mixed teaching of college physical education based on the health big data and blockchain technology.

In the era of health big data, with the continuous development of information technology, students' physical health management also relies more on various information technologies. Blockchain, as an emerging technology in recent years, has the characteristics of high efficiency and intelligence. College physical education is an important part of college students' health big data. Unlike cultural classes, physical education with its rich movements and activities, leaves teachers no time to monitor students' real classroom performance. Therefore, we propose a human pose estimation method based on cross-attention-based Transformer multi-scale representation learning to monitor students' class concentration. Firstly, the feature maps with different resolution are obtained by deep convolutional network and these feature maps are transformed into multi-scale visual markers. Secondly, we propose a cross-attention module with the multi-scales. The module reduces the redundancy of key point markers and the number of cross fusion operations through multiple interactions between feature markers with different resolutions and the strategy of moving key points for key point markers. Finally, the cross-attention fusion module extracts feature information of different scales from feature tags to form key tags. We can confirm the performance of the cross-attention module and the fusion module by the experimental results conducting on MSCOCO datasets, which can effectively promote the Transformer encoder to learn the association relationship between key points. Compared with the completive TokenPose method, our method can reduce the computational cost by 11.8% without reducing the performance.

ViTAEv2: Vision Transformer Advanced by Exploring Inductive Bias for Image Recognition and Beyond

Vision transformers have shown great potential in various computer vision tasks owing to their strong capability to model long-range dependency using the self-attention mechanism. Nevertheless, they treat an image as a 1D sequence of visual tokens, lacking an intrinsic inductive bias (IB) in modeling local visual structures and dealing with scale variance, which is instead learned implicitly from large-scale training data with longer training schedules. In this paper, we leverage the two IBs and propose the ViTAE transformer, which utilizes a reduction cell for multi-scale feature and a normal cell for locality. The two kinds of cells are stacked in both isotropic and multi-stage manners to formulate two families of ViTAE models, i.e., the vanilla ViTAE and ViTAEv2. Experiments on the ImageNet dataset as well as downstream tasks on the MS COCO, ADE20K, and AP10K datasets validate the superiority of our models over the baseline and representative models. Besides, we scale up our ViTAE model to 644 M parameters and obtain the state-of-the-art classification performance, i.e., 88.5% Top-1 classification accuracy on ImageNet validation set and the best 91.2% Top-1 classification accuracy on ImageNet Real validation set, without using extra private data. It demonstrates that the introduced inductive bias still helps when the model size becomes large. The source code and pretrained models are publicly available atcode.

Multi-scale image–text matching network for scene and spatio-temporal images

In recent years, with the development of deep learning technology, computer vision and natural language processing have made significant progress, and establishing the relationship between computer vision and natural language processing has attracted more and more attention. The spatio-temporal images taken by satellites or aircrafts and scene images with people and other things are the main focus area. Existing methods have yielded excellent results in image–text matching, but there is still room for improvement in effectively using coarse and fine-grained information. We propose a method to solve this problem using multi-scale graph convolutional neural networks. We extracted the multi-scale features of images and texts for matching separately. Global and local matching are used to calculate the overall image sentence and local image–word similarity. Local matching is divided into two stages, first, the node level matches the correspondence between the learning region and the word. Next, the structure level matches the correspondence between the learning region and the phrase to make the matching more comprehensive. Finally, we verified our model on Flickr30k, MSCOCO and RSICD datasets.

Full-Memory Transformer for Image Captioning

The Transformer-based approach represents the state-of-the-art in image captioning. However, existing studies have shown Transformer has a problem that irrelevant tokens with overlapping neighbors incorrectly attend to each other with relatively large attention scores. We believe that this limitation is due to the incompleteness of the Self-Attention Network (SAN) and Feed-Forward Network (FFN). To solve this problem, we present the Full-Memory Transformer method for image captioning. The method improves the performance of both image encoding and language decoding. In the image encoding step, we propose the Full-LN symmetric structure, which enables stable training and better model generalization performance by symmetrically embedding Layer Normalization on both sides of the SAN and FFN. In the language decoding step, we propose the Memory Attention Network (MAN), which extends the traditional attention mechanism to determine the correlation between attention results and input sequences, guiding the model to focus on the words that need to be attended to. Our method is evaluated on the MS COCO dataset and achieves good performance, improving the result in terms of BLEU-4 from 38.4 to 39.3.

Decoupled Metric Network for Single-Stage Few-Shot Object Detection.

Within the last few years, great efforts have been made to study few-shot learning. Although general object detection is advancing at a rapid pace, few-shot detection remains a very challenging problem. In this work, we propose a novel decoupled metric network (DMNet) for single-stage few-shot object detection. We design a decoupled representation transformation (DRT) and an image-level distance metric learning (IDML) to solve the few-shot detection problem. The DRT can eliminate the adverse effect of handcrafted prior knowledge by predicting objectness and anchor shape. Meanwhile, to alleviate the problem of representation disagreement between classification and location (i.e., translational invariance versus translational variance), the DRT adopts a decoupled manner to generate adaptive representations so that the model is easier to learn from only a few training data. As for a few-shot classification in the detection task, we design an IDML tailored to enhance the generalization ability. This module can perform metric learning for the whole visual feature, so it can be more efficient than traditional DML due to the merit of parallel inference for multiobjects. Based on the DRT and IDML, our DMNet efficiently realizes a novel paradigm for few-shot detection, called single-stage metric detection. Experiments are conducted on the PASCAL VOC dataset and the MS COCO dataset. As a result, our method achieves state-of-the-art performance in few-shot object detection. The codes are available at https://github.com/yrqs/DMNet.

Knowing What it is: Semantic-Enhanced Dual Attention Transformer

Attention has become an indispensable component of the models of various multimedia tasks like Image Captioning (IC) and Visual Question Answering (VQA). However, most existing attention modules are designed for capturing the spatial dependency, and are still insufficient in semantic understanding, e.g.}, the categories of objects and their attributes, which is also critical for image captioning. To compensate for this defect, we propose a novel attention module termed Channel-wise Attention Block (CAB) to model channel-wise dependency for both visual modality and linguistic modality, thereby improving semantic learning and multi-modal reasoning simultaneously. Specifically, CAB has two novel designs to tickle with the high overhead of channel-wise attention, which are the reduction-reconstruction block structure and the gating-based attention prediction. Based on CAB, we further propose a novel Semantic-enhanced Dual Attention Transformer (termed SDATR), which combines the merits of spatial and channel-wise attentions. To validate SDATR, we conduct extensive experiments on the MS COCO dataset and yield new state-of-the-art performance of 134.5 CIDEr score on COCO Karpathy test split and 136.0 CIDEr score on the official online testing server. To examine the generalization of SDATR, we also apply it to the task of visual question answering, where superior performance gains are also witnessed.

Image Caption Generation Using Contextual Information Fusion With Bi-LSTM-s

The image caption generation algorithm necessitates the expression of image content using accurate natural language. Given the existing encoder-decoder algorithm structure, the decoder solely generates words one by one in a front-to-back order and is unable to analyze integral contextual information. This paper employs a Bi-LSTM (Bi-directional Long Short-Term Memory) structure, which not only draws on past information but also captures subsequent information, resulting in the prediction of image content subject to the context clues. The visual information is respectively fed into the F-LSTM decoder (forward LSTM decoder) and B-LSTM decoder (backward LSTM decoder) to extract semantic information, along with complementing semantic output. Specifically, the subsidiary attention mechanism S-Att acts between F-LSTM and B-LSTM, while the semantic information of B-LSTM and F-LSTM is extracted using the attention mechanism. Meanwhile, the semantic interaction is extracted pursuant to the similarity while aligning the hidden states, resulting in the output of the fused semantic information. We adopt a Bi-LSTM-s model capable of extracting contextual information and realizing finer-grained image captioning effectively. In the end, our model improved by 9.7% on the basis of the original LSTM. In addition, our model effectively solves the problem of inconsistent semantic information in the forward and backward direction of the simultaneous order, and gets a score of 37.5 on BLEU-4. The superiority of this approach is experimentally demonstrated on the MSCOCO dataset.