MSCOCO Dataset Research Articles

Transformer-Based Context Condensation for Boosting Feature Pyramids in Object Detection

Current object detectors typically have a feature pyramid (FP) module for multi-level feature fusion (MFF) which aims to mitigate the gap between features from different levels and form a comprehensive object representation to achieve better detection performance. However, they usually require heavy cross-level connections or iterative refinement to obtain better MFF result, making them complicated in structure and inefficient in computation. To address these issues, we propose a novel and efficient context modeling mechanism that can help existing FPs deliver better MFF results while reducing the computational costs effectively. In particular, we introduce a novel insight that comprehensive contexts can be decomposed and condensed into two types of representations for higher efficiency. The two representations include a locally concentrated representation and a globally summarized representation, where the former focuses on extracting context cues from nearby areas while the latter extracts general contextual representations of the whole image scene as global context cues. By collecting the condensed contexts, we employ a Transformer decoder to investigate the relations between them and each local feature from the FP and then refine the MFF results accordingly. As a result, we obtain a simple and light-weight Transformer-based Context Condensation (TCC) module, which can boost various FPs and lower their computational costs simultaneously. Extensive experimental results on the challenging MS COCO dataset show that TCC is compatible to four representative FPs and consistently improves their detection accuracy by up to 7.8% in terms of average precision and reduce their complexities by up to around 20% in terms of GFLOPs, helping them achieve state-of-the-art performance more efficiently. Code will be released at https://github.com/zhechen/TCC.

In-Domain Transfer Learning Strategy for Tumor Detection on Brain MRI.

Transfer learning has gained importance in areas where there is a labeled data shortage. However, it is still controversial as to what extent natural image datasets as pre-training sources contribute scientifically to success in different fields, such as medical imaging. In this study, the effect of transfer learning for medical object detection was quantitatively compared using natural and medical image datasets. Within the scope of this study, transfer learning strategies based on five different weight initialization methods were discussed. A natural image dataset MS COCO and brain tumor dataset BraTS 2020 were used as the transfer learning source, and Gazi Brains 2020 was used for the target. Mask R-CNN was adopted as a deep learning architecture for its capability to effectively handle both object detection and segmentation tasks. The experimental results show that transfer learning from the medical image dataset was found to be 10% more successful and showed 24% better convergence performance than the MS COCO pre-trained model, although it contains fewer data. While the effect of data augmentation on the natural image pre-trained model was 5%, the same domain pre-trained model was measured as 2%. According to the most widely used object detection metric, transfer learning strategies using MS COCO weights and random weights showed the same object detection performance as data augmentation. The performance of the most effective strategies identified in the Mask R-CNN model was also tested with YOLOv8. Results showed that even if the amount of data is less than the natural dataset, in-domain transfer learning is more efficient than cross-domain transfer learning. Moreover, this study demonstrates the first use of the Gazi Brains 2020 dataset, which was generated to address the lack of labeled and qualified brain MRI data in the medical field for in-domain transfer learning. Thus, knowledge transfer was carried out from the deep neural network, which was trained with brain tumor data and tested on a different brain tumor dataset.

Transformer-Based Approach Via Contrastive Learning for Zero-Shot Detection.

Zero-shot detection (ZSD) aims to locate and classify unseen objects in pictures or videos by semantic auxiliary information without additional training examples. Most of the existing ZSD methods are based on two-stage models, which achieve the detection of unseen classes by aligning object region proposals with semantic embeddings. However, these methods have several limitations, including poor region proposals for unseen classes, lack of consideration of semantic representations of unseen classes or their inter-class correlations, and domain bias towards seen classes, which can degrade overall performance. To address these issues, the Trans-ZSD framework is proposed, which is a transformer-based multi-scale contextual detection framework that explicitly exploits inter-class correlations between seen and unseen classes and optimizes feature distribution to learn discriminative features. Trans-ZSD is a single-stage approach that skips proposal generation and performs detection directly, allowing the encoding of long-term dependencies at multiple scales to learn contextual features while requiring fewer inductive biases. Trans-ZSD also introduces a foreground-background separation branch to alleviate the confusion of unseen classes and backgrounds, contrastive learning to learn inter-class uniqueness and reduce misclassification between similar classes, and explicit inter-class commonality learning to facilitate generalization between related classes. Trans-ZSD addresses the domain bias problem in end-to-end generalized zero-shot detection (GZSD) models by using balance loss to maximize response consistency between seen and unseen predictions, ensuring that the model does not bias towards seen classes. The Trans-ZSD framework is evaluated on the PASCAL VOC and MS COCO datasets, demonstrating significant improvements over existing ZSD models.

Improving scene text image captioning using transformer-based multilevel attention

Many existing image captioning methods only focus on image objects and their relationships for generating image captions, ignoring the text present in an image. Scene text (ST) contains crucial information to understand an image and facilitating reasoning. The existing methods fail to establish strong correlations between optical character recognition (OCR) tokens, as they have limited OCR representation power. Further, these methods have not efficiently used the positional information of the text. In this work, we have proposed an ST-based image captioning model (Trans-MAtt) based on a multilevel attention mechanism and relation network. We have used relation networks to enhance the connections between ST tokens. We have employed a multi-level attention method, which comprises of spatial, semantic, and appearance attention modules that precisely define the image. To represent context-enriched ST tokens, we use a combination of appearance, location, FastText, and PHOC features. We predict the ST location in the image, which is further integrated with the generated word embeddings for final caption generation. Experiments on the TextCaps dataset demonstrate the effectiveness of the proposed Trans-MAtt model, where it outperforms the current best model by 3.4% on B-4, 2.9% on METEOR, 3.3% on ROUGE-L, 3.1% on CIDEr-D, and 4.1% on SPICE metric scores. Our experiments on the Flickr30k and MSCOCO datasets demonstrated the superiority of our proposed model over existing methods.

LKC-Net: large kernel convolution object detection network

Deep learning-based object detection methods have achieved great performance improvement. However, since small kernel convolution has been widely used, the semantic feature is difficult to obtain due to the small receptive fields, and the key information cannot be highlighted, resulting in a series of problems such as wrong detection, missing detection, and repeated detection. To overcome these problems, we propose a large kernel convolution object detection network based on feature capture enhancement and vast receptive field attention, called LKC-Net. Firstly, a feature capture enhancement block based on large kernel convolution is proposed to improve the semantic feature capturing ability, and depth convolution is used to reduce the number of parameters. Then, the vast receptive filed attention mechanism is constructed to enhance channel direction information extraction ability, and it is more compatible with the proposed backbone than other existing attention mechanisms. Finally, the loss function is improved by introducing the SIoU, which can overcome the angle mismatch problem between the ground truth and prediction box. Experiments are conducted on Pascal VOC and MS COCO datasets for demonstrating the performance of LKC-Net.

Context-aware transformer for image captioning

Recently, image captioning models have made remarkable progress by introducing transformer architecture, which utilizes self-attention to explore intra- and inter-modal interactions. However, most existing methods only consider region-level characteristic during the attention weight calculation and ignore the image-level information. This seriously hinders the whole model from understanding the scene content. In this paper, we propose a Context-Aware Transformer (CATNet) with two novel designs, namely Context Augmented Attention (CAA) and Dual Way Controller (DWC). Concretely, CAA in encoder enables the extraction of more comprehensive visual representation through modeling the communications between multi-level visual features. DWC in decoder is used to enhance the fusion between visual features and language representation through utilizing complementarity of global context and local regions. Extensive experiments conducted on MSCOCO dataset show that the proposed CATNet has achieved state-of-the-art performance on both Karpathy test set and online test.

An Improved YOLOv5 Underwater Detector Based on an Attention Mechanism and Multi-Branch Reparameterization Module

Underwater target detection is a critical task in various applications, including environmental monitoring, underwater exploration, and marine resource management. As the demand for underwater observation and exploitation continues to grow, there is a greater need for reliable and efficient methods of detecting underwater targets. However, the unique underwater environment often leads to significant degradation of the image quality, which results in reduced detection accuracy. This paper proposes an improved YOLOv5 underwater-target-detection network to enhance accuracy and reduce missed detection. First, we added the global attention mechanism (GAM) to the backbone network, which could retain the channel and spatial information to a greater extent and strengthen cross-dimensional interaction so as to improve the ability of the backbone network to extract features. Then, we introduced the fusion block based on DAMO-YOLO for the neck, which enhanced the system’s ability to extract features at different scales. Finally, we used the SIoU loss to measure the degree of matching between the target box and the regression box, which accelerated the convergence and improved the accuracy. The results obtained from experiments on the URPC2019 dataset revealed that our model achieved an mAP@0.5 score of 80.2%, representing a 1.8% and 2.3% increase in performance compared to YOLOv7 and YOLOv8, respectively, which means our method achieved state-of-the-art (SOTA) performance. Moreover, additional evaluations on the MS COCO dataset indicated that our model’s mAP@0.5:0.95 reached 51.0%, surpassing advanced methods such as ViDT and RF-Next, demonstrating the versatility of our enhanced model architecture.

Optimized real-time parking management framework using deep learning

Today’s modern world has seen massive population growth in the past few decades, causing the automobile industry to expand rapidly. The exponential increase has disrupted the seamless flow of traffic globally and has fashioned significant problems such as traffic congestion and its mismanagement. To tackle these issues, some hardware and data-driven-based solutions have been previously proposed. The hardware-based solutions utilize different types of sensors installed at relevant sites to monitor the status of parking slots which makes them less scalable, complex, costly to install and maintain. In contrast, the data-driven solutions utilize the already available infrastructure of surveillance cameras installed at parking lots, thus overcoming the limitations of sensor-based solutions. However, currently only classification-based approaches are prevalent and adopted to monitor the status of the parking slots which makes the systems less generalized for scalability with slower performance. This paper proposes an intelligent parking management system which employs deep learning to alleviate the limitations in the data driven solutions by leveraging the high performance and fast inference capability of YOLO v5 for vehicles detection instead of parking slot classification. The model was evaluated using the PKLot dataset which is a benchmark for identifying the status of the parking lot with state of the art performance achieved having an accuracy of 99.5%. To augment the performance of the algorithm in real-time, a pretrained model of YOLO v5 on MS COCO dataset was employed to detect and assign vacant parking slots and generate vehicle statistics. The performance of the proposed system was evaluated on our custom dataset with an accuracy of 96.8% achieved and a nearly real-time performance of 45 Fps which makes it more efficient, scalable, and generalized.

Active learning in multi-label image classification with graph convolutional network embedding

Active learning has achieved considerable success in sample selection for deep learning models and has been widely used to address the issue of high-cost sample annotation. However, most of the existing active learning methods focus on single-label image classification and have limited use in multi-label scenarios. To address this issue and take advantage of label associations, we propose an active learning model based on the graph convolutional network (GCN) embedding and loss prediction network. Specifically, we construct a heterogeneous information network (HIN) that uses GCN embeddings to learn multiple label associations, as well as associations between images and labels. We also use a loss prediction network to predict target losses of unlabeled inputs. In addition, we propose a dynamic active coefficient to adjust the proportion of active learning gradually in the training process. Comprehensive multi-label image classification experiments with limited training labels are conducted on the MS-COCO, VOC 2007, and NUS-WIDE datasets. The comparison results demonstrate the superiority of our method compared with conventional methods in terms of both classification accuracy and convergence speed.

Computer vision models application in the current system on object detection tasks

The implementation of object detection algorithms would be helpful to the various fields of the current time. When object detection is applied to the surveillance camera system, it will be more efficient to locate crimes or find lost kids. This paper will investigate the performance of different object detection algorithms in a real-world scenario. With experimentation, CenterNet++ outperforms YOLO and MaskRCNN, two traditional and classic object detection algorithms, on the MS COCO dataset, which concludes that CenterNet++ can ensure both accuracy and speed.

Visual Paraphrase Generation with Key Information Retained

Visual paraphrase generation task aims to rewrite a given image-related original sentence into a new paraphrase, where the paraphrase needs to have the same expressed meaning as the original sentence but have a difference in expression form. Existing studies mainly extract two semantic vectors to represent the entire image and the entire original sentence, respectively, for paraphrase generation. However, these semantic vectors for an image or a sentence may lead to the model failing to focus on some key objects in the original sentence, which may generate semantically inconsistent sentences by changing key object information. In this article, we propose an object-level paraphrase generation model, which generates paraphrases by adjusting the permutation of key objects and modifying their associated descriptions. To adjust the permutation of key objects, an object-sorting module aims to obtain new object sequences based on the key object information and original sentences. Then, a sequence generation module sequentially generates paraphrases based on the permutation of the newly object sequences. Each generation step focuses on different image features associated with different key objects to generate descriptions with differences. Furthermore, we use a semantic discriminator module to promote the generated paraphrase to be semantically close to the original sentence. Specifically, the loss function of the discriminator penalizes the excessive distance between the paraphrase and the original sentence. Extensive experiments on the MS COCO dataset show that the proposed model outperforms the baselines.

Adversarial Attack for Deep Steganography Based on Surrogate Training and Knowledge Diffusion

Deep steganography (DS), using neural networks to hide one image in another, has performed well in terms of invisibility, embedding capacity, etc. Current steganalysis methods for DS can only detect or remove secret images hidden in natural images and cannot analyze or modify secret content. Our technique is the first approach to not only effectively prevent covert communications using DS, but also analyze and modify the content of covert communications. We proposed a novel adversarial attack method for DS considering both white-box and black-box scenarios. For the white-box attack, several novel loss functions were applied to construct a gradient- and optimizer-based adversarial attack that could delete and modify secret images. As a more realistic case, a black-box method was proposed based on surrogate training and a knowledge distillation technique. All methods were tested on the Tiny ImageNet and MS COCO datasets. The experimental results showed that the proposed attack method could completely remove or even modify the secret image in the container image while maintaining the latter’s high quality. More importantly, the proposed adversarial attack method can also be regarded as a new DS approach.

PMST: A parallel and miniature Swin transformer for logo detection

With the popularity of online shopping and the rise of various brands, logo detection is gradually coming to the forefront of researchers' minds. However, accurately detecting multiscale, similar, diverse and shape-shifting logos poses a challenge for this technology. The Swin transformer has created a milestone as a high-performance deep learning method across modalities, domains and tasks in various tasks of computer vision. We improve the Swin transformer for better image feature extraction and enhance the robustness for its application to logo detection. In this paper, we propose the PMST as the backbone of logo detection, and design a bypass-parallelizable shift module and a miniature window tandem shift strategy to further enhance image feature fusion and transfer between windows. The PMST achieves a 79.2 box AP on the LogoData dataset, surpassing most detection models. According to existing known methods, in logo detection, the PMST is the first backbone to put transformer to the detection. The results achieved state-of-the-art on the FlickrLogos-32 and FoodLogoDet-1500 datasets. It also achieves excellent efficiency on the ImageNet-1K, OpenBrands-80 and MS-COCO datasets. The code is available athttps://github.com/blowhen/PMST.

A Small Object Detection Network Based on Multiple Feature Enhancement and Feature Fusion

Due to the small size, high resolution, and complex background, small object detection has become a difficult point in computer vision. Making full use of high-resolution features and reducing information loss in the process of information propagation is of great significance to improve small object detection. In this article, to achieve the above two points, this work proposes a small object detection network based on multiple feature enhancement and feature fusion based on RetinaNet (MFEFNet). First, this work designs a densely connected dilated convolutions to adequately extract high-resolution features from C2. Then, this work utilizes subpixel convolution to avoid the loss of channel information caused by channel dimension reduction in the lateral connection. Finally, this article introduces a bidirectional fusion feature pyramid structure to shorten the propagation path of high-resolution features and reduce the loss of high-resolution features. Experiments show that our proposed MFEFNet achieves stable performance gains in object detection task. Specifically, the improved method improves RetinaNet from 34.4AP to 36.2AP on the challenging MS COCO dataset, and especially achieves excellent results in small object detection with an improvement of 2.9%.

An Improved Feature Pyramid Network and Metric Learning Approach for Rail Surface Defect Detection

When deep learning methods are used to detect rail surface defects, the training accuracy declines due to small defects and an insufficient number of samples. This paper investigates the problem of rail surface defect detection by using an improved feature pyramid network (FPN) and the metric learning approach. Firstly, the FPN is improved by adding deformable convolution and convolutional block attention modules to improve the accuracy of detecting defects of different scales, and it is pretrained on the MS COCO dataset. Secondly, a new model is established to extract network features based on the transfer learning model and learned network parameters. Thirdly, a multimodal network structure is constructed, and the distance between each modal representative and the embedded feature vector is calculated to classify the defects. Finally, experiments are carried out on the miniImageNet dataset and the rail surface defect dataset. The results show that the mAP (five-way five-shot) of our method is 73.42% on the miniImageNet dataset and 63.29% on the rail defect dataset. Our experiments show the effectiveness of the proposed method, and the results of the rail surface defect detection are satisfactory. As there are few sample classification studies of rail surface defects, this work provides a different approach and lays a foundation for further research.

Dual-Spatial Normalized Transformer for image captioning

Self-attention modules have shown dominance in image captioning. However, current self-attention modules insufficiently consider spatial correlations between objects in an image and easily suffer from distribution shifts. In this work, we aim to improve self-attention modules in two ways to solve the above problems. We first propose a Spatial-Enhanced Attention (SEA) module, which incorporates dual spatial encodings to enhance geometric correlations between visual instances. We further introduce a Gated-Normalized Attention (GNA) module to fix the distributions inside self-attention module. GNA module also smooths gradients and reduces misleading information. To prove our designs, we apply SEA module and GNA module to the standard Transformer to construct a novel Dual-Spatial Normalized Transformer (DSNT), which captures a more accurate spatial representations and improves performance of image captioning. Extensive experiments are conducted to verify the effectiveness of our proposal. Compared to non-pretrained state-of-the-art methods, quantitative results on the MSCOCO dataset show that our DSNT achieves competitive performance on both the offline and online test sets, i.e., 134.8% CIDEr score on the Karpathy split and 134.0% CIDEr score on the official split. Our source code is available at https://github.com/TBI805/DSNT.

HA-FPN: Hierarchical Attention Feature Pyramid Network for Object Detection.

The goals of object detection are to accurately detect and locate objects of various sizes in digital images. Multi-scale processing technology can improve the detection accuracy of the detector. Feature pyramid networks (FPNs) have been proven to be effective in extracting multi-scaled features. However, most existing object detection methods recognize objects in isolation, without considering contextual information between objects. Moreover, for the sake of computational efficiency, a significant reduction in the channel dimension may lead to the loss of semantic information. This study explores the utilization of attention mechanisms to augment the representational power and efficiency of features, ultimately improving the accuracy and efficiency of object detection. The study proposed a novel hierarchical attention feature pyramid network (HA-FPN), which comprises two key components: transformer feature pyramid networks (TFPNs) and channel attention modules (CAMs). In TFPNs, multi-scaled convolutional features are embedded as tokens and self-attention is applied to across both the intra- and inter-scales to capture contextual information between the tokens. CAMs are employed to select the channels with rich channel information to alleviate massive channel information losses. By introducing contextual information and attention mechanisms, the HA-FPN significantly improves the accuracy of bounding box detection, leading to more precise identification and localization of target objects. Extensive experiments conducted on the challenging MS COCO dataset demonstrate that the proposed HA-FPN outperforms existing multi-object detection models, while incurring minimal computational overhead.

A crowd clustering prediction and captioning technique for public health emergencies.

The COVID-19 pandemic has come to the end. People have started to consider how quickly different industries can respond to disasters due to this public health emergency. The most noticeable aspect of the epidemic regarding news text generation and social issues is detecting and identifying abnormal crowd gatherings. We suggest a crowd clustering prediction and captioning technique based on a global neural network to detect and caption these scenes rapidly and effectively. We superimpose two long convolution lines for the residual structure, which may produce a broad sensing region and apply our model's fewer parameters to ensure a wide sensing region, less computation, and increased efficiency of our method. After that, we can travel to the areas where people are congregating. So, to produce news material about the present occurrence, we suggest a double-LSTM model. We train and test our upgraded crowds-gathering model using the ShanghaiTech dataset and assess our captioning model on the MSCOCO dataset. The results of the experiment demonstrate that using our strategy can significantly increase the accuracy of the crowd clustering model, as well as minimize MAE and MSE. Our model can produce competitive results for scene captioning compared to previous approaches.

Globally Guided Confidence Enhancement Network for Image-Text Matching

Image-text matching is a crucial aspect of multi-modal intelligence. The main challenge in this area is accurately measuring the relevance between the image and text, using evidence obtained through matching. Previous studies either concentrated on obtaining a well-represented global feature to measure similarity directly or on investigating complex matching patterns at a local level before aggregating them, with little attention paid to combining them. We propose a Globally Guided Confidence Enhancement Network that combines both approaches by obtaining a good global representation to guide fine-grained local interactions. In this process, content that better matches the text from a global perspective is enhanced and represented with confidence scores. Extensive experiments demonstrate that the approach we have employed achieves superior performance on Flickr30K and MSCOCO datasets.

Adaptive adjustment with semantic embedding for zero-shot object detection

Traditional zero-shot object-detection algorithms detect images of untrained classes in the model with the help of semantic embedding. However, these approaches may perform poorly due to the limitations of fixed semantic embedding. Given that fixed semantic attributes lead to a lack of generalization capabilities in the model, a semantic enhancement mechanism is proposed to update the semantic embedding, which is used to serve the needs of the visual space. Specifically, considering that the original semantic space is not enough to construct a visual-semantic mapping relationship, an augmented semantic embedding (ASE) approach is designed to supplement semantic attribute information. Then, a semantic channel attention mechanism is used to adjust the ASE. The adjustment strategy retains adequate attribute information, which is highly relevant to visual features. Finally, to alleviate the domain shift problem, a clustering association strategy is introduced to establish an inferred relationship, which ensures that the predictor is generalized to the unseen domain during training. The superiority of the proposed method is demonstrated by the MS-COCO and PASCAL VOC datasets.