MSCOCO Dataset Research Articles

YOLO-FL: A target detection algorithm for reflective clothing wearing inspection

In recent years, target detection has been widely used in specific dress detection, to reduce the probability of accidents in certain fields such as construction sites, traffic management, etc. However, the detection of professional dress, especially reflective clothing, is mainly relied on manual detection so far, with the disadvantages of low efficiency and high time consumption. To this end, we focus on reflective clothing and propose the YOLO-FL detection algorithm based on YOLOV4-tiny. Firstly, a new intersection and union ratio calculation method KIoU is proposed and introduced into the original K-means++ clustering method to avoid the influence brought by distance in the original K-means++ clustering method. Secondly, the SPPF module is introduced into the neck network to make the feature information of different scales fully integrated and obtain a richer feature representation. Finally, the label smoothing strategy is used to optimize the loss function of the network model to prevent the network from overfitting and to improve the generalization ability of the model to a certain extent. We tested on the self-made reflective clothing dataset, and the mAP and F1 values of the YOLO-FL algorithm were 91.14% and 0.85, respectively, which improved by 8.29% and 0.12, respectively, compared with the original algorithm. In addition, experiments on PASCAL VOC and MS-COCO datasets show that the accuracy of the YOLO-FL algorithm has significantly improved (+4.77% on the Pascal VOC benchmark and +2.49% on the MS-COCO benchmark), and has excellent migration generalization performance.

Global-local fusion based on adversarial sample generation for image-text matching

In the increasingly popular era of adversarial machine learning (AML), developing more robust and generalized algorithms has become a key research topic. Image-text matching as the foundation of tasks such as video Q&A and text-image generation also faces various attacks in AML. Current image-text matching based on the similarity of matching fragments only focuses on the local matching results, which does not establish a comprehensive cognition of content in text and image, resulting in mismatching of the abstract scene when facing complex attacks. Meanwhile, existing methods are not sensitive enough to identify the internal relationship between objects in different local areas, which also confuse matching. Therefore, aiming at the above problems, a global similarity matching module is proposed, which is dynamically fused with local similarity to measure the matching results flexibly and improve the understanding of abstract scenes. Furthermore, a global-local cognition fusion training mechanism based on relationship adversarial sample generation is proposedto enhance understanding of internal relationships between objects in different local area through adversarial sample generation. Global loss is introduced to train the overall model, and adjust the proportion of global-local loss in the training process to better identified the relationships between objects in different local areas, and avoided confusion and matching caused by the similarity of matching objects. Experimental results show that the proposed method is 7.4 % (rSum) better than the SOTA method on the Flickr30K dataset, and 4.0 % (rSum using the 1K test set) better on the MS-COCO dataset. The proposed global-local fusion (GLF) based on adversarial sample generation for image-text matching algorithm improves the accuracy and robustness of image-text matching performs well in facing some security challenges, promoting the development of visual and linguistic modal fusion.

EDKSANet: An Efficient Dual-Kernel Split Attention Neural Network for the Classification of Tibetan Medicinal Materials

Tibetan medicine has received wide acclaim for its unique diagnosis and treatment methods. The identification of Tibetan medicinal materials, which are a vital component of Tibetan medicine, is a key research area in this field. However, traditional deep learning-based visual neural networks face significant challenges in efficiently and accurately identifying Tibetan medicinal materials due to their large number, complex morphology, and the scarcity of public visual datasets. To address this issue, we constructed a computer vision dataset with 300 Tibetan medicinal materials and proposed a lightweight and efficient cross-dimensional attention mechanism, the Dual-Kernel Split Attention (DKSA) module, which can adaptively share parameters of the kernel in both spatial and channel dimensions. Based on the DKSA module, we achieve efficient unification of convolution and self-attention under the CNN architecture and develop a new lightweight backbone architecture, EDKSANet, to provide enhanced performance for various computer vision tasks. As compared to RedNet, the top-1 accuracy is improved by 1.2% on an ImageNet dataset, and a larger margin of +1.5 box AP for object detection and an improvement of +1.3 mask AP for instance segmentation on MS-COCO dataset are obtained. Moreover, EDKSANet achieved excellent classification performance on the Tibetan medicinal materials dataset, with an accuracy of up to 96.85%.

FenceTalk: Exploring False Negatives in Moving Object Detection

Deep learning models are often trained with a large amount of labeled data to improve the accuracy for moving object detection in new fields. However, the model may not be robust enough due to insufficient training data in the new field, resulting in some moving objects not being successfully detected. Training with data that is not successfully detected by the pre-trained deep learning model can effectively improve the accuracy for the new field, but it is costly to retrieve the image data containing the moving objects from millions of images per day to train the model. Therefore, we propose FenceTalk, a moving object detection system, which compares the difference between the current frame and the background image based on the structural similarity index measure (SSIM). FenceTalk automatically selects suspicious images with moving objects that are not successfully detected by the Yolo model, so that the training data can be selected at a lower labor cost. FenceTalk can effectively define and update the background image in the field, reducing the misjudgment caused by changes in light and shadow, and selecting images containing moving objects with an optimal threshold. Our study has demonstrated its performance and generality using real data from different fields. For example, compared with the pre-trained Yolo model using the MS COCO dataset, the overall recall of FenceTalk increased from 72.36% to 98.39% for the model trained with the data picked out by SSIM. The recall of FenceTalk, combined with Yolo and SSIM, can reach more than 99%.

Spatial Bias for attention-free non-local neural networks

In this paper, we introduce the Spatial Bias to learn global knowledge without self-attention in convolutional neural networks. Owing to the limited receptive field, conventional convolutional neural networks suffer from learning long-range dependencies. Non-local neural networks have struggled to learn global knowledge, but unavoidably have too heavy a network design due to the self-attention operation. Therefore, we propose a fast and lightweight Spatial Bias that efficiently encodes global knowledge without self-attention on convolutional neural networks. Spatial Bias is stacked on the feature map and convolved together to adjust the spatial structure of the convolutional features. Because we only use the convolution operation in this process, ours is lighter and faster than traditional methods based on the heavy self-attention operation. Therefore, we learn the global knowledge on the convolution layer directly with very few additional resources. Our method is very fast and lightweight due to the attention-free non-local method while improving the performance of neural networks considerably. Compared to non-local neural networks, the Spatial Bias use about ×10 times fewer parameters while achieving comparable performance with 1.6∼3.3 times more throughput on a very little budget. Furthermore, the Spatial Bias can be used with conventional non-local neural networks to further improve the performance of the backbone model. We show that the Spatial Bias achieves competitive performance that improves the classification accuracy by +0.79% and +1.5% on ImageNet-1K and CIFAR-100 datasets. Additionally, we validate our method on the MS-COCO and ADE20K datasets for downstream tasks involving object detection and semantic segmentation.

Parallel encoder–decoder framework for image captioning

Recent progress in deep learning has led to successful utilization of encoder–decoder frameworks inspired by machine translation in image captioning models. The stacking of layers in encoders and decoders has made it possible to use several modules in encoders and decoders. However, just one type of module in encoder or decoder has been used in stacked models. In this research, we propose a parallel encoder–decoder framework that aims to take advantage of multiple of types modules in encoders and decoders, simultaneously. This framework contains augmented parallel blocks, which include stacking modules or non-stacked ones. Then, the results of the blocks are integrated to extract higher-level semantic concepts. This general idea is not limited to image captioning and can be customized for many applications that utilize encoder–decoder frameworks. We evaluated our proposed method on the MS-COCO dataset and achieved state-of-the-art results. We got 149.92 for CIDEr-D metric outperforming state-of-the-art image captioning models.

Multi-Branch Dilation Convolution CenterNet for Object Detection of Underwater Vehicles

Object detection occupies a very important position in the fishing operation and autonomous navigation of underwater vehicles. At present, most deep-learning object detection approaches, such as R-CNN, SPPNet, R-FCN, etc., have two stages and are based on anchors. However, the previous methods generally have the problems of weak generalization ability and not high enough computational efficiency due to the generation of anchors. As a well-known one-stage anchor-free method, CenterNet can accelerate the inference speed by omitting the step of generating anchors, whereas it is difficult to extract sufficient global information because of the residual structure at the bottom layer, which leads to low detection precision for the overlapping targets. Dilation convolution makes the kernel obtain a larger reception field and access more information. Multi-branch structure can not only preserve the whole area information, but also efficiently separate foreground and background. By combining the dilation convolution and multi-branch structure, multi-branch dilation convolution is proposed and applied to the Hourglass backbone network in CenterNet, then an improved CenterNet named multi-branch dilation convolution CenterNet (MDC-CenterNet) is built, which has a stronger ability of object detection. The proposed method is successfully utilized for detection of underwater organisms including holothurian, scallop, echinus and starfish, and the comparison result shows that it outperforms the original CenterNet and the classical object detection network. Moreover, with the MS-COCO and PASCAL VOC datasets, a number of comparative experiments are performed for showing the advancement of our method compared to other best methods.

CycMixer: A simplified and rapidly converging object detection network of query based on cycle mixing

The query-based object detection models are improvement of transformer. Using the learnable query, they complete the transition from traditional dense detection to sparse detection. However, such models have the weaknesses of slow convergence speed, poor adaptability to target changes, and additional networks. The CycMixer proposed in this paper uses multi-scale granule cluster sampling for encoding, avoiding a large number of parameters brought by the explicit structure. It adopts the cycle mixing module to improve the adaptability and increase the receptive field to cope with the change in detection targets by means of adaptive channel sampling and cycle spatial sampling. Firstly, the query extracted by backbone is decoupled into content vector and positional vector. Then the multi-head attention mechanism generates offsets. Secondly, the multi-scale feature map obtained by backbone and the transformed position enter the multi-scale granule cluster sampling stage. The stage consists of multi-scale feature space generation and granule cluster sampling. In addition, offsets and original positional vectors are converted to a new positional vector, which is decoded to get the bounding box. Finally, the content vector and the sampled matrix enter the cycle mixing. Cycle mixing consists of adaptive channel mixing and cycle spatial mixing. The mixed content vector is updated by the post feed-forward network (FFN) to receive a new content vector. Another FFN outputs the predicted categories. Compared with the existing detectors, the AP of our method on MS COCO dataset can reach 44.0 in 12 epochs and 46.0 in 36 epochs of training, which has obvious advantages in the convergence speed and the simplicity of the model. Experimental results on the cityscapes dataset also demonstrated the superiority of the method.

PiGLET: Pixel-Level Grounding of Language Expressions With Transformers.

This paper proposes Panoptic Narrative Grounding, a spatially fine and general formulation of the natural language visual grounding problem. We establish an experimental framework for the study of this new task, including new ground truth and metrics. We propose PiGLET, a novel multi-modal Transformer architecture to tackle the Panoptic Narrative Grounding task, and to serve as a stepping stone for future work. We exploit the intrinsic semantic richness in an image by including panoptic categories, and we approach visual grounding at a fine-grained level using segmentations. In terms of ground truth, we propose an algorithm to automatically transfer Localized Narratives annotations to specific regions in the panoptic segmentations of the MS COCO dataset. PiGLET achieves a performance of 63.2 absolute Average Recall points. By leveraging the rich language information on the Panoptic Narrative Grounding benchmark on MS COCO, PiGLET obtains an improvement of 0.4 Panoptic Quality points over its base method on the panoptic segmentation task. Finally, we demonstrate the generalizability of our method to other natural language visual grounding problems such as Referring Expression Segmentation. PiGLET is competitive with previous state-of-the-art in RefCOCO, RefCOCO+ and RefCOCOg.

RelNet-MAM: Relation Network with Multilevel Attention Mechanism for Image Captioning

Text present in an image contains rich semantic information which is crucial for the understanding of an image. For example, a signboard having the text “deep water” conveys the danger involved in the image. The current image captioning models do not effectively utilize this useful semantic information due to their limited representation capabilities of scene-text tokens. Our work presents a novel image captioning model called RelNet-MAM, which utilizes a multilevel attention mechanism and relation network. To improve the appearance feature representation, RelNet-MAM uses multilevel attention which consists of spatial attention, channel-wise attention, and semantic attention. For representing the scene-text token effectively, RelNet-MAM uses appearance, FastText, location, and PHOC features for each token. Further, the proposed RelNet-MAM uses the relation network to establish the relationships between the objects and scene-text tokens. Finally, the transformer model together with dynamic pointer networks is used as a decoder in the caption generation process. The proposed RelNet-MAM model outperforms the state-of-the-art models on TextCaps, Flickr30k, and MS COCO datasets. TextCaps requires models to read and reason about the texts in an image for caption generation. MSCOCO and Flickr30k contain diverse images; persons, animals, automobiles, indoor and outdoor scenes. Remarkably, the proposed RelNet-MAM model outperforms the current best model by 2.3% on B-4, 1.8% on METEOR, 2.2% on ROUGE-L, 2.0% on CIDEr-D and 3.0% on SPICE metric scores on TextCaps dataset.

Few-Shot Object Detection with Memory Contrastive Proposal Based on Semantic Priors

Few-shot object detection (FSOD) aims to detect objects belonging to novel classes with few training samples. With the small number of novel class samples, the visual information extracted is insufficient to accurately represent the object itself, presenting significant intra-class variance and confusion between classes of similar samples, resulting in large errors in the detection results of the novel class samples. We propose a few-shot object detection framework to achieve effective classification and detection by embedding semantic information and contrastive learning. Firstly, we introduced a semantic fusion (SF) module, which projects semantic spatial information into visual space for interaction, to compensate for the lack of visual information and further enhance the representation of feature information. To further improve the classification performance, we embed the memory contrastive proposal (MCP) module to adjust the distribution of the feature space by calculating the contrastive loss between the class-centered features of previous samples and the current input features to obtain a more discriminative embedding space for better intra-class aggregation and inter-class separation for subsequent classification and detection. Extensive experiments on the PASCAL VOC and MS-COCO datasets show that the performance of our proposed method is effectively improved. Our proposed method improves nAP50 over the baseline model by 4.5% and 3.5%.

Background Instance-Based Copy-Paste Data Augmentation for Object Detection

In supervised deep learning object detection, the quantity of object information and annotation quality in a dataset affect model performance. To augment object detection datasets while maintaining contextual information between objects and backgrounds, we proposed a Background Instance-Based Copy-Paste (BIB-Copy-Paste) data augmentation model. We devised a method to generate background pseudo-labels for all object classes by calculating the similarity between object background features and image region features in Euclidean space. The background classifier, trained with these pseudo-labels, can guide copy-pasting to ensure contextual relevance. Several supervised object detectors were evaluated on the PASCAL VOC 2012 dataset, achieving a 1.1% average improvement in mean average precision. Ablation experiments with the BlitzNet object detector on the PASCAL VOC 2012 dataset showed an improvement of mAP by 1.19% using the proposed method, compared to a 0.18% improvement with random copy-paste. Images from the MS COCO dataset containing objects of the same classes as in PASCAL VOC 2012 were also selected for object pasting experiments. The contextual relevance of pasted objects demonstrated our model’s effectiveness and transferability between datasets with same class of objects.

Weakly supervised fine-grained semantic segmentation via spatial correlation-guided learning

Weakly supervised semantic segmentation (WSSS) mainly adopts class activation map (CAM) to recognize different categories with generated pseudo masks of only image-level labels. Recently, many advanced works focus on learning the semantic correlation to refine the conventional CAM, which only identify sparse and discriminative semantic regions, severely weakening the further learning ability of spatial features. To copy with the above problem, a spatial correlation-guided learning framework is proposed to exploit the spatial and semantic correlation between adjacent pixels for weakly supervised fine-grained semantic segmentation (WSFGSS). From the spatial perspective, self-supervised multi-view clustering (SMC) is designed to fully mine spatial correlation by clustering of multiple views including scale, angle and position. Moreover, a hybrid self-supervised (HS) loss function is used to further promote the optimized speed and accuracy of three spatial representations. From the semantic perspective, the affinity matrix is applied to describe the semantic similarity between different pixels by building a weighted graph, and combine above robust pseudo label into a probability transition matrix. Therefore, the initial CAM is gradually corrected during the iterative optimization by the random walk algorithm. Finally, the refined CAM is utilized as supervision information for training the standard segmentation network effectively. Sufficient experimental results on BSDS500, PASCAL VOC 2012 and MS COCO datasets show that the proposed SMC method obtains more accurate pseudo labels than the recent unsupervised segmentation models. Meantime, with these pseudo labels, the proposed fine-grained framework achieves the state-of-the-art performance for WSFGSS.

Deep internally connected transformer hashing for image retrieval

Transformer based on self-attention mechanism has made remarkable achievements in natural language processing, which inspired the application research of Transformer in computer vision. The current deep hashing algorithms extract image features through the convolutional neural network (CNN). CNN concentrates on local information, and features lack global dependency information, which has an impact on image retrieval accuracy. To remedy the above defects, this paper proposes deep internally connected Transformer hashing for image retrieval (DICTH). DICTH has designed an improved Transformer block: internally connected Transformer block (ICT). ICT performs an embedded transformation on the feature maps, splices the generated Keys and Queries, to explore the rich context information between Query–Key pairs, and then dynamically encodes through multi-layer convolution to learn the context multi-head self-attention matrix. By combining ICT and ResNet18 to achieve self-attention injection, a long-distance dependency is established in the feature space to make up for the shortcomings of pure CNN in the feature extraction process and guide the algorithm to learn more accurate hash codes. At the same time, in the face of complex label information in big data sets, this paper uses an improved cross-entropy loss function: T-cross-entropy loss, to promote network learning of hash codes with more ability to distinguish between classes. In this paper, a lot of experiments have been conducted on CIFAR10, NUS-WIDE and MS-COCO datasets to verify the performance of DICTH.

Deformable Part Region Learning and Feature Aggregation Tree Representation for Object Detection.

Region-based object detection infers object regions for one or more categories in an image. Due to the recent advances in deep learning and region proposal methods, object detectors based on convolutional neural networks (CNNs) have been flourishing and provided promising detection results. However, the accuracy of the convolutional object detectors can be degraded often due to the low feature discriminability caused by geometric variation or transformation of an object. In this article, we propose a deformable part region (DPR) learning in order to allow decomposed part regions to be deformable according to the geometric transformation of an object. Because the ground truth of the part models is not available in many cases, we design part model losses for the detection and segmentation, and learn the geometric parameters by minimizing an integral loss including those part losses. As a result, we can train our DPR network without extra supervision, and make multi-part models deformable according to object geometric variation. Moreover, we propose a novel feature aggregation tree (FAT) so as to learn more discriminative region of interest (RoI) features via bottom-up tree construction. The FAT can learn the stronger semantic features by aggregating part RoI features along the bottom-up pathways of the tree. We also present a spatial and channel attention mechanism for the aggregation between different node features. Based on the proposed DPR and FAT networks, we design a new cascade architecture that can refine detection tasks iteratively. Without bells and whistles, we achieve impressive detection and segmentation results on MSCOCO and PASCAL VOC datasets. Our Cascade D-PRD achieves the 57.9 box AP with the Swin-L backbone. We also provide an extensive ablation study to prove the effectiveness and usefulness of the proposed methods for large-scale object detection.

AiShifu: AI Karate Pose Trainer Using Human Pose Estimation

One application of the artificial intelligence (AI) technology is self-guided physical activities where a computing device acts as a trainer. One key challenge for these applications is how to measure the performance of such an AI trainer, especially when the AI trainer is run on a generic PC or a mobile device. In the spirit of the Turing test, an AI trainer should mimic the behavior of a human trainer. A good human trainer generally considers the training history and the level of the trainee when providing feedback, which requires more than body position analysis. In this work, we built a Martial Art trainer application called AIShifu that helps users practice martial art poses using Human Pose Estimation (HPE). We chose an open-source neural network called HRNET trained with MS-COCO dataset as the core of the HPE. The joint coordinates and angles were used to identify the pose being practiced by the trainee, whether the active side is left or right, and how close is the key joint angle to that from a “golden” image. We collected data from both a black belt martial artist and a novice trainee and on three Karate poses. Based on the data, it is clear that the blackbelt performed the poses more consistently. A much larger sample size was required to test how well an AI trainer can discern the difference between trainees with different levels of proficiency. This understanding forms the foundation to customize AI trainer softwares for different users.

Word self-update contrastive adversarial networks for text-to-image synthesis

Synthesizing realistic fine-grained images from text descriptions is a significant computer vision task. Although many GANs-based methods have been proposed to solve this task, generating high-quality images consistent with text information remains a difficult problem. These existing GANs-based methods ignore important words due to the use of fixed initial word features in generator, and neglect to learn semantic consistency between images and texts for discriminators. In this article, we propose a novel attentional generation and contrastive adversarial framework for fine-grained text-to-image synthesis, termed as Word Self-Update Contrastive Adversarial Networks (WSC-GAN). Specifically, we introduce a dual attention module for modeling color details and semantic information. With a new designed word self-update module, the generator can leverage visually important words to compute attention maps in the feature synthesis module. Furthermore, we contrive multi-branch contrastive discriminators to maintain better consistency between the generated image and text description. Two novel contrastive losses are proposed for our discriminators to impose image-sentence and image-word consistency constraints. Extensive experiments on CUB and MS-COCO datasets demonstrate that our method achieves better performance compared with state-of-the-art methods.

Semantic-Enhanced Cross-Modal Fusion for Improved Unsupervised Image Captioning

Unsupervised image captioning often grapples with challenges such as image–text mismatches and modality gaps, resulting in suboptimal captions. This paper introduces a semantic-enhanced cross-modal fusion model (SCFM) to address these issues. The SCFM integrates three innovative components: a text semantic enhancement network (TSE-Net) for nuanced semantic representation; contrast learning for optimizing similarity measures between text and images; and enhanced visual selection decoding (EVSD) for precise captioning. Unlike existing methods that struggle with capturing accurate semantic relationships and flexibility across scenarios, the proposed model provides a robust solution for unbiased and diverse captioning. Through experimental evaluations on the MS COCO and Flickr30k datasets, SCFM demonstrates significant improvements over the benchmark model, enhancing the CIDEr and BLEU-4 metrics by 3.6% and 3.2%, respectively. Visualization analysis further reveals the model’s superiority in increasing variability between hidden features and its potential in cross-domain and stylized image captioning. The findings not only contribute to the advancement of image captioning techniques but also open avenues for future research. Further investigations will explore SCFM’s adaptability to other multimodal tasks and refine it for more intricate image–text relationships.

Mining graph-based dynamic relationships for object detection

Since the propagation of deep neural networks results in the loss of detailed feature information, the performance of most object detection methods is limited due to their tendency to learn regional features in visual space while neglecting relationships between objects. Therefore, this study proposes the Graph Relational Decision Network (GRDN), which mines relationships between objects in a dataset. The GRDN consists of a graph decision network, decision coefficient, and step-wise relation deduction module. The graph decision network comprises an edge decision network, and a node decision network, wherein a data-driven technique is employed to obtain implicit relationships between labels in a dataset. These relationships are expressed through an adaptive dynamic graph, which is subsequently recoded by means of the decision coefficient, which can enhance semantic information. In the step-wise relation deduction module, semantic information is employed as a guide to prevent distraction. A series of experiments were conducted on the MS COCO dataset. The proposed method achieves 52.8% box AP on object detection, which is 2.3% box AP higher than Cascade Mask R-CNN. The experimental results show that the addition of dynamic semantic information in this study can make up for the loss of detailed information and focus on key information, thereby improving the detection ability of small objects and occluded objects. In summary, this study extracts inter-object relationships to obtain more complete semantic information, which enriches the research of object detection.

Feature disentanglement in one-stage object detection

In this paper, an enhanced disentanglement module is proposed to address feature misalignment caused by inherently irreconcilable conflicts between classification and regression tasks in Convolutional Neural Network-based object detectors. The proposed method disentangles features in the feature pyramid network (FPN) at the neck of the architecture. In addition, a response alignment strategy is proposed to reduce inconsistent responses and suppress inferior predictions. Extensive experiments are performed on the MS COCO and PASCAL VOC datasets with different backbones, confirming that the proposed method improves performance significantly. The proposed method exhibits two main advantages over existing solutions—features are disentangled at the neck instead of at the head, enabling comprehensive resolution of feature misalignment, and independent outputs of the two tasks after feature disentanglement are avoided, thereby preventing response inconsistencies.