MSCOCO Dataset Research Articles

Disentangling the intrinsic feature from the related feature in image classification using knowledge distillation and object replacement

In our previous work, the features extracted by deep learning models are conceptually divided into the intrinsic features and the related features, where the object-oriented features are the intrinsic features and the background-oriented features are the related features. And the intrinsic feature-based classification results are regarded more reliable. In this work, we have improved our previously used metric and propose a new metric called Reliable Classification Accuracy (RCA) to more effectively quantify the dependency of classification results on the intrinsic features. Compared to our previous method that heightens the intrinsic feature exaction, we propose a novel method to further disentangle the intrinsic features from the related features. The proposed method comprises specially designed knowledge distillation and unique data augmentation. Unlike traditional knowledge distillation, the teacher model shares an identical network structure with the student model but has a special training process: the teacher model is pre-trained on a carefully designed dataset for pure intrinsic feature extraction and then guides the student model to focus on the category-related objects while ignoring the backgrounds in general datasets. And in the unique data augmentation, the category-related objects and the backgrounds from different images are re-matched in a graph-based method to further disentangle the intrinsic features from the related features in the guided feature extraction by specially designed knowledge distillation. The experiment results prove that the proposed method significantly improves both RCA and classification accuracy on the filtered datasets from MS COCO, Pascal VOC and Open Image Dataset.

SDSDet: A real-time object detector for small, dense, multi-scale remote sensing objects

Object detection in remote sensing images (RSIs) plays a crucial role in aerial and satellite image analysis. Existing methods lack the capability to effectively detect small and multi-scale objects in RSIs. Consequently, achieving an optimal trade-off between speed and accuracy remains unattainable. Extensive investigation reveals that state-of-the-art detectors have largely overlooked two critical aspects: Spatial artifacts from convolution operations and gradient confusion caused by neighboring levels in the Feature Pyramid Network. To address the first problem, we propose adopting a non-reorganized patch-embedding layer in the downsampling stage and a dual-path learning network (DPLNet) as the backbone, which can effectively mitigate the adverse effects of the edge pixel feature bias in feature maps. Additionally, using DPLNet as the backbone network can minimize costs while learning the intrinsic feature information of objects in RSI. For the second aspect, we propose a neighbor-erasing module with only one gradient flow (OGF-NEM). This module utilizes deep features to erase large objects to highlight small objects in shallow features and changes the backpropagation path to prevent the backflow of unreasonable gradients and the erosion of information from neighbor scales. Thus, a novel detector, called SDSDet, is proposed, which achieves excellent performance for small, dense, and multi-scale objects in RSIs. We have conducted exhaustive experiments on DOTA and MS COCO datasets. Specifically, the SDSDet achieves 42.8% AP on DOTA and 33.3% AP on MS COCO, together with nearly 4.87 M model size and 95 FPS.

Small target detection algorithm based on multi-branch stacking and new sampling transition module.

Aiming at the problem that the SSD algorithm does not fully extract the feature information contained in each feature layer, as well as the feature information is easily lost during the sampling process, which makes the feature expression ineffective and leads to insufficient performance in small target detection. In this paper, AMT-SSD is proposed, a small target detection algorithm that incorporates the multi-branch stacking and new sampling transition module of the attention mechanism. In this algorithm, the composite attention mechanism is utilized to improve the correlation of features of the samples to be detected in terms of spatial and channels, and the efficiency of the algorithm; secondly, multi-branch stacking module is used to extract multi-size features for each feature layer, and different sizes of convolution kernels are utilized in parallel to fully extract their features and improve the expression of features; meanwhile, during the sampling process, the problem of missing features is solved by applying inverse subpixel convolution in the new sampling transition module. Experimentally, the AMT-SSD algorithm achieves 84.6% and 53.4% mAP metrics on the PASCAL VOC dataset and MS COCO dataset, respectively. This indicates that the AMT-SSD algorithm can effectively extract feature information that is beneficial to detection samples, and also performs well in reducing feature loss, which is effective for the algorithm to improve the algorithm on small targets.

Relation Knowledge Distillation by Auxiliary Learning for Object Detection.

Balancing the trade-off between accuracy and speed for obtaining higher performance without sacrificing the inference time is a challenging topic for object detection task. Knowledge distillation, which serves as a kind of model compression techniques, provides a potential and feasible way to handle above efficiency and effectiveness issue through transferring the dark knowledge from the sophisticated teacher detector to the simple student one. Despite demonstrating promising solutions to make harmonies between accuracy and speed, current knowledge distillation for object detection methods still suffer from two limitations. Firstly, most of the methods are inherited or refereed from the frameworks in image classification task, and deploy an implicit manner by imitating or constraining the features from the intermediate layers or the output predictions between the teacher and student models. While little consideration has been raised to the intrinsic relevance of the classification and localization predictions in object detection task. Besides, these methods fail to investigate the relationship between detection and distillation tasks in knowledge distillation pipeline, and they train the whole network by simply integrating losses from these two different tasks through hand-crafted designation parameters. For addressing the aforementioned issues, we propose a novel Relation Knowledge Distillation by Auxiliary Learning for Object Detection (ReAL) method in this paper. Specifically, we first design a prediction relation distillation module which makes the student model directly mimic the output predictions from the teacher one, and conduct self and mutual relation distillation losses to excavate the relation information between teacher and student models. Moreover, for better devolving into the relationship between different tasks in distillation pipeline, we introduce the auxiliary learning into knowledge distillation for object detection and develop a dynamic weight adaptation strategy. Through regarding detection task as primary task and treating distillation task as auxiliary task in auxiliary learning framework, we dynamically adjust and regularize the corresponding weights of the losses for these tasks during the training process. Experiments on MS COCO dataset are conducted using various detector combinations of teacher and student models and the results show that our proposed ReAL can achieve obvious improvement on different distillation model configurations, while performing favorably against state-of-the-arts.

Image Captioning via Dynamic Path Customization.

This article explores a novel dynamic network for vision and language (V&L) tasks, where the inferring structure is customized on the fly for different inputs. Most previous state-of-the-art (SOTA) approaches are static and handcrafted networks, which not only heavily rely on expert knowledge but also ignore the semantic diversity of input samples, therefore resulting in suboptimal performance. To address these issues, we propose a novel Dynamic Transformer Network (DTNet) for image captioning, which dynamically assigns customized paths to different samples, leading to discriminative yet accurate captions. Specifically, to build a rich routing space and improve routing efficiency, we introduce five types of basic cells and group them into two separate routing spaces according to their operating domains, i.e., spatial and channel. Then, we design a Spatial-Channel Joint Router (SCJR), which endows the model with the capability of path customization based on both spatial and channel information of the input sample. To validate the effectiveness of our proposed DTNet, we conduct extensive experiments on the MS-COCO dataset and achieve new SOTA performance on both the Karpathy split and the online test server. The source code is publicly available at https://github.com/xmu-xiaoma666/DTNet.

Joint-Guided Distillation Binary Neural Network via Dynamic Channel-Wise Diversity Enhancement for Object Detection

Through truncating the weights and activations of a deep neural network, conventional binary quantization imposes limitations on the representation capability of the network parameters, which hence deteriorates the detection performance of the network. In this paper, a joint-guided distillation binary neural network via dynamic channel-wise diversity enhancement for object detection (JDBNet) is proposed to mitigate the gap of quantization errors. Our JDBNet includes a dynamic channel-wise diversity scheme and real-valued joint-guided teacher assistance to enhance the representation capability of the binary neural network in the object detection tasks. In the dynamic diversity scheme, the learning channel-wise bias (LCB) layer supports adjusting the magnitude of the parameters in which the sensitivity of the model parameters to the arbitrary quantization method is reduced, thereby improving the diversity expression ability of the feature parameters. In the joint-guided strategy, the single-precision implicit knowledge from the guiding teacher in the multilevel layer is utilized to supervise and penalize the quantitative model, enhancing the fitting performance of parameters in the binary quantized model. Extensive experiments on the PASCAL VOC, MS COCO, and VisDrone-DET datasets demonstrate that our JDBNet outperforms the state-of-the-art binary object detection networks in terms of mean Average Precision.

USER: Unified Semantic Enhancement With Momentum Contrast for Image-Text Retrieval.

As a fundamental and challenging task in bridging language and vision domains, Image-Text Retrieval (ITR) aims at searching for the target instances that are semantically relevant to the given query from the other modality, and its key challenge is to measure the semantic similarity across different modalities. Although significant progress has been achieved, existing approaches typically suffer from two major limitations: (1) It hurts the accuracy of the representation by directly exploiting the bottom-up attention based region-level features where each region is equally treated. (2) It limits the scale of negative sample pairs by employing the mini-batch based end-to-end training mechanism. To address these limitations, we propose a Unified Semantic Enhancement Momentum Contrastive Learning (USER) method for ITR. Specifically, we delicately design two simple but effective Global representation based Semantic Enhancement (GSE) modules. One learns the global representation via the self-attention algorithm, noted as Self-Guided Enhancement (SGE) module. The other module benefits from the pre-trained CLIP module, which provides a novel scheme to exploit and transfer the knowledge from an off-the-shelf model, noted as CLIP-Guided Enhancement (CGE) module. Moreover, we incorporate the training mechanism of MoCo into ITR, in which two dynamic queues are employed to enrich and enlarge the scale of negative sample pairs. Meanwhile, a Unified Training Objective (UTO) is developed to learn from mini-batch based and dynamic queue based samples. Extensive experiments on the benchmark MSCOCO and Flickr30K datasets demonstrate the superiority of both retrieval accuracy and inference efficiency. For instance, compared with the existing best method NAAF, the metric R@1 of our USER on the MSCOCO 5K Testing set is improved by 5% and 2.4% on caption retrieval and image retrieval without any external knowledge or pre-trained model while enjoying over 60 times faster inference speed. Our source code will be released at https://github.com/zhangy0822/USER.

Semantic Distance Adversarial Learning for Text-to-Image Synthesis

Text-to-Image (T2I) synthesis is a cross-modality task that requires a text description as input to generate a realistic and semantically consistent image. To guarantee semantic consis- tency, previous studies regenerate text descriptions from synthetic images and align them with the given descriptions. However, the existing redescription modules lack explicit modeling of their training objectives, which is crucial for reliable measurement of semantic distance between redescriptions and given text inputs. Consequently, the aligned text redescriptions suffer from training bias caused by the emergence of adversarial image samples, unseen semantics, and mistaken contents from low- quality synthesized images. To this end, we propose a SEMantic distance Adversarial learning (SEMA) framework for Text-to- Image synthesis which strengthens semantic consistency from two aspects: 1) We introduce adversarial learning between the image generator and the text redescription module to mutually promote or demote the quality of generated image or text instances. This learning model ensures accurate redescription of image contents, thus diminishing the generation of adversarial image samples. 2) We introduce two-fold semantic distance discrimination (SEM distance) to characterize semantic relevance between matching text or image pairs. The unseen semantics and mistaken contents will be penalized with a large SEM distance. The proposed discrimination method also simplifies the model training process with no need to optimize multiple discriminators. Experimental results on CUB Birds 200 and MS-COCO datasets show that the proposed model outperforms the state-of-the-art methods. Code is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/NJUPT-MCC/SEMA</uri> .

Visual Image Captioning through Transformer

Abstract: The convergence of computer vision and natural language processing in Artificial Intelligence has sparked significant interest in recent years, largely propelled by the advancements in deep learning. One notable application born from this synergy is the automatic description of images in English. Image captioning involves the computer's ability to interpret visual information from an image and translate it into one or more descriptive phrases. Generating meaningful descriptions requires understanding the state, properties, and relationships between the depicted objects, demanding a grasp of high-level picture semantics. Automatically captioning images is a complex task that intertwines image analysis with text generation. Central to this process is the concept of attention, determining what to describe and in what sequence. While transformer architectures have shown success in text analysis and translation, adapting them for image captioning presents unique challenges due to structural differences between semantic units in images (usually identified regions from object detection models) and sentences (composed of individual words). Little effort has been devoted to tailoring transformer architectures to suit images' structural characteristics. In this study, we introduce the Image Transformer, a novel architecture comprising a modified encoding transformer and an implicit decoding transformer. Our approach involves expanding the inner architecture of the original transformer layer to better accommodate the structural nuances of images. By utilizing only region features as inputs, our model achieves state-of-the-art performance on the MSCOCO Dataset. This research employing CNN-Transformer architectural models for image captioning aims to detect objects within images and convey information through textual messages. The envisioned application of this method extends to aiding individuals with visual impairments, using text-to-speech messages to facilitate their access to information and nurture their cognitive abilities. This paper meticulously explores fundamental concepts in image captioning and its standardized procedures, introducing a generative CNN-Transformer model as a significant advancement in this field.

Image Caption Generator for Sinhala Using Deep Learning

In this study, for the image caption generation in the Sinhala language, we have implemented a Recurrent Neural Network based model consisting of an InceptionV3 model as an image feature extraction model and a Long Short Term Memory network for the language model by referring to the literature. The different variations of Sinhala versions of the Flickr8K and MS COCO datasets have been constructed and used to train experimental models. Evaluation of the generated captions has been done using both automated and manual approaches. The model trained on the MS COCO dataset with Google translated Sinhala captions has achieved the highest BLEU score of 0.592 and the highest METEOR score of 0.281. After doing the manual caption analysis, it was observed that there could be generated captions which could provide a good idea to the reader while having lower BLEU and METEOR scores.

GhostFormer: Efficiently amalgamated CNN-transformer architecture for object detection

The lightweight network model has gradually evolved into an important research direction in object detection. Network lightweight design has a variety of research methods, such as quantization, knowledge distillation, and neural architecture search. However, these methods either fail to break through the performance bottleneck of the model itself or require massive training costs. In order to solve these problems, a new object detection model based on CNN-Transformer hybrid feature extraction network called GhostFormer is proposed from the perspective of lightweight network structure design. GhostFormer makes full use of the advantages of local modeling of CNN and global modeling of Transformer, not only effectively reducing the complexity of the convolution model but also breaking through the limitation of Transformer’s lack of inductive bias. Finally, better transfer results are obtained in downstream tasks. Experiments show that the model is less than half as computationally expensive as YOLOv7 on the Pascal VOC dataset, with only about 3 % mAP@0.5 loss, and 9.7% mAP@0.5:0.95 improvement on the MS COCO dataset compared with GhostNet.

Powerful-IoU: More straightforward and faster bounding box regression loss with a nonmonotonic focusing mechanism

Bounding box regression (BBR) is one of the core tasks in object detection, and the BBR loss function significantly impacts its performance. However, we have observed that existing IoU-based loss functions suffer from unreasonable penalty factors, leading to anchor boxes expanding during regression and significantly slowing down convergence. To address this issue, we intensively analyzed the reasons for anchor box enlargement. In response, we propose a Powerful-IoU (PIoU) loss function, which combines a target size-adaptive penalty factor and a gradient-adjusting function based on anchor box quality. The PIoU loss guides anchor boxes to regress along efficient paths, resulting in faster convergence than existing IoU-based losses. Additionally, we investigate the focusing mechanism and introduce a non-monotonic attention layer that was combined with PIoU to obtain a new loss function PIoU v2. PIoU v2 loss enhances the capability to focus on anchor boxes of medium quality. By incorporating PIoU v2 into popular object detectors such as YOLOv8 and DINO, we achieved an increase in average precision (AP) and improved performance compared to their original loss functions on the MS COCO and PASCAL VOC datasets, thus validating the effectiveness of our proposed improvement strategies.

A Novel Lightweight Object Detection Network with Attention Modules and Hierarchical Feature Pyramid

Object detection methods based on deep learning typically require devices with ample computing capabilities, which limits their deployment in restricted environments such as those with embedded devices. To address this challenge, we propose Mini-YOLOv4, a lightweight real-time object detection network that achieves an excellent trade-off between speed and accuracy. Based on CSPDarknet-Tiny as the backbone network, we enhance the detection performance of the network in three ways. We use a multibranch structure embedded in an attention module for simultaneous spatial and channel attention calibration. We design a group self-attention block with a symmetric structure consisting of a pair of complementary self-attention modules to mine contextual information, thereby ensuring that the detection accuracy is improved without increasing the computational cost. Finally, we introduce a hierarchical feature pyramid network to fully exploit multiscale feature maps and promote the extraction of fine-grained features. The experimental results demonstrate that Mini-YOLOv4 requires only 4.7 M parameters and has a billion floating point operations (BFLOPs) value of 3.1. Compared with YOLOv4-Tiny, our approach achieves a 3.2% improvement in mean accuracy precision (mAP) for the PASCAL VOC dataset and obtains a significant improvement of 3.5% in overall detection accuracy for the MS COCO dataset. In testing with an embedded platform, Mini-YOLOv4 achieves a real-time detection speed of 25.6 FPS on the NVIDIA Jetson Nano, thus meeting the demand for real-time detection in computationally limited devices.

Vital information is only worth one thumbnail: Towards efficient human pose estimation

In pursuit of impressive performance, existing DCNN-based approaches of human pose estimation usually use massive networks and large-size images to train a deep model. When applying these deep based methods in real-time systems, current works try to compress the deep network by reducing the number of layers and channels, but such approaches are complex and poorly generalized since they require elaborate design of small-scale network structures. Based on the fact that large-size images contain redundant information, in this paper, we explore the influence of image-size on system complexity and propose a novel framework called ThumbPose to accelerate and compress deep models by inferring on thumbnail representations in the task of human pose estimation. In our framework, we first propose a style supervised online downscaler to reduce an input image into a thumbnail image. Furthermore, a training strategy of dual-branch auto-encoding is designed to obtain effective and accurate thumbnail representation in a knowledge distillation manner, which is further used to maintain the performance of thumbnail images as the original-size input images. For heat-map based human pose estimation, ThumbPose is an orthogonal and implementation-friendly method, that can not only compress and accelerate the inference network but also obtain an image downscaler in a supervised manner that can be used in other high-level tasks (e.g. detection, segmentation, etc. in practical applications). Extensive experiments on MS COCO dataset demonstrate the effectiveness of our proposed method, and ThumbPose achieves superior performance (＋ 1.3% AP and ＋ 0.7% AR) with negligible additional cost (<0.2 GFLOPs) compared to previous state-of-the-art methods when using small-size images as inputs. Moreover, experiments on MPII show that our model achieves higher accuracy (＋ 0.2% Mean@0.5) with minimal computation (2.5 GFLOPs) compared to superior lightweight models obtained by the network compression methods.

Boosting Few-shot Object Detection with Discriminative Representation and Class Margin

Classifying and accurately locating a visual category with few annotated training samples in computer vision has motivated the few-shot object detection technique, which exploits transfering the source-domain detection model to the target domain. Under this paradigm, however, such transferred source-domain detection model usually encounters difficulty in the classification of the target domain because of the low data diversity of novel training samples. To combat this, we present a simple yet effective few-shot detector, Transferable RCNN. To transfer general knowledge learned from data-abundant base classes to data-scarce novel classes, we propose a weight transfer strategy to promote model transferability and an attention-based feature enhancement mechanism to learn more robust object proposal feature representations. Further, we ensure strong discrimination by optimizing the contrastive objectives of feature maps via a supervised spatial contrastive loss. Meanwhile, we introduce an angle-guided additive margin classifier to augment instance-level inter-class difference and intra-class compactness, which is beneficial for improving the discriminative power of the few-shot classification head under a few supervisions. Our proposed framework outperforms the current works in various settings of PASCAL VOC and MSCOCO datasets; this demonstrates the effectiveness and generalization ability.

External knowledge-assisted Transformer for image captioning

Internal relationship exploring based on the descriptive region features of image objects and grid features have contributed significantly to the development of image captioning, especially when combined with Transformer architecture. However, when conducting self-attention calculation, most of these methods only consider the relationship intra-objects and ignore the connection between entities and background. Besides, the way of exploring the relation information inside the image can also be extended. In this paper, we introduce a novel Mixed Knowledge Relation Transformer (MKRT) to explore the relationship between objects from both internal attribute relationship and external the object-verb-subject relationship. Furthermore we embed the important image background information into the relation module. In MKRT, the semantic relation obtained from the external knowledge is incorporated into the relation modeling in the novel Mixed Knowledge Relation Attention (MKRA). To validate the effectiveness of our model, we conduct extensive experiments on the most popular MSCOCO dataset, and achieve 134.5 CIDEr score on the offline test split and 133.5 CIDEr (c40) score on the official online testing server.

A New Multinetwork Mean Distillation Loss Function for Open-World Domain Incremental Object Detection

The development of object detection networks has reached a high point, and there have been significant improvements in accuracy and detection speed. Object detection is widely used in intelligent robots, self-driving cars, and other edge-intelligent terminals. Unfortunately, when a detector is allowed to learn new objects in an unfamiliar environment, it can catastrophically forget the objects it has already learned. In particular, reliable and stable knowledge cannot be extracted from old models. Based on this, a new multinetwork mean distillation loss function for open-world domain incremental object detection is presented. To better extract reliable and stable knowledge from old models, we enhanced the distillation output of the detector with a ResNet50 backbone and an output RoI head. The distillation output of the intermediate RPN is softened by adaptive distillation. To obtain more stable results, the ResNet50 backbone and RPN on the channel are zero-averaged. Various incremental steps and stability experiments are performed on two benchmark datasets, PASCAL VOC and MS COCO. The experimental results show the excellent performance of our method in different experimental scenarios, and it is superior to the most advanced methods. For example, in the setting of the batch task, incremental object detection on the PASCAL VOC and MS COCO datasets is improved by 3.4% and 2.1%, respectively.

SSDLiteX: Enhancing SSDLite for Small Object Detection

Object detection in many real applications requires the capability of detecting small objects in a system with limited resources. Convolutional neural networks (CNNs) show high performance in object detection, but they are not adequate to resource-limited environments. The combination of MobileNet V2 and SSDLite is one of the common choices in such environments, but it has a problem in detecting small objects. This paper analyzes the structure of SSDLite and proposes variations leading to small object detection improvement. The feature maps with the higher resolution are utilized more, and the base CNN is modified to have more layers in the high resolution. Experiments have been performed for the various configurations and the results show the proposed CNN, SSDLiteX, improves the detection accuracy AP of small objects by 1.5 percent points in the MS COCO data set.

YOLOD: A Task Decoupled Network Based on YOLOv5

Object detection includes three subtasks of predicting target position, classification, and confidence. In the mainstream object detection model, the model pursues internal structure refinement, and each subtask shares almost the same structure, which is a task-coupled structure. The task-coupled structure of the model reduces the training parameters, but it cannot be tuned on the network structure for each task separately, which can limit the model performance. We designed a task decoupled object detection network (YOLOD) based on YOLOv5, where YOLOD is decoupled immediately after the backbone network. By observing the loss convergence of each subtask, three network structures are designed separately and the branch size is controlled so that the model has fewer training parameters. At the same time, some experimental adjustments were made to YOLOD to accelerate the convergence speeds of the model. In addition, we add image contour information to the original three-channel image to assist model training and improve detection accuracy. The experiments demonstrate that the modified model is smaller in size and has the largest accuracy improvement on the small-scale model. without introducing any attention-based modules, YOLOD-S achieves a mAP improvement of 1.1% on the MS COCO dataset and 2.29% on the VOC dataset, and the larger model YOLOD-L achieves an accuracy of 48.8% on the COCO dataset.

Research on Human Pose Estimation Based on High-resolution Network

The problem addressed in this paper is the increase in the parameter quantity of existing human body pose estimation network models when the depth of the network is scaled up to improve forecast precision. To tackle this issue, an optimization network model of the new human body pose estimation called BDENet is proposed, inspired by high-resolution detection networks. This model incorporates a bottleneck structure and dilated convolution to reduce parameters and incorporates the ECA lightweight attention mechanism to enhance precision. Compared with HRNet, the proposed model achieves a 21.4% reduction in parameter quantity on the MSCOCO dataset while scaling up precision by 1.4%. The experimental findings strongly suggest that the modified network significantly improves network precision along with lowering the number of parameters.