MS COCO Research Articles

YOLO vs CNN : A Comparative Study on Object Detection

Abstract—With the availability of large amount of data and increasing demand of digitizing the visual systems, lot of research on object detection techniques has been the foremost requirement since last ten years. As the technological advancements are pacing at such a higher rate with increasing computational power and new architectures for CNN has already been proposed in a successive manner to enhance the detection techniques. For Real time Object identification, CNN has proposed several architectures but YOLO proves itself better than others based on the speed of identification. This paper presents the brief overview and comparative analysis of various object detection techniques like SSD, RCNN, YOLO and their subsequent versions for different applications. On comparing, it is found that single stage detectors outperform two stage detectors. Keywords—Pascal VOC, MS COCO, YOLO, CNN, Object detection, Object Recognition, SSD.

Pesticide Residue Coverage Estimation on Citrus Leaf Using Image Analysis Assisted by Machine Learning

Globally, the agricultural industry has benefited from using pesticides to minimize crop losses. Nevertheless, the indiscriminate overuse of pesticides has led to significant risks associated with a detrimental impact on the environment and human health. Therefore, emerging concerns of pesticide residue found in crops, food, and livestock are a pressing issue. To address the above challenges, there have been many efforts made towards implementing machine learning to enable precision agricultural practices to reduce pesticide overuse. As of today, there are no guiding digital tools available for citrus growers to provide pesticide residue leaf coverage analysis after foliar applications. Herein, we are the first to report software assisted by lightweight machine learning (ML) to determine the Kocide 3000 and Oxytetracycline (OTC) residue coverage on citrus leaves based on image data analysis. This tool integrates a foundational Segment Anything Model (SAM) for image preprocessing to isolate the area of interest. In addition, Kocide 3000 and Oxytetracycline (OTC) residue coverage analysis was carried out using a specialized Mask Region-Based Convolutional Neural Network (CNN). This CNN was pre-trained on the MS COCO dataset and fine-tuned by training with acquired datasets in laboratory and field conditions. The developed software demonstrated excellent performance on both pesticides’ accuracy, precision, and recall, and F1 score metrics. In summary, this tool has the potential to assist growers with the decision-making process for controlling pesticide use rate and frequency, minimizing pesticide overuse.

A Multi-Scale and Multi-Stage Human Pose Recognition Method Based on Convolutional Neural Networks for Non-Wearable Ergonomic Evaluation

In the context of industrial robot maintenance and assembly, workers often suffer from work-related musculoskeletal disorders (WRMSDs). This paper proposes a multi-scale, multi-stage pose recognition method (MMARM-CNN) based on convolutional neural networks to provide ergonomic intervention. The method leverages computer vision technology to enable non-contact data acquisition, reducing the interference of physiological and psychological factors on assessment results. Built upon the baseline yolov8-pose framework, the method addresses complex maintenance environments, which are prone to occlusion, by introducing the Lightweight Shared Convolutional Detection Head-pose (LSCD-pose) module, Multi-Scale Channel Attention (MSCA) mechanism, and Efficient Multi-Scale Patch Convolution (EMSPC) module, enhancing the model’s feature extraction capabilities. The MMARM-CNN model was validated using the MS COCO 2017 dataset and robot assembly data collected under laboratory conditions. The experimental results show that the MMARM-CNN achieved an accuracy improvement, reaching 0.875 in the mAP@0.5 evaluation. Overall, this method demonstrates significant potential in advancing the automation and intelligence of ergonomic interventions.

Illuminating Salient Contributions in Neuron Activation with Attribution Equilibrium.

With the remarkable success of deep neural networks, there is a growing interest in research aimed at providing clear interpretations of their decision-making processes. In this paper, we introduce Attribution Equilibrium, a novel method to decompose output predictions into fine-grained attributions, balancing positive and negative relevance for clearer visualization of the evidence behind a network decision. We carefully analyze conventional approaches to decision explanation and present a different perspective on the conservation of evidence. We define the evidence as a gap between positive and negative influences among gradient-derived initial contribution maps. Then, we incorporate antagonistic elements and a user-defined criterion for the degree of positive attribution during propagation. Additionally, we consider the role of inactivated neurons in the propagation rule, thereby enhancing the discernment of less relevant elements such as the background. We conduct various assessments in a verified experimental environment with PASCAL VOC 2007, MS COCO 2014, and ImageNet datasets. The results demonstrate that our method outperforms existing attribution methods both qualitatively and quantitatively in identifying the key input features that influence model decisions.

Beyond clean data: Exploring the effects of label noise on object detection performance

In recent years, the growth of large-scale datasets has significantly propelled the progress of deep learning applications. Yet, annotating these datasets remains a labor-intensive endeavor, pushing the reliance on cost-effective but less specialized data collection methods and internet data sources. This often results in noisy and inaccurate labels, compromising data quality. Traditional machine learning models assume clean data, but real-world datasets often exhibit significant label noise. This paper examines the impact of such noise on object detection performance, a pivotal aspect of computer vision. We analyze the influence of noisy labels using three renowned object detection frameworks: YOLOv5, Faster R-CNN, and the recent YOLOv8, alongside established datasets: MS COCO, VOC, and ExDARK. Additionally, experiments with the UVM dataset explore domain-specific tasks in dense object scenarios. Two new metrics — Model Health and Detection Capability — were introduced to evaluate the results. Findings indicate that models maintain over 80% of their health (a 20% decline in mAP from the baseline) with up to 40% label corruption. However, Detection Capability deteriorates more sharply under the same conditions. The research also employs the D-RISE method for model explainability, highlighting crucial image regions affecting detection outcomes. Despite the noise, critical detection areas in models remain similar to those in clean data up to the 40% corruption level, as verified by similarity metrics. This study underscores the resilience of object detection models to label noise and provides insights into maintaining performance amidst data quality challenges.

NAS-BNN: Neural Architecture Search for Binary Neural Networks

Binary Neural Networks (BNNs) have gained extensive attention for their superior inferencing efficiency and compression ratio compared to traditional full-precision networks. However, due to the unique characteristics of BNNs, designing a powerful binary architecture is challenging and often requires significant manpower. A promising solution is to utilize Neural Architecture Search (NAS) to assist in designing BNNs, but current NAS methods for BNNs are relatively straightforward and leave a performance gap between the searched models and manually designed ones. To address this gap, we propose a novel neural architecture search scheme for binary neural networks, named NAS-BNN. We first carefully design a search space based on the unique characteristics of BNNs. Then, we present three training strategies, which significantly enhance the training of supernet and boost the performance of all subnets. Our discovered binary model family outperforms previous BNNs for a wide range of operations (OPs) from 20M to 200M. For instance, we achieve 68.20% top-1 accuracy on ImageNet with only 57M OPs. In addition, we validate the transferability of these searched BNNs on the object detection task, and our binary detectors with the searched BNNs achieve a novel state-of-the-art result, e.g., 31.6% mAP with 370M OPs, on MS COCO dataset. The source code and models will be released at https://github.com/VDIGPKU/NAS-BNN.

Coupling Global Context and Local Contents for Weakly-Supervised Semantic Segmentation.

Thanks to the advantages of the friendly annotations and the satisfactory performance, weakly-supervised semantic segmentation (WSSS) approaches have been extensively studied. Recently, the single-stage WSSS (SS-WSSS) was awakened to alleviate problems of the expensive computational costs and the complicated training procedures in multistage WSSS. However, the results of such an immature model suffer from problems of background incompleteness and object incompleteness. We empirically find that they are caused by the insufficiency of the global object context and the lack of local regional contents, respectively. Under these observations, we propose an SS-WSSS model with only the image-level class label supervisions, termed weakly supervised feature coupling network (WS-FCN), which can capture the multiscale context formed from the adjacent feature grids, and encode the fine-grained spatial information from the low-level features into the high-level ones. Specifically, a flexible context aggregation (FCA) module is proposed to capture the global object context in different granular spaces. Besides, a semantically consistent feature fusion (SF2) module is proposed in a bottom-up parameter-learnable fashion to aggregate the fine-grained local contents. Based on these two modules, WS-FCN lies in a self-supervised end-to-end training fashion. Extensive experimental results on the challenging PASCAL VOC 2012 and MS COCO 2014 demonstrate the effectiveness and efficiency of WS-FCN, which can achieve state-of-the-art results by 65.02% and 64.22% mIoU on PASCAL VOC 2012 val set and test set, 34.12% mIoU on MS COCO 2014 val set, respectively. The code and weight have been released at:WS-FCN.

Deep convolutional neural networks for ship detection using refined DOTA and TGRS-HRRSD high-resolution image datasets

Automated identification of ships in satellite imagery is of utmost significance in many military and civilian applications, including monitoring maritime traffic and maintaining maritime security. However, ship detection from remotely sensed imagery is a challenging task because of their varying sizes, orientations, types, and various interferences. In recent years, deep learning algorithms have become a powerful tool in image processing applications, including image classification and object detection due to their strong feature representation capabilities. In this study, a refined ship dataset, containing a total of 13,735 instances and representing different ship types, was generated using state-of-the-art datasets for the ship detection task. In a comparative framework, the performance of five deep learning algorithms (Faster R-CNN, YOLOv8, CenterNet, EfficientDet, and SSD) in ship detection was investigated using the created dataset. The performance of the deep learning models was compared using MS COCO accuracy metrics. In addition, the generalization capabilities of the models were tested on an independent image acquired by Göktürk-1 having a mucilage effect that undermines the spectral discrimination. Results revealed that the YOLOv8 model had the greatest ship detection performance (AP@0.50 = 96.33 %), followed by CenterNet (AP@0.50 = 88.80 %), Faster R-CNN (AP@0.50 = 85.30 %), EfficientDet (AP@0.50 = 78.91 %) and SSD (AP@0.5 = 75.98 %), and obtained the highest accuracies in terms of other COCO metrics. In addition, generalization capabilities on the test image confirmed the superiority of the YOLOv8 model with AP@0.50 score of 63.93 %. Overall, the results of this study validated the effectiveness of the YOLOv8 model in identifying ship targets in remotely sensed images.

LVD-YOLO: An efficient lightweight vehicle detection model for intelligent transportation systems

Vehicle detection is a fundamental component of intelligent transportation systems. However, current algorithms often encounter issues such as high computational complexity, long execution times, and significant resource demands, making them unsuitable for resource-limited environments. To overcome these challenges, we propose LVD-YOLO, a Lightweight Vehicle Detection Model based on YOLO. This model incorporates the EfficientNetv2 network structure as its backbone, which reduces parameters and enhances feature extraction capabilities. By utilizing a bidirectional feature pyramid structure along with a dual attention mechanism, we enable efficient information exchange across feature layers, thereby improving multiscale feature fusion. Additionally, we refine the model's loss function with SIoU loss to boost regression and prediction performance. Experimental results on the PASCAL VOC and MS COCO datasets show that LVD-YOLO outperforms YOLOv5s, achieving a 0.5% increase in accuracy while reducing FLOPs by 64.6% and parameters by 48.6%. These improvements highlight its effectiveness for use in resource-constrained environments.

Innovation in public health surveillance for social distancing during the COVID-19 pandemic: A deep learning and object detection based novel approach.

The Corona Virus Disease (COVID-19) has a huge impact on all of humanity, and people's disregard for COVID-19 regulations has sped up the disease's spread. Our study uses a state-of-the-art object detection model like YOLOv4 (You Only Look Once, version 4), a very effective tool, on real-time 25fps, 1920 X 1080 video data streamed live by a camera-mounted Unmanned Aerial Vehicle (UAV) quad-copter to observe proper maintenance of social distance in an area of 35m range in this study. The model has demonstrated remarkable efficacy in identifying and quantifying instances of social distancing, with an accuracy of 82% and little latency. It has been able to work efficiently with real-time streaming at 25-30 ms. Our model is based on CSPDarkNet-53, which was trained on the MS COCO dataset for image classification. It includes additional layers to capture feature maps from different phases. Additionally, the model's neck is made up of PANet, which is used to aggregate the parameters from various CSPDarkNet-53 layers. The CSPDarkNet-53's 53 convolutional layers are followed by 53 more layers in the model head, for a total of 106 completely convolutional layers in the design. This architecture is further integrated with YOLOv3, resulting in the YOLOv4 model, which will be used by our detection model. Furthermore, to differentiate humans The aforementioned method was used to evaluate drone footage and count social distance violations in real time. Our findings show that our model was reliable and successful at detecting social distance violations in real-time with an average accuracy of 82%.

Decoupling foreground and background with Siamese ViT networks for weakly-supervised semantic segmentation

Due to the coarse granularity of information extraction in image-level annotation-based weakly supervised semantic segmentation algorithms, there exists a significant gap between the generated pseudo-labels and the real pixel-level labels. In this paper, we propose the DeFB-SV framework, which consists of a dual-branch Siamese network structure. This framework separates the foreground and background of images by generating unified resolution and mixed resolution class activation maps, which are then fused to obtain pseudo-labels. The mixed-resolution class activation maps are produced by a new mixed-resolution patch partition method, where we introduce a semantically heuristic patch scorer to divide the image into patches of different sizes based on semantics. Additionally, a novel multi-confidence region division mechanism is proposed to enable the adaptive extraction of the effective parts of pseudo-labels, further enhancing the accuracy of weakly supervised semantic segmentation algorithms. The proposed semantic segmentation framework, DeFB-SV, is evaluated on the PASCAL VOC 2012 and MS COCO 2014 datasets, demonstrating comparable segmentation performance with state-of-the-art methods.

Bridging the gap between object detection in close-up and high-resolution wide shots

Recent years have seen a significant rise in gigapixel-level image/video capture systems and benchmarks with high-resolution wide (HRW) shots. Different from close-up shots like MS COCO, the higher resolution and wider field of view raise new research and application problems, such as how to perform accurate and efficient object detection with such large input in low-power edge devices like UAVs. There are several unique challenges in HRW shots. (1) Sparse information: the objects of interest cover less area. (2) Various scale: there is 10 to 100× object scale change in one single image. (3) Incomplete objects: the sliding window strategy to handle the large input leads to truncated objects at the window edge. (4) Multi-scale information: it is unclear how to use multi-scale information in training and inference. Consequently, directly using a close-up detector leads to inaccuracy and inefficiency. In this paper, we systematically investigate this problem and bridge the gap between object detection in close-up and HRW shots, by introducing a novel sparse architecture that can be integrated with common networks like ConvNet and Transformer. It leverages alternative sparse learning to complementarily fuse coarse-grained and fine-grained features to (1) adaptively extract valuable information from (2) different object scales. We also propose a novel Cross-window Non-Maximum Suppression (C-NMS) algorithm to (3) improve the box merge from different windows. Furthermore, we propose a (4) simple yet effective multi-scale training and inference strategy to improve accuracy. Experiments on two benchmarks with HRW shots, PANDA and DOTA-v1.0, demonstrate that our methods significantly improve accuracy (up to 5.8%) and speed (up to 3×) over SotAs, for both ConvNet or Transformer based detectors, on edge devices. Our code is open-sourced and available at https://github.com/liwenxi/SparseFormer.

Probability based dynamic soft label assignment for object detection

By defining effective supervision labels for network training, the performance of object detectors can be improved without incurring additional inference costs. Current label assignment strategies generally require two steps: first, constructing a positive sample candidate bag, and then designing labels for these samples. However, the construction of candidate bag of positive samples may result in some noisy samples being introduced into the label assignment process. We explore a single-step label assignment approach: directly generating a probability map as labels for all samples. We design the label assignment approach from the following perspectives: Firstly, it should be able to reduce the impact of noise samples. Secondly, each sample should be treated differently because each one matches the target to a different extent, which assists the network to learn more valuable information from high-quality samples. We propose a probability-based dynamic soft label assignment method. Instead of dividing the samples into positive and negative samples, a probability map, which is calculated based on prediction quality and prior knowledge, is used to supervise all anchor points of the classification branch. The weight of prior knowledge in the labels decreases as the network improves the quality of instance predictions, as a way to reduce noise samples introduced by prior knowledge. By using continuous probability values as labels to supervise the classification branch, the network is able to focus on high-quality samples. As demonstrated in the experiments on the MS COCO benchmark, our label assignment method achieves 40.9% AP in the ResNet-50 under 1x schedule, which improves FCOS performance by approximately 2.0% AP. The code has been available at https://github.com/Liyi4578/PDSLA.

M[formula omitted]ixup: A multi-modal data augmentation approach for image captioning

Despite the great success, most models in image captioning (IC) are still stuck in the dilemma of generating simple and non-discriminative captions. In this paper, we study this problem from the perspective of data augmentation and propose a novel method called Multi-modal Mixup (M3ixup). Compared with the original Mixup strategy designed for image classification, the proposed M3ixup has three novel designs to mix IC samples from the aspects of visual features, sentence embeddings and loss values, respectively. In practice, M3ixup can not only enrich the diversity of IC training data, but also enforce the model to focus more on visual information for captioning, thereby alleviating the negative effect of dataset bias and addressing the issue of simple captioning. To validate M3ixup, we apply it to three baseline models and conduct extensive experiments on MS COCO. The experimental results demonstrate that our proposed M3ixup can not only improve the discriminability and quality of generated captions, but also help the baseline models obtain obvious performance gains, i.e., improving the CIDEr scores of the state-of-the-art model from 133.8 to 135.3 on off-line testing and 135.4 to 137.1 on online testing.

Learning General and Specific Embedding with Transformer for Few-Shot Object Detection

AbstractFew-shot object detection (FSOD) studies how to detect novel objects with few annotated examples effectively. Recently, it has been demonstrated that decent feature embeddings, including the general feature embeddings that are more invariant to visual changes and the specific feature embeddings that are more discriminative for different object classes, are both important for FSOD. However, current methods lack appropriate mechanisms to sensibly cooperate both types of feature embeddings based on their importance to detecting objects of novel classes, which may result in sub-optimal performance. In this paper, to achieve more effective FSOD, we attempt to explicitly encode both general and specific feature embeddings using learnable tensors and apply a Transformer to help better incorporate them in FSOD according to their relations to the input object features. We thus propose a Transformer-based general and specific embedding learning (T-GSEL) method for FSOD. In T-GSEL, learnable tensors are employed in a three-stage pipeline, encoding feature embeddings in general level, intermediate level, and specific level, respectively. In each stage, we apply a Transformer to first model the relations of the corresponding embedding to input object features and then apply the estimated relations to refine the input features. Meanwhile, we further introduce cross-stage connections between embeddings of different stages to make them complement and cooperate with each other, delivering general, intermediate, and specific feature embeddings stage by stage and utilizing them together for feature refinement in FSOD. In practice, a T-GSEL module is easy to inject. Extensive empirical results further show that our proposed T-GSEL method achieves compelling FSOD performance on both PASCAL VOC and MS COCO datasets compared with other state-of-the-art approaches.

Transformer-CNN for small image object detection

Object recognition in computer vision technology has been a popular research field in recent years. Although the detection success rate of regular objects has achieved impressive results, small object detection (SOD) is still a challenging issue. In the Microsoft Common Objects in Context (MS COCO) public dataset, the detection rate of small objects is typically half that of regular-sized objects. The main reason is that small objects are often affected by multi-layer convolution and pooling, leading to insufficient details to distinguish them from the background or similar objects, resulting in poor recognition rates or even no results. This paper presents a network architecture, Transformer-CNN, that combines a self-attention mechanism-based transformer and a convolutional neural network (CNN) to improve the recognition rate of SOD. It captures global information through a transformer and uses the translation invariance and translation equivalence of CNN to maximize the retention of global and local features while improving the reliability and robustness of SOD. Our experiments show that the proposed model improves the small object recognition rate by 2∼5 % than the general transformer architectures.

Object detection with a dynamic interactive network based on relational graph routing

Combinatorial relational reasoning in neural networks used for object detection is usually static; therefore, it cannot selectively fuse visual information and semantic relations, which limits their performance. To address this problem, we propose a relational graph routing network (RGRN) that enables the dynamic interaction of visual and semantic features. The network consists of a dynamic graph network, dual path-sharing module, and relational routing interaction module. First, we used a data-driven technique to obtain the semantic information between tags from the dataset. Rich semantic information was obtained by calculating the similarity between tags. Second, the two types of semantic information were fused using a dynamic graph network to capture high-level semantic information. The visual and semantic features are then filtered and encoded through the dual path-sharing module to obtain enhanced visual and semantic features. Finally, three units were used to dynamically fuse visual and semantic information in the relational routing interaction module, which densely links the three units and routers to construct a routing space that can autonomously decide on the optimal fusion path through model learning. A series of experiments was conducted on the MS COCO dataset. RGRN achieved 54.7% box AP on object detection, which was 2.8% box AP higher than that of the Cascade Mask R-CNN. The experimental results show that the routing space enables better interaction between visual and semantic information. Therefore, our method can achieve better performance than many state-of-the-art methods.

Dynamic Feature Focusing Network for small object detection

Deep learning has driven research in object detection and achieved proud results. Despite its significant advancements in object detection, small object detection still struggles with low recognition rates and inaccurate positioning, primarily attributable to their miniature size. The location deviation of small objects induces severe feature misalignment, and the disequilibrium between classification and regression tasks hinders accurate recognition. To address these issues, we propose a Dynamic Feature Focusing Network (DFFN), which contains a duo of crucial modules: Visual Perception Enhancement Module (VPEM) and Task Association Module (TAM). Drawing upon the deformable convolution and attention mechanism, the VPEM concentrates on sparse key features and perceives the misalignment via positional offset. We aggregate multi-level features at identical spatial locations via layer average operation for learning a more discriminative representation. Incorporating class alignment and bounding box alignment parts, the TAM promotes classification ability, refines bounding box regression, and facilitates the joint learning of classification and localization. We conduct diverse experiments, and the proposed method considerably enhances the small object detection performance on four benchmark datasets of MS COCO, VisDrone, VOC, and TinyPerson. Our method has improved by 3.4 and 2.2 in mAP and APs, making solid improvements on COCO. Compared to other classic detection models, DFFN exhibits a high level of competitiveness in precision.

Multi-scale feature correspondence and pseudo label retraining strategy for weakly supervised semantic segmentation

Recently, the performance of semantic segmentation using weakly supervised learning has significantly improved. Weakly supervised semantic segmentation (WSSS) that uses only image-level labels has received widespread attention, it employs Class Activation Maps (CAM) to generate pseudo labels. Compared to traditional use of pixel-level labels, this technique greatly reduces annotation costs by utilizing simpler and more readily available image-level annotations. Besides, due to the local perceptual ability of Convolutional Neural Networks (CNN), the generated CAM cannot activate the entire object area. Researchers have found that this CNN limitation can be compensated for by using Vision Transformer (ViT). However, ViT also introduces an over-smoothing problem. Recent research has made good progress in solving this issue, but when discussing CAM and its related segmentation predictions, it is easy to overlook their intrinsic information and the interrelationships between them. In this paper, we propose a Multi-Scale Feature Correspondence (MSFC) method. Our MSFC can obtain the feature correspondence of CAM and segmentation predictions at different scales, re-extract useful semantic information from them, enhancing the network's learning of feature information and improving the quality of CAM. Moreover, to further improve the segmentation precision, we design a Pseudo Label Retraining Strategy (PLRS). This strategy refines the accuracy in local regions, elevates the quality of pseudo labels, and aims to enhance segmentation precision. Experimental results on the PASCAL VOC 2012 and MS COCO 2014 datasets show that our method achieves impressive performance among end-to-end WSSS methods.

PatchNet: Maximize the Exploration of Congeneric Semantics for Weakly Supervised Semantic Segmentation.

With the increase in the number of image data and the lack of corresponding labels, weakly supervised learning has drawn a lot of attention recently in computer vision tasks, especially in the fine-grained semantic segmentation problem. To alleviate human efforts from expensive pixel-by-pixel annotations, our method focuses on weakly supervised semantic segmentation (WSSS) with image-level labels, which are much easier to obtain. As a considerable gap exists between pixel-level segmentation and image-level labels, how to reflect the image-level semantic information on each pixel is an important question. To explore the congeneric semantic regions from the same class to the maximum, we construct the patch-level semantic augmentation network (PatchNet) based on the self-detected patches from different images that contain the same class labels. Patches can frame the objects as much as possible and include as little background as possible. The patch-level semantic augmentation network that is established with patches as the nodes can maximize the mutual learning of similar objects. We regard the embedding vectors of patches as nodes and use a transformer-based complementary learning module to construct weighted edges according to the embedding similarity between different nodes. Moreover, to better supplement semantic information, we propose softcomplementary loss functions matched with the whole network structure. We conduct experiments on the popular PASCAL VOC 2012 and MS COCO 2014 benchmarks, and our model yields the state-of-the-art performance.