Improvement In Average Precision Research Articles

YH-RTYO: an end-to-end object detection method for crop growth anomaly detection in UAV scenarios

Background Small object detection via unmanned Aerial vehicle (UAV) is crucial for smart agriculture, enhancing yield and efficiency. Methods This study addresses the issue of missed detections in crowded environments by developing an efficient algorithm tailored for precise, real-time small object detection. The proposed Yield Health Robust Transformer-YOLO (YH-RTYO) model incorporates several key innovations to advance conventional convolutional models. The model features an efficient convolutional expansion module that captures additional feature information through extended branches while maintaining parameter efficiency by consolidating features into a single convolution during validation. It also includes a local feature pyramid module designed to suppress background interference during feature interaction. Furthermore, the loss function is optimized to accommodate various object scales in different scenes by adjusting the regression box size and incorporating angle factors. These enhancements collectively contribute to improved detection performance and address the limitations of traditional methods. Result Compared to YOLOv8-L, the YH-RTYO model achieves superior performance in all key accuracy metrics, with a 13% reduction in the scale of model. Experimental results demonstrate that the YH-RTYO model outperforms others in key detection metrics. The model reduces the number of parameters by 13%, facilitating deployment while maintaining accuracy. On the OilPalmUAV dataset, it achieves a 3.97% improvement in average precision (AP). Additionally, the model shows strong generalization on the RFRB dataset, with AP50 and AP values exceeding those of the YOLOv8 baseline by 3.8% and 2.7%, respectively.

Bi-directional information interaction for multi-modal 3D object detection in real-world traffic scenes

Multimodal 3D object detection methods are poorly adapted to real-world traffic scenes due to sparse distribution of point clouds and misalignment multimodal data during actual collection. Among the existing methods, they focus on high-quality open-source datasets, with performance relying on the accurate structural representation of point clouds and the precise mapping relationship between point clouds and images. To solve the above challenges, this paper proposes a multimodal feature-level fusion method based on the bi-directional interaction between image and point cloud. To overcome the sparsity issue in asynchronous multi-modal data, a point cloud densification scheme based on visual guidance and point cloud density guidance is proposed. This scheme can generate object-level virtual point clouds even when the point cloud and image are misaligned. To deal with the unalignment issue between point cloud and image, a bi-directional interaction module based on image-guided interaction with key points of point clouds and point cloud-guided interaction with image context information is proposed. It achieves effective feature fusion even when the point cloud and image are misaligned. The experiments on the VANJEE and KITTI datasets demonstrated the effectiveness of the proposed method, with average precision improvements of 6.20% and 1.54% compared to the baseline.

AMW-YOLOv8n: Road Scene Object Detection Based on an Improved YOLOv8

This study introduces an improved YOLOv8 model tailored for detecting objects in road scenes. To overcome the limitations of standard convolution operations in adapting to varying targets, we introduce Adaptive Kernel Convolution (AKconv). AKconv dynamically adjusts the convolution kernel’s shape and size, enhancing the backbone network’s feature extraction capabilities and improving feature representation across different scales. Additionally, we employ a Multi-Scale Dilated Attention (MSDA) mechanism to focus on key target features, further enhancing feature representation. To address the challenge posed by YOLOv8’s large down sampling factor, which limits the learning of small target features in deeper feature maps, we add a small target detection layer. Finally, to improve model training efficiency, we introduce a regression loss function with a Wise-IoU dynamic non-monotonic focusing mechanism. With these enhancements, our improved YOLOv8 model excels in road scene object detection tasks, achieving a 5.6 percentage point improvement in average precision over the original YOLOv8n on real road datasets.

A framework for automatically generating composite keywords for geo-tagged street images

Due to the ubiquity of sensor-equipped cameras such as smartphones, images are associated with spatial metadata, including camera’s geographical location and viewing orientation, which can be used for automatically generating better semantic keywords about geo-tagged urban street images in addition to visual keywords extracted from image analysis. This study introduces a novel framework for auto-tagging images that integrates both spatial and visual properties to generate comprehensive and accurate tags. The framework operates through four phases: extraction, abstraction, composition, and assessment. Our research highlights the benefits of combining visual and spatial analyses, demonstrated through a case study using geo-tagged urban street images from Orlando, Pittsburgh, and Manhattan. Experimental results show that the proposed framework significantly enhances the accuracy of keyword-based searches compared to conventional methods. In particular, based on our experiments, image search using the tags generated by our proposed framework, referred to as descriptive tags, achieved an average precision improvement factor of 0.9 compared to conventional tags. Additionally, our proposed ranking algorithm, which extends the term frequency-inverse document frequency (TF-IDF) algorithm, resulted in improvement factors of 0.86 for mean average precision (MAP) and 0.57 for mean reciprocal rank (MRR). Moreover, our framework’s flexibility and robustness make it suitable for diverse applications, from smart cities to online shopping. The paper also includes a detailed evaluation and user study, confirming the precision and reliability of the generated tags.

TP-Transfiner: high-quality segmentation network for tea pest.

Detecting and controlling tea pests promptly are crucial for safeguarding tea production quality. Due to the insufficient feature extraction ability of traditional CNN-based methods, they face challenges such as inaccuracy and inefficiency of detecting pests in dense and mimicry scenarios. This study proposes an end-to-end tea pest detection and segmentation framework, TeaPest-Transfiner (TP-Transfiner), based on Mask Transfiner to address the challenge of detecting and segmenting pests in mimicry and dense scenarios. In order to improve the feature extraction inability and weak accuracy of traditional convolution modules, this study proposes three strategies. Firstly, a deformable attention block is integrated into the model, which consists of deformable convolution and self-attention using the key content only term. Secondly, the FPN architecture in the backbone network is improved with a more effective feature-aligned pyramid network (FaPN). Lastly, focal loss is employed to balance positive and negative samples during the training period, and parameters are adapted to the dataset distribution. Furthermore, to address the lack of tea pest images, a dataset called TeaPestDataset is constructed, which contains 1,752 images and 29 species of tea pests. Experimental results on the TeaPestDataset show that the proposed TP-Transfiner model achieves state-of-the-art performance compared with other models, attaining a detection precision (AP50) of 87.211% and segmentation performance of 87.381%. Notably, the model shows a significant improvement in segmentation average precision (mAP) by 9.4% and a reduction in model size by 30% compared to the state-of-the-art CNN-based model Mask R-CNN. Simultaneously, TP-Transfiner's lightweight module fusion maintains fast inference speeds and a compact model size, demonstrating practical potential for pest control in tea gardens, especially in dense and mimicry scenarios.

Classification of Toraja Wood Carving Motif Images Using Convolutional Neural Network (CNN)

Wood carving is a cultural heritage with deep meaning and significance for the Toraja ethnic group's culture. By understanding the meaning of each Toraja carving, both tourists and the local community can gain knowledge about Toraja culture, thereby preserving and maintaining the culture amidst modern developments. Image processing approaches, particularly the development of Convolutional Neural Networks (CNN), offer a solution for extracting information from the diverse and intricate patterns of Toraja wood carvings. This study is highly significant as it implements a deep learning model using the CNN algorithm optimized with the ResNet50 architecture. The methodology in this study involves adjusting the batch size during the model training phase and applying weak-to-strong pixel transformation during the double threshold hysteresis phase in the Canny Feature Extraction process on the edges of Toraja carving images, resulting in ResNet50 architecture accurately recognizing the patterns of Toraja wood carvings. The results demonstrate significant improvements in the performance of the ResNet50 architecture with the preprocessed dataset. average precision, recall, precision, and F1-Score improvements in each Toraja carving class. For the Pa' Lulun Pao class, it was found that the precision and recall values were 0.94, and the F1-Score was 0.94. The Pa’ Somba class also showed good results, with a precision value of 0.9697, a recall of 0.96, and an F1-Score of 0.9648. The Pa’ Tangke Lumu class showed even better results, with a precision value of 0.9898, a recall of 0.97, and an F1-Score of 0.9798. The Pa’ Tumuru class also demonstrated good performance, with a precision value of 0.9327, a recall of 0.97, and an F1-Score of 0.9500. This study not only underscores the effectiveness of processing in enhancing CNN capabilities but also opens opportunities for further research in applying these methods to various image types and exploring different CNN architectures.

The Target Detection of Wear Particles in Ferrographic Images Based on the Improved YOLOv8

An enhanced YOLOv8 algorithm is proposed in the following paper to address challenging issues encountered in ferrographic image target detection, such as the identification of complex-shaped wear particles, overlapping and intersecting wear particles, and small and edge-wear particles. This aim is achieved by integrating the main body network with the improved Deformable Convolutional Network v3 to enhance feature extraction capabilities. Additionally, the Dysample method is employed to optimize the upsampling technique in the neck network, resulting in a clearer fused feature image and improved precision for detecting small and edge-wear particles. In the head network, parameter sharing simplifies the detection head while enhancing convergence speed and precision through improvements made to the loss function. The experimental results of the present study demonstrate that compared to the original algorithm, this enhanced approach achieves an average precision improvement of 5.6% without compromising the detection speed (111.6FPS), therefore providing valuable support for online monitoring device software foundations.

DILA: Dynamic Gaussian Distribution Fitting and Imitation Learning-Based Label Assignment for tiny object detection

With the rapid advancement of deep learning technology, significant achievements have been made in the field of general object detection. However, challenges still remain in the detection of tiny objects. There are two main drawbacks: (1) static prior information makes the localization of tiny objects relatively fixed; (2) Intersection over Union (IoU) is highly sensitive to deviations in tiny objects. To this end, we propose a Dynamic Gaussian Distribution Fitting and Imitation Learning-Based Label Assignment (DILA) strategy for Tiny Object Detection. Specifically, to address the positional deviation of effective receptive fields at different network layers during Gaussian modeling, DILA first designs an Adaptive Dynamic Calculation Strategy (ADCS) to compute estimation factors for the effective receptive field in different feature spaces, dynamically modeling prior information using Gaussian distribution. Then, DILA introduces a new Balance of Gaussian Scaling-aware Metric (BGSM) to measure the similarity between tiny bounding boxes and predefined anchors, instead of using IoU, which is highly sensitive to tiny pixel shifts, for sample assignment, thereby providing a more accurate basis for label assignment. Finally, a Detail Information Imitation Compensation Module (DIM) is presented to improve and compensate for the detailed information of tiny objects that troubles label assignment, achieving balanced learning for tiny objects. The proposed DILA strategy can be seamlessly integrated into various anchor-based detectors. Extensive experiments were conducted on three publicly available datasets for tiny object detection. The results indicate that when DILA is embedded into Faster RCNN, it outperforms other state-of-the-art methods in terms of detection performance for tiny objects, achieving an improvement in average precision of 10.3%, 1.5%, and 4.2% on AI-TOD, SODA-D, and VisDrone2019, respectively. The source codes and results are available at: https://github.com/chnu-cpl/DILA.

ID-Det: Insulator Burst Defect Detection from UAV Inspection Imagery of Power Transmission Facilities

The global rise in electricity demand necessitates extensive transmission infrastructure, where insulators play a critical role in ensuring the safe operation of power transmission systems. However, insulators are susceptible to burst defects, which can compromise system safety. To address this issue, we propose an insulator defect detection framework, ID-Det, which comprises two main components, i.e., the Insulator Segmentation Network (ISNet) and the Insulator Burst Detector (IBD). (1) ISNet incorporates a novel Insulator Clipping Module (ICM), enhancing insulator segmentation performance. (2) IBD leverages corner extraction methods and the periodic distribution characteristics of corners, facilitating the extraction of key corners on the insulator mask and accurate localization of burst defects. Additionally, we construct an Insulator Defect Dataset (ID Dataset) consisting of 1614 insulator images. Experiments on this dataset demonstrate that ID-Det achieves an accuracy of 97.38%, a precision of 97.38%, and a recall rate of 94.56%, outperforming general defect detection methods with a 4.33% increase in accuracy, a 5.26% increase in precision, and a 2.364% increase in recall. ISNet also shows a 27.2% improvement in Average Precision (AP) compared to the baseline. These results indicate that ID-Det has significant potential for practical application in power inspection.

AFE-YOLOv8: A Novel Object Detection Model for Unmanned Aerial Vehicle Scenes with Adaptive Feature Enhancement

Object detection in unmanned aerial vehicle (UAV) scenes is a challenging task due to the varying scales and complexities of targets. To address this, we propose a novel object detection model, AFE-YOLOv8, which integrates three innovative modules: the Multi-scale Nonlinear Fusion Module (MNFM), the Adaptive Feature Enhancement Module (AFEM), and the Receptive Field Expansion Module (RFEM). The MNFM introduces nonlinear mapping by exploiting the property that deformable convolution can dynamically adjust the shape of the convolution kernel according to the shape of the target, and it effectively enhances the feature extraction capability of the backbone network by integrating multi-scale feature maps from different mapping branches. Meanwhile, the AFEM introduces an adaptive fusion factor, and through the fusion factor, it adaptively integrates the small-target features contained in the feature maps of different detection branches into the small-target detection branch, thus enhancing the expression of the small-target features contained in the feature maps of the small-target detection branch. Furthermore, the RFEM expands the receptive field of the feature maps of the large- and medium-scale target detection branches through stacked convolution, so as to make the model’s receptive field cover the whole target, and thereby learn more rich and comprehensive features of the target. The experimental results demonstrate the superior performance of the proposed model compared to the baseline in detecting objects of various scales. On the VisDrone dataset, the proposed model achieves a 4.5% enhancement in mean average precision (mAP) and a 5.45% improvement in average precision at an IOU threshold of 0.5 (AP50). Additionally, ablation experiments conducted on the challenging DOTA dataset showcase the model’s robustness and generalization capabilities.

Oversized ore detection using rotated CCAD-YOLOv3 and ellipse fitting

The early detection and processing of oversized ores on the conveyor belt can effectively prevent accidents such as material blockage, ensuring transportation efficiency and production safety. The long-used manual detection method, however, has drawbacks such as high risk and poor accuracy. To address these challenges, this study proposes an automatic detection system for oversized ores based on rotated CCAD-YOLOv3 and ellipse fitting. Firstly, to more accurately represent the shape of adhesive ores and estimate their area, the rotated ellipse object detection is employed instead of conventional horizontal box object detection. Secondly, to enhance feature extraction capability and compensate for the low accuracy arising from insufficient data, the more sophisticated ConvNeXtv2-tiny with pre-trained parameters from ImageNet is adopted to replace DarkNet53 in the backbone. Then, to better integrate full-scale feature information, a structure combining the path aggregation network and adaptively spatial feature fusion with channel attention (PAN+CA-ASFF) is developed to substitute the feature pyramid network (FPN) in the neck. Lastly, to mitigate the inherent conflict between classification and regression tasks, a decoupling strategy is applied in the head. Experimental results demonstrate that the rotated ellipse is more conducive to fitting the shape of adhesive oversized ores than the segmentation mask, horizontal box, rotated box, and horizontal ellipse. Further, compared with the benchmark YOLOv3, CCAD-YOLOv3 exhibits a notable improvement in average precision (AP) by 6.52%, reaching 68.43%. This positions it as a leading solution, outperforming 21 other object detection algorithms.

Improving remote sensing object detection by using feature extraction and rotational equivariant attention

ABSTRACT In recent years, computer vision has witnessed significant attention in the research on object detection in remote-sensing images. Unlike objects in traditional natural images, those in remote sensing images are captured vertically by spacecraft, introducing arbitrary directionality, substantial scale variation, and a more complex background. We propose a rotation-equivariant detector enhanced with feature fusion and attention modules to address remote sensing image object detection challenges. Specifically, we introduce a Rotation-Enhanced Feature Extraction (REFE) module and a Rotational Equivariant Attention (REA) module. These enhancements empower the detector to extract object information more effectively from remotely sensed images, filtering out complex background information and improving detection accuracy and stability. Through extensive experiments on diverse and challenging remote sensing image datasets, our method outperforms the task of object detection. Remarkably, our network achieves 1.0, 2.91, and 1.11 mean average precision (mAP) improvements on the DOTA-v1.5, HRSC2016 and DIOR-R datasets. These experimental results robustly demonstrate the effectiveness and superiority of our proposed method.

Enhanced Water Surface Object Detection with Dynamic Task-Aligned Sample Assignment and Attention Mechanisms.

The detection of objects on water surfaces is a pivotal technology for the perceptual systems of unmanned surface vehicles (USVs). This paper proposes a novel real-time target detection system designed to address the challenges posed by indistinct bottom boundaries and foggy imagery. Our method enhances the YOLOv8s model by incorporating the convolutional block attention module (CBAM) and a self-attention mechanism, examining their impact at various integration points. A dynamic sample assignment strategy was introduced to enhance the precision of our model and accelerate its convergence. To address the challenge of delineating bottom boundaries with clarity, our model employs a two-strategy approach: a threshold filter and a feedforward neural network (FFN) that provides targeted guidance for refining these boundaries. Our model demonstrated exceptional performance, achieving a mean average precision (mAP) of 47.1% on the water surface object dataset, which represents a 1.7% increase over the baseline YOLOv8 model. The dynamic sample assignment strategy contributes a 1.0% improvement on average precision at the intersection over union (IoU) threshold of 0.5 (AP0.5), while the FFN strategy fine-tunes the bottom boundaries and achieves an additional 0.8% improvement in average precision at IoU threshold of 0.75 (AP0.75). Furthermore, ablation studies have validated the versatility of our approach, confirming its potential for integration into various detection frameworks.

GP-HTNLoc: A graph prototype head-tail network-based model for multi-label subcellular localization prediction of ncRNAs

Numerous research results demonstrated that understanding the subcellular localization of non-coding RNAs (ncRNAs) is pivotal in elucidating their roles and regulatory mechanisms in cells. Despite the existence of over ten computational models dedicated to predicting the subcellular localization of ncRNAs, a majority of these models are designed solely for single-label prediction. In reality, ncRNAs often exhibit localization across multiple subcellular compartments. Furthermore, the existing multi-label localization prediction models are insufficient in addressing the challenges posed by the scarcity of training samples and class imbalance in ncRNA dataset. To address these limitations, this study proposes a novel multi-label localization prediction model for ncRNAs, named GP-HTNLoc. To mitigate class imbalance, GP-HTNLoc adopts separate training approaches for head and tail location labels. Additionally, GP-HTNLoc introduces a pioneering graph prototype module to enhance its performance in small-sample, multi-label scenarios. The experimental results based on 10-fold cross-validation on benchmark datasets demonstrate that GP-HTNLoc achieves competitive predictive performance. The average results from 10 rounds of testing on an independent dataset show that GP-HTNLoc outperforms the best existing models on the human lncRNA, human snoRNA, and human miRNA subsets, with average precision improvements of 31.5%, 14.2%, and 5.6%, respectively, reaching 0.685, 0.632, and 0.704. A user-friendly online GP-HTNLoc server is accessible at https://56s8y85390.goho.co.

Domain Feature Decomposition for Efficient Object Detection in Aerial Images

Object detection in UAV aerial images faces domain-adaptive challenges, such as changes in shooting height, viewing angle, and weather. These changes constitute a large number of fine-grained domains that place greater demands on the network’s generalizability. To tackle these challenges, we initially decompose image features into domain-invariant and domain-specific features using practical imaging condition parameters. The composite feature can improve domain generalization and single-domain accuracy compared to the conventional fine-grained domain-detection method. Then, to solve the problem of the overfitting of high-frequency imaging condition parameters, we mixed images from different imaging conditions in a balanced sampling manner as input for the training of the detection network. The data-augmentation method improves the robustness of training and reduces the overfitting of high-frequency imaging parameters. The proposed algorithm is compared with state-of-the-art fine-grained domain detectors on the UAVDT and VisDrone datasets. The results show that it achieves an average detection precision improvement of 5.7 and 2.4, respectively. The airborne experiments validate that the algorithm achieves a 20 Hz processing performance for 720P images on an onboard computer with Nvidia Jetson Xavier NX.

TENET: Triple-enhancement based graph neural network for cell-cell interaction network reconstruction from spatial transcriptomics

Cellular communication relies on the intricate interplay of signaling molecules, forming the Cell–cell Interaction network (CCI) that coordinates tissue behavior. Researchers have shown the capability of shallow neural networks in reconstructing CCI, given molecules’ abundance in the Spatial Transcriptomics (ST) data. When encountering situations such as sparse connections in CCI and excessive noise, the susceptibility of shallow networks to these factors significantly impacts the accuracy of CCI reconstruction, resulting in subpar results. To reconstruct a more comprehensive and accurate CCI, we propose a novel method named Triple-Enhancement based Graph Neural Network (TENET). In TENET, three progressive enhancement mechanisms build upon each other, creating a cumulative effect. This approach can ensure the ability to capture valuable features in limited data and amplify the noise signal to facilitate the denoising effect. Additionally, the whole architecture guides the decoding reconstruction phase with integrated knowledge, which leverages the accumulated insights from each stage of enhancement to ensure a refined and comprehensive CCI reconstruction. The presented TENET has been implemented and tested on both real and synthetic ST datasets. Averagely, the CCI reconstruction using TENET achieves a 9.61% improvement in Average Precision (AP) and a 7.32% improvement in Area Under the Receiver Operating Characteristic (AUROC) compared to the existing state-of-the-art (SOTA) method. The source code and data are available at https://github.com/Yujian-Lee/TENET.

Defect Detection Algorithm for Battery Cell Casings Based on Dual-Coordinate Attention and Small Object Loss Feedback

To address the issue of low accuracy in detecting defects of battery cell casings with low space ratio and small object characteristics, the low space ratio feature and small object feature are studied, and an object detection algorithm based on dual-coordinate attention and small object loss feedback is proposed. Firstly, the EfficientNet-B1 backbone network is employed for feature extraction. Secondly, a dual-coordinate attention module is introduced to preserve more positional information through dual branches and embed the positional information into channel attention for precise localization of the low space ratio features. Finally, a small object loss feedback module is incorporated after the bidirectional feature pyramid network (BiFPN) for feature fusion, balancing the contribution of small object loss to the overall loss. Experimental comparisons on a battery cell casing dataset demonstrate that the proposed algorithm outperforms the EfficientDet-D1 object detection algorithm, with an average precision improvement of 4.23%. Specifically, for scratches with low space ratio features, the improvement is 13.21%; for wrinkles with low space ratio features, the improvement is 9.35%; and for holes with small object features, the improvement is 3.81%. Moreover, the detection time of 47.6 ms meets the requirements of practical production.

Road surface crack detection based on improved YOLOv5s.

In response to the issues of low efficiency and high cost in traditional manual methods for road surface crack detection, an improved YOLOv5s (you only look once version 5 small) algorithm was proposed. Based on this improvement, a road surface crack object recognition model was established using YOLOv5s. First, based on the Res2Net (a new multi-scale backbone architecture) network, an improved multi-scale Res2-C3 (a new multi-scale backbone architecture of C3) module was suggested to enhance feature extraction performance. Second, the feature fusion network and backbone of YOLOv5 were merged with the GAM (global attention mechanism) attention mechanism, reducing information dispersion and enhancing the interaction of global dimensions features. We incorporated dynamic snake convolution into the feature fusion network section to enhance the model's ability to handle irregular shapes and deformation problems. Experimental results showed that the final revision of the model dramatically increased both the detection speed and the accuracy of road surface identification. The mean average precision (mAP) reached 93.9%, with an average precision improvement of 12.6% compared to the YOLOv5s model. The frames per second (FPS) value was 49.97. The difficulties of low accuracy and slow speed in road surface fracture identification were effectively addressed by the modified model, demonstrating that the enhanced model achieved relatively high accuracy while maintaining inference speed.

Improved Deep Learning Model for Workpieces of Rectangular Pipeline Surface Defect Detection

This study introduces a novel approach to address challenges in workpiece surface defect identification. It presents an enhanced Single Shot MultiBox Detector model, incorporating attention mechanisms and multi-feature fusion. The research methodology involves carefully curating a dataset from authentic on-site factory production, enabling the training of a model with robust real-world generalization. Leveraging the Single Shot MultiBox Detector model lead to improvements integrating channel and spatial attention mechanisms in the feature extraction network. Diverse feature extraction methods enhance the network’s focus on crucial information, improving its defect detection efficacy. The proposed model achieves a significant Mean Average Precision (mAP) improvement, reaching 99.98% precision, a substantial 3% advancement over existing methodologies. Notably, the proposed model exhibits a tendency for the values of the P-R curves in object detection for each category to approach 1, which allows a better balance between the requirements of real-time detection and precision. Within the threshold range of 0.2 to 1, the model maintains a stable level of precision, consistently remaining between 0.99 and 1. In addition, the average running speed is 2 fps lower compared to other models, and the reduction in detection speed after the model improvement is kept within 1%. The experimental results indicate that the model excels in pixel-level defect identification, which is crucial for precise defect localization. Empirical experiments validate the algorithm’s superior performance. This research represents a pivotal advancement in workpiece surface defect identification, combining technological innovation with practical efficacy.

Multi-Label Action Anticipation for Real-World Videos With Scene Understanding.

With human action anticipation becoming an essential tool for many practical applications, there has been an increasing trend in developing more accurate anticipation models in recent years. Most of the existing methods target standard action anticipation datasets, in which they could produce promising results by learning action-level contextual patterns. However, the over-simplified scenarios of standard datasets often do not hold in reality, which hinders them from being applied to real-world applications. To address this, we propose a scene-graph-based novel model SEAD that learns the action anticipation at the high semantic level rather than focusing on the action level. The proposed model is composed of two main modules, 1) the scene prediction module, which predicts future scene graphs using a grammar dictionary, and 2) the action anticipation module, which is responsible for predicting future actions with an LSTM network by taking as input the observed and predicted scene graphs. We evaluate our model on two real-world video datasets (Charades and Home Action Genome) as well as a standard action anticipation dataset (CAD-120) to verify its efficacy. The experimental results show that SEAD is able to outperform existing methods by large margins on the two real-world datasets and can also yield stable predictions on the standard dataset at the same time. In particular, our proposed model surpasses the state-of-the-art methods with mean average precision improvements consistently higher than 65% on the Charades dataset and an average improvement of 40.6% on the Home Action Genome dataset.