Weakly-supervised Method Research Articles

PPDistiller: Weakly-supervised 3D point cloud semantic segmentation framework via point-to-pixel distillation

Despite the significant growth in the availability of 3D light detection and ranging (LiDAR) point cloud data in recent years, annotation remains expensive and time-consuming. This has led to an increasing demand for weakly-supervised semantic segmentation (WSSS) methods in applications such as autonomous driving, mapping, and robotics. Existing approaches typically rely solely on LiDAR point cloud data for WSSS, which often results in lower segmentation accuracy due to the sparsity of point clouds. To address these challenges, we propose a novel architecture, PPDistiller, which employs multiple teacher networks from different modalities. Compared to other WSSS and multimodal approaches, PPDistiller achieves superior segmentation accuracy with fewer annotations. This is facilitated through the novel Mean Multi-Teacher Framework (MMT), which incorporates multiple modalities and teachers. To address the issue of lacking 2D labels, we propose the Distance-CAM Self-Training (DCAM-ST) module, which utilizes sparse 3D weak annotations to produce accurate 2D pixel-level annotations. To enable adaptive fusion of 2D and 3D data, we introduce the Attention Point to Pixel Fusion (APPF) module, facilitating bidirectional transfer of cross-modal knowledge. Additionally, to fully leverage the spatial semantic information in point cloud, we propose the Pyramid Semantic-context Neighbor Aggregation (PSNA) module, aiming to exploit spatial and semantic correlations to improve performance. Extensive experimentation on SemanticKITTI, ScribbleKITTI and nuScenes datasets demonstrates the superiority of our proposed method. Compared to state-of-the-art fusion and weakly-supervised methods, PPDistiller achieves the highest mean Intersection over Union (mIoU) scores under both fully-supervised and weakly-supervised settings.

Exploring better sparsely annotated shadow detection

Sparsely annotated image segmentation has attracted increasing attention due to its low labeling cost. However, existing weakly-supervised shadow detection methods require complex training procedures, and there is still a significant performance gap compared to fully-supervised methods. This paper summarizes two current challenges in sparsely annotated shadow detection, i.e., weak supervision diffusion and poor structure recovery, and attempts to alleviate them. To this end, we propose a one-stage weakly-supervised learning framework to facilitate sparsely annotated shadow detection. Specifically, we first design a simple yet effective semantic affinity module (SAM) that adaptively propagates scribble supervision to unlabeled regions using a gradient diffusion scheme. Then, to better recover shadow structures, we introduce a feature-guided edge-aware loss, which leverages higher-level semantic relations to perceive shadow boundaries, while avoiding interference from ambiguous regions. Finally, we present an intensity-guided structure consistency loss to ensure that the same images with different brightness are predicted to be consistent shadow masks, which can be regarded as a self-consistent mechanism to improve the model’s generalization ability. Experimental results on three benchmark datasets demonstrate that our approach significantly outperforms previous weakly-supervised methods and achieves competitive performance in comparison to recent state-of-the-art fully-supervised methods.

PA-Net: Trustworthy weakly supervised point cloud semantic segmentation with primary–auxiliary structure

Weakly supervised point cloud segmentation is a hot research topic with significant implications for practical applications such as autonomous driving. However, existing weakly supervised approaches primarily focus on improving prediction results while neglecting the reliability of the inference outcomes. To address this issue, this paper proposes a primary–auxiliary network PA-Net for point cloud segmentation and uncertainty estimation. PA-Net achieves excellent segmentation results and enables effective uncertainty assessment. Specifically, we first train PA-Net on sparsely annotated point cloud data, which allows our network to capture multi-scale information of labeled points and provide additional supervisory signals. Subsequently, we utilize the trained model for inference. We mainly conducted experiments on the indoor point cloud dataset S3DIS in a 1% labeled setting. PA-Net achieved a mean Intersection over Union (mIoU) of 64.5%, which surpasses most weakly-supervised methods. In particular, it exceeded PSD by 3.5% mIoU and SQN by 0.8% mIoU. Similarly, on the outdoor dataset Toronto3D, PA-Net also outperformed most weakly supervised methods. With only 0.01% labeling, it surpassed PSD by 11.99% mIoU and SQN by 20.34% mIoU. Lastly, to validate the effectiveness of uncertainty estimation, we visualized scene uncertainty, thereby demonstrating the practicality of our uncertainty estimation approach. Code will be made publicly available at https://github.com/NiuYingchun/PA-Net.

Source separation and classification using generative adversarial networks and weak class supervision

In this paper, we propose a decomposition-based weakly-supervised model that utilizes the class labels of the sources present in mixtures. We apply this weak class supervision approach to superimposed handwritten digit images using both non-negative matrix factorization (NMF) and generative adversarial networks (GANs). In this way, we can learn non-linear representations of the sources. The results of our experiments demonstrate that the proposed weakly-supervised methods are viable and mostly on par with the fully supervised baselines. The proposed joint classification and separation approach achieves a weakly-supervised source classification performance of 90.3 in terms of F1 score and outperforms the multi-label source classifier baseline of 68.2 when there are two sources. The separation performance of the proposed method is measured in terms of peak-signal-to-noise-ratio (PSNR) as 16 dB, outperforming the class-informed sparse NMF which achieves separation of two sources with a PSNR value of 13.9 dB. We show that it is possible to replace supervised training with weakly-supervised methods without performance penalty.

From image-level to pixel-level labeling: A weakly-supervised learning method for identifying aquaculture ponds using iterative anti-adversarial attacks guided by aquaculture features

Aquaculture mapping is essential for monitoring and managing aquaculture resources. However, accurately geotargeting individual aquaculture ponds from medium-resolution remote sensing imagery remains challenging, and convolutional deep learning methods for identifying aquaculture ponds require labor-intensive pixel-level annotations. This paper presents a novel weakly-supervised learning method to derive pixel-level labels from image-level annotations for aquaculture ponds. Our approach uses iterative anti-adversarial attacks to refine localization results from multi-scale class activation maps (CAMs). The improved method integrates two regularization methods guided by aquaculture features to form a joint loss function for adversarial samples: discriminative water region suppression and non-aquaculture water class suppression. We also propose an aquaculture feature termed CFNDWI to constrain the localization results and generate high-quality pseudo-labels. As a result, the pseudo-labels are used to train semantic segmentation networks for accurately identifying aquaculture ponds. We evaluated the performance of our method using commonly-used backbones on 10 m Sentinel-2 imagery. Our method achieves Intersection over Union (IoU) values of 0.618–0.655 for pseudo-label generation, and IoU values of 0.664–0.708 for semantic segmentation, outperforming state-of-the-art weakly-supervised methods and public datasets. The effectiveness of each module of our method was also testified through ablation experiments. Our method leverages knowledge-driven aquaculture features to guide the data-driven adversarial learning process, addressing the lack of high-quality aquaculture datasets for model training. The code for implementing our method will be accessible at https://github.com/designer1024/WSLM-AQ.

SHRINKAGE-BASED WEAKLY-SUPERVISED FEATURE LEARNING TO ENHANCE IoT ANOMALY DETECTION

IoT anomaly detection faces challenges due to the rarity of IoT anomalies and the limited availability of labels. Recent weakly-supervised approaches, like Feature Encoding with AutoEncoder and Weakly-supervised Anomaly Detection (FeaWAD) and an improvement on FeaWAD (iFWAD), address this scarcity by constructing detectors from a combination of unlabeled data and a small labeled anomalous set. While effective, these methods lack constraints during the feature learning stage to delineate normal regions from anomalies. Notably, the Shrink AutoEncoder promotes clustering of normal data around the origin while preserving space for anomalies. Drawing inspiration from the Shrink AutoEncoder, the study aims to introduce Shrink iFWAD (called sFWAD), embedding a shrink regularizer into iFWAD. This term compels the feature encoder of sFWAD to learn penalizing normal data that is close to zero, while simultaneously pushing IoT anomalies further away from zero. This process facilitates the anomalous score generator of sFWAD in efficiently identifying IoT anomalies. The proposed method is evaluated against state-of-the-art weakly-supervised techniques and other common anomaly detection methods using the N-BaIoT dataset. Experimental results indicate that sFWAD often surpasses recent weakly-supervised methods as well as the common techniques in IoT anomaly detection performance. For identifying unknown/new IoT anomalies, Missed Detection Rate from sFWAD (0.008) is much lower than those from iFWAD (0.026) and RoSAS (0.015).

DAAL-WS: A weakly-supervised method integrated with data augmentation and active learning strategies for MLS point cloud semantic segmentation

Mobile laser scanning (MLS) point clouds have increasingly been a significant data source for acquiring accurate three-dimensional (3D) semantic information from complex scenes. However, most current point cloud semantic segmentation methods heavily depend on a huge number of manually labeled training samples, which is labor-intensive and time-consuming. To address the above challenge, a weakly-supervised MLS point cloud semantic segmentation method integrated with data augmentation and active learning strategies is proposed (termed DAAL-WS). By taking advantage of our previously presented multi-branch weakly-supervised network (WSPointNet) in weakly supervisory signals and ensemble predictions, the DAAL-WS integrates WSPointNet with two essential components, i.e., an elevation-calibrated Mix3D (EC-Mix3D) data augmentation strategy and a point-level ensemble prediction-based (PLEP) active learning strategy. Specifically, the EC-Mix3D data augmentation strategy leverages elevation information to calibrate sub-point clouds and generates new contextual scenes by mixing the elevation-calibrated sub-point clouds, thereby augmenting the training point cloud distribution. Designed to reduce labeling redundancy, the PLEP active learning strategy selects the most important labeled points for model training. This strategy first measures the uncertainty for each unlabeled point by ensemble predictions and then employs a feature-distance suppression module to select the significant and discriminating unlabeled points for manual labeling. The proposed DAAL-WS method was evaluated on three public MLS datasets, including Toronto3D, Paris-Lille-3D, and WHU-MLS datasets, on which DAAL-WS obtained a competitive performance over fully-supervised baselines using only 0.015 %, 0.03 %, and 0.03 % labeled points, with mean Intersection over Union (mIoU) scores of 81.91 %, 82.59 %, and 60.36 %, respectively.

Wavelet Dynamic Selection Network for Inertial Sensor Signal Enhancement

As attitude and motion sensing components, inertial sensors are widely used in various portable devices, covering consumer electronics, sports health, aerospace, etc. But the severe intrinsic errors of inertial sensors heavily restrain their function implementation, especially the advanced functionality, including motion trajectory recovery and motion semantic recognition, which attracts considerable attention. As a mainstream signal processing method, wavelet is hailed as the mathematical microscope of signal due to the plentiful and diverse wavelet basis functions. However, complicated noise types and application scenarios of inertial sensors make selecting wavelet basis perplexing. To this end, we propose a wavelet dynamic selection network (WDSNet), which intelligently selects the appropriate wavelet basis for variable inertial signals. In addition, existing deep learning architectures excel at extracting features from input data but neglect to learn the characteristics of target categories, which is essential to enhance the category awareness capability, thereby improving the selection of wavelet basis. Therefore, we propose a category representation mechanism (CRM), which enables the network to extract and represent category features without increasing trainable parameters. Furthermore, CRM transforms the common fully connected network into category representations, which provide closer supervision to the feature extractor than the far and trivial one-hot classification labels. We call this process of imposing interpretability on a network and using it to supervise the feature extractor the feature supervision mechanism, and its effectiveness is demonstrated experimentally and theoretically in this paper. The enhanced inertial signal can perform impracticable tasks with regard to the original signal, such as trajectory reconstruction. Both quantitative and visual results show that WDSNet outperforms the existing methods. Remarkably, WDSNet, as a weakly-supervised method, achieves the state-of-the-art performance of all the compared fully-supervised methods.

A weakly-supervised deep learning model for end-to-end detection of airfield pavement distress

Accurate and timely surveying of airfield pavement distress is crucial for cost-effective airport maintenance management. Deep learning (DL) approaches, leveraging advancements in computer science and image acquisition techniques, have become the mainstream for automated airfield pavement distress detection. However, fully-supervised DL methods require a large number of manually annotated ground truth labels to achieve high accuracy. To address the challenge of limited high-quality manual annotations, we propose a novel end-to-end distress detection model called WSDD-CAM (Class Activation Map informed Weakly-Supervised Distress Detection). Based on YOLOv5, WSDD-CAM consists of an efficient backbone, a classification branch, and a localization network. By utilizing CAM (Class Activation Map) information, our model significantly reduces the need for manual annotations, automatically generating pseudo bounding boxes with a 71% overlap with the ground truth. To evaluate WSDD-CAM, we tested it on a self-made dataset and compared it with other weakly-supervised and fully-supervised models. The results show that our model achieves 49.2% mean average precision (mAP), outperforming other weakly-supervised methods and even approaching state-of-the-art fully-supervised methods. Additionally, ablation experiments confirm the effectiveness of our architecture design. In conclusion, our WSDD-CAM model offers a promising solution for airfield pavement distress detection, reducing manual annotation time while maintaining high accuracy. This efficient and effective approach can significantly contribute to cost-effective airport maintenance management.

Exploring weakly-supervised image manipulation localization with tampering Edge-based class activation map

In recent investigations concerning manipulated image localization, prevailing methodologies have relied on Convolutional Neural Networks (CNNs) to facilitate supervised learning predicated upon manipulated images and their corresponding ground-truth annotations. Nevertheless, these methods produce high false detection rates when detecting unaltered images due to the limited availability of authentic images. In this paper, we initiate our inquiry by scrutinizing the distribution characteristics of class activation maps within the context of image manipulation detection and the inherent patterns of alterations within this domain. We then introduce the novel concept of a tampering Edge-based Class Activation Map (EdgeCAM) tailored for detecting tampering within manipulated images. Leveraging EdgeCAM, we formulate a framework for localizing tampered regions in images through a weakly-supervised approach, relying solely on image-level annotations. Ultimately, we establish a fundamental benchmark for weakly-supervised image manipulation localization by utilizing genuine tampered images collected from real-world scenarios. The empirical results highlight the notable efficacy of our proposed method, outperforming current weakly-supervised methods for detecting image manipulation by a significant margin of 12.7% on the average combined F1 metrics across various generic datasets. This result also achieves performance comparable to supervised methods for image manipulation localization.

CCTwins: A Weakly Supervised Transformer-Based Crowd Counting Method With Adaptive Scene Consistency Attention

Recently, crowd counting has attracted significant attention, particularly in the context of the COVID-19 pandemic, due to its ability to automatically provide accurate crowd numbers in images. To address the challenges of location-level labeling, several transformer-based crowd counting methods have been proposed with only count-level supervision. However, these methods directly use the transformer as an encoder without considering the uneven crowd distribution. To address this issue, we propose CCTwins, a novel transformer-based crowd counting method with only count-level supervision. Specifically, we introduce an adaptive scene consistency attention mechanism to enhance the transformer-based model Twins-SVT-L for feature extraction in crowded scenes. Additionally, we design a multi-level weakly-supervised loss function that generates estimated crowd numbers in a coarse-to-fine manner, making it more appropriate for weakly-supervised settings. Moreover, intermediate features supervised by count-level labels are utilized to fuse multi-scale features. Experimental results on four public datasets demonstrate that our proposed method outperforms the state-of-the-art weakly-supervised methods, achieving up to a 16.6% improvement in MAE and up to a 13.8% improvement in RMSE across all evaluation settings. Moreover, the proposed CCTwins obtains competitive counting performance, even when compared to the state-of-the-art fully-supervised methods.

CrowdGraph: Weakly supervised Crowd Counting via Pure Graph Neural Network

Most existing weakly supervised crowd counting methods utilize Convolutional Neural Networks (CNN) or Transformer to estimate the total number of individuals in an image. However, both CNN-based (grid-to-count paradigm) and Transformer-based (sequence-to-count paradigm) methods take images as inputs in a regular form. This approach treats all pixels equally but cannot address the uneven distribution problem within human crowds. This challenge would lead to a decline in the counting performance of the model. Compared with grid and sequence, the graph structure could better explore the relationship among features. In this article, we propose a new graph-based crowd counting method named CrowdGraph, which reinterprets the weakly supervised crowd counting problem from a graph-to-count perspective. In the proposed CrowdGraph, each image is constructed as a graph, and a graph-based network is designed to extract features at the graph level. CrowdGraph comprises three main components: a dynamic graph convolutional backbone, a multi-scale dilated graph convolution module, and a regression head. To the best of our knowledge, CrowdGraph is the first method that is completely formulated based on the Graph Neural Network (GNN) for the crowd counting task. Extensive experiments demonstrate that the proposed CrowdGraph outperforms pure CNN-based and pure Transformer-based weakly supervised methods comprehensively and achieves highly competitive counting performance.

BSTS: A Weakly-Supervised Method for Semantic Learning of 3D Point Clouds

BSTS: A Weakly-Supervised Method for Semantic Learning of 3D Point Clouds

Single-Frame Supervision for Spatio-Temporal Video Grounding.

Spatio-Temporal Video Grounding (STVG) aims at localizing the spatio-temporal tube of a specific object in an untrimmed video given a free-form natural language query. As the annotation of tubes is labor intensive, researchers are motivated to explore weakly supervised approaches in recent works, which usually results in significant performance degradation. To achieve a less expensive STVG method with acceptable accuracy, this work investigates the "single-frame supervision" paradigm that requires a single frame labeled with a bounding box within the temporal boundary of the fully supervised counterpart as the supervisory signal. Based on the characteristics of the STVG problem, we propose a Two-Stage Multiple Instance Learning (T-SMILE) method, which creates pseudo labels by expanding the annotated frame to its contextual frames, thereby establishing a fully-supervised problem to facilitate further model training. The innovations of the proposed method are three-folded, including 1) utilizing multiple instance learning to dynamically select instances in positive bags for the recognition of starting and ending timestamps, 2) learning highly discriminative query features by incorporating spatial prior constraints in cross-attention, and 3) designing a curriculum learning-based strategy that iterative assigns dynamic weights to spatial and temporal branches, thereby gradually adapting to the learning branch with larger difficulty. To facilitate future research on this task, we also contribute a large-scale benchmark containing 12,469 videos on complex scenes with single-frame annotation. The extensive experiments on two benchmarks demonstrate that T-SMILE significantly outperforms all weakly-supervised methods. Remarkably, it also performs better than some fully-supervised methods associated with much more annotation labor costs. The dataset and codes are available at https://github.com/qumengxue/T-SMILE.

Evaluation of mediastinal lymph node segmentation of heterogeneous CT data with full and weak supervision

Accurate lymph node size estimation is critical for staging cancer patients, initial therapeutic management, and assessing response to therapy. Current standard practice for quantifying lymph node size is based on a variety of criteria that use uni-directional or bi-directional measurements. Segmentation in 3D can provide more accurate evaluations of the lymph node size. Fully convolutional neural networks (FCNs) have achieved state-of-the-art results in segmentation for numerous medical imaging applications, including lymph node segmentation. Adoption of deep learning segmentation models in clinical trials often faces numerous challenges. These include lack of pixel-level ground truth annotations for training, generalizability of the models on unseen test domains due to the heterogeneity of test cases and variation of imaging parameters. In this paper, we studied and evaluated the performance of lymph node segmentation models on a dataset that was completely independent of the one used to create the models. We analyzed the generalizability of the models in the face of a heterogeneous dataset and assessed the potential effects of different disease conditions and imaging parameters. Furthermore, we systematically compared fully-supervised and weakly-supervised methods in this context. We evaluated the proposed methods using an independent dataset comprising 806 mediastinal lymph nodes from 540 unique patients. The results show that performance achieved on the independent test set is comparable to that on the training set. Furthermore, neither the underlying disease nor the heterogeneous imaging parameters impacted the performance of the models. Finally, the results indicate that our weakly-supervised method attains 90%− 91% of the performance achieved by the fully supervised training.

Weakly-Supervised 3D Medical Image Segmentation using Geometric Prior and Contrastive Similarity.

Medical image segmentation is almost the most important pre-processing procedure in computer-aided diagnosis but is also a very challenging task due to the complex shapes of segments and various artifacts caused by medical imaging, (i.e., low-contrast tissues, and non-homogenous textures). In this paper, we propose a simple yet effective segmentation framework that incorporates the geometric prior and contrastive similarity into the weakly-supervised segmentation framework in a loss-based fashion. The proposed geometric prior built on point cloud provides meticulous geometry to the weakly-supervised segmentation proposal, which serves as better supervision than the inherent property of the bounding-box annotation (i.e., height and width). Furthermore, we propose the contrastive similarity to encourage organ pixels to gather around in the contrastive embedding space, which helps better distinguish low-contrast tissues. The proposed contrastive embedding space can make up for the poor representation of the conventionally-used gray space. Extensive experiments are conducted to verify the effectiveness and the robustness of the proposed weakly-supervised segmentation framework. The proposed framework are superior to state-of-the-art weakly-supervised methods on the following publicly accessible datasets: LiTS 2017 Challenge, KiTS 2021 Challenge and LPBA40. We also dissect our method and evaluate the performance of each component.

Weakly supervised joint whole-slide segmentation and classification in prostate cancer.

The identification and segmentation of histological regions of interest can provide significant support to pathologists in their diagnostic tasks. However, segmentation methods are constrained by the difficulty in obtaining pixel-level annotations, which are tedious and expensive to collect for whole-slide images (WSI). Though several methods have been developed to exploit image-level weak-supervision for WSI classification, the task of segmentation using WSI-level labels has received very little attention. The research in this direction typically require additional supervision beyond image labels, which are difficult to obtain in real-world practice. In this study, we propose WholeSIGHT, a weakly-supervised method that can simultaneously segment and classify WSIs of arbitrary shapes and sizes. Formally, WholeSIGHT first constructs a tissue-graph representation of WSI, where the nodes and edges depict tissue regions and their interactions, respectively. During training, a graph classification head classifies the WSI and produces node-level pseudo-labels via post-hoc feature attribution. These pseudo-labels are then used to train a node classification head for WSI segmentation. During testing, both heads simultaneously render segmentation and class prediction for an input WSI. We evaluate the performance of WholeSIGHT on three public prostate cancer WSI datasets. Our method achieves state-of-the-art weakly-supervised segmentation performance on all datasets while resulting in better or comparable classification with respect to state-of-the-art weakly-supervised WSI classification methods. Additionally, we assess the generalization capability of our method in terms of segmentation and classification performance, uncertainty estimation, and model calibration. Our code is available at: https://github.com/histocartography/wholesight.

Automatic labeling of Parkinson's Disease gait videos with weak supervision.

Motor dysfunction in Parkinson's Disease (PD) patients is typically assessed by clinicians employing the Movement Disorder Society's Unified Parkinson's Disease Rating Scale (MDS-UPDRS). Such comprehensive clinical assessments are time-consuming, expensive, semi-subjective, and may potentially result in conflicting labels across different raters. To address this problem, we propose an automatic, objective, and weakly-supervised method for labeling PD patients' gait videos. The proposed method accepts videos of patients and classifies their gait scores as normal (Gait score in MDS-UPDRS = 0) or PD (MDS-UPDRS ≥ 1). Unlike previous work, the proposed method does not require a priori MDS-UPDRS ratings for training, utilizing only domain-specific knowledge obtained from neurologists. We propose several labeling functions that classify patients' gait and use a generative model to learn the accuracy of each labeling function in a self-supervised manner. Since results depended upon the estimated values of the patients' 3D poses, and existing pre-trained 3D pose estimators did not yield accurate results, we propose a weakly-supervised 3D human pose estimation method for fine-tuning pre-trained models in a clinical setting. Using leave-one-out evaluations, the proposed method obtains an accuracy of 89% on a dataset of 29 PD subjects - a significant improvement compared to previous work by 7%-10% depending upon the dataset. The method obtained state-of-the-art results on the Human3.6M dataset. Our results suggest that the use of labeling functions may provide a robust means to interpret and classify patient-oriented videos involving motor tasks.

Self-Supervised Adversarial Video Summarizer With Context Latent Sequence Learning

Video summarization attempts to create concise and complete synopsis of a video through identifying the most informative and explanatory parts while removing redundant video frames, which facilitates retrieving, managing and browsing video efficiently. Most existing video summarization approaches heavily rely on enormous high-quality human-annotated labels or fail to produce semantically meaningful video summaries with the guidance of prior information. Without any supervised labels, we propose Self-supervised Adversarial Video Summarizer (2SAVS) that exploits context latent sequence learning to generate satisfying video summary. To implement it, our model elaborates a novel pretext task of identifying latent sequences and normal frames by training self-supervised generative adversarial network (GAN) with several well-designed losses. As the core components of 2SAVS, Clip Consistency Representation (CCR) and Hybrid Feature Refinement (HFR) are developed to ensure semantic consistency and continuity of clips. Furthermore, a novel separation loss is designed to explicitly enlarge the distance between prediction frame scores to effectively enhance the model’s discriminative ability. Differently, latent sequences, additional finetune operations and generators are not required when inferring video summary. Experiments on two challenging and diverse datasets demonstrate that our approach outperforms other state-of-the-art unsupervised and weakly-supervised methods, and even produces comparable results with several excellent supervised methods.

Weakly-supervised content-based video moment retrieval using low-rank video representation

Content-based video moment retrieval (CVMR) aims to localize a successive sequence of frames in an untrimmed reference video, called target moment, that is semantically corresponding to a given query video. Current state-of-the-art CVMR methods are mainly developed using frame-level annotation, which is often quite expensive to collect. In this paper, we aim to develop a weakly-supervised CVMR method, which uses coarse-grained video-level annotations during training. Under weak supervision, video localizers require more discriminative frame-level video features. To achieve this goal, we proposed a novel prior, termed low-rank prior, based on an observation that the frame-level feature of a video should have low-rank properties. We demonstrated that the low-rank features are more discriminative and are beneficial to accurately localize the action boundaries. To produce a low-rank feature, we designed a low-rank feature reconstruction (LFR) operator. A new differentiable matrix decomposition approach is proposed to generate the low-rank reconstruction of the input matrix, meanwhile ensuring that the matrix decomposition process is differentiable. Based on the LFR, we developed a new weakly-supervised CVMR model which produces low-rank video representation and performs semantic consistency measures to discover the semantically matched segment in the reference video to the query video. Extensive experiments demonstrate that our method outperforms state-of-the-art weakly-supervised methods consistently and even achieves competing performance to fully-supervised baselines.