Large-scale Benchmark Research Articles

CLUE: Contrastive language-guided learning for referring video object segmentation

Referring video object segmentation (R-VOS), the task of separating the object described by a natural language query from the video frames, has become increasingly critical with recent advances in multi-modal understanding. Existing approaches are mainly visual-dominant in both representation-learning and decision-making process, and are less sensitive to fine-grained clues in text description. To address this, we propose a language-guided contrastive learning and data augmentation framework to enhance the model sensitivity to the fine-grained textual clues (i.e., color, location, subject) in the text that relate heavily to the video information. By substituting key information of the original sentences and paraphrasing them with a text-based generation model, our approach conducts contrastive learning through automatically building diverse and fluent contrastive samples. We further enhance the multi-modal alignment with a sparse attention mechanism, which can find the most relevant video information by optimal transport. Experiments on a large-scale R-VOS benchmark show that our method significantly improves strong Transformer-based baselines, and further analysis demonstrates the better ability of our model in identifying textual semantics.

Put Your Voice on Stage: Personalized Headline Generation for News Articles

In this article, we study the problem of personalized news headline generation, which aims to produce not only concise and fact-consistent titles for news articles but also decorate these titles as personalized irresistible reading invitations by incorporating readers’ preferences. We propose an approach named PNG ( P ersonalized N ews headline G enerator) by utilizing distant supervision in readers’ past click behaviors to resolve. First, user preference representations are learned through a knowledge-aware user encoder that comprehensively captures the genuine, sequential, and flash interests of users reflected in their historical clicked news. Then, a user-perturbed pointer-generator network is devised to accomplish the headline generation in which the learned user representations implicitly affect the word prediction. The proposed model is optimized by reinforcement learning solvers where indicators on factual, personalized, and linguistic aspects of the generated headline are regarded as rewards. Extensive experiments are conducted on the real-world dataset PENS, 1 which is a large-scale benchmark collected from Microsoft News. Both the quantitative and qualitative results validate the effectiveness of our approach.

Multi-grained clip focus for skeleton-based action recognition

Joint-level and part-level information are crucial for modeling actions with different granularity. In addition, the relevant information on different joints between consecutive frames is very useful for skeleton-based action recognition. To effectively capture the action information, a new multi-grained clip focus network (MGCF-Net) is proposed. Firstly, the skeleton sequence is divided into multiple clips, each containing several consecutive frames. According to the structure of the human body, each clip is divided into several tuples. Then an intra-clip attention module is proposed to capture intra-clip action information. Specifically, multi-head self-attention is divided into two parts, obtaining relevant information at the joint and part levels, and integrating the information captured from these two parts to obtain multi-grained contextual features. In addition, an inter-clip focus module is used to capture the key information of several consecutive sub-actions, which will help to distinguish similar actions. On two large-scale benchmarks for skeleton-based action recognition, our method achieves the most advanced performance, and its effectiveness has been verified.

Learning Gait Representation from Massive Unlabelled Walking Videos: A Benchmark.

Gait depicts individuals' unique and distinguishing walking patterns and has become one of the most promising biometric features for human identification. As a fine-grained recognition task, gait recognition is easily affected by many factors and usually requires a large amount of completely annotated data that is costly and insatiable. This paper proposes a large-scale self-supervised benchmark for gait recognition with contrastive learning, aiming to learn the general gait representation from massive unlabelled walking videos for practical applications via offering informative walking priors and diverse real-world variations. Specifically, we collect a large-scale unlabelled gait dataset GaitLU-1M consisting of 1.02M walking sequences and propose a conceptually simple yet empirically powerful baseline model GaitSSB. Experimentally, we evaluate the pre-trained model on four widely-used gait benchmarks, CASIA-B, OU-MVLP, GREW and Gait3D with or without transfer learning. The unsupervised results are comparable to or even better than the early model-based and GEI-based methods. After transfer learning, GaitSSB outperforms existing methods by a large margin in most cases, and also showcases the superior generalization capacity. Further experiments indicate that the pre-training can save about 50% and 80% annotation costs of GREW and Gait3D. Theoretically, we discuss the critical issues for gait-specific contrastive framework and present some insights for further study. As far as we know, GaitLU-1M is the first large-scale unlabelled gait dataset, and GaitSSB is the first method that achieves remarkable unsupervised results on the aforementioned benchmarks. The source code of GaitSSB and anonymous data of GaitLU-1M is available at https://github.com/ShiqiYu/OpenGait.

QDTrack: Quasi-Dense Similarity Learning for Appearance-Only Multiple Object Tracking.

Similarity learning has been recognized as a crucial step for object tracking. However, existing multiple object tracking methods only use sparse ground truth matching as the training objective, while ignoring the majority of the informative regions in images. In this paper, we present Quasi-Dense Similarity Learning, which densely samples hundreds of object regions on a pair of images for contrastive learning. We combine this similarity learning with multiple existing object detectors to build Quasi-Dense Tracking (QDTrack), which does not require displacement regression or motion priors. We find that the resulting distinctive feature space admits a simple nearest neighbor search at inference time for object association. In addition, we show that our similarity learning scheme is not limited to video data, but can learn effective instance similarity even from static input, enabling a competitive tracking performance without training on videos or using tracking supervision. We conduct extensive experiments on a wide variety of popular MOT benchmarks. We find that, despite its simplicity, QDTrack rivals the performance of state-of-the-art tracking methods on all benchmarks and sets a new state-of-the-art on the large-scale BDD100K MOT benchmark, while introducing negligible computational overhead to the detector.

Learning from partially labeled data for multi-organ and tumor segmentation.

Medical image benchmarks for the segmentation of organs and tumors suffer from the partially labeling issue due to its intensive cost of labor and expertise. Current mainstream approaches follow the practice of one network solving one task. With this pipeline, not only the performance is limited by the typically small dataset of a single task, but also the computation cost linearly increases with the number of tasks. To address this, we propose a Transformer based dynamic on-demand network (TransDoDNet) that learns to segment organs and tumors on multiple partially labeled datasets. Specifically, TransDoDNet has a hybrid backbone that is composed of the convolutional neural network and Transformer. A dynamic head enables the network to accomplish multiple segmentation tasks flexibly. Unlike existing approaches that fix kernels after training, the kernels in the dynamic head are generated adaptively by the Transformer, which employs the self-attention mechanism to model long-range organ-wise dependencies and decodes the organ embedding that can represent each organ. We create a large-scale partially labeled Multi-Organ and Tumor Segmentation benchmark, termed MOTS, and demonstrate the superior performance of our TransDoDNet over other competitors on seven organ and tumor segmentation tasks. This study also provides a general 3D medical image segmentation model, which has been pre-trained on the large-scale MOTS benchmark and has demonstrated advanced performance over current predominant self-supervised learning methods. Code and data are available at https://github.com/jianpengz/DoDNet.

StudioGAN: A Taxonomy and Benchmark of GANs for Image Synthesis.

Generative Adversarial Network(GAN) is one of the state-of-the-art generative models for realistic image synthesis. While training and evaluating GAN becomes increasingly important, the current GAN research ecosystem does not provide reliable benchmarks for which the evaluation is conducted consistently and fairly. Furthermore, because there are few validated GAN implementations, researchers devote considerable time to reproducing baselines. We study the taxonomy of GAN approaches and present a new open-source library named StudioGAN. StudioGAN supports 7 GAN architectures, 9 conditioning methods, 4 adversarial losses, 12 regularization modules, 3 differentiable augmentations, 7 evaluation metrics, and 5 evaluation backbones. With our training and evaluation protocol, we present a large-scale benchmark using various datasets (CIFAR10, ImageNet, AFHQv2, FFHQ, and Baby/Papa/Granpa-ImageNet) and 3 different evaluation backbones (InceptionV3, SwAV, and Swin Transformer). Unlike other benchmarks used in the GAN community, we train representative GANs, including BigGAN and StyleGAN series in a unified training pipeline and quantify generation performance with 7 evaluation metrics. The benchmark evaluates other cutting-edge generative models(e.g., StyleGAN-XL, ADM, MaskGIT, and RQ-Transformer). StudioGAN provides GAN implementations, training, and evaluation scripts with the pre-trained weights. StudioGAN is available at https://github.com/POSTECH-CVLab/PyTorch-StudioGAN.

Anchor Ball Regression Model for large-scale 3D skull landmark detection

Recent deep learning models have exhibited impressive performance in the area of 3D skull landmark detection, but most of them aimed to detect a fixed number of landmarks. This paper focuses on automatically detecting an arbitrary number of landmarks on CT volumes, which meets the real clinical needs. To achieve robust performance for detecting arbitrary molar landmarks, we propose a novel 3D landmark detection model named Anchor Ball Regression Model (ABRM), which combines landmark detection and landmark classification losses for network training. For landmark detection, a novel landmark regression loss is proposed by predicting offsets to anchor balls instead of directly predicting landmarks. For landmark classification, an online hard negative mining loss is used to reduce absent landmarks’ learning errors, and a small regularization constraint loss is performed for voxels outside the anchor balls. The network backbone of ABRM is obtained by manually pruning popular 3D-CNNs. We also present an available large-scale benchmark dataset in this paper, which, to the best of our knowledge, is the largest dataset for 3D skull landmark detection. The dataset comprises of 658 CT volumes, with 14 landmarks labeled by two junior and one senior doctors. The ABRM presents a good robust performance and outperforms other models on this dataset. The codes and dataset are accessible at https://github.com/ithet1007/mmld_code.

Directed quick search guided evolutionary framework for large-scale multi-objective optimization problems

For large-scale multi-objective evolutionary algorithms (LSMOEAs), obtaining efficient evolutionary directions in an ultrahigh-dimensional decision space to produce high-quality offspring is a major challenge. This study proposes a novel framework called the directed quick search-guided large-scale evolutionary framework (QSLMOF) to address multi-objective optimization problems on a large scale. The framework contains two innovative strategies: the bidirectional vector-based sampling strategy (BDVS) and the quick search-guided directed reproduction strategy (QS-DRS). BDVS is introduced as an approach to swiftly discern promising solutions that can steer the exploration in the large-scale decision space. This is achieved by formulating two distinct types of sampling directions to rapidly reduce the search space and strike a delicate balance between convergence and diversity. In QS-DRS, we introduced the concept of the potential convergence gradient (PCG), incorporating directional information from historical searches and convergence directions indicated byelite solutions in the current population. With this property, inferior solutions can obtain excellent search directions to explore the decision space, which can improve the convergence rate and prevent the search from falling into a local optimum. The proposed large-scale evolutionary framework incorporates an existing environmental selection mechanism. Comprehensive experiments show that the two novel strategies improve the search efficiency and evolutionary quality of LSMOEAs in ultrahigh-dimensional decision spaces. Moreover, the proposed framework outperformed seven state-of-the-art LSMOEAs for nine large-scale multi-objective benchmark problems LSMOP1-LSMOP9 with up to three objectives and 10000 decision variables.

A Memorizing and Generalizing Framework for Lifelong Person Re-Identification.

In this paper, we introduce a challenging yet practical setting for person re-identification (ReID) task, named lifelong person re-identification (LReID), which aims to continuously train a ReID model across multiple domains and the trained model is required to generalize well on both seen and unseen domains. It is therefore critical to learn a ReID model that can learn a generalized representation without forgetting knowledge of seen domains. In this paper, we propose a new MEmorizing and GEneralizing framework (MEGE) for LReID, which can jointly prevent the model from forgetting and improve its generalization ability. Specifically, our MEGE is composed of two novel modules, i.e., Adaptive Knowledge Accumulation (AKA) and differentiable Ranking Consistency Distillation (RCD). Taking inspiration from the cognitive processes in the human brain, we endow AKA with two special capacities, knowledge representation and knowledge operation by graph convolution networks. AKA can effectively mitigate catastrophic forgetting on seen domains while improving the generalization ability to unseen domains. By considering the ranking factor that is specifically important in ReID, RCD is designed to distill the ranking knowledge in a differentiable manner, which can further prevent the catastrophic forgetting. To supporting the study of LReID, we build a new and large-scale benchmark with two practical evaluation protocols that consider the metrics of non-forgetting and generalization. Experiments demonstrate that 1) our MEGE framework can effectively improve the performance on seen and unseen domains under the domain-incremental learning constraint, and that 2) the proposed MEGE outperforms state-of-the-art competitors by large margins.

FakePoI: A Large-Scale Fake Person of Interest Video Detection Benchmark and a Strong Baseline

Deepfake technique can synthesize realistic images, audios, and videos, facilitating the thriving of entertainment, education, healthcare, and other industries. However, its abuse may pose potential threats to personal privacy, social stability, and even national security. Therefore, the development of deepfake detection methods is attracting more and more attention. Existing works mainly focus on the detection of common videos for entertainment purposes. In contrast, fake videos maliciously synthesized for Person of Interest (PoI, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e</i> ., who is in an authoritative position and has broadly public influences) are much more harmful to society because of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">celebrity endorsement</i> . However, there is no particular benchmark for driving related research in the community. Motivated by this observation, we present the first large-scale benchmark dataset, named <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">FakePoI</i> , to enable the research on fake PoI detection. It contains numerous fake videos of important people from all walks of life, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">e.g</i> ., police chiefs, city mayors, famous artists, and well-known Internet bloggers. In summary, our <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">FakePoI</i> includes 11,092 synthesized videos where only a few clips rather than the entire are fake. Previous fake detection algorithms deteriorate heavily or even fail on our <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">FakePoI</i> due to two main challenges. On the one hand, the rich diversity of our fake videos makes it pretty difficult to find universally applicable patterns for detection. On the other hand, the high credibility contributed by the presence of real frames easily confuses a common detector. To tackle these challenges, we present an amplifier framework, highlighting the feature gap between real and generated video frames. Specifically, we present a quadruplet loss to narrow the distance of all real PoIs and meanwhile push away each real and fake PoI in embedding space. We implement our framework and conduct extensive experiments on the proposed benchmark. The quantitative results demonstrate that our approach outperforms existing methods significantly, setting a strong baseline on <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">FakePoI</i> . The qualitative analysis also shows its superiority. We will release our dataset and code at https://github.com/cslltian/deepfake-detection to encourage future research on this valuable area.

Memory Network With Pixel-Level Spatio-Temporal Learning for Visual Object Tracking

Making full use of temporal and spatial information is critical to cope with the appearance changes of objects in visual object tracking. However, existing methods in the tracking field, which employ a memory network at frame level to learn this information, bring redundancy and cannot build long-term relationships among historical frames due to the limited memory size. In this paper, we propose a novel memory network, Pixel-level Spatio-Temporal Memory (PSTM), which organizes object features in an efficient way to leverage temporal and spatial context information. Specifically, PSTM is constructed and updated by a memory writer, which includes a pixel-level updating strategy to maintain the temporal consistency and dynamically memorize the noteworthy variations. Furthermore, in order to exploit relationships between the object and search region and precisely estimate the state of the object, we propose a memory reader, Pixel-wise Matching and Refinement module (PMR), and model spatial context without a complex manual-designed mechanism. Comprehensive experiments and comparisons on challenging large-scale benchmarks, including GOT-10k, TrackingNet, LaSOT, OTB2015, VOT2020, and NfS, have demonstrated the effectiveness of our proposed method, which performs favorably against state-of-the-art trackers.

A large scale benchmark for session-based recommendations on the legal domain

A large scale benchmark for session-based recommendations on the legal domain

ICG: A Machine Learning Benchmark Dataset and Baselines for Inline Code Comments Generation Task

As a fundamental component of software documentation, code comments could help developers comprehend and maintain programs. Several datasets of method header comments have been proposed in previous studies for machine learning-based code comment generation. As part of code comments, inline code comments are also crucial for code understanding activities. However, unlike method header comments written in a standard format and describing the whole method code, inline comments are often written in arbitrary formats by developers due to timelines pressures and different aspects of code snippets in the method are described. Currently, there is no large-scale dataset used for inline comments generation considering these. Hence, this naturally inspires us to explore whether we can construct a dataset to foster machine learning research that not only performs fine-grained noise-cleaning but conducts a taxonomy of inline comments. To this end, we first collect inline comments and code snippets from 8000 Java projects on GitHub. Then, we conduct a manual review to obtain heuristic rules, which could be used to clean the data noise in a fine-grained manner. As a result, we construct a large-scale benchmark dataset named ICG with 5,740,770 pairs of inline comments and code snippets. We then build a comprehensive taxonomy and conduct a statistical and manual analysis to explore the performances of different categories of inline comments, such as helpfulness in code understanding. After that, we provide and compare several baseline models to automatically generate inline comments, such as CodeBERT, to enhance the usability of the benchmark for researchers. The availability of our benchmark and baselines can help develop and validate new inline comment generation methods, which would also further facilitate code understanding activities.

IEFM and IDS: Enhancing 3D environment perception via information encoding in indoor point cloud semantic segmentation

The point cloud semantic segmentation plays a significant role in the understanding of 3D environment. However, current 3D point cloud segmentation methods pay little attention in the inevitable blurring and loss of point information within deep network, which considerably impairs the segmentation performance, particularly in the complex and colorful indoor scenes. To overcome the inevitable loss of point information (position, color and normal vector), in this paper, we propose a brand-new Information Encoding and Fusion Module (IEFM), comprising a point Information Encoding Method (IEM) and an optimized Multi-Encoding Fusion Method (EFM). In terms of constructing the innovative bias standardization of point information and effectively merging other information encodings into position encoding, IEFM adaptively complements the loss of the point information and thereby achieving enhanced environment perception ability. Additionally, although the proposed IEFM is capable of managing multi-class point information, the encoding interference resulting from the coexistence of multiple information is quite unavoidable, leading to detrimental consequences for the overall semantic segmentation performance. Therefore, to reduce the information encoding interference problem, we further innovatively propose the Information Distribution Strategy (IDS), in terms of hierarchically distribute kinds of point information, so that the interference of multiple information encodings will be extremely mitigated and achieving more accurate indoor point cloud semantic segmentation. Benefiting from the modular design, the proposed IEFM and IDS can be easily inserted in existing point-based point cloud segmentation models. The experimental results have shown the effectiveness of our proposed methods across multiple state-of-the-art models and benchmarks (S3DIS and ScanNet), achieving competitive performance of 77.4% mIoU on the large-scale S3DIS benchmark.

CMGNet: Collaborative multi-modal graph network for video captioning

In video captioning, it is very challenging to comprehensively describe multi-modal content information of a video, such as appearance, motion, and object. Prior arts often neglect interactions among multiple modalities and thus their video representations may not fully depict scene contents. In this paper, we propose a collaborative multi-modal graph network (CMGNet) to explore the interactions among multi-modal features in video captioning. Our CMGNet is composed of an encoder–decoder structure: a Compression-driven Intra-inter Attentive Graph (CIAG) encoder and an Adaptive Multi-modal Selection (AMS) decoder. Specifically, in our CIAG encoder, we first design a Basis Vector Compression (BVC) module to reduce the redundant nodes in graphs and thus improve the efficiency in coping with a large number of nodes. Then we propose an Intra-inter Attentive Graph (IAG) to improve the graph representation by sharing information across intra-and-inter nodes. Afterwards, we present an AMS decoder to generate video captions from the encoded video res presentations. In particular, we let the proposed AMS decoder learn to produce words by adaptively focusing on different modality information, thus leading to comprehensive and accurate captions. Extensive experiments on the large-scale benchmarks, i.e., MSR-VTT and TGIF, demonstrate that our proposed CMGNet achieves the state-of-the-art.

Sparse mixed attention aggregation network for multimodal images fusion tracking

Recent years have witnessed the exciting performance of trackers based on Transformer. However, they usually separate the process of information extraction and integration, weakening the information interaction between the target and search region. In addition, they depend on traditional Transformer to model the long range dependency, which leads to a lack of focus on the primary information needed by high-accuracy trackers. In this paper, a sparse mixed attention aggregation model is proposed for robust tracking based on visible and thermal infrared images. To be specific, a backbone network composed of sparse mixed attention is designed to achieve information extraction and integration. This is helpful to obtain specific discriminative feature information and enhance their communication. To give full play to the complementary visible and thermal information, a confidence aware aggregation network is designed, which can learn the reliable confidence of visible and thermal branches. Finally, a corner-based localization head is introduced to estimate the target state. Extensive experiments on three large-scale multimodal tracking benchmarks demonstrate the superior tracking ability of the proposed tracker over other advanced trackers.

Learning convolutional multi-level transformers for image-based person re-identification

As a vital vision task, person re-identification (Re-ID) aims to retrieve the same person under non-overlapping cameras. It is a very challenging task due to the presence of complex backgrounds, diverse illuminations and different perspectives. In this work, we integrate the advantages of convolutional neural networks (CNNs) and transformers, and propose a novel learning framework named convolutional multi-level transformer (CMT) for image-based person Re-ID. More specifically, we first propose a scale-aware feature enhancement (SFE) module to extract multi-scale local features from a pre-trained CNN backbone. Then, we introduce a part-aware transformer encoder (PTE) to further mine discriminative local information guided by global semantics. Finally, a deeply-supervised learning (DSL) technique is adopted to optimize the proposed CMT and improve its training efficiency. Extensive experiments on four large-scale Re-ID benchmarks demonstrate that our method performs favorably against several state-of-the-art methods.

Pol-InSAR-Island - A benchmark dataset for multi-frequency Pol-InSAR data land cover classification

This paper presents Pol-InSAR-Island, the first publicly available multi-frequency Polarimetric Interferometric Synthetic Aperture Radar (Pol-InSAR) dataset labeled with detailed land cover classes, which serves as a challenging benchmark dataset for land cover classification. In recent years, machine learning has become a powerful tool for remote sensing image analysis. While there are numerous large-scale benchmark datasets for training and evaluating machine learning models for the analysis of optical data, the availability of labeled SAR or, more specifically, Pol-InSAR data is very limited. The lack of labeled data for training, as well as for testing and comparing different approaches, hinders the rapid development of machine learning algorithms for Pol-InSAR image analysis. The Pol-InSAR-Island benchmark dataset presented in this paper aims to fill this gap. The dataset consists of Pol-InSAR data acquired in S- and L-band by DLR's airborne F-SAR system over the East Frisian island Baltrum. The interferometric image pairs are the result of a repeat-pass measurement with a time offset of several minutes. The image data are given as 6 × 6 coherency matrices in ground range on a 1 m × 1m grid. Pixel-accurate class labels, consisting of 12 different land cover classes, are generated in a semi-automatic process based on an existing biotope type map and visual interpretation of SAR and optical images. Fixed training and test subsets are defined to ensure the comparability of different approaches trained and tested prospectively on the Pol-InSAR-Island dataset. In addition to the dataset, results of supervised Wishart and Random Forest classifiers that achieve mean Intersection-over-Union scores between 24% and 67% are provided to serve as a baseline for future work. The dataset is provided via KITopenData: https://doi.org/10.35097/1700.

GATraj: A graph- and attention-based multi-agent trajectory prediction model

Trajectory prediction has been a long-standing problem in intelligent systems like autonomous driving and robot navigation. Models trained on large-scale benchmarks have made significant progress in improving prediction accuracy. However, the importance on efficiency for real-time applications has been less emphasized. This paper proposes an attention-based graph model, named GATraj, which achieves a good balance of prediction accuracy and inference speed. We use attention mechanisms to model the spatial–temporal dynamics of agents, such as pedestrians or vehicles, and a graph convolutional network to model their interactions. Additionally, a Laplacian mixture decoder is implemented to mitigate mode collapse and generate diverse multimodal predictions for each agent. GATraj achieves state-of-the-art prediction performance at a much higher speed when tested on the ETH/UCY datasets for pedestrian trajectories, and good performance at about 100Hz inference speed when tested on the nuScenes dataset for autonomous driving. We conduct extensive experiments to analyze the probability estimation of the Laplacian mixture decoder and compare it with a Gaussian mixture decoder for predicting different multimodalities. Furthermore, comprehensive ablation studies demonstrate the effectiveness of each proposed module in GATraj.