MARS Datasets Research Articles

In-Depth Analysis of GAF-Net: Comparative Fusion Approaches in Video-Based Person Re-Identification

This study provides an in-depth analysis of GAF-Net, a novel model for video-based person re-identification (Re-ID) that matches individuals across different video sequences. GAF-Net combines appearance-based features with gait-based features derived from skeletal data, offering a new approach that diverges from traditional silhouette-based methods. We thoroughly examine each module of GAF-Net and explore various fusion methods at the both score and feature levels, extending beyond initial simple concatenation. Comprehensive evaluations on the iLIDS-VID and MARS datasets demonstrate GAF-Net’s effectiveness across scenarios. GAF-Net achieves state-of-the-art 93.2% rank-1 accuracy on iLIDS-VID’s long sequences, while MARS results (86.09% mAP, 89.78% rank-1) reveal challenges with shorter, variable sequences in complex real-world settings. We demonstrate that integrating skeleton-based gait features consistently improves Re-ID performance, particularly with long, more informative sequences. This research provides crucial insights into multi-modal feature integration in Re-ID tasks, laying a foundation for the advancement of multi-modal biometric systems for diverse computer vision applications.

Light4Mars: A lightweight transformer model for semantic segmentation on unstructured environment like Mars

Auto-semantic segmentation is important for robots on unstructured and dynamic environments like planets where ambient conditions cannot be controlled and the scale is larger than that found indoors. Current learning-based methods have achieved breathtaking improvements on this topic. For onboard applications, however, all those methods still suffer from huge computational costs and are difficult to deploy on edge devices. In this paper, unlike previous transformer-based SOTA approaches that heavily relied on complex design, we proposed Light4Mars, a lightweight model with minimal computational complexity while maintaining high segmenting accuracy. We designed a lightweight squeeze window transformer module that focuses on window-scale feature extraction and is more effective in learning global and local contextual information. The aggregated local attention decoder is utilized to fuse semantic information at different scales, especially for unstructured scenes. Since there are few all-terrain datasets for semantic segmentation of unstructured scenes like Mars, we built a synthetic dataset SynMars-TW, referencing images collected by the ZhuRong rover on the Tianwen-1 mission and the Curiosity rover. Extensive experiments on SynMars-TW and the real Mars dataset, MarsScapes show that our approach achieves state-of-the-art performance with favorable computational simplicity. To the best of our knowledge, the proposed Light4Mars-T network is the first segmentation model for Mars image segmentation with parameters lower than 0.1M. Code and datasets are available at https://github.com/CVIR-Lab/Light4Mars.

Situation Awareness of Mars helicopter based on YOLOv7

Situation awareness on Mars is indispensable for various downstream applications such as navigation and path planning, mapping and scientific discover. However, current Mars rover platform suffered from the absence of global information, leading to mission reduction and the failure of global planning. Compared to the rover, Mars helicopter is a new type of attempt for Mars exploration, which can provide a wide range view of scenes to well support the above missions. To this end, in this paper we proposed a novel learning-based situation awareness model for Mars helicopter. We modified the Yolov7 model by embedding a new channel attention on its head to improve the accuracy of detection. Besides, we designed a Tiny-Scale Detection Module (TSDM) to capture small or hidden rocks in complex Martian surface. To achieve further refinement and validate our proposal, we built an open-source synthetic dataset in air view. After extensive experiments, our model achieved excellent outperforms than the baseline model in our Mars dataset.

Weakly Supervised Tracklet Association Learning With Video Labels for Person Re-Identification.

Supervised person re-identification (re-id) methods require expensive manual labeling costs. Although unsupervised re-id methods can reduce the requirement of the labeled datasets, the performance of these methods is lower than the supervised alternatives. Recently, some weakly supervised learning-based person re-id methods have been proposed, which is a balance between supervised and unsupervised learning. Nevertheless, most of these models require another auxiliary fully supervised datasets or ignore the interference of noisy tracklets. To address this problem, in this work, we formulate a weakly supervised tracklet association learning (WS-TAL) model only leveraging the video labels. Specifically, we first propose an intra-bag tracklet discrimination learning (ITDL) term. It can capture the associations between person identities and images by assigning pseudo labels to each person image in a bag. And then, the discriminative feature for each person is learned by utilizing the obtained associations after filtering the noisy tracklets. Based on that, a cross-bag tracklet association learning (CTAL) term is presented to explore the potential tracklet associations between bags by mining reliable positive tracklet pairs and hard negative pairs. Finally, these two complementary terms are jointly optimized to train our re-id model. Extensive experiments on the weakly labeled datasets demonstrate that WS-TAL achieves 88.1% and 90.3% rank-1 accuracy on the MARS and DukeMTMC-VideoReID datasets respectively. The performance of our model surpasses the state-of-the-art weakly supervised models by a large margin, even outperforms some fully supervised re-id models.

QI-IRA: Quantum-Inspired Interactive Ranking Aggregation for Person Re-identification

Ranking aggregation (RA), the process of aggregating multiple rankings derived from multiple search strategies, has been proved effective in person re-identification (re-ID) because of a single re-ID method can not always achieve consistent superiority for different scenarios. Existing RA research mainly focus on unsupervised and fully-supervised methods. The former lack external supervision to optimize performance, while the latter are costly because of expensive labeling effort required for training. To address the above challenges, this paper proposes a quantum-inspired interactive ranking aggregation (QI-IRA) method, which (1) utilizes quantum theory to interpret and model the generation and aggregation of multiple basic rankings, (2) approximates or even exceeds the performance of fully-supervised RA methods with much less labeling cost, even as low as only two feedbacks per query on Market1501, MARS and DukeMTMC-VideoReID datasets. Comparative experiments conducted on six public re-ID datasets validate the superiority of the proposed QI-IRA method over existing unsupervised, interactive, and fully-supervised RA approaches.

Rethink Motion Information for Occluded Person Re-Identification

Person re-identification aims to identify the same pedestrians captured by various cameras from different viewpoints in multiple scenarios. Occlusion is the toughest problem for practical applications. In video-based ReID tasks, motion information can be easily obtained from sampled frames, and provide discriminative human part representations. However, most motion-based methodologies are designed for video frames which are not suitable for processing single static image input. In this paper, we propose a Motion-Aware Fusion (MAF) network, aiming to acquire motion information from static images in order to improve the performance of ReID tasks. Specifically, a visual adapter is introduced to enable visual feature extraction, either from image or video data. We design a motion consistency task to guide the motion-aware transformer to learn representative human-part motion information and greatly improve the learning quality of features of occluded pedestrians. Extensive experiments on popular holistic, occluded, and video datasets demonstrate the effectiveness of our proposed method. This method outperforms state-of-the-art approaches by improving the mean average precision (mAP) by 1.5% and rank-1 accuracy by 1.2% on the challenging Occluded-REID dataset. At the same time, it surpasses other methods on the MARS dataset with an improvement of 0.2% in mAP and 0.1% in rank-1 accuracy.

Multi-Context Grouped Attention for Unsupervised Person Re-Identification

Recent advancements like multiple contextual analysis, attention mechanisms, distance-aware optimization, and multi-task guidance have been widely used for supervised person re-identification (ReID), but the implementation and effects of such methods in unsupervised ReID frameworks are non-trivial and unclear, respectively. Moreover, with increasing size and complexity of image-and video-based ReID datasets, manual or semi-automated annotation procedures for supervised ReID are becoming labor intensive and cost prohibitive, which is undesirable especially considering the likelihood of annotation errors increase with scale/complexity of data collections. Therefore, we propose a new iterative clustering framework that is insensitive to annotation errors and over-fitting ReID annotations (i.e., labels). Our proposed unsupervised framework incorporates (a) a novel multi-context group attention architecture that learns a holistic attention map from multiple local and global contexts, (b) an unsupervised clustering loss function that down-weights easily discriminative identities, and (c) a background diversity term that helps cluster persons across different cross-camera views without leveraging any identification or camera labels. We perform extensive analysis using the DukeMTMC-VideoReID and MARS video-based ReID datasets and the MSMT17 image-based ReID dataset. Our approach is shown to provide a new state-of-the-art performance for unsupervised ReID, reducing the rank-1 performance gap between supervised and unsupervised ReID to 1.1%, 12.1%, and 21.9% from 6.1%, 17.9%, and 22.6% for DukeMTMC, MARS, and MSMT17 datasets, respectively.

Multitask Multigranularity Aggregation With Global-Guided Attention for Video Person Re-Identification

The goal of video-based person re-identification (Re-ID) is to identify the same person across multiple non-overlapping cameras. The key to accomplishing this challenging task is to sufficiently exploit both spatial and temporal cues in video sequences. However, most current methods are incapable of accurately locating semantic regions or efficiently filtering discriminative spatio-temporal features; so it is difficult to handle issues such as spatial misalignment and occlusion. Thus, we propose a novel feature aggregation framework, multi-task and multi-granularity aggregation with global-guided attention (MMA-GGA), which aims to adaptively generate more representative spatio-temporal aggregation features. Specifically, we develop a multi-task multi-granularity aggregation (MMA) module to extract features at different locations and scales to identify key semantic-aware regions that are robust to spatial misalignment. Then, to determine the importance of the multi-granular semantic information, we propose a global-guided attention (GGA) mechanism to learn weights based on the global features of the video sequence, allowing our framework to identify stable local features while ignoring occlusions. Therefore, the MMA-GGA framework can efficiently and effectively capture more robust and representative features. Extensive experiments on four benchmark datasets demonstrate that our MMA-GGA framework outperforms current state-of-the-art methods. In particular, our method achieves a rank-1 accuracy of 91.0% on the MARS dataset, the most widely used database, significantly outperforming existing methods.

Video Person Re-Identification Using Attribute-Enhanced Features

In this work we propose to boost video-based person re-identification (Re-ID) by using attribute-enhanced feature presentation. To this end, we not only try to use the ID-relevant attributes more effectively, but also for the first time in literature harness the ID-irrelevant attributes to help model training. The former mainly include gender, age, clothing characteristics, etc., which contain rich and supplementary information about the pedestrian; the latter include viewpoint, action, etc., which are seldom used for identification previously. In particular, we use the attributes to enhance the significant areas of the image with a novel Attribute Salient Region Enhance (ASRE) module that can attend more accurately to the body of the pedestrian, so as to better separate the target from the background. Furthermore, we find that many ID-irrelevant but subject-relevant factors, like the view angle and movement of the target pedestrian, have great impact on the two-dimensional appearance of a pedestrian. We then propose to exploit both the ID-relevant and the ID-irrelevant attributes via a novel triplet loss called the Viewpoint and Action-Invariant (VAI) triplet loss. Based on the above, we design an Attribute Salience Assisted Network (ASA-Net) to perform attribute recognition along with identity recognition, and use the attributes for feature enhancement and hard sample mining. Extensive experiments on MARS and DukeMTMC-VideoReID datasets show that our method outperforms the state-of-the-arts. Also, the visualizations of learning results further prove the effectiveness of the proposed method.

MSTN: A Multi-granular Spatial–Temporal Network for video-based person re-identification

With the popularization of surveillance cameras, public-safety related applications requiring the functionality of video-based person re-identification (Re-ID) thrive. Re-ID aims at accurately identifying a person-of-interest across video sequences from multiple cameras. Existing methods usually focus on either spatially salient regions, or temporal features among frames of fixed intervals (i.e., either short- or long-term temporal features), resulting in the under-utilization of neglected features and hence moderate identification accuracy. To achieve high Re-ID accuracy, we propose a novel framework termed Multi-granular Spatial–Temporal Network (MSTN), that facilitates full utilization of spatial–temporal features for video-based person Re-ID. Within MSTN, a Temporal Kernel Attention (TKA) module is proposed to adaptively capture both short- and long-term temporal relationships; a Feature Disentanglement Spatial Attention (FDSA) module is further proposed to mine spatially salient and subtle features. Extensive experiments on the MARS dataset demonstrate that MSTN can achieve high identification accuracy, exhibiting 86.1% in terms of mAP and 91.0% in terms of Rank-1, notably higher than state-of-the-art comparison schemes.

MRRNet: Learning multiple region representation for video person re-identification

Video person re-identification is a crucial component of a robust surveillance system. Within a video clip, different human regions exhibit unique stability characteristics, which would be harmful to generating a discriminative representation. Unfortunately, prior works cannot effectively deal with the stability characteristics of different regions. To tackle this problem, we propose a Multiple Region Representation Network (MRRNet) that aims to discover the discriminative information from different human regions. Firstly, a Stable Region Representation (SRR) layer is proposed to capture important clues from the stable regions and exchange temporal information by cross-relation aware operation. Secondly, a Multiple Region Representation (MRR) layer is designed to address the unstable regions and preserve the attention on stable regions. Thirdly, SRR and MRR can be conveniently inserted into multiple stages of the deep residual networks and significantly improve the performance of the network. Comprehensive experiments validate the effectiveness of our network. Particularly, MRRNet achieves 86.7% mAP and 91.1% Rank-1 accuracy on the MARS dataset, which outperforms state-of-the-arts.

Keypoint Message Passing for Video-Based Person Re-identification

Video-based person re-identification~(re-ID) is an important technique in visual surveillance systems which aims to match video snippets of people captured by different cameras. Existing methods are mostly based on convolutional neural networks~(CNNs), whose building blocks either process local neighbor pixels at a time, or, when 3D convolutions are used to model temporal information, suffer from the misalignment problem caused by person movement. In this paper, we propose to overcome the limitations of normal convolutions with a human-oriented graph method. Specifically, features located at person joint keypoints are extracted and connected as a spatial-temporal graph. These keypoint features are then updated by message passing from their connected nodes with a graph convolutional network~(GCN). During training, the GCN can be attached to any CNN-based person re-ID model to assist representation learning on feature maps, whilst it can be dropped after training for better inference speed. Our method brings significant improvements over the CNN-based baseline model on the MARS dataset with generated person keypoints and a newly annotated dataset: PoseTrackReID. It also defines a new state-of-the-art method in terms of top-1 accuracy and mean average precision in comparison to prior works.

An Attention-Guided Framework for Explainable Biometric Presentation Attack Detection.

Despite the high performances achieved using deep learning techniques in biometric systems, the inability to rationalise the decisions reached by such approaches is a significant drawback for the usability and security requirements of many applications. For Facial Biometric Presentation Attack Detection (PAD), deep learning approaches can provide good classification results but cannot answer the questions such as “Why did the system make this decision”? To overcome this limitation, an explainable deep neural architecture for Facial Biometric Presentation Attack Detection is introduced in this paper. Both visual and verbal explanations are produced using the saliency maps from a Grad-CAM approach and the gradient from a Long-Short-Term-Memory (LSTM) network with a modified gate function. These explanations have also been used in the proposed framework as additional information to further improve the classification performance. The proposed framework utilises both spatial and temporal information to help the model focus on anomalous visual characteristics that indicate spoofing attacks. The performance of the proposed approach is evaluated using the CASIA-FA, Replay Attack, MSU-MFSD, and HKBU MARs datasets and indicates the effectiveness of the proposed method for improving performance and producing usable explanations.

Person Re-Identification Using LBPH and K-Reciprocal Encoding

Individual re-identification proof (Re-ID) targets recovering an individual of interest across different non-covering cameras. With the recent development of technological algorithm and expanding request of intelligence video observation, it has acquired fundamentally expanded interest in the computer vision. Person re-identification is characterized as the issue of perceiving an individual caught in different occasions and additionally areas more than a few nonoverlapping camera sees, thinking about a huge arrangement of up-and-comers. This issue influences essentially the administration of disseminated, multiview observation frameworks, in which subjects should be followed across better places, either deduced or on-the-fly when they travel through various areas. Re-identification proof is a truly challenging issue, as more often than not individuals can be caught by a few low goal cameras, under impediment conditions, severely (and not quite the same as view to see) enlightened, and in differing presents. In this context an encoding technique K-reciprocal results using the LBPH (Local Binary Patterns Histogram) Algorithm has been proposed. This work aims to obtain a genuine image match more prone to the probe in the K-corresponding closest neighbour. When probe image is given, complementary is encoded with the k-equal nearest neighbours into a vector to rerank using the Jaccard matrix. The obtained result is a combination of a Mahalanobis metric, the Jaccard metric, and the LBPH algorithm. The reranking activity needs no Human interference in producing an appropriate enormous scale dataset. The performance of rank-1 metrics 77.27, 61.90, 76.34 &.55.11 percentage is achieved for large-scale Market-1501, CUHK03, MARS, and PRW datasets. The other metrics used for person re-id named mAP recorded 65.01%, 61.21%, 68.21% and 38.13% for the same dataset in that order.

Comparative Study of Markerless Vision-Based Gait Analyses for Person Re-Identification

The model-based gait analysis of kinematic characteristics of the human body has been used to identify individuals. To extract gait features, spatiotemporal changes of anatomical landmarks of the human body in 3D were preferable. Without special lab settings, 2D images were easily acquired by monocular video cameras in real-world settings. The 2D and 3D locations of key joint positions were estimated by the 2D and 3D pose estimators. Then, the 3D joint positions can be estimated from the 2D image sequences in human gait. Yet, it has been challenging to have the exact gait features of a person due to viewpoint variance and occlusion of body parts in the 2D images. In the study, we conducted a comparative study of two different approaches: feature-based and spatiotemporal-based viewpoint invariant person re-identification using gait patterns. The first method is to use gait features extracted from time-series 3D joint positions to identify an individual. The second method uses a neural network, a Siamese Long Short Term Memory (LSTM) network with the 3D spatiotemporal changes of key joint positions in a gait cycle to classify an individual without extracting gait features. To validate and compare these two methods, we conducted experiments with two open datasets of the MARS and CASIA-A datasets. The results show that the Siamese LSTM outperforms the gait feature-based approaches on the MARS dataset by 20% and 55% on the CASIA-A dataset. The results show that feature-based gait analysis using 2D and 3D pose estimators is premature. As a future study, we suggest developing large-scale human gait datasets and designing accurate 2D and 3D joint position estimators specifically for gait patterns. We expect that the current comparative study and the future work could contribute to rehabilitation study, forensic gait analysis and early detection of neurological disorders.

Image-to-video person re-identification using three-dimensional semantic appearance alignment and cross-modal interactive learning

Image-to-video person re-identification (I2V ReID), which aims to retrieve human targets between image-based queries and video-based galleries, has recently become a new research focus. However, the appearance misalignment and modality misalignment in both images and videos caused by pose variations, camera views, misdetections, and different data types, make I2V ReID still challenging. To this end, we propose a deep I2V ReID pipeline based on three-dimensional semantic appearance alignment (3D-SAA) and cross-modal interactive learning (CMIL) to address the aforementioned two challenges. Specifically, in the 3D-SAA module, the aligned local appearance images extracted by dense 3D human appearance estimation are in conjunction with global image and video embedding streams to learn more fine-grained identity features. The aligned local appearance images are further semantically aggregated by the proposed multi-branch aggregation network to weaken the negligible body parts. Moreover, to overcome the influence of modality misalignment, a CMIL module enables the communication between global image and video streams by interactively propagating the temporal information in videos to the channels of image feature maps. Extensive experiments on challenging MARS, DukeMTMC-VideoReID and iLIDS-VID datasets, show the superiority of our approach.

Automatic Rocks Segmentation Based on Deep Learning for Planetary Rover Images

Accurate detection and segmentation of obstacles is the key to the smooth operation of the planetary rovers and the basic guarantee of scientific exploration mission. The traditional method of rock segmentation based on boundary detector is affected by the change of illumination and dust storms. To address this problem, this paper proposes an improved U-net-based architecture combined with Visual Geometry Group (VGG) and dilated convolutional neural network for the segmentation of rocks from images of planetary exploration rovers. The proposed method also has a contracting path and an expansive path to get high-resolution output similar with U-Net. In the contracting path, the convolution layers in U-Net are replaced by the convolutional layers of VGG16. Inspired by the dilated convolution, the multiscale dilated convolution in the expansive path is proposed. Furthermore, our method is further optimized in the expansive path. To evaluate the proposed method, extensive experiments on segmentation with the Mars dataset have been conducted. The experimental results demonstrate that the proposed method produces accurate semantic segmentation and identification results automatically and outperforms state-of-the-art methods.

Interactive Image Segmentation of MARS Datasets Using Bag of Features

In this article, we propose a slice-based interactive segmentation of spectral CT datasets using a bag of features method. The data are acquired from a MARS scanner that divides up the X-ray spectrum into multiple energy bins for imaging. In literature, most existing segmentation methods are limited to performing a specific task or tied to a particular imaging modality. Therefore, when applying generalized methods to MARS datasets, the additional energy information acquired from the scanner cannot be sufficiently utilized. We describe a new approach that circumvents this problem by effectively aggregating the data from multiple channels. Our method solves a classification problem to get the solution for segmentation. Starting with a set of labeled pixels, we partition the data using superpixels. Then, a set of local descriptors, extracted from each superpixel, are encoded into a codebook and pooled together to create a global superpixel-level descriptor (bag of features representation). We propose to use the vector of locally aggregated descriptors as our encoding/pooling strategy, as it is efficient to compute and leads to good results with simple linear classifiers. A linear support vector machine is then used to classify the superpixels into different labels. The proposed method was evaluated on multiple MARS datasets. Experimental results show that our method achieved an average of more than 10% increase in the accuracy over other state-of-the-art methods.

Self‐training with one‐shot stepwise learning method for person re‐identification

SummaryPerson re‐identification (Re‐ID) aims at identifying the same person across multiple non‐overlapping camera views. A number of existing methods have been presented for this task in a fully‐supervised manner that requires a large amount of training annotations. However, obtaining high quality labels is extremely time consuming and expensive. In this article, we focus on the semi‐supervised person Re‐ID and propose a one‐shot stepwise learning method to address the above issue. It exploits only one labeled data along with additional unlabeled samples to gradually but steadily improving the discriminative capability of the feature representation. Specifically, we first construct labeled data portion to train Re‐ID model. Then we fine‐tune the overall system by the following two steps iteratively: (1) assigning the estimated labels to the unlabeled portion; (2) updating the network parameters according to the selected data. During the propagation process, different from conventional sampling method, we propose a novel dynamic sampling strategy to enlarge the pseudo‐labeled subset step by step to make the pseudo labels more reliable. On Market‐1501, DukeMTMC‐ReID and MARS datasets, we conducted extensively experiments to demonstrate that our proposed method contributes indispensably and achieves a very competitive Re‐ID performance.

Critique of “Planetary Normal Mode Computation: Parallel Algorithms, Performance, and Reproducibility” by SCC Team From National Tsing Hua University

As a special activity of the Student Cluster Competition at SC19 conference, we made an attempt to reproduce the scalability evaluations of a highly paralleled polynomial filtering eigensolver for computing planetary interior normal modes. Our experiments were conducted on a Mars dataset using a small scale 4-node cluster with Intel Skylake CPU architecture, while the original article's were conducted on a Moon dataset using a large scale 256-node supercomputer with Intel CPU Skylake and KNL architectures. This article shares our experiences and observations from our reproducibility activity and discusses our findings on three main sections: the weak scalability, the strong scalability, and the relationships between variables. The results of weak scalability and strong scalability were successfully reproduced. But due to the differences on the problem scale, input dataset, and system architecture, different behaviors regarding the polynomial degree were observed.