Transformer-based Method Research Articles

Weighted window attention and recover feature resolution-based network for deformable abdominal image registration

The Swin Transformer has attracted attention in the field of medical image analysis since its introduction in 2021 owing to its computational efficiency and long-range modeling capabilities. Because of these characteristics, the Swin Transformer can establish distant relationships between corresponding voxels at distant locations for complex abdominal image registration tasks. Transformer-based registration methods model the features of multiple fused voxels and output coarse-grained features of the same size. To enable the Transformer to output fine-grained information and strengthen the contribution of the Transformer in the registration model, we propose the recover feature resolution network (RFRNet), which converts the features outputted by the Transformer into fine-grained spatial information. In addition, fixed-window partitioning restricts the Transformer from modeling the global connection of semantic information with uncertain distances. Therefore, we proposed a weighted window attenuation (WWA) mechanism to achieve automatic global-scale interaction of window information after a window partitioning operation. Based on these improvements, we proposed a single-modal unsupervised deformable abnormal image registration model called RFR-WWANet. Qualitative and quantitative results showed that RFR-WWANet achieved significant improvements compared with the current state-of-the-art methods. Ablation experiments demonstrated the effectiveness of the RFRNet and WWA designs. Our code is available at https://github.com/MingR-Ma/RFR-WWANet.

Abstract PO5-12-07: Analyzing Patient Experiences with Adjuvant Hormone Therapy for Early Breast Cancer Treatment via NLP/ML on Social Media: Symptoms and Their Mitigation

Abstract Understanding the early breast cancer (eBC) patient experience of disease and treatment is critical to addressing unmet patient needs, improving outcomes, and informing the next wave of drug development. Traditional approaches, such as interviews or focus group studies, are resource intensive and have limitations in reflecting the diverse experiences of patients with different disease characteristics, treatments, and sociodemographic backgrounds, and in detecting rare severe events that could potentially lead to therapy discontinuation. To overcome these limitations, we have developed a semi-automated approach utilizing Natural Language Processing (NLP) and Machine Learning (ML) methods in application to public patient forums to extract patient insights. Specifically, we analyzed 500,000 anonymized posts from the BreastCancer.org forum, utilizing Transformer-based methods to identify posts related to eBC patient experiences treated with standard hormonal therapies (HT). First, a novel ML-based model identified posts related to eBC adjuvant HT. For those, patient demographics, symptoms and their severity, and symptom management strategies such as exercise and yoga were subject to further analysis. Out of 362,074 relevant eBC posts, 32 symptoms and 9 impacts were significantly associated with at least one of the six hormonal therapies we considered. Hot flushes (relative risk [RR]: 6.70; 95% CI: 3.36–13.36), arthralgia (RR: 6.67; 95% CI: 3.53–12.59), weight increase (RR: 4.83; 95% CI: 3.20–7.28), mood swings (RR: 7.36; 95% CI: 5.75–9.42), insomnia (RR: 4.76; 95% CI: 3.14–7.22), and depression (RR: 3.05; 95% CI: 1.71–5.44) demonstrated the strongest associations. Severe headache, dizziness, back pain, and muscle spasms showed significant associations with ≥1 HT despite their low overall prevalence in eBC posts. Notably, posts contained a wide range of symptoms severity, including rarely seen symptoms in smaller patient pools, such as severe night hot flashes leading to treatment discontinuation. Symptoms/impacts mentioned in association with exercise and yoga as a way for symptom mitigation included hot flashes, arthralgia, fatigue, anxiety, depression, and pain. Of those posts, 53.3% to 82.5% reported positive or neutral sentiments for exercise, and 22.2% to 100.0% for yoga. For posts specifically mentioning the management of side effects of HT, between 50.0% and 63.5% had a positive sentiment associated with exercise concerning each of the symptoms/impacts with a lower positive impact related to yoga (between 11.1% and 43.0% of posts). Posts discussing pain management and exercise included a higher proportion of negative feedback, versus positive sentiment for depression. Posts related to yoga as an HT symptom mitigation strategy had a higher prevalence of negative sentiments for anxiety and depression. Our proposed ML/NLP-based approach to collecting patient feedback from publicly available forums complements standard qualitative methods by enabling the scalable extraction of insights from hundreds of thousands of online conversations that could span a broad diversity of symptoms and well-being reflections. Furthermore, our findings suggest the potential value of informing clinical outcome assessment (COA) measurement for clinical trials, alone and in combination with standard qualitative research approaches to guide the next clinical development. Overall, our study highlights the significant potential of NLP and ML to address the critical need to better understand the eBC patient experience of disease and treatment to improve outcomes for patients. Citation Format: Sameet Sreenivasan, Chao Fang, Emuella Flood, Natasha Markuzon, Jasmine Sze. Analyzing Patient Experiences with Adjuvant Hormone Therapy for Early Breast Cancer Treatment via NLP/ML on Social Media: Symptoms and Their Mitigation [abstract]. In: Proceedings of the 2023 San Antonio Breast Cancer Symposium; 2023 Dec 5-9; San Antonio, TX. Philadelphia (PA): AACR; Cancer Res 2024;84(9 Suppl):Abstract nr PO5-12-07.

Drug-target Interaction Prediction By Combining Transformer and Graph Neural Networks

Background: The prediction of drug-target interactions (DTIs) plays an essential role in drug discovery. Recently, deep learning methods have been widely applied in DTI prediction. However, most of the existing research does not fully utilize the molecular structures of drug compounds and the sequence structures of proteins, which makes these models unable to obtain precise and effective feature representations. Methods: In this study, we propose a novel deep learning framework combining transformer and graph neural networks for predicting DTIs. Our model utilizes graph convolutional neural networks to capture the global and local structure information of drugs, and convolutional neural networks are employed to capture the sequence feature of targets. In addition, the obtained drug and protein representations are input to multi-layer transformer encoders, respectively, to integrate their features and generate final representations. Results: The experiments on benchmark datasets demonstrated that our model outperforms previous graph-based and transformer-based methods, with 1.5% and 1.8% improvement in precision and 0.2% and 1.0% improvement in recall, respectively. The results indicate that the transformer encoders effectively extract feature information of both drug compounds and proteins. Conclusion: Overall, our proposed method validates the applicability of combining graph neural networks and transformer architecture in drug discovery, and due to the attention mechanisms, it can extract deep structure feature data of drugs and proteins.

IMVS: Integrating multi-view information on multiple scales for 3D object recognition

3D object recognition is a fundamental task in 3D computer vision. View-based methods have received considerable attention due to their high efficiency and superior performance. To better capture the long-range dependencies among multi-view images, Transformer has recently been introduced into view-based 3D object recognition and achieved excellent performance. However, the information among views on multiple scales is not utilized sufficiently in the existing Transformer-based methods. To address this limitation, we proposed a 3D object recognition method named iMVS to integrate Multi-View information on multiple Scales. Specifically, for the single-view image/features at each scale, we adopt a hybrid feature extraction module consisting of CNN and Transformer to jointly capture local and non-local information. For the extracted multi-view image features at each scale, we develop a feature transfer module including a view Transformer block to achieve the information transfer across views. Following a sequential process of the single-view feature extraction and multi-view feature transfer on multiple scales, the multi-view information is sufficiently interacted. Subsequently, the multi-scale features with multi-view information are fed into our designed feature aggregation module to generate a category-specific descriptor, where the adopted channel Transformer block facilitates the descriptor to be more expressive. Coupling with these designs, our method can fully exploit the information embedded within multi-view images. Experimental results on ModelNet40, ModelNet10 and a real-world dataset MVP-N demonstrate the superior performance of our method.

Inferring bifurcation diagrams with transformers.

The construction of bifurcation diagrams is an essential component of understanding nonlinear dynamical systems. The task can be challenging when one knows the equations of the dynamical system and becomes much more difficult if only the underlying data associated with the system are available. In this work, we present a transformer-based method to directly estimate the bifurcation diagram using only noisy data associated with an arbitrary dynamical system. By splitting a bifurcation diagram into segments at bifurcation points, the transformer is trained to simultaneously predict how many segments are present and to minimize the loss with respect to the predicted position, shape, and asymptotic stability of each predicted segment. The trained model is shown, both quantitatively and qualitatively, to reliably estimate the structure of the bifurcation diagram for arbitrarily generated one- and two-dimensional systems experiencing a codimension-one bifurcation with as few as 30 trajectories. We show that the method is robust to noise in both the state variable and the system parameter.

Assessing Face Image Quality: A Large-Scale Database and a Transformer Method.

The amount of face images has been witnessing an explosive increase in the last decade, where various distortions inevitably exist on transmitted or stored face images. The distortions lead to visible and undesirable degradation on face images, affecting their quality of experience (QoE). To address this issue, this paper proposes a novel Transformer-based method for quality assessment on face images (named as TransFQA). Specifically, we first establish a large-scale face image quality assessment (FIQA) database, which includes 42,125 face images with diversifying content at different distortion types. Through an extensive crowdsource study, we obtain 712,808 subjective scores, which to the best of our knowledge contribute to the largest database for assessing the quality of face images. Furthermore, by investigating the established database, we comprehensively analyze the impacts of distortion types and facial components (FCs) on the overall image quality. Accordingly, we propose the TransFQA method, in which the FC-guided Transformer network (FT-Net) is developed to integrate the global context, face region and FC detailed features via a new progressive attention mechanism. Then, a distortion-specific prediction network (DP-Net) is designed to weight different distortions and accurately predict final quality scores. Finally, the experiments comprehensively verify that our TransFQA method significantly outperforms other state-of-the-art methods for quality assessment on face images.

PL-DINO: An Improved Transformer-Based Method for Plant Leaf Disease Detection

Agriculture is important for ecology. The early detection and treatment of agricultural crop diseases are meaningful and challenging tasks in agriculture. Currently, the identification of plant diseases relies on manual detection, which has the disadvantages of long operation time and low efficiency, ultimately impacting the crop yield and quality. To overcome these disadvantages, we propose a new object detection method named “Plant Leaf Detection transformer with Improved deNoising anchOr boxes (PL-DINO)”. This method incorporates a Convolutional Block Attention Module (CBAM) into the ResNet50 backbone network. With the assistance of the CBAM block, the representative features can be effectively extracted from leaf images. Next, an EQualization Loss (EQL) is employed to address the problem of class imbalance in the relevant datasets. The proposed PL-DINO is evaluated using the publicly available PlantDoc dataset. Experimental results demonstrate the superiority of PL-DINO over the related advanced approaches. Specifically, PL-DINO achieves a mean average precision of 70.3%, surpassing conventional object detection algorithms such as Faster R-CNN and YOLOv7 for leaf disease detection in natural environments. In brief, PL-DINO offers a practical technology for smart agriculture and ecological monitoring.

The evolution of object detection methods

Object detection is one of the most important domains in computer vision tasks, which is an important branch of artificial intelligence. It aims at finding and locating the accurate position of objects in given pictures or videos. With the development of deep learning techniques, more powerful and robust algorithms have emerged to deal with multi-scale, high-level features to overcome the limitations of traditional pipeline of object detectors. The popularity of transformer framework enables larger capacity datasets by processing self-attention mechanism, and the object detection methods have evolved into a new era. This paper first reviews traditional object detection pipeline and brief history of deep learning, afterwards it focuses on the classification of deep learning-based object detection methods covering Convolution Neural Network based and transformer-based methods. Commonly used datasets and metrics are also covered in the next part. The Convolution Neural Network based methods mainly contain two-stage and one-stage detectors, Convolution Neural Network is the underlying structure of these methods convolutional stages are fundamental parts. Transformer-based models convert traditional object detection issues into end-to-end detection, which is widely used in dealing with images. Finally, the promising future of object detection areas are listed to show guidance on future work.

Hepatic and portal vein segmentation with dual-stream deep neural network.

Liver lesions mainly occur inside the liver parenchyma, which are difficult to locate and have complicated relationships with essential vessels. Thus, preoperative planning is crucial for the resection of liver lesions. Accurate segmentation of the hepatic and portal veins (PVs) on computed tomography (CT) images is of great importance for preoperative planning. However, manually labeling the mask of vessels is laborious and time-consuming, and the labeling results of different clinicians are prone to inconsistencies. Hence, developing an automatic segmentation algorithm for hepatic and PVs on CT images has attracted the attention of researchers. Unfortunately, existing deep learning based automatic segmentation methods are prone to misclassifying peripheral vessels into wrongcategories. This study aims to provide a fully automatic and robust semantic segmentation algorithm for hepatic and PVs, guiding subsequent preoperative planning. In addition, to address the deficiency of the public dataset for hepatic and PV segmentation, we revise the annotations of the Medical Segmentation Decathlon (MSD) hepatic vessel segmentation dataset and add the masks of the hepatic veins (HVs) andPVs. We proposed a structure with a dual-stream encoder combining convolution and Transformer block, named Dual-stream Hepatic Portal Vein segmentation Network, to extract local features and long-distance spatial information, thereby extracting anatomical information of hepatic and portal vein, avoiding misdivisions of adjacent peripheral vessels. Besides, a multi-scale feature fusion block based on dilated convolution is proposed to extract multi-scale features on expanded perception fields for local features, and a multi-level fusing attention module is introduced for efficient context information extraction. Paired t-test is conducted to evaluate the significant difference in dice between the proposed methods and the comparingmethods. Two datasets are constructed from the original MSD dataset. For each dataset, 50 cases are randomly selected for model evaluation in the scheme of 5-fold cross-validation. The results show that our method outperforms the state-of-the-art Convolutional Neural Network-based and transformer-based methods. Specifically, for the first dataset, our model reaches 0.815, 0.830, and 0.807 at overall dice, precision, and sensitivity. The dice of the hepatic and PVs are 0.835 and 0.796, which also exceed the numeric result of the comparing methods. Almost all the p-values of paired t-tests on the proposed approach and comparing approaches are smaller than 0.05. On the second dataset, the proposed algorithm achieves 0.749, 0.762, 0.726, 0.835, and 0.796 for overall dice, precision, sensitivity, dice for HV, and dice for PV, among which the first four numeric results exceed comparingmethods. The proposed method is effective in solving the problem of misclassifying interlaced peripheral veins for the HV and PV segmentation task and outperforming the comparing methods on the relabeleddataset.

Adjustable patch and feature prior token-based transformer for weakly supervised semantic segmentation

Weakly supervised semantic segmentation is a challenging task, utilizing only low-cost weak supervision to produce pixel-level predictions. Existing transformer-based methods for weakly supervised semantic segmentation have some limitations: (1) Fixing patch size might destroy the structured semantics, which is unfriendly to objects of different scales, and (2) Ignoring the prior features when using multi-head attention mechanisms might lead to inaccurate segmentation localization. To tackle these issues, we proposed an effective transformer framework coupled with the adjustable patch and prior feature tokens, termed as APFPformer, in which an adjustable patch module is developed to split the image according to the area of the salient object for preserving the structured semantics in patches. A prior feature token module is devised to exploit the edge and texture information as prior tokens, ensuring the gain of discriminative representation. Additionally, a single-stage scheme is applied to reduce the computation and rapid segmentation process. Our experiments demonstrate the superiority of our approach over early methods, gaining competitive mean Intersection-over-Union scores of 67.9% on the PASCAL VOC2012 dataset and 40.2% on the MS COCO dataset.

A transformer-based method for correcting significant wave height numerical forecasting errors

Accurate significant wave height (SWH) forecasting is essential for various marine activities. While traditional numerical and mathematical-statistical methods have made progress, there is still room for improvement. This study introduces a novel transformer-based approach called the 2D-Geoformer to enhance SWH forecasting accuracy. The 2D-Geoformer combines the spatial distribution capturing capabilities of SWH numerical models with the ability of mathematical-statistical methods to identify intrinsic relationships among datasets. Using a comprehensive long time series of SWH numerical hindcast datasets as the numerical forecasting database and ERA5 reanalysis SWH datasets as the observational proxies database, with a focus on a 72-hour forecasting window, the 2D-Geoformer is designed. By training the potential connections between SWH numerical forecasting fields and forecasting errors, we can retrieve SWH forecasting errors for each numerical forecasting case. The corrected forecasting results can be obtained by subtracting the retrieved SWH forecasting errors from the original numerical forecasting fields. During long-term validation periods, this method consistently and effectively corrects numerical forecasting errors for almost every case, resulting in a significant reduction in root mean square error compared to the original numerical forecasting fields. Further analysis reveals that this method is particularly effective for numerical forecasting fields with higher errors compared to those with relatively smaller errors. This integrated approach represents a substantial advancement in SWH forecasting, with the potential to improve the accuracy of operational SWH forecasts. The 2D-Geoformer combines the strengths of numerical models and mathematical-statistical methods, enabling better capture of spatial distributions and intrinsic relationships in the data. The method's effectiveness in correcting numerical forecasting errors, particularly for cases with higher errors, highlights its potential for enhancing SWH forecasting accuracy in operational settings.

Two-stage image colorization via color codebook

Image colorization has a wide range of applications, but it remains a challenging task due to it is an inherently ill-posed problem with multi-modal uncertainty. The advancement of deep learning techniques has provided extensive avenues for addressing image colorization. However, current works mainly suffer from two problems: inaccurate colorization leading to biased color tones (e.g., cool or warm bias) and undersaturation of images. Existing Transformer-based methods can produce impressive results, but they often come with high training costs and may result in color overflow effects. In this paper, we propose a two-stage image colorization strategy based on a color codebook. Clustering methods in the three-dimensional CIE Lab color space is proposed to integrate brightness information so that the colors in the codebook can be lifelike. In the first stage, we treat the colorization task as a classification problem based on a color codebook, and a high-quality codebook is advantageous for enhancing color classification accuracy. In the second stage, different from the traditional Transformer-based method, a pyramid-type Transformer structure is used to extract rich image features to refine the colors, which can solve potential color bands, color errors and color overflow. In addition, the parameters and FLOPs are significantly smaller than other traditional Transformer-based methods. Extensive experiments demonstrate that our method outperforms state-of-the-art approaches. On the ImageNet validation set, the achieved values are 4.60, 25.23, 0.19, and 39.82 in terms of FID, PSNR, LPIPS, and CF, respectively. On the COCO-Stuff validation set, the achieved values are 5.62, 25.15, 0.19, and 36.25 in terms of FID, PSNR, LPIPS, and CF, respectively. The codes are available at https://github.com/Tanghui2000/Two-stage_Image_Colorization_via_Color_Codebook.

A simple transformer-based baseline for crowd tracking with Sequential Feature Aggregation and Hybrid Group Training

Tracking pedestrians in crowded scenes is a challenging task. Existing transformer-based tracking methods integrate detection and tracking into a unified model, which simplifies the tracking process. However, these methods also introduce complicated attention mechanisms that increase the model complexity and cost. To address this issue, we propose SOTTrack, a simple online transformer-based method for crowd tracking. Our method enhances feature learning and training strategies without sacrificing simplicity and efficiency. Specifically, we introduce the Sequential Feature Aggregation (SFA) module and the Hybrid Group Training (HGT) approach. The SFA module fuses features from sequential images to improve the temporal consistency of visual features within short time intervals. The HGT approach assigns different queries to multiple guided tasks, such as label assignment and de-noising, which are only used during training and do not incur any inference cost. We evaluate our method on the MOT17 and MOT20 datasets and demonstrate its competitive performance.

Face anti-spoofing with cross-stage relation enhancement and spoof material perception

Face Anti-Spoofing (FAS) seeks to protect face recognition systems from spoofing attacks, which is applied extensively in scenarios such as access control, electronic payment, and security surveillance systems. Face anti-spoofing requires the integration of local details and global semantic information. Existing CNN-based methods rely on small stride or image patch-based feature extraction structures, which struggle to capture spatial and cross-layer feature correlations effectively. Meanwhile, Transformer-based methods have limitations in extracting discriminative detailed features. To address the aforementioned issues, we introduce a multi-stage CNN-Transformer-based framework, which extracts local features through the convolutional layer and long-distance feature relationships via self-attention. Based on this, we proposed a cross-attention multi-stage feature fusion, employing semantically high-stage features to query task-relevant features in low-stage features for further cross-stage feature fusion. To enhance the discrimination of local features for subtle differences, we design pixel-wise material classification supervision and add a auxiliary branch in the intermediate layers of the model. Moreover, to address the limitations of a single acquisition environment and scarcity of acquisition devices in the existing Near-Infrared dataset, we create a large-scale Near-Infrared Face Anti-Spoofing dataset with 380k pictures of 1040 identities. The proposed method could achieve the state-of-the-art in OULU-NPU and our proposed Near-Infrared dataset at just 1.3GFlops and 3.2M parameter numbers, which demonstrate the effective of the proposed method.

CoT: a transformer-based method for inferring tumor clonal copy number substructure from scDNA-seq data.

Single-cell DNA sequencing (scDNA-seq) has been an effective means to unscramble intra-tumor heterogeneity, while joint inference of tumor clones and their respective copy number profiles remains a challenging task due to the noisy nature of scDNA-seq data. We introduce a new bioinformatics method called CoT for deciphering clonal copy number substructure. The backbone of CoT is a Copy number Transformer autoencoder that leverages multi-head attention mechanism to explore correlations between different genomic regions, and thus capture global features to create latent embeddings for the cells. CoT makes it convenient to first infer cell subpopulations based on the learned embeddings, and then estimate single-cell copy numbers through joint analysis of read counts data for the cells belonging to the same cluster. This exploitation of clonal substructure information in copy number analysis helps to alleviate the effect of read counts non-uniformity, and yield robust estimations of the tumor copy numbers. Performance evaluation on synthetic and real datasets showcases that CoT outperforms the state of the arts, and is highly useful for deciphering clonal copy number substructure.

Count What You Want: Exemplar Identification and Few-Shot Counting of Human Actions in the Wild

This paper addresses the task of counting human actions of interest using sensor data from wearable devices. We propose a novel exemplar-based framework, allowing users to provide exemplars of the actions they want to count by vocalizing predefined sounds ``one'', ``two'', and ``three''. Our method first localizes temporal positions of these utterances from the audio sequence. These positions serve as the basis for identifying exemplars representing the action class of interest. A similarity map is then computed between the exemplars and the entire sensor data sequence, which is further fed into a density estimation module to generate a sequence of estimated density values. Summing these density values provides the final count. To develop and evaluate our approach, we introduce a diverse and realistic dataset consisting of real-world data from 37 subjects and 50 action categories, encompassing both sensor and audio data. The experiments on this dataset demonstrate the viability of the proposed method in counting instances of actions from new classes and subjects that were not part of the training data. On average, the discrepancy between the predicted count and the ground truth value is 7.47, significantly lower than the errors of the frequency-based and transformer-based methods. Our project, code and dataset can be found at https://github.com/cvlab-stonybrook/ExRAC.

Dual-Window Multiscale Transformer for Hyperspectral Snapshot Compressive Imaging

Coded aperture snapshot spectral imaging (CASSI) system is an effective manner for hyperspectral snapshot compressive imaging. The core issue of CASSI is to solve the inverse problem for the reconstruction of hyperspectral image (HSI). In recent years, Transformer-based methods achieve promising performance in HSI reconstruction. However, capturing both long-range dependencies and local information while ensuring reasonable computational costs remains a challenging problem. In this paper, we propose a Transformer-based HSI reconstruction method called dual-window multiscale Transformer (DWMT), which is a coarse-to-fine process, reconstructing the global properties of HSI with the long-range dependencies. In our method, we propose a novel U-Net architecture using a dual-branch encoder to refine pixel information and full-scale skip connections to fuse different features, enhancing the extraction of fine-grained features. Meanwhile, we design a novel self-attention mechanism called dual-window multiscale multi-head self-attention (DWM-MSA), which utilizes two different-sized windows to compute self-attention, which can capture the long-range dependencies in a local region at different scales to improve the reconstruction performance. We also propose a novel position embedding method for Transformer, named con-abs position embedding (CAPE), which effectively enhances positional information of the HSIs. Extensive experiments on both the simulated and the real data are conducted to demonstrate the superior performance, stability, and generalization ability of our DWMT. Code of this project is at https://github.com/chenx2000/DWMT.

Context Enhanced Transformer for Single Image Object Detection in Video Data

With the increasing importance of video data in real-world applications, there is a rising need for efficient object detection methods that utilize temporal information. While existing video object detection (VOD) techniques employ various strategies to address this challenge, they typically depend on locally adjacent frames or randomly sampled images within a clip. Although recent Transformer-based VOD methods have shown promising results, their reliance on multiple inputs and additional network complexity to incorporate temporal information limits their practical applicability. In this paper, we propose a novel approach to single image object detection, called Context Enhanced TRansformer (CETR), by incorporating temporal context into DETR using a newly designed memory module. To efficiently store temporal information, we construct a class-wise memory that collects contextual information across data. Additionally, we present a classification-based sampling technique to selectively utilize the relevant memory for the current image. In the testing, We introduce a test-time memory adaptation method that updates individual memory functions by considering the test distribution. Experiments with CityCam and ImageNet VID datasets exhibit the efficiency of the framework on various video systems. The project page and code will be made available at: https://ku-cvlab.github.io/CETR.

Multitarget Device-Free Localization via Cross-Domain Wi-Fi RSS Training Data and Attentional Prior Fusion

Device-free localization (DFL) using easily-obtained Wi-Fi received signal strength (RSS) has wide real-world applications for not requiring people to carry trackable devices. However, accurate multitarget DFL remains challenging due to the unknown number of targets, multipath interference (MPI), especially between nearby targets, and limited real-world data. In this study, we pioneeringly propose a transformer-based learning method with Wi-Fi RSS as input, and an attentional prior fusion module, to simultaneously locate an unknown number of people at random positions. To overcome the multitarget data collection challenges, we contribute a large-scale cross-domain real-simulation-augmentation training dataset with one and two real-world nearby non-person objects at limited positions and up to five simulated and augmented randomly distributed targets. Experimental results demonstrate our method's improved accuracy, generalization ability, and robustness with fewer Wi-Fi nodes than previous methods.

TOP-ReID: Multi-Spectral Object Re-identification with Token Permutation

Multi-spectral object Re-identification (ReID) aims to retrieve specific objects by leveraging complementary information from different image spectra. It delivers great advantages over traditional single-spectral ReID in complex visual environment. However, the significant distribution gap among different image spectra poses great challenges for effective multi-spectral feature representations. In addition, most of current Transformer-based ReID methods only utilize the global feature of class tokens to achieve the holistic retrieval, ignoring the local discriminative ones. To address the above issues, we step further to utilize all the tokens of Transformers and propose a cyclic token permutation framework for multi-spectral object ReID, dubbled TOP-ReID. More specifically, we first deploy a multi-stream deep network based on vision Transformers to preserve distinct information from different image spectra. Then, we propose a Token Permutation Module (TPM) for cyclic multi-spectral feature aggregation. It not only facilitates the spatial feature alignment across different image spectra, but also allows the class token of each spectrum to perceive the local details of other spectra. Meanwhile, we propose a Complementary Reconstruction Module (CRM), which introduces dense token-level reconstruction constraints to reduce the distribution gap across different image spectra. With the above modules, our proposed framework can generate more discriminative multi-spectral features for robust object ReID. Extensive experiments on three ReID benchmarks (i.e., RGBNT201, RGBNT100 and MSVR310) verify the effectiveness of our methods. The code is available at https://github.com/924973292/TOP-ReID.