Pre-trained Backbone Research Articles

ESCAPE: Energy-based Selective Adaptive Correction for Out-of-distribution 3D Human Pose Estimation

Despite recent advances in human pose estimation (HPE), poor generalization to out-of-distribution (OOD) data remains a difficult problem. While previous works have proposed Test-Time Adaptation (TTA) to bridge the train-test domain gap by refining network parameters at inference, the absence of ground-truth annotations makes it highly challenging and existing methods typically increase inference times by one or more orders of magnitude. We observe that (1) not every test time sample is OOD, and (2) HPE errors are significantly larger on distal keypoints (wrist, ankle). To this end, we propose ESCAPE: a lightweight correction and selective adaptation framework which applies a fast, forward-pass correction on most data while reserving costly TTA for OOD data. The free energy function is introduced to separate OOD samples from incoming data and a correction network is trained to estimate the errors of pretrained backbone HPE predictions on the distal keypoints. For OOD samples, we propose a novel self-consistency adaptation loss to update the correction network by leveraging the constraining relationship between distal keypoints and proximal keypoints (shoulders, hips), via a second “reverse” network. ESCAPE improves the distal MPJPE of five popular HPE models by up to 7% on unseen data, achieves state-of-the-art results on two popular HPE benchmarks, and is significantly faster than existing adaptation methods.

Leveraging SAM for automatic prostate segmentation on micro-ultrasound images

Purpose: Medical image segmentation models are essential for efficiently diagnosing prostate cancer. Current models for segmentation of micro ultrasound images do not utilize foundation models, or geometric and spatial data from the ultrasound image. We aim to leverage the Segment Anything Model (SAM) as a pretrained backbone to create a fully automatic prostate segmentation model and investigate prompting with positonal information such as the slice frame number and previous segmentations. Methods: The Segment Anything Model (SAM) was fine-tuned on Micro-Ultrasound images1 and adapted for the context of clinical prostate segmentation. We explored different fine-tuning methods and prompting combinations fit for automatic segmentation. Dice coefficient (DSC) and Hausdorff 95% distance (HD95) were used as evaluation metrics to compare our model against past models. Results: In comparison to the state of the art models, SliceTrack-SAM improved the mean Dice coefficient from the MicroSegNet2 model (93.1) to 94.3, and reduced the Hausdorff distance from the nnU-Net model (1.09 mm) to 0.81 mm. Conclusion: In summary, using a fully-finetuned SAM, improves prostate segmentation accuracy. In addition, prompting with geometric and spatial data derived from ultrasound images shows potential to enhance segmentation accuracy, particularly on difficult datasets.

Mask optimization for optical lithography based on UNet

Abstract Traditional photolithography methods are increasingly unable to meet the ever-stringent demands for pattern resolution and alignment accuracy, and developing efficient mask optimization techniques has become an urgent need in the industry. This paper delves into the application of Inverse Lithography Technology (ILT) in nanoscale photolithography and proposes an improved mask optimization algorithm. The backbone network of this algorithm employs a UNet architecture, trained initially on a prepared dataset comprising original masks and the ones optimized through traditional methods. The pre-trained backbone network generates a coarse mask, which is then inputted into a correction layer to refine the mask, enhancing pattern accuracy and processing efficiency. Compared to traditional gradient-based mask optimization methods, neural ILT demonstrates superior effectiveness, which enhances the efficiency and pattern quality of the lithography process while reducing production costs.

UDANet: An unsupervised domain adaptive vehicle density estimation network based on joint adversarial learning

Unsupervised domain adaptive vehicle density estimation aims to transfer the knowledge learned from the labeled source domain to the unlabeled target domain, which has received extensive attention due to its practicality and effectiveness. However, the existing methods often rely on a large amount of expensive labeled data, and ignore the large differences between different data domains, making the network trained in one scene difficult to apply to the target scene. To solve these problems, this paper proposes an unsupervised domain adaptive vehicle density estimation network based on joint adversarial learning (UDANet), which effectively extends the model to unlabeled target domain data by adding two domain adaptive components to the pre-trained backbone. Specifically, a multi-level feature discriminator is designed to align vehicle features from the image level and refine features from the pixel level by joint adversarial learning, thus achieving cross-domain transfer of features from the source domain to the target domain. In addition, to address the shared convolutional structure that incorrectly highlights irrelevant information, a feature correction module is designed to improve the adaptability of the network to target domain data by re-modeling the source and target data according to channel independence. Several well-known backbones are used to prove the superiority of the backbone and the mobility of the two domain adaptive components. The domain adaptation experimental results on four vehicle datasets show that UDANet achieves high estimation accuracy compared to current state-of-the-art unsupervised methods, and is suitable for the task of vehicle density estimation in complex scenes.

Deep Convolutional Backbone Comparison for Automated PET Image Quality Assessment.

Pretraining deep convolutional network mappings using natural images helps with medical imaging analysis tasks; this is important given the limited number of clinically-annotated medical images. Many two-dimensional pretrained backbone networks, however, are currently available. This work compared 18 different backbones from 5 architecture groups (pretrained on ImageNet) for the task of assessing [18F]FDG brain Positron Emission Transmission (PET) image quality (reconstructed at seven simulated doses), based on three clinical image quality metrics (global quality rating, pattern recognition, and diagnostic confidence). Using two-dimensional randomly sampled patches, up to eight patients (at three dose levels each) were used for training, with three separate patient datasets used for testing. Each backbone was trained five times with the same training and validation sets, and with six cross-folds. Training only the final fully connected layer (with ~6,000-20,000 trainable parameters) achieved a test mean-absolute-error of ~0.5 (which was within the intrinsic uncertainty of clinical scoring). To compare "classical" and over-parameterized regimes, the pretrained weights of the last 40% of the network layers were then unfrozen. The mean-absolute-error fell below 0.5 for 14 out of the 18 backbones assessed, including two that previously failed to train. Generally, backbones with residual units (e.g. DenseNets and ResNetV2s), were suited to this task, in terms of achieving the lowest mean-absolute-error at test time (~0.45 - 0.5). This proof-of-concept study shows that over-parameterization may also be important for automated PET image quality assessments.

Few-shot learning with long-tailed labels

Few-Shot Learning (FSL) is a challenging classification task in machine learning, and it aims to recognize unseen examples of new classes with only a few labeled reference examples (i.e., the support set). The training phase of FSL typically requires a large amount of labeled examples (i.e., the base set) to effectively learn transferable knowledge, but it is usually difficult to obtain sufficient data annotation in practical applications. Existing semi-supervised FSL approaches can learn generalizable representations from partly labeled data, yet they do not sufficiently consider the real distribution of those labeled data. In this paper, we propose a new problem setting termed Few-Shot Learning with Long-Tailed Labels (FSL-LTL) to further consider a more practical semi-supervised scenario where the labeled examples are long-tailed. To effectively address this new problem, we build a novel two-stage training framework dubbed Reweighted Contrastive Embedding (RCE). In the first stage of RCE, we adopt the popular contrastive learning framework to pre-train a reliable network in a self-supervised manner. In the second stage, we integrate the semi-supervised empirical risk into a Weighted Random Sampling (WRS) strategy to fine-tune the pre-trained backbone with the aid of a consistency regularization. Experimental results demonstrate the feasibility of the proposed FSL-LTL problem setting and the superiority of our new RCE method over existing FSL approaches and semi-supervised learning methods. These results also suggest that the RCE approach is a promising solution for addressing the new FSL-LTL problem.

DiffDD: A surface defect detection framework with diffusion probabilistic model

Automated defect detection is critical to improve product quality, reduce costs, and enhance productivity in the manufacturing process. More recently, diffusion-based generative models have entered a new realm of vision tasks by offering high-quality and diverse images. However, the diffusion-based defect detection model fails to be researched, and generative models typically require postprocessing modules to achieve the segmentation of defective regions. Furthermore, generic diffusion models struggle to reconstruct defective regions while preserving the semantic information of the inspection images. To address these issues, we propose a Diffusion-based Defect Detection (DiffDD) framework, comprising a pre-trained backbone (PvTv2) and diffusion probabilistic model (DPM), for surface defect detection. The backbone is employed to extract multi-scale feature maps from input images supervised by pixel-wise labels. These pyramid maps interact with DPM to guide the diffusion process, refining the semantic information of the reconstructed image. Additionally, a semantic restoration (SR) module is used to restore the output of the backbone for loss computation with the pixel-wise label. A conditional image prior method is implemented to constrain the randomness of DPM, aiding the model in generating segmentation masks corresponding to the inspection images. Experimental results on NEU_SEG and MT demonstrate that our model outperforms state-of-the-art (SOTA) methods by a significant margin, indicating the generalization and effectiveness of the proposed DiffDD framework.

Self-supervised learning for RGB-D object tracking

Recently, there has been a growing interest in RGB-D object tracking thanks to its promising performance achieved by combining visual information with auxiliary depth cues. However, the limited volume of annotated RGB-D tracking data for offline training has hindered the development of a dedicated end-to-end RGB-D tracker design. Consequently, the current state-of-the-art RGB-D trackers mainly rely on the visual branch to support the appearance modelling, with the depth map utilised for elementary information fusion or failure reasoning of online tracking. Despite the achieved progress, the current paradigms for RGB-D tracking have not fully harnessed the inherent potential of depth information, nor fully exploited the synergy of vision-depth information. Considering the availability of ample unlabelled RGB-D data and the advancement in self-supervised learning, we address the problem of self-supervised learning for RGB-D object tracking. Specifically, an RGB-D backbone network is trained on unlabelled RGB-D datasets using masked image modelling. To train the network, the masking mechanism creates a selective occlusion of the input visible image to force the corresponding aligned depth map to help with discerning and learning vision-depth cues for the reconstruction of the masked visible image. As a result, the pre-trained backbone network is capable of cooperating with crucial visual and depth features of the diverse objects and background in the RGB-D image. The intermediate RGB-D features output by the pre-trained network can effectively be used for object tracking. We thus embed the pre-trained RGB-D network into a transformer-based tracking framework for stable tracking. Comprehensive experiments and the analysis of the results obtained on several RGB-D tracking datasets demonstrate the effectiveness and superiority of the proposed RGB-D self-supervised learning framework and the following tracking approach.

Keep DRÆMing: Discriminative 3D anomaly detection through anomaly simulation

Recent surface anomaly detection methods rely on pretrained backbone networks for efficient anomaly detection. On standard RGB anomaly detection benchmarks these methods achieve excellent results but fail on 3D anomaly detection due to a lack of pretrained backbones that suit this domain. Additionally, there is a lack of industrial depth data that would enable the backbone network training that could be used in 3D anomaly detection models. Discriminative anomaly detection methods do not require pretrained networks and are trained using simulated anomalies. The process of simulating anomalies that fit the domain of industrial depth data is not trivial and is necessary for training discriminative methods. We propose a novel 3D anomaly simulation process that follows the natural characteristics of industrial depth data and generates diverse deformations, making it suitable for training discriminative anomaly detection methods. We demonstrate its effectiveness by adapting the DRÆM method to work on 3D anomaly detection, thus obtaining 3DRÆM, a strong discriminative 3D anomaly detection model. The proposed approach achieves excellent results on the MVTec3D anomaly detection benchmark where it achieves state-of-the-art results on both 3D and RGB+3D problem setups, significantly outperforming competing methods.

BERT4ST:: Fine-tuning pre-trained large language model for wind power forecasting

Accurate forecasting of wind power generation is essential for ensuring power safety, scheduling various energy sources, and improving energy utilization. However, the elusive nature of wind, influenced by various meteorological and geographical factors, greatly complicates the wind power forecasting task. To improve the forecasting accuracy of wind power (WP), we propose a BERT-based model for spatio-temporal forecasting (BERT4ST), which is the first approach to fine-tune a large language model for the spatio-temporal modeling of WP. To deal with the inherent characteristics of WP, BERT4ST exploits the individual spatial and temporal dependency of patches and redesigns a set of spatial and temporal encodings. By well analyzing the connection between bidirectional attention networks and WP spatio-temporal data, BERT4ST employs a pre-trained BERT encoder as the backbone network to learn the individual spatial and temporal dependency of patches of WP data. Additionally, BERT4ST fine-tunes the pre-trained backbone in a multi-stage manner, i.e., first aligning the language model with the spatio-temporal data and then fine-tuning the downstream tasks while maintaining the stability of the backbone network. Experimental results demonstrate that our BERT4ST achieves desirable performance compared to some state-of-the-art methods.

XKD: Cross-Modal Knowledge Distillation with Domain Alignment for Video Representation Learning

We present XKD, a novel self-supervised framework to learn meaningful representations from unlabelled videos. XKD is trained with two pseudo objectives. First, masked data reconstruction is performed to learn modality-specific representations from audio and visual streams. Next, self-supervised cross-modal knowledge distillation is performed between the two modalities through a teacher-student setup to learn complementary information. We introduce a novel domain alignment strategy to tackle domain discrepancy between audio and visual modalities enabling effective cross-modal knowledge distillation. Additionally, to develop a general-purpose network capable of handling both audio and visual streams, modality-agnostic variants of XKD are introduced, which use the same pretrained backbone for different audio and visual tasks. Our proposed cross-modal knowledge distillation improves video action classification by 8% to 14% on UCF101, HMDB51, and Kinetics400. Additionally, XKD improves multimodal action classification by 5.5% on Kinetics-Sound. XKD shows state-of-the-art performance in sound classification on ESC50, achieving top-1 accuracy of 96.5%.

LION: Implicit Vision Prompt Tuning

Despite recent promising performances across a range of vision tasks, vision Transformers still have an issue of high computational costs. Recently, vision prompt learning has provided an economical solution to this problem without fine-tuning the whole large-scale model. However, the efficiency and effectiveness of existing models are still far from satisfactory due to the parameter cost of extensive prompt blocks and tricky prompt framework designs. In this paper, we propose a light-weight prompt framework named impLicit vIsion prOmpt tuNing (LION), which is motivated by deep implicit models with stable low memory costs for various complex tasks. In particular, we merely insect two equilibrium implicit layers in two ends of the pre-trained backbone with parameters frozen. Moreover, according to the lottery hypothesis, we further prune the parameters to relieve the computation burden in implicit layers. Various experiments have validated that our LION obtains promising performances on a wide range of datasets. Most importantly, LION reduces up to 11.5 % of training parameter numbers while obtaining higher performance than the state-of-the-art VPT, especially under challenging scenes. Furthermore, we find that our proposed LION has an excellent generalization performance, making it an easy way to boost transfer learning in the future.

Robust change detection for remote sensing images based on temporospatial interactive attention module

Change Detection (CD) is a vital monitoring method in Earth observation, especially pertinent for land-use analysis, city management, and disaster damage assessment. However, in the era of constellation interconnection and air-sky collaboration, the changes in the Regions Of Interest (ROI) cause many false detections due to geometric perspective rotation and temporal style difference. In response to these challenges, we introduce CDNeXt, this framework elucidates a robust and efficient method for combining Siamese networks based on the pre-trained backbone with the innovative Temporospatial Interactive Attention Module (TIAM) for remote sensing imagery. The CDNeXt can be categorized into four primary components: Encoder, Interactor, Decoder, and Detector. Notably, the Interactor, powered by TIAM, queries and rebuilds spatial perspective dependencies and temporal style correlations from binary temporal features extracted by the Encoder to enlarge the difference of ROI change. Culminating the process, the Detector integrates the hierarchical features generated by the Decoder, subsequently producing a binary change mask. We have achieved new State-Of-The-Art (SOTA) performance in change detection, with our method surpassing existing techniques on four benchmark datasets: an F1 score of 82.63% on SYSU-CD, 87.14% on LEVIR-CD+, 66.71% on S2Looking, and 71.11% BANDON. To further validate the effectiveness of the TIAM, we compared it to other attention modules in both interactive and non-interactive modes. Our code is available on GitHub: https://github.com/wjj282439449/CDNeXt.

Multi-Scale Feature Fusion Network with Symmetric Attention for Land Cover Classification Using SAR and Optical Images

The complementary characteristics of SAR and optical images are beneficial in improving the accuracy of land cover classification. Deep learning-based models have achieved some notable results. However, how to effectively extract and fuse the unique features of multi-modal images for pixel-level classification remains challenging. In this article, a two-branch supervised semantic segmentation framework without any pretrained backbone is proposed. Specifically, a novel symmetric attention module is designed with improved strip pooling. The multiple long receptive fields can better perceive irregular objects and obtain more anisotropic contextual information. Meanwhile, to solve the semantic absence and inconsistency of different modalities, we construct a multi-scale fusion module, which is composed of atrous spatial pyramid pooling, varisized convolutions and skip connections. A joint loss function is introduced to constrain the backpropagation and reduce the impact of class imbalance. Validation experiments were implemented on the DFC2020 and WHU-OPT-SAR datasets. The proposed model achieved the best quantitative values on the metrics of OA, Kappa and mIoU, and its class accuracy was also excellent. It is worth mentioning that the number of parameters and the computational complexity of the method are relatively low. The adaptability of the model was verified on RGB–thermal segmentation task.

A Reconfigurable Framework for Neural Network Based Video In-Loop Filtering

This article proposes a reconfigurable framework for neural network based video in-loop filtering to guide large-scale models for content-aware processing. Specifically, the backbone neural model is decomposed into several convolutional groups and the encoder systematically traverses all candidate configurations combined by these groups to find the best one. The selected configuration index is then encapsulated as side information and passed to the decoder, enabling dynamic model reconfiguration during the decoding stage. The preceding reconfiguration process is only deployed in the inference stage on top of a pre-trained backbone model. Furthermore, we devise WMSPFormer , a wavelet multi-scale Poolformer, as the backbone network structure. WMSPFormer utilizes a wavelet-based multi-scale structure to losslessly decompose the input into multiple scales for spatial-spectral features aggregation. Moreover, it uses multi-scale pooling operations ( MSPoolformer ) instead of complicated matrix calculations to substitute the attention process. We also extend MSPoolformer to a large-scale version using more parameters, referred to as MSPoolformerExt . Extensive experiments demonstrate that the proposed WMSPFormer+Reconfig. and WMSPFormerExt+Reconfig. achieve a remarkable 7.13% and 7.92% BD-Rate reduction over the anchor H.266/VVC, outperforming most existing methods evaluated under the same training and testing conditions. In addition, the low-complexity nature of the WMSPFormer series makes it attractive for practical applications.

Exploring biometric domain adaptation in human action recognition models for unconstrained environments

AbstractIn conventional machine learning (ML), a fundamental assumption is that the training and test sets share identical feature distributions, a reasonable premise drawn from the same dataset. However, real-world scenarios often defy this assumption, as data may originate from diverse sources, causing disparities between training and test data distributions. This leads to a domain shift, where variations emerge between the source and target domains. This study delves into human action recognition (HAR) models within an unconstrained, real-world setting, scrutinizing the impact of input data variations related to contextual information and video encoding. The objective is to highlight the intricacies of model performance and interpretability in this context. Additionally, the study explores the domain adaptability of HAR models, specifically focusing on their potential for re-identifying individuals within uncontrolled environments. The experiments involve seven pre-trained backbone models and introduce a novel analytical approach by linking domain-related (HAR) and domain-unrelated (re-identification (re-ID)) tasks. Two key analyses addressing contextual information and encoding strategies reveal that maintaining the same encoding approach during training results in high task correlation while incorporating richer contextual information enhances performance. A notable outcome of this study is the comprehensive evaluation of a novel transformer-based architecture driven by a HAR backbone, which achieves a robust re-ID performance superior to state-of-the-art (SOTA). However, it faces challenges when other encoding schemes are applied, highlighting the role of the HAR classifier in performance variations.

Feature Mixture on Pre-Trained Model for Few-Shot Learning.

Few-shot learning (FSL) aims at recognizing a novel object under limited training samples. A robust feature extractor (backbone) can significantly improve the recognition performance of the FSL model. However, training an effective backbone is a challenging issue since 1) designing and validating structures of backbones are time-consuming and expensive processes, and 2) a backbone trained on the known (base) categories is more inclined to focus on the textures of the objects it learns, which is hard to describe the novel samples. To solve these problems, we propose a feature mixture operation on the pre-trained (fixed) features: 1) We replace a part of the values of the feature map from a novel category with the content of other feature maps to increase the generalizability and diversity of training samples, which avoids retraining a complex backbone with high computational costs. 2) We use the similarities between the features to constrain the mixture operation, which helps the classifier focus on the representations of the novel object where these representations are hidden in the features from the pre-trained backbone with biased training. Experimental studies on five benchmark datasets in both inductive and transductive settings demonstrate the effectiveness of our feature mixture (FM). Specifically, compared with the baseline on the Mini-ImageNet dataset, it achieves 3.8% and 4.2% accuracy improvements for 1 and 5 training samples, respectively. Additionally, the proposed mixture operation can be used to improve other existing FSL methods based on backbone training.

MAS-CL: An End-to-end Multi-atlas Supervised Contrastive Learning Framework for Brain ROI Segmentation.

Brain region-of-interest (ROI) segmentation with magnetic resonance (MR) images is a basic prerequisite step for brain analysis. The main problem with using deep learning for brain ROI segmentation is the lack of sufficient annotated data. To address this issue, in this paper, we propose a simple multi-atlas supervised contrastive learning framework (MAS-CL) for brain ROI segmentation with MR images in an end-to-end manner. Specifically, our MAS-CL framework mainly consists of two steps, including 1) a multi-atlas supervised contrastive learning method to learn the latent representation using a limited amount of voxel-level labeling brain MR images, and 2) brain ROI segmentation based on the pre-trained backbone using our MSA-CL method. Specifically, different from traditional contrastive learning, in our proposed method, we use multi-atlas supervised information to pre-train the backbone for learning the latent representation of input MR image, i.e., the correlation of each sample pair is defined by using the label maps of input MR image and atlas images. Then, we extend the pre-trained back-bone to segment brain ROI with MR images. We perform our proposed MAS-CL framework with five segmentation methods on LONI-LPBA40, IXI, OASIS, ADNI, and CC359 datasets for brain ROI segmentation with MR images. Various experimental results suggested that our proposed MAS-CL framework can significantly improve the segmentation performance on these five datasets.

Incrementally Adapting Pretrained Model Using Network Prior for Multi-Focus Image Fusion.

Multi-focus image fusion can fuse the clear parts of two or more source images captured at the same scene with different focal lengths into an all-in-focus image. On the one hand, previous supervised learning-based multi-focus image fusion methods relying on synthetic datasets have a clear distribution shift with real scenarios. On the other hand, unsupervised learning-based multi-focus image fusion methods can well adapt to the observed images but lack the general knowledge of defocus blur that can be learned from paired data. To avoid the problems of existing methods, this paper presents a novel multi-focus image fusion model by considering both the general knowledge brought by the supervised pretrained backbone and the extrinsic priors optimized on specific testing sample to improve the performance of image fusion. To be specific, the Incremental Network Prior Adaptation (INPA) framework is proposed to incrementally integrate features extracted from the pretrained strong baselines into a tiny prior network (6.9% parameters of the backbone network) to boost the performance for test samples. We evaluate our method on both synthetic and real-world public datasets (Lytro, MFI-WHU, and Real-MFF) and show that our method outperforms existing supervised learning-based methods and unsupervised learning based methods.

Iris Recognition Using an Enhanced Pre-Trained Backbone Based on Anti-Aliased CNNs

Iris Recognition Using an Enhanced Pre-Trained Backbone Based on Anti-Aliased CNNs