Auxiliary Image Research Articles

ACGRHA-Net: Accelerated Multi-Contrast MR Imaging with Adjacency Complementary Graph assisted Residual Hybrid Attention Network

Multi-contrast magnetic resonance (MR) imaging is an advanced technology used in medical diagnosis, but the long acquisition process can lead to patient discomfort and limit its broader application. Shortening acquisition time by undersampling k-space data introduces noticeable aliasing artifacts. To address this, we propose a method that reconstructs multi-contrast MR images from zero-filled data by utilizing a fully-sampled auxiliary contrast MR image as a prior to learn an adjacency complementary graph. This graph is then combined with a residual hybrid attention network, forming the adjacency complementary graph assisted residual hybrid attention network (ACGRHA-Net) for multi-contrast MR image reconstruction. Specifically, the optimal structural similarity is represented by a graph learned from the fully sampled auxiliary image, where the node features and adjacency matrices are designed to precisely capture structural information among different contrast images. This structural similarity enables effective fusion with the target image, improving the detail reconstruction. Additionally, a residual hybrid attention module is designed in parallel with the graph convolution network, allowing it to effectively capture key features and adaptively emphasize these important features in target contrast MR images. This strategy prioritizes crucial information while preserving shallow features, thereby achieving comprehensive feature fusion at deeper levels to enhance multi-contrast MR image reconstruction. Extensive experiments on the different datasets, using various sampling patterns and accelerated factors demonstrate that the proposed method outperforms the current state-of-the-art reconstruction methods.

Removing thick clouds in landsat 8 OLI images with cloudy time-series auxiliary data

Abstract Thick cloud occlusion severely limits the subsequent application of optical remote-sensing images. Existing mainstream cloud removal methods often use cloud-free homologous temporal images to provide auxiliary information. However, due to the limitation of cloud occlusion and satellite revisit cycle, the time interval between the cloud-free auxiliary image and the target cloudy image is often too large, and the possible land cover changes during the period increase the uncertainty of effective information on the auxiliary images. In this case, more accurate reconstructions may be obtained by using temporally closer and partially cloud-contaminated images. In this paper, a deep network, convLSTM, is developed for cloud removal (i.e., CR-convLSTM), which makes full use of the complementary effective information on partially cloud-contaminated time-series images. Experiments based on simulated clouds indicated that CR-convLSTM can obtain more accurate predictions than the classical cloud removal method MNSPI. CR-convLSTM is an effective cloud removal method with high computational efficiency while ensuring predicting accuracy.

Unveiling image source: Instance-level camera device linking via context-aware deep Siamese network

Unveiling the source of an image is one of the most effective ways to validate the originality, authenticity, and reliability in the field of digital forensics. Source camera device identification can identify the specific camera device used to take a photo under investigation. While great progress has been made by the photo-response non-uniformity (PRNU)-based methods over the past decade, the challenge of instance-level source camera device linking, which verifies whether two images in question were captured by the same camera device, remains significant. This challenge is mainly due to the absence of auxiliary images to construct a clean camera fingerprint for each camera, particularly dealing with small image sizes. To overcome this limitation, in this paper, we formulate the task of source device linking as a binary classification problem and propose a simple yet effective framework based on a context-aware deep Siamese network. We take advantage of a Siamese architecture to extract the intrinsic camera device-related noise patterns from a pair of image patches in parallel for comparisons without any auxiliary images. Moreover, a recurrent criss-cross group is utilized to aggregate contextual information in the noise residual maps to alleviate the problem that PRNU noise maps are easily contaminated by the additive noises from image contents. For reliable device linking, we employ a patch-selection strategy on a pair of test images to adaptively choose suitable image patch pairs according to image contents. The final decision of a pair of test images is obtained from the average similarity score of the selected image patch pairs. Compared with existing state-of-the-art methods, our proposed framework can achieve better performance on both the tasks of source camera identification and source device linking without any prior knowledge, i.e., reliable camera fingerprints, regardless of whether the camera devices are “seen” or “unseen” in the training stage. The experimental results on two standard image forensic datasets demonstrate that the proposed method not only shows robustness with respect to different image patch sizes and image quality degenerations, but also has a generalization ability across digital camera and smartphone devices.

SwiftWater: A lightweight SwiftFormer-based underwater image enhancement network with auxiliary prompt module

Due to variations in light attenuation underwater and the complexities of light propagation in aquatic environments, underwater images commonly suffer from degradation, such as color distortion, blur, and low contrast. These challenges pose significant obstacles to researchers studying marine resources. In recent years, numerous solutions have been proposed for underwater image enhancement (UIE) tasks, with deep learning-based methods, including those employing convolution and transformer structures, demonstrating promising performance. However, deep learning approaches often encounter issues related to large parameter sizes and substantial computational demands. Addressing these concerns, this paper introduces SwiftWater, a lightweight transformer-based network specifically designed for UIE tasks, featuring a U-Net style architecture. Leveraging the efficient SwiftFormer encoder as the network backbone, we introduce a Hybrid Squeeze and Excitation Block (HSEB) to dynamically adjust representations within skip connections. Additionally, we introduce an Auxiliary Prompt Module (APM) with a Lightweight Prompt Block (LPB) that utilizes the HSV color space of underwater images to enhance training. LPB within APM comprises two modules: the Prompt Generation Module (PGM) for prompt generation and the Lightweight Prompt Interaction Module (LPIM) for prompt-feature interaction, generating rich auxiliary image representations. Compared to existing mainstream UIE methods employing transformer structures, our proposed approach offers reduced parameter counts while achieving state-of-the-art (SOTA) performance across both quantitative and qualitative metrics.

Aligning the domains in cross domain model inversion attack

Model Inversion Attack reconstructs confidential training dataset from a target deep learning model. Most of the existing methods assume the adversary has an auxiliary dataset that has similar distribution with the private dataset. However, this assumption does not always hold in real-world scenarios. Since the private dataset is unknown, the domain divergence between the auxiliary dataset and the private dataset is inevitable. In this paper, we use Cross Domain Model Inversion Attack to represent the distribution divergence scenario in MIA. With the distribution divergence between the private images and auxiliary images, the distribution between the feature vectors of the private images and those of the auxiliary images is also different. Moreover, the outputted prediction vectors of the auxiliary images are also misclassified. The inversion attack is thus hard to be performed. We perform both the feature vector inversion task and prediction vector inversion task in this cross domain setting. For feature vector inversion, Domain Alignment MIA (DA-MIA) is proposed. While performing the reconstruction task, DA-MIA aligns the feature vectors of auxiliary images with the feature vectors of private images in an adversarial manner to mitigate the domain divergence between them. Thus, semantically meaningful images can be reconstructed. For prediction vector inversion, we further introduce an auxiliary classifier and propose Domain Alignment MIA with Auxiliary Classifier (DA-MIA-AC). The auxiliary classifier is pretrained by the auxiliary dataset and fine-tuned during the adversarial training stage. Thus, the misclassification problem caused by domain divergence can be solved, and the images can be reconstructed correctly. Various experiments are performed to show the advancement of our methods, the results show that DA-MIA can improve the SSIM score of the reconstructed images for up to 191%, DA-MIA-AC can increase the classification accuracy score of the reconstructed images from 9.18% to 81.32% in Cross Domain Model Inversion Attack.

CNN-Based Multi-Factor Authentication System for Mobile Devices Using Faces and Passwords

Multi-factor authentication (MFA) is a system for authenticating an individual’s identity using two or more pieces of data (known as factors). The reason for using more than two factors is to further strengthen security through the use of additional data for identity authentication. Sequential MFA requires a number of steps to be followed in sequence for authentication; for example, with three factors, the system requires three authentication steps. In this case, to proceed with MFA using a deep learning approach, three artificial neural networks (ANNs) are needed. In contrast, in parallel MFA, the authentication steps are processed simultaneously. This means that processing is possible with only one ANN. A convolutional neural network (CNN) is a method for learning images through the use of convolutional layers, and researchers have proposed several systems for MFA using CNNs in which various modalities have been employed, such as images, handwritten text for authentication, and multi-image data for machine learning of facial emotion. This study proposes a CNN-based parallel MFA system that uses concatenation. The three factors used for learning are a face image, an image converted from a password, and a specific image designated by the user. In addition, a secure password image is created at different bit-positions, enabling the user to securely hide their password information. Furthermore, users designate a specific image other than their face as an auxiliary image, which could be a photo of their pet dog or favorite fruit, or an image of one of their possessions, such as a car. In this way, authentication is rendered possible through learning the three factors—that is, the face, password, and specific auxiliary image—using the CNN. The contribution that this study makes to the existing body of knowledge is demonstrating that the development of an MFA system using a lightweight, mobile, multi-factor CNN (MMCNN), which can even be used in mobile devices due to its low number of parameters, is possible. Furthermore, an algorithm that can securely transform a text password into an image is proposed, and it is demonstrated that the three considered factors have the same weight of information for authentication based on the false acceptance rate (FAR) values experimentally obtained with the proposed system.

TTMRI: Multislice texture transformer network for undersampled MRI reconstruction

AbstractMagnetic resonance imaging (MRI) is a non‐interposition imaging technique that provides rich anatomical and physiological information. Yet it is limited by the long imaging time. Recently, deep neural networks have shown potential to significantly accelerate MRI. However, most of these approaches ignore the correlation between adjacent slices in MRI image sequences. In addition, the existing deep learning‐based methods for MRI are mainly based on convolutional neural networks (CNNs). They fail to capture long‐distance dependencies due to the small receptive field. Inspired by the feature similarity in adjacent slices and impressive performance of Transformer for exploiting the long‐distance dependencies, a novel multislice texture transformer network is presented for undersampled MRI reconstruction (TTMRI). Specifically, the proposed TTMRI is consisted of four modules, namely the texture extraction, correlation calculation, texture transfer and texture synthesis. It takes three adjacent slices as inputs, in which the middle one is the target image to be reconstructed, and the other two are auxiliary images. The multiscale features are extracted by the texture extraction module and their inter‐dependencies are calculated by the correlation calculation module, respectively. Then the relevant features are transferred by the texture transfer module and fused by the texture synthesis module. By considering inter‐slice correlations and leveraging the Transformer architecture, the joint feature learning across target and adjacent slices are encouraged. Moreover, TTMRI can be stacked with multiple layers to recover more texture information at different levels. Extensive experiments demonstrate that the proposed TTMRI outperforms other state‐of‐the‐art methods in both quantitative and qualitative evaluationsions.

Learning Multi-Modal Cross-Scale Deformable Transformer Network for Unregistered Hyperspectral Image Super-resolution

Hyperspectral image super-resolution (HSI-SR) is a technology to improve the spatial resolution of HSI. Existing fusion-based SR methods have shown great performance, but still have some problems as follows: 1) existing methods assume that the auxiliary image providing spatial information is strictly registered with the HSI, but images are difficult to be registered finely due to the shooting platforms, shooting viewpoints and the influence of atmospheric turbulence; 2) most of the methods are based on convolutional neural networks (CNNs), which is effective for local features but cannot utilize the global features. To this end, we propose a multi-modal cross-scale deformable transformer network (M2DTN) to achieve unregistered HSI-SR. Specifically, we formulate a spectrum-preserving based spatial-guided registration-SR unified model (SSRU) from the view of the realistic degradation scenarios. According to SSRU, we propose multi-modal registration deformable module (MMRD) to align features between different modalities by deformation field. In order to efficiently utilize the unique information between different modals, we design multi-scale feature transformer (MSFT) to emphasize the spatial-spectral features at different scales. In addition, we propose the cross-scale feature aggregation module (CSFA) to accurately reconstruct the HSI by aggregating feature information at different scales. Experiments show that M2DTN outperforms the-state-of-the-art HSI-SR methods. Code is obtainable at https://github.com/Jiahuiqu/M2DTN.

Triplet-constrained deep hashing for chest X-ray image retrieval in COVID-19 assessment

Radiology images of the chest, such as computer tomography scans and X-rays, have been prominently used in computer-aided COVID-19 analysis. Learning-based radiology image retrieval has attracted increasing attention recently, which generally involves image feature extraction and finding matches in extensive image databases based on query images. Many deep hashing methods have been developed for chest radiology image search due to the high efficiency of retrieval using hash codes. However, they often overlook the complex triple associations between images; that is, images belonging to the same category tend to share similar characteristics and vice versa. To this end, we develop a triplet-constrained deep hashing (TCDH) framework for chest radiology image retrieval to facilitate automated analysis of COVID-19. The TCDH consists of two phases, including (a) feature extraction and (b) image retrieval. For feature extraction, we have introduced a triplet constraint and an image reconstruction task to enhance discriminative ability of learned features, and these features are then converted into binary hash codes to capture semantic information. Specifically, the triplet constraint is designed to pull closer samples within the same category and push apart samples from different categories. Additionally, an auxiliary image reconstruction task is employed during feature extraction to help effectively capture anatomical structures of images. For image retrieval, we utilize learned hash codes to conduct searches for medical images. Extensive experiments on 30,386 chest X-ray images demonstrate the superiority of the proposed method over several state-of-the-art approaches in automated image search. The code is now available online.

Style-KD: Class-imbalanced medical image classification via style knowledge distillation

Problem:In the medical domain, obtaining training images on a large scale is difficult due to privacy and cost issues, and as further disease incidence varies widely, class imbalance has been a critical problem. Importantly, securing samples for rare diseases will always be difficult, but a diagnosis model should be able to predict the diseased samples accurately. When a model is trained on imbalanced data, it is expected that (1) the model only exhibits biased results for the majority classes, and (2) the recognition performance for minority classes will decline rapidly, resulting in low specificity. Objective:We aim to improve the prediction performance of such few sample classes using our auxiliary data from an efficient generation model and style knowledge distillation. In this paper, we introduce a style knowledge distillation method to address the class imbalance problem. Methods:The proposed method generates auxiliary images in which the style of a training sample is transferred. We train our model with balanced training samples by distilling the styles of auxiliary images. Conclusion:We demonstrate that it considerably increases the accuracy of classes with few samples, notably without adversely impacting classes with large samples, using the APTOS2019, and ISIC2018. This approach is straightforward but very effective for medical image datasets with skewed class distributions.

The article analyzes female images, auxiliary images and the anti-heroic pole: the main and auxiliary images in the Koroglu epic

The article analyzes female images, auxiliary images and the anti-heroic pole: the main and auxiliary images in the Koroglu epic

Single-Image SVBRDF Estimation Using Auxiliary Renderings as Intermediate Targets.

Recently, single-image SVBRDF capture is formulated as a regression problem, which uses a network to infer four SVBRDF maps from a flash-lit image. However, the accuracy is still not satisfactory since previous approaches usually adopt endto-end inference strategies. To mitigate the challenge, we propose "auxiliary renderings" as the intermediate regression targets, through which we divide the original end-to-end regression task into several easier sub-tasks, thus achieving better inference accuracy. Our contributions are threefold. First, we design three (or two pairs of) auxiliary renderings and summarize the motivations behind the designs. By our design, the auxiliary images are bumpiness-flattened or highlight-removed, containing disentangled visual cues about the final SVBRDF maps and can be easily transformed to the final maps. Second, to help estimate the auxiliary targets from the input image, we propose two mask images including a bumpiness mask and a highlight mask. Our method thus first infers mask images, then with the help of the mask images infers auxiliary renderings, and finally transforms the auxiliary images to SVBRDF maps. Third, we propose backbone UNets to infer mask images, and gated deformable UNets for estimating auxiliary targets. Thanks to the well designed networks and intermediate images, our method outputs better SVBRDF maps than previous approaches, validated by the extensive comparisonal and ablation experiments.

Subspace Dynamic Combined Sparsity-Based Hyper-Sharpening for Diverse Auxiliary Images

Subspace Dynamic Combined Sparsity-Based Hyper-Sharpening for Diverse Auxiliary Images

PDA: Progressive Domain Adaptation for Semantic Segmentation

The unsupervised domain adaptation semantic segmentation task is challenging due to the distribution shift problem between the source and the target domains. In this paper, we provide a novel perspective to address this problem. Currently, in the literature, it is shown that output-level domain adaptation networks (OL-DAN) can generate outputs with smaller distribution shift. Motivated by this phenomenon, a Progressive Domain Adaptation (PDA) framework is proposed, which uses the outputs generated by OL-DAN as the auxiliary domain images to progressively align the distribution between the source and target domains. Unlike existing two-stage input-level domain adaptation methods which use an image translation network as a standalone model to generate the auxiliary domain images, the PDA is an end-to-end framework that contains an OL-DAN and a domain fusion domain adaptation network (DF-DAN). The OL-DAN aims to gradually generate the outputs with smaller distribution shift and more accurate semantic structures as the auxiliary domain images in every iteration optimization. The DF-DAN is proposed to further mine the domain-invariant information from the auxiliary images and then fuse the features learned from the original images and the auxiliary images to obtain a richer representation. Finally, the distribution between the source and target domains is aligned to optimize the OL-DAN and DF-DAN. Experiments demonstrate that the proposed PDA achieves superior performance on three cross-domain semantic segmentation benchmarks.

Asymmetric Dual-Direction Quasi-Recursive Network for Single Hyperspectral Image Super-Resolution

Single hyperspectral image super-resolution aims to reconstruct a high-resolution hyperspectral image from a low-resolution one, which does not use any auxiliary images. For now, existing super-resolution methods often ignore the difference between the features of neighbor and non-neighbor spectral bands, leaving the feature exploration untargeted. As a result, the complementary information of such bands has not been effectively exploited. To do so, we propose an asymmetric dual-direction quasi-recursive network for single hyperspectral image super-resolution, which separately explores the features among neighbor and non-neighbor bands via forward and backward units. By considering the high similarity among neighbor bands, each forward unit thoroughly exploits spatial-spectral features among such bands through two kinds of correspondence aggregation modules. It also preserves a spectral structure by a spectral band grouping strategy and a spatial-spectral consistency module. Owning to the inconsecutive spectra among non-neighbor bands, backward units focus on extracting spatial features in such bands. With the aid of a global feature context fusion module, the information of global non-neighbor context and neighbor bands are adaptively fused, thus improving information completeness and complementarity. Experimental results reported for natural and remote sensing hyperspectral image datasets demonstrate the proposed network not only outperforms the state-of-the-art methods in terms of reconstruction quality and noise suppression, but also requires a smaller memory footprint.

Deep rigid registration for slice-to-volume in real time

Slice-to-volume registration that achieves high accuracy in real time is one of the enabling technologies for clinical imaging scenarios. Recently, deep learning methods have been explored to significantly improve the accuracy and efficiency of the registration. Nevertheless, such a 2D/3D registration problem is very challenging due to several considerable barriers including highly computational cost brought by dense sampling in six dimensional parameter space and significant modal appearance differences. We proposed a Differentiable resampling based Slice-to-Volume Registration network, which can achieve real-time and accurate registration in both mono- and multi-modal scenarios. The proposed network learns the out-of-plane transformation parameters in the spherical coordinates that decide the content and is invariant to the remaining in-plane parameters by feeding 2D rigid transformed data on-the-fly, thus reducing the training sample size remarkably. To further improve the performance, we introduce an auxiliary image similarity connected by differentiable resampling. For multi-modal scenarios, we optionally include the enhanced modality independent neighborhood descriptor to unify different modalities into common space. Training size is considerably reduced to 30k per dataset with half of all possible orientations and 80% of the volume space covered while the average registration error drops to 1.84 mm and 6.49 mm respectively for mono- and multi-modal experiments. Extensive experiments show the proposed methods’ superiority over existing deep learning methods for real-time 2D/3D rigid registration in both mono- and multi-modal settings.

Image recognition of sports dance teaching and auxiliary function data verification based on neural network algorithm

Nowadays, physical dance is widely spread in the society as an emerging sport. Dance movement is favored by people because of its unique social function and fitness effect. For dance teaching, dance movement analysis can help optimize and improve the existing dance movements and the understanding and inheritance of traditional dance movements. With the rise of online teaching, intelligent identification and analysis of dance movements can promote the better development of sports dance teaching. However, the relevant research in this area is still very scarce. As the basis of this kind of research, there is an urgent need for dance motion recognition technology. Based on this background, this paper by introducing neural network algorithm for dance teaching sports image special recognition design, the algorithm can combine feature extraction technology to process video, extract the dance movements in the target data set, then for the extraction of cumulative feature extraction operation, in order to accumulate all the collected target features, so as to further complete the gradient histogram acquisition. Through the design experimental test, the cumulative feature image extraction results obtained through the algorithm are obviously better than the traditional image recognition results, so the design rationality and effectiveness of the algorithm are proved, and the sports dance teaching can be specially assisted. This paper designs an effective auxiliary image recognition algorithm by introducing the neural network algorithm into the field of sports dance teaching.

Incorporating inconsistent auxiliary images in haze removal of very high resolution images

Incorporating inconsistent auxiliary images in haze removal of very high resolution images

Combining global receptive field and spatial spectral information for single-image hyperspectral super-resolution

Single-image hyperspectral super-resolution has poorer reconstruction performance in the spatial dimension than fused-image hyperspectral super-resolution due to the lack of auxiliary images. Some studies have attempted to use 3D convolution to explore hidden features between spatial spectra to enhance spatial details. However, either 2D or 3D convolution, the obtained receptive fields are limited, ignoring the effect of global spatial information on hyperspectral image reconstruction, and cannot be used for long-range dependent modeling. Therefore, we make the first attempt to combine Transformer with 3D convolution in single-image hyperspectral super-resolution and propose a 3D convolution and Transformer hyperspectral super-resolution (3D-THSR) network, which explores the hidden information between space and spectra while obtaining the global receptive field of space. Specifically, the Transformer module is used for feature extraction to enhance the learning ability of global spatial information and long-distance features. In addition, the 3D convolution module is embedded in the Transformer module to extract the information between different spectral bands by fusing the spectral and spatial dimensions. Finally, we design to train the network by using three loss functions to reduce the distortion spectrum and ensure the spectral band purity. Compared with other single-image hyperspectral methods by six hyperspectral evaluation metrics, spatial detail image, spectral error line map, and ablation study, it is proved that the proposed method achieves better hyperspectral super-resolution reconstruction performance.

Hyperspectral Image Super-resolution via Knowledge-Driven Deep Unrolling and Transformer Embedded Convolutional Recurrent Neural Network.

Hyperspectral (HS) imaging has been widely used in various real application problems. However, due to the hardware limitations, the obtained HS images usually have low spatial resolution, which could obviously degrade their performance. Through fusing a low spatial resolution HS image with a high spatial resolution auxiliary image (e.g., multispectral, RGB or panchromatic image), the so-called HS image fusion has underpinned much of recent progress in enhancing the spatial resolution of HS image. Nonetheless, a corresponding well registered auxiliary image cannot always be available in some real situations. To remedy this issue, we propose in this paper a newly single HS image super-resolution method based on a novel knowledge-driven deep unrolling technique. Precisely, we first propose a maximum a posterior based energy model with implicit priors, which can be solved by alternating optimization to determine an elementary iteration mechanism. We then unroll such iteration mechanism with an ingenious Transformer embedded convolutional recurrent neural network in which two structural designs are integrated. That is, the vision Transformer and 3D convolution learn the implicit spatial-spectral priors, and the recurrent hidden connections over iterations model the recurrence of the iterative reconstruction stages. Thus, an effective knowledge-driven, end-to-end and data-dependent HS image super-resolution framework can be successfully attained. Extensive experiments on three HS image datasets demonstrate the superiority of the proposed method over several state-of-the-art HS image super-resolution methods.