Multiple Reference Images Research Articles

Temporal relation transformer for robust visual tracking with dual-memory learning

Recently, transformer trackers mostly associate multiple reference images with the search area to adapt to the changing appearance of the target. However, they ignore the learned cross-relations between the target and surrounding, leading to difficulties in building coherent contextual models for specific target instances. This paper presents a Temporal Relation Transformer Tracker (TRTT) for robust visual tracking, providing a concise approach to modeling temporal relations by dual target memory learning. Specifically, a temporal relation transformer network generates paired memories based on static and dynamic templates, which are reinforced interactively. The memory contains implicit relation hints that capture the relations between the tracked object and its immediate surroundings. More importantly, to ensure consistency of target instance identities between frames, the relation hints from previous frames are transferred to the current frame for merging temporal contextual attention. Our method also incorporates mechanisms for reusing favorable cross-relations and instance-specific features, thereby overcoming background interference in complex spatio-temporal interactions through a sequential constraint. Furthermore, we design a memory token sparsification method that leverages the key points of the target to eliminate interferences and optimize attention calculations. Extensive experiments demonstrate that our method surpasses advanced trackers on 8 challenging benchmarks while maintaining real-time running speed.

Centre‐loss—A preferred class verification approach over sample‐to‐sample in self‐checkout products datasets

AbstractSiamese networks excel at comparing two images, serving as an effective class verification technique for a single‐per‐class reference image. However, when multiple reference images are present, Siamese verification necessitates multiple comparisons and aggregation, often unpractical at inference. The Centre‐Loss approach, proposed in this research, solves a class verification task more efficiently, using a single forward‐pass during inference, than sample‐to‐sample approaches. Optimising a Centre‐Loss function learns class centres and minimises intra‐class distances in latent space. The authors compared verification accuracy using Centre‐Loss against aggregated Siamese when other hyperparameters (such as neural network backbone and distance type) are the same. Experiments were performed to contrast the ubiquitous Euclidean against other distance types to discover the optimum Centre‐Loss layer, its size, and Centre‐Loss weight. In optimal architecture, the Centre‐Loss layer is connected to the penultimate layer, calculates Euclidean distance, and its size depends on distance type. The Centre‐Loss method was validated on the Self‐Checkout products and Fruits 360 image datasets. Centre‐Loss comparable accuracy and lesser complexity make it a preferred approach over sample‐to‐sample for the class verification task, when the number of reference image per class is high and inference speed is a factor, such as in self‐checkouts.

FPRF: Feed-Forward Photorealistic Style Transfer of Large-Scale 3D Neural Radiance Fields

We present FPRF, a feed-forward photorealistic style transfer method for large-scale 3D neural radiance fields. FPRF stylizes large-scale 3D scenes with arbitrary, multiple style reference images without additional optimization while preserving multi-view appearance consistency. Prior arts required tedious per-style/-scene optimization and were limited to small-scale 3D scenes. FPRF efficiently stylizes large-scale 3D scenes by introducing a style-decomposed 3D neural radiance field, which inherits AdaIN’s feed-forward stylization machinery, supporting arbitrary style reference images. Furthermore, FPRF supports multi-reference stylization with the semantic correspondence matching and local AdaIN, which adds diverse user control for 3D scene styles. FPRF also preserves multi-view consistency by applying semantic matching and style transfer processes directly onto queried features in 3D space. In experiments, we demonstrate that FPRF achieves favorable photorealistic quality 3D scene stylization for large-scale scenes with diverse reference images.

A siamese-based verification system for open-set architecture attribution of synthetic images

Despite the wide variety of methods developed for synthetic image attribution, most of them can only attribute images generated by models or architectures included in the training set and do not work with unknown architectures, hindering their applicability in real-world scenarios. In this paper, we propose a verification framework that relies on a Siamese Network to address the problem of open-set attribution of synthetic images to the architecture that generated them. We consider two different settings. In the first setting, the system determines whether two images have been produced by the same generative architecture or not. In the second setting, the system verifies a claim about the architecture used to generate a synthetic image, utilizing one or multiple reference images generated by the claimed architecture. The main strength of the proposed system is its ability to operate in both closed and open-set scenarios so that the input images, either the query and reference images, can belong to the architectures considered during training or not. Experimental evaluations encompassing various generative architectures such as GANs, diffusion models, and transformers, focusing on synthetic face image generation, confirm the excellent performance of our method in both closed and open-set settings, as well as its strong generalization capabilities.

Ultracold atomic imaging based on enhanced fringe removal method

Absorption imaging is the foundation for quantitative measurements in experiments on ultracold atoms. This technique mainly involves capturing images of both the probing light field and the atom absorption light field. In this process, the unavoidable jitter of the probing light introduces imaging noise of fringe patterns into the atomic optical density distribution OD. In conventional fringe removal algorithms, this type of noise can be normalized by constructing an optimal reference image from multiple reference images that have been actually taken, which shares similar fringe patterns to an absorption image (Fig. (a)). Although this method works well in the region without atomic signal, they often overlook the modulation of the noise signal due to atomic absorption effects, leading to persistent residual fringes on the atom clouds. This problem becomes more pronounced with atomic density increasing. Here, we propose an enhanced fringe removal algorithm that takes into account the effects of atomic absorption, and actively modulates the intensity of the noise signal in the reference image constructed by conventional fringe removal algorithms (Fig. (b)), effectively preventing the residual fringes from forming, thus significantly improving the signal-to-noise ratio of the atomic images. When applied to the absorption imaging of homogeneous Fermi gases with high density, as shown in Fig. (d), this new algorithm successfully reduces the relative standard deviation of optical depth characterizing atomic density fluctuations by approximately 37%, which is about 3 times the relative standard deviation by conventional algorithm. Three subgraphs in Fig. (e) show the optical depth distribution at corresponding data points labeled by square boxes in Fig. (d). Furthermore, we also use this technique to quantitatively determine the second sound in the unitary Fermi superfluid of <sup>6</sup>Li atoms. Compared with conventional fringe removal methods, our new algorithm increases the correlation function’s contrast of the density wave nearly 4 times, therefore enhancing the intensity of the density response spectrum by approximately 15% with half the measured standard error, paving the way for quantitatively determining the speed and attenuation of the second sound. These results demonstrate that the enhanced fringe removal algorithm not only effectively suppresses fringe noise, but also facilitates the identification and detection of important physical phenomena in high-density atomic systems, such as some collective excitations and new quantum phases.

A Flexible Spatiotemporal Thick Cloud Removal Method with Low Requirements for Reference Images

Thick cloud and shadows have a significant impact on the availability of optical remote sensing data. Although various methods have been proposed to address this issue, they still have some limitations. First, most approaches rely on a single clear reference image as complementary information, which becomes challenging when the target image has large missing areas. Secondly, the existing methods that can utilize multiple reference images require the complementary data to have high temporal correlation, which is not suitable for situations where the difference between the reference image and the target image is large. To overcome these limitations, a flexible spatiotemporal deep learning framework based on generative adversarial networks is proposed for thick cloud removal, which allows for the use of three arbitrary temporal images as references. The framework incorporates a three-step encoder that can leverage the uncontaminated information from the target image to assimilate the reference images, enhancing the model’s ability to handle reference images with diverse temporal differences. A series of simulated and real experiments on Landsat 8 and Sentinel 2 data is performed to demonstrate the effectiveness of the proposed method. The proposed method is especially applicable to small/large-scale regions with reference images that are significantly different from the target image.

Semi-supervised super-resolution of diffusion-weighted images based on multiple references.

Spatial resolution of diffusion tensor images is usually compromised to accelerate the acquisitions, and the state-of-the-art (SOTA) image super-resolution (SR) reconstruction methods are commonly based on supervised learning models. Considering that matched low-resolution (LR) and high-resolution (HR) diffusion-weighted (DW) image pairs are not readily available, we propose a semi-supervised DW image SR reconstruction method based on multiple references (MRSR) extracted from other subjects. In MRSR, the prior information of multiple HR reference images is migrated into a residual-like network to assist SR reconstruction of DW images, and a CycleGAN-based semi-supervised strategy is used to train the network with 30% matched and 70% unmatched LR-HR image pairs. We evaluate the performance of the MRSR by comparing against SOTA methods on an HCP dataset in terms of the quality of reconstructed DW images and diffusion metrics. MRSR achieves the best performance, with the mean PSNR/SSIM of DW images being improved by at least 14.3%/28.8% and 1%/1.4% respectively relative to SOTA unsupervised and supervised learning methods, and with the fiber orientations deviating from the ground truth by about 6.28° on average, the RMSEs of fractional anisotropy, mean diffusivity, axial diffusivity and radial diffusivity being 3.0%, 4.6%, 5.7% and 4.5% respectively relative to the ground truth. We validate the effectiveness of the proposed network structure, multiple-reference and CycleGAN-based semi-supervised learning strategies for SR reconstruction of diffusion tensor images through the ablation studies. The proposed method allows us to achieve SR reconstruction for diffusion tensor images with a limited number of matched image pairs.

Advanced Reference-Constrained Image Restoration Algorithm

Image restoration is always a long-term problem in low-level computer vision, which has many practical applications. In this paper, we devote to reconstructing a single image from compressive measurements with the assistance of one or multiple reference images, by using the similarity prior information of them. More specifically, we propose to bring in the concept of Regularization by Denoising, which leverages existing denoising engine to regularize inverse problems, and illustrate how the reference images and this powerful algorithm can be merged into a new highly effective recovery approach, which is named Reference Image Constrained Regularization by Denoising for solving various image restoration tasks. Since the similarity between the reference and the ground truth is not guaranteed, or the degree of the similarity is different, we extend the proposed algorithm by adding adaptive weighting matrix to the difference between the target image and the reference images to further improve the performance. We evaluate experimentally the proposed algorithm in the image deblurring and single image super-resolution problems. Furthermore, we also carry out our method on video super-resolution task, in the case of multiple reference images available. We compare the advanced reference-constrained image restoration algorithm with previous methods about reconstruction accuracy and computation cost. The experimental results show that the proposed method outperforms state-of-the-art algorithms, including Regularization by Denoising, in terms of both the Peak Signal to Noise Ratio and the visual performance.

Robust Isometric Non-Rigid Structure-From-Motion.

Non-Rigid Structure-from-Motion (NRSfM) reconstructs a deformable 3D object from keypoint correspondences established between monocular 2D images. Current NRSfM methods lack statistical robustness, which is the ability to cope with correspondence errors. This prevents one to use automatically established correspondences, which are prone to errors, thereby strongly limiting the scope of NRSfM. We propose a three-step automatic pipeline to solve NRSfM robustly by exploiting isometry. Step (i) computes the optical flow from correspondences, step (ii) reconstructs each 3D point's normal vector using multiple reference images and integrates them to form surfaces with the best reference and step (iii) rejects the 3D points that break isometry in their local neighborhood. Importantly, each step is designed to discard or flag erroneous correspondences. Our contributions include the robustification of optical flow by warp estimation, new fast analytic solutions to local normal reconstruction and their robustification, and a new scale-independent measure of 3D local isometric coherence. Experimental results show that our robust NRSfM method consistently outperforms existing methods on both synthetic and real datasets.

Light Field Image Super-Resolution Based on Multilevel Structures

Light field (LF) images suffer from low spatial resolution due to the trade-off between angular and spatial resolutions. Thus, spatial super-resolution (SR) of LF images is an essential task to obtain high-quality LF images. However, the existing SR networks still have limitations, since they exploit only single-level features to use sub-pixel information in LF images. In this paper, we proposed a light field super-resolution (LFSR) network to effectively improve the spatial resolution of light field images. The proposed network takes one target image and its 8-neighboring images for references. We construct multi-level structures for the proposed network to effectively estimate and mix sub-pixel information in reference images. The proposed network is composed of a feature extractor, a feature warping module, a feature mixing module, and a upscaling module. The feature extractor provides multi-level features for SR and offsets to the feature warping module to obtain aligned features for multiple reference images. The feature mixing module mixes multiple aligned features based on the similarity between the target and reference images to obtain multi-level mixed features. Finally, the upscaling module generates a high-resolution residual image using the multi-level mixed features. Experimental results demonstrate the proposed network outperforms the state-of-the-art methods on various light field datasets. The pre-trained model and source codes are available at https://github.com/Hwa-Jong/LF_MLS.

Barbershop

Seamlessly blending features from multiple images is extremely challenging because of complex relationships in lighting, geometry, and partial occlusion which cause coupling between different parts of the image. Even though recent work on GANs enables synthesis of realistic hair or faces, it remains difficult to combine them into a single, coherent, and plausible image rather than a disjointed set of image patches. We present a novel solution to image blending, particularly for the problem of hairstyle transfer, based on GAN-inversion. We propose a novel latent space for image blending which is better at preserving detail and encoding spatial information, and propose a new GAN-embedding algorithm which is able to slightly modify images to conform to a common segmentation mask. Our novel representation enables the transfer of the visual properties from multiple reference images including specific details such as moles and wrinkles, and because we do image blending in a latent-space we are able to synthesize images that are coherent. Our approach avoids blending artifacts present in other approaches and finds a globally consistent image. Our results demonstrate a significant improvement over the current state of the art in a user study, with users preferring our blending solution over 95 percent of the time. Source code for the new approach is available at https://zpdesu.github.io/Barbershop.

Co-attention dictionary network for weakly-supervised semantic segmentation

In this paper, we propose the co-attention dictionary network (CODNet) for weakly-supervised semantic segmentation using only image-level class labels. The CODNet model exploits extra semantic information by jointly leveraging a pair of samples with common semantics through co-attention rather than processing them independently. The inter-sample similarities of spatially distributed deep features are computed to merge reference features through non-local connections. To discover similar patterns regardless of appearance variations, we propose to extract image representations by equipping the neural networks with dictionary learning which provides the universal basis elements for different images. Based on the CODNet model, we propose a multi-reference class activation map (MR-CAM) algorithm which generates semantic segmentation masks for a target image by jointly merging semantic cues from multiple reference images. Experimental results on the PASCAL VOC 2012 and MSCOCO benchmark datasets for weakly-supervised semantic segmentation show that the proposed algorithm performs favorably against the state-of-the-art methods.

Exploiting High Geopositioning Accuracy of SAR Data to Obtain Accurate Geometric Orientation of Optical Satellite Images

Accurate geopositioning of optical satellite imagery is a fundamental step for many photogrammetric applications. Considering the imaging principle and data processing manner, SAR satellites can achieve high geopositioning accuracy. Therefore, SAR data can be a reliable source for providing control information in the orientation of optical satellite images. This paper proposes a practical solution for an accurate orientation of optical satellite images using SAR reference images to take advantage of the merits of SAR data. Firstly, we propose an accurate and robust multimodal image matching method to match the SAR and optical satellite images. This approach includes the development of a new structural-based multimodal applicable feature descriptor that employs angle-weighted oriented gradients (AWOGs) and the utilization of a three-dimensional phase correlation similarity measure. Secondly, we put forward a general optical satellite imagery orientation framework based on multiple SAR reference images, which uses the matches of the SAR and optical satellite images as virtual control points. A large number of experiments not only demonstrate the superiority of the proposed matching method compared to the state-of-the-art methods but also prove the effectiveness of the proposed orientation framework. In particular, the matching performance is improved by about 17% compared with the latest multimodal image matching method, namely, CFOG, and the geopositioning accuracy of optical satellite images is improved, from more than 200 to around 8 m.

Noise Reduction for Single-Shot Grating-Based Phase-Contrast Imaging at an X-ray Backlighter.

X-ray backlighters allow the capture of sharp images of fast dynamic processes due to extremely short exposure times. Moiré imaging enables simultaneously measuring the absorption and differential phase-contrast (DPC) of these processes. Acquiring images with one single shot limits the X-ray photon flux, which can result in noisy images. Increasing the photon statistics by repeating the experiment to gain the same image is not possible if the investigated processes are dynamic and chaotic. Furthermore, to reconstruct the DPC and transmission image, an additional measurement captured in absence of the object is required. For these reference measurements, shot-to-shot fluctuations in X-ray spectra and a source position complicate the averaging of several reference images for noise reduction. Here, two approaches of processing multiple reference images in combination with one single object image are evaluated regarding the image quality. We found that with only five reference images, the contrast-to-noise ratio can be improved by approximately 13% in the DPC image. This promises improvements for short-exposure single-shot acquisitions of rapid processes, such as laser-produced plasma shock-waves in high-energy density experiments at backlighter X-ray sources such as the PHELIX high-power laser facility.

Multi-exposure X-ray image fusion quality evaluation based on CSF and gradient amplitude similarity.

Due to the limitation of dynamic range of the imaging device, the fixed-voltage X-ray images often produce overexposed or underexposed regions. Some structure information of the composite steel component is lost. This problem can be solved by fusing the multi-exposure X-ray images taken by using different voltages in order to produce images with more detailed structures or information. Due to the lack of research on multi-exposure X-ray image fusion technology, there is no evaluation method specially for multi-exposure X-ray image fusion. For the multi-exposure X-ray fusion images obtained by different fusion algorithms may have problems such as the detail loss and structure disorder. To address these problems, this study proposes a new multi-exposure X-ray image fusion quality evaluation method based on contrast sensitivity function (CSF) and gradient amplitude similarity. First, with the idea of information fusion, multiple reference images are fused into a new reference image. Next, the gradient amplitude similarity between the new reference image and the test image is calculated. Then, the whole evaluation value can be obtained by weighting CSF. In the experiments of MEF Database, the SROCC of the proposed algorithm is about 0.8914, and the PLCC is about 0.9287, which shows that the proposed algorithm is more consistent with subjective perception in MEF Database. Thus, this study demonstrates a new objective evaluation method, which generates the results that are consistent with the subjective feelings of human eyes.

Intelligent Reference Curation for Visual Place Recognition Via Bayesian Selective Fusion

A key challenge in visual place recognition (VPR) is recognizing places despite drastic visual appearance changes due to factors such as time of day, season, weather or lighting conditions. Numerous approaches based on deep-learnt image descriptors, sequence matching, domain translation, and probabilistic localization have had success in addressing this challenge, but most rely on the availability of carefully curated representative reference images of the possible places. In this letter, we propose a novel approach, dubbed Bayesian Selective Fusion, for actively selecting and fusing informative reference images to determine the best place match for a given query image. The selective element of our approach avoids the counterproductive fusion of every reference image and enables the dynamic selection of informative reference images in environments with changing visual conditions (such as indoors with flickering lights, outdoors during sunshowers or over the day-night cycle). The probabilistic element of our approach provides a means of fusing multiple reference images that accounts for their varying uncertainty via a novel training-free likelihood function for VPR. On difficult query images from two benchmark datasets, we demonstrate that our approach matches and exceeds the performance of several alternative fusion approaches along with state-of-the-art techniques that are provided with prior (unfair) knowledge of the best reference images. Our approach is well suited for long-term robot autonomy where dynamic visual environments are commonplace since it is training-free, descriptor-agnostic, and complements existing techniques such as sequence matching.

Studies on deformation measurement with non-fixed camera using digital image correlation method

A digital image correlation technique with non-fixed camera is developed for deformation measurement in the field since it is difficult to fix cameras when the deformations of a large number of areas need to be measured. Multiple reference and target images of the specimen surface and a feature template are captured from different orientations and corrected before the correlation calculation. The relative positions of the specimen surface and the camera are obtained by the camera calibration method. The specimen surface is divided to several meshes to reconstruct the speckle patterns. The technique is more flexible and convenient than traditional digital image correlation method in measuring the deformation of high altitude facilities such as bridges. In order to test the accuracy of this method, the deformations of an aluminum plate under different loading forces are measured and compared to the values recorded by the tensile tester.

Identifying Individual Jaguars and Ocelots via Pattern‐Recognition Software: Comparing HotSpotter and Wild‐ID

ABSTRACTCamera‐trapping is widespread in wildlife studies, especially for species with individually unique markings to which capture–recapture analytical techniques can be applied. The large volume of data such studies produce have encouraged researchers to increasingly look to computer‐assisted pattern‐recognition software to expedite individual identifications, but little work has been done to formally assess such software for camera‐trap data. We used 2 sets of camera‐trap images—359 images of jaguars (Panthera onca) and 332 images of ocelots (Leopardus pardalis) collected from camera traps deployed in 4 study sites in Orange Walk District, Belize, in 2015 and 2016—to compare the accuracy of 2 such programs, HotSpotter and Wild‐ID, and assess the effect of image quality on matching success. Overall, HotSpotter selected a correct match as its top rank 71–82% of the time, whereas the rate for Wild‐ID was 58–73%. Positive matching rates for both programs were highest for high‐quality images (85–99%) and lowest for low‐quality images (28–52%). False match rates were very low for HotSpotter (0–2%) but these were greater in Wild‐ID (6–28%). When lower ranks were also considered, both programs performed similarly (overall 22–24% nonmatches for HotSpotter, 17–26% nonmatches for Wild‐ID). We found that in both programs, images more often matched to other images of the same quality; therefore, including multiple reference images of an individual, of different qualities, improves matching success. These programs do not provide fully automatic identification of individuals and human involvement is still required to confirm matches, but we found that they are effective tools to expedite processing of camera‐trap data. We also offer usage recommendations for researchers to maximize the benefits of these tools. © 2020 The Wildlife Society.

Blind Image Quality Estimation via Distortion Aggravation

Traditional blind image quality assessment (IQA) measures generally predict quality from a sole distorted image directly. In this paper, we first introduce multiple pseudo reference images (MPRIs) by further degrading the distorted image in several ways and to certain degrees, and then compare the similarities between the distorted image and the MPRIs. Via such distortion aggravation, we can have some references to compare with, i.e., the MPRIs, and utilize the full-reference IQA framework to compute the quality. Specifically, we apply four types and five levels of distortion aggravation to deal with the commonly encountered distortions. Local binary pattern features are extracted to describe the similarities between the distorted image and the MPRIs. The similarity scores are then utilized to estimate the overall quality. More similar to a specific pseudo reference image (PRI) indicates closer quality to this PRI. Owning to the availability of the created multiple PRIs, we can reduce the influence of image content, and infer the image quality more accurately and consistently. Validation is conducted on four mainstream natural scene image and screen content image quality assessment databases, and the proposed method is comparable to or outperforms the state-of-the-art blind IQA measures. The MATLAB source code of the proposed measure will be publicly available.

Novel Image Set Compression Algorithm Using Rate-Distortion Optimized Multiple Reference Image Selection

Image set compression has recently become an active research topic due to the explosion of digital photographs. In order to efficiently compress the image sets of similar images with moving objects, in this paper, we propose a novel algorithm for image set compression using multiple reference images. First, for an image set, its depth-constrained minimum arborescence is generated. We then present a reference image candidate determination method to build the reference image candidates for the images of the set. Furthermore, we propose a rate-distortion optimized multiple reference image selection method. This method compares the correlation between every image and each of its reference image candidates to produce its multiple reference images. Finally, compressed image data are achieved by employing block-based motion compensation and residue coding. In addition, we also give a new way of access to images to keep the same access delay with single reference image-based schemes. Compared with the state-of-the-art image compression algorithms, experimental results show that our proposed algorithm can significantly improve the image compression performance.