PatchMatch Algorithm Research Articles

Attenuative Unified U-Net with Conditional PatchMatch Algorithm for Edge Preserving Salient Object Detection

The rise of Deep Neural Networks (DNNs) has significantly boosted salient object detection in computer vision tasks. However, downsampling methods like striding and pooling often introduce bluish artifacts near edges, hindering detection accuracy. To address this, the "Attenuative Unified U-Net with Conditional PatchMatch Algorithm" is proposed. The method employs Gaussian smoothing for noise reduction and depth refinement in preprocessing. In existing methods, the Edge-preserving salient object detection struggles with cluttered backgrounds and connected objects, like a bird on a rock, making it difficult to accurately distinguish edges and salient regions due to shared boundaries. Hence, an Attenuative Unified Backpropagated Entropy UNet is proposed, which integrates an Attention Mechanism that enhance feature map spatial weight assignment, emphasizing features or regions that are judged most relevant for edge and salient region detection. Then, the Cascaded Laplace pyramid is incorporated into the Unified U-Net’s design for multi-scale information processing, capturing contextual details effectively. The Backpropagated Entropy Loss Function is then created to ensure accurate and confident fused image output. The depth map, edge map, and salient area map are combined in the U-Net’s final output layer to create an overall fused image. Next, to addresses depth discontinuity problem a noteworthy novel idea the Conditional PatchMatch Random Field Algorithm (CPMRF), which combines PatchMatch efficiency and CRFs’ contextual modeling. PatchMatch iteratively propagates matches from neighboring patches to find similar patches in the remaining portion of the image and Conditional Random Field (CRF) is the next step, which improves the PatchMatch results by taking into account the connections between the matched patches and their neighboring areas. Hence, the experimental outcomes of the proposed model effectively show the improved detection of edge in salient object detection with better accuracy, precision, recall, MaxF, and F1 score with minimized MAE.

COMPARISON OF STATE-OF-THE-ART MULTI-VIEW STEREO SOLUTIONS FOR CLOSE RANGE HERITAGE DOCUMENTATION

Abstract. In recent years novel 3D reconstruction methods have been developed to improve the conventional image-based point cloud generation techniques. These novel methods generally attempt to address various challenges encountered in conventional methods, namely, the reconstruction of reflective surfaces and the amount of processing time required, both of which are major bottlenecks in heritage documentation and especially those related to large and complex objects. In this paper, we identified three types of 3D image-based reconstruction techniques and tested their usage on heritage datasets, namely (1) conventional multi-view stereo (MVS), (2) learning-based MVS, and (3) neural radiance fields (NeRF). The aim of this study is to determine the capabilities of these methods in reconstruction of three different heritage-related datasets with different challenges. Our results show that conventional MVS is nowadays a reliable solution for 3D reconstruction, in many instances recording good results relative to the reference terrestrial laser scans (TLS) when properly deployed. When applied to a challenging highly reflective scene, conventional MVS fared well using the PatchMatch algorithm (reaching an object completeness rate of 99.05%), while NeRF’s best performance was 99.98%. However, NeRF suffered from noisy data, some of which may stem from its radiance field-to-point cloud conversion method. The results show that there is great potential in using specific methods for specific cases, and research in combining them may yield interesting results in the future.

Unsupervised 3D reconstruction method based on multi-view propagation

In this paper, an end-to-end deep learning framework for reconstructing 3D models by computing depth maps from multiple views is proposed. An unsupervised 3D reconstruction method based on multi-view propagation is introduced, which addresses the issues of large GPU memory consumption caused by most current research methods using 3D convolution for 3D cost volume regularization and regression to obtain the initial depth map, as well as the difficulty in obtaining true depth values in supervised methods due to device limitations. The method is inspired by the Patchmatch algorithm, and the depth is divided into n layers within the depth range to obtain depth hypotheses through multi-view propagation. What's more, a multi-metric loss function is constructed based on luminosity consistency, structural similarity, and depth smoothness between multiple views to serve as a supervisory signal for learning depth predictions in the network. The experimental results show our proposed method has a very competitive performance and generalization on the DTU, Tanks & Temples and our self-made dataset; Specifically, it is at least 1.7 times faster and requires more than 75% less memory than the method that utilizes 3D cost volume regularization.

A survey on conventional and learning‐based methods for multi‐view stereo

Abstract3D reconstruction of scenes using multiple images, relying on robust correspondence search and depth estimation, has been thoroughly studied for the two‐view and multi‐view scenarios in recent years. Multi‐view stereo (MVS) algorithms aim to generate a rich, dense 3D model of the scene in the form of a dense point cloud or a triangulated mesh. In a typical MVS pipeline, the robust estimations for the camera poses along with the sparse points obtained from structure from motion (SfM) are used as input. During this process, the depth of generally every pixel of the scene is to be calculated. Several methods, either conventional or, more recently, learning‐based have been developed for solving the correspondence search problem. A vast amount of research exists in the literature using local, global or semi‐global stereomatching approaches, with the PatchMatch algorithm being among the most popular and efficient conventional ones in the last decade. Yet, and despite the widespread evolution of the algorithms, yielding complete, accurate and aesthetically pleasing 3D representations of a scene remains an open issue in real‐world and large‐scale photogrammetric applications. This work aims to provide a concrete survey on the most widely used MVS methods, investigating underlying concepts and challenges. To this end, the theoretical background and relative literature are discussed for both conventional and learning‐based approaches, with a particular focus on close‐range 3D reconstruction applications.

An efficient texture-structure conserving patch matching algorithm for inpainting mural images

An efficient texture-structure conserving patch matching algorithm for inpainting mural images

Nearest-neighbor field algorithm based on patchMatch for myocardial perfusion motion estimation/correction

Deformation correction and recovery of dynamic magnetic resonance images (DMRI) with low complexity algorithms without compromising image quality is a challenging problem. We proposed a motion estimation deformation-correction compressive sensing (DC-CS) scheme to recover dynamic images from its undersampled measurements. We simplify the complex optimization problem into three sub-problems. The contributions of this research are: introducing a global search strategy instead of the DC registration step, guaranteeing a non-explicit motion estimation that avoids any spatial alignment or registration of the images, and lowering the computational cost to the minimum by using PatchMatch (PM). The simulation result shows that the PM algorithm accelerates the recovery time without losing the quality in comparison with the DC algorithm.

Multi-View Stereo Vision Patchmatch Algorithm Based on Data Augmentation

In this paper, a multi-view stereo vision patchmatch algorithm based on data augmentation is proposed. Compared to other works, this algorithm can reduce runtime and save computational memory through efficient cascading of modules; therefore, it can process higher-resolution images. Compared with algorithms utilizing 3D cost volume regularization, this algorithm can be applied on resource-constrained platforms. This paper applies the data augmentation module to an end-to-end multi-scale patchmatch algorithm and adopts adaptive evaluation propagation, avoiding the substantial memory resource consumption characterizing traditional region matching algorithms. Extensive experiments on the DTU and Tanks and Temples datasets show that our algorithm is very competitive in completeness, speed and memory.

Optimal Transport-Based Patch Matching for Image Style Transfer

State-of-the-art image style transfer methods have achieved impressive results by using neural networks. However, neural style transfer (NST) methods either ignore the local details of the style image by using the global statistics for style modeling or cannot fully use shallow features of neural networks, leading to the synthesized image having fewer details. In this study, we proposed a new patch-based style transfer method that directly operates in the image pixel domain without using any neural networks, achieving fascinating style transfer results with rich image details. The proposed method was derived from classic texture synthesis methods. Most previous methods rely on nearest neighbor search (NNS) for patch matching. However, this greedy strategy cannot guarantee the similarity of patch distributions between the synthesized image and the style image, which limits the expressiveness of textures. We solved this problem by proposing an optimal patch matching algorithm formed on the Optimal Transport (OT) theory, which theoretically guarantees the similarity of the patch distributions and gives a flexible style modeling method. Various qualitative and quantitative experiments demonstrated that the proposed method achieves better synthesized results than state-of-the-art style transfer methods, including NST and classic methods based on texture synthesis.

A Non-Local Tensor Completion Algorithm Based on Weighted Tensor Nuclear Norm

In this paper, we proposed an image inpainting algorithm, including an interpolation step and a non-local tensor completion step based on a weighted tensor nuclear norm. Specifically, the proposed algorithm adopts the triangular based linear interpolation algorithm firstly to preform the observation image. Second, we extract the non-local similar patches in the image using the patch match algorithm and rearrange them to a similar tensor. Then, we use the tensor completion algorithm based on the weighted tensor nuclear norm to recover the similar tensors. Finally, we regroup all these recovered tensors to obtain the final recovered image. From the image inpainting experiments on color RGB images, we can see that the performance of the proposed algorithm on the peak signal-to-noise ratio (PSNR) and the relative standard error (RSE) are significantly better than other image inpainting methods.

A Novel Approach to Detect Copy Move Forgery Using Deep Learning

With the development of readily available image editing tools, manipulating an image has become a universal issue. To check the authenticity, it is necessary to identify how various images might be forged and the way they might be detected using various forgery detection approaches. The importance of detecting copy-move forgery is that it identifies the integrity of an image, which helps in fraud detection at various places such as courtrooms, news reports. This article presents an appropriate technique to detect Copy-Move forgery in which to some extent an image is copied and pasted onto an equivalent image to hide some object or to make duplication. The input image is segmented using the real-time superpixel segmentation algorithm DBSCAN (Density based spatial clustering of application with noise). Due to the high accuracy rate of the VGGNet 16 architecture, it is utilized for feature extraction of segmented images, which will also enhance the efficiency of the overall technique while matching the extracted patches using adaptive patch matching algorithm. The experimental results reveal that the proposed deep learning-based architecture is more accurate in identifying the tempered area even when the images are noisy and can save computational time as compared to existing architectures. For future research, the technique can be enhanced to work on other forgery detection techniques such as image splicing and multi-cloned images.

Continuous digital zooming using generative adversarial networks for dual camera system

This paper presents a generative adversarial network (GAN) with patch match algorithm to realize a high-quality digital zooming using two camera modules with different focal lengths. In dual camera system, shorter focal length module produces the wide-view image with the low resolution. On the other hand, the longer focal length module produces the tele-view image via optical zooming. The long-focal image contains more details than short-focal image and can be used to guide short-focal image to reconstruct high frequency part. Firstly, a feature extraction block (FEB) is advanced to extract feature of long-focal image and short focal-image to reconstruct a wide-view image with different resolutions. Next, a patch match algorithm is integrated into convolution neural networks (CNN) to fuse information of long-focal with short-focal image and generate a new fused image. Finally, the fused image and short-focal image are merged with a feature fusion block (FFB) to predict high-resolution images. In addition, generative adversarial networks are used for filtering information integrated by previous network and output the zoomed image. Extensive experiments on benchmark datasets show that our algorithm achieves favorable performance against state-of-the-art methods.

A Novel Protein Sequence Alignment-Based Patch Similarity Estimation for Two-Level Data Aggregation in WMSNs

In the digital era, massive growth in Wireless Multimedia Sensor Networks (WMSNs) can be used in various applications such as environmental and industrial monitoring, disaster management. WMSN comprises many low-power camera sensor nodes that may have overlapping field of view for enhancing the accuracy of the monitored environment. Hence, huge volume of redundant data which make it inefficient in terms of network lifetime and energy. It needs to establish a tremendous suitable approach to communicate images over WMSN that handle critical factors such as maximize QoS and minimize energy consumption efficiently. There is a need for energy efficient image processing and transmission over WMSN. So aggregating multi-view images using cluster based distributed routing approaches with QoS constraints have received significant research interest. The Distributed Two-Layer Cluster framework is used in this paper for transmitting aggregated data by local cluster head (LCH) and master cluster head from various clusters to base station in multi-hop basis. This aggregated data are generated from multi-view images captured by correlated camera sensor nodes of local cluster (LC) and master cluster (MC) using first level data aggregation (FLDA) and second level data aggregation (SLDA) algorithms respectively. In FLDA, less complex novel protein sequence alignment based patch matching algorithm is used to reduce inter-view redundancy of multi-view images. Then, this first level aggregated data transmission of LC is reduced using Varying Bit Encoding based on Arithmetic Operations algorithm by combining multi-view images together and achieves its compression ratio of about 2.54. In SLDA, Higher-Order SVD is used to aggregate FLDA data from LCH for reducing MC transmission rate and achieves its compression ratio of about 9.1. The comparison of network performance results with existing cluster based routing algorithms exhibit the improvement of the proposed system in terms of performance metrices such as energy consumption, packet delivery ratio, end-to-end delay and network lifetime. This results show that PSA based TLDA performs better and average energy consumption of processing and transmitting multi-view images per round is considerably minimized to 0.320 J. Thus the proposed system prolongs the network lifetime by 35%, 6% when compared with DFRP and GRAD-CORR based TLDA in terms of number of rounds.

Use of Generative Adversarial Networks to Altering Remote Sensing Data

The paper investigates the use of generative adversarial networks (GAN) for intentional modification of Earth remote sensing data. A generative neural network that includes a special subnet for object boundary inpainting is considered. The network comprises two GAN: the first completes the object boundary, and the second repaints blank areas. Actual remote sensing data are used to test the generative network under consideration. The exemplar-based Patch-Match algorithm is taken as a reference for comparison purposes. The experimental results allow the conclusion that the approach is an effective tool for the intentional modification of large terrestrial area images in falsification of Earth remote sensing data.

Reduced-Dimensional Capture of High-Dynamic Range Images with Compressive Sensing

The range of light illumination in real scenes is very large, and ordinary cameras can only record a small part of this range, which is far lower than the range of human eyes’ perception of light. High-dynamic range (HDR) imaging technology that has appeared in recent years can record a wider range of illumination than the perceptual range of the human eye. However, the current mainstream HDR imaging technology is to capture multiple low-dynamic range (LDR) images of the same scene with different exposures and then merge them into one HDR image, which greatly increases the amount of data captured. The advent of single-pixel cameras (compressive imaging system) has proved the feasibility of obtaining and restoring image data based on compressive sensing. Therefore, this paper proposes a method for reduced-dimensional capture of high dynamic range images with compressive sensing, which includes algorithms for front end (capturing) and back end (processing). At the front end, the K-SVD dictionary is used to compressive sensing the input multiple-exposure image sequence, thereby reducing the amount of data transmitted to the back end. At the back end, the Orthogonal Matching Pursuit (OMP) algorithm is used to reconstruct the input multiple-exposure image sequence. A low-rank PatchMatch algorithm is proposed to merge the reconstructed image sequence to obtain an HDR image. Simulation results show that, under the premise of reducing the complexity of the front-end equipment and the amount of communication data between the front end and the back end, the overall system achieves a good balance between the amount of calculation and the quality of the HDR image obtained.

Patch-match based detection of copy-move forgeries using rotation invariant features

As society increasingly relies on computers and digital data, the judiciary and other businesses are highly dependent on digital forensics for identification, preservation, examination and analysis of digital evidence such as digital photographs and digital documents. The wide range of powerful image editing tools that are evolving day by day helps to modify the image content, or to tamper the image with ease and does not require any expertise. The most challenging form of tampering to be detected is the copy – move forgery. The process of tampering images by copying one portion of the and pasting it in some other region of the same image is referred to as copy-move forgery. To make the copy-move forgery detection problem further complex, variations like scaling, rotation, flipping, etc. may be made over the copied region before it is pasted. Even those of us with the sharpest eyes may easily be fooled with such fake images and this is one of the major problems in courtrooms and in press publications. As such the integrity as well of authenticity of images is highly important where decision making or judgements have to be made based on the information obtained through images.Rotation- invariant features computed densely on the image and patch-match algorithm is proposed in this paper, to detect copy-move forgeries. The proposed detection algorithm is executed in three steps. Firstly, Circular Harmonic Transform based rotation invariant Zernike Moment and FMT features are computed for each pixel of interest. Secondly matching of dense features, is carried out using patch matching algorithm and finally post-processing is carried out to minimize the false positives. Experimental results demonstrate that the proposed approach is efficient in detecting copy move forgeries subjected to various types of geometric distortions.

Authentication of medical images using passive approach

Telemedicine is getting a lot of popularity. It is because of the way that the medical images are sent over the Internet to share the restorative state of a patient, making the analysis simple for the medical staff. There is, however, a good possibility that the contents of a medical image may be altered due to the openness of the Internet. The watermarking techniques are utilised to keep the integrity of medical images intact. These techniques, however, need some additional information (watermark) to be embedded into the medical images, which may deteriorate their visual quality, leading to false diagnosis. In this study, a non-intrusive procedure for verification of the medical images is proposed in which the polar cosine transform is used for feature extraction and the PatchMatch algorithm to locate the forged regions. Experimentally, the authors demonstrate that the proposed technique is robust against the noise, blurring, scaling, rotation and JPEG compression.

Visual Correspondences for Unsupervised Domain Adaptation on Electron Microscopy Images.

We present an Unsupervised Domain Adaptation strategy to compensate for domain shifts on Electron Microscopy volumes. Our method aggregates visual correspondences-motifs that are visually similar across different acquisitions-to infer changes on the parameters of pretrained models, and enable them to operate on new data. In particular, we examine the annotations of an existing acquisition to determine pivot locations that characterize the reference segmentation, and use a patch matching algorithm to find their candidate visual correspondences in a new volume. We aggregate all the candidate correspondences by a voting scheme and we use them to construct a consensus heatmap: a map of how frequently locations on the new volume are matched to relevant locations from the original acquisition. This information allows us to perform model adaptations in two different ways: either by a) optimizing model parameters under a Multiple Instance Learning formulation, so that predictions between reference locations and their sets of correspondences agree, or by b) using high-scoring regions of the heatmap as soft labels to be incorporated in other domain adaptation pipelines, including deep learning ones. We show that these unsupervised techniques allow us to obtain high-quality segmentations on unannotated volumes, qualitatively consistent with results obtained under full supervision, for both mitochondria and synapses, with no need for new annotation effort.

Fast Finger Vein Recognition Based on Sparse Matching Algorithm under a Multicore Platform for Real-Time Individuals Identification

Nowadays, individual identification is a problem in many private companies, but also in governmental and public order entities. Currently, there are multiple biometric methods, each with different advantages. Finger vein recognition is a modern biometric technique, which has several advantages, especially in terms of security and accuracy. However, image deformations and time efficiency are two of the major limitations of state-of-the-art contributions. In spite of affine transformations produced during the acquisition process, the geometric structure of finger vein images remains invariant. This consideration of the symmetry phenomena presented in finger vein images is exploited in the present work. We combine an image enhancement procedure, the DAISY descriptor, and an optimized Coarse-to-fine PatchMatch (CPM) algorithm under a multicore parallel platform, to develop a fast finger vein recognition method for real-time individuals identification. Our proposal provides an effective and efficient technique to obtain the displacement between finger vein images and considering it as discriminatory information. Experimental results on two well-known databases, PolyU and SDUMLA, show that our proposed approach achieves results comparable to deformation-based techniques of the state-of-the-art, finding statistical differences respect to non-deformation-based approaches. Moreover, our method highly outperforms the baseline method in time efficiency.

Individuals Identification Based on Palm Vein Matching under a Parallel Environment

Biometric identification and verification are essential mechanisms in modern society. Palm vein recognition is an emerging biometric technique, which has several advantages, especially in terms of security against forgery. Contactless palm vein systems are more suitable for real-world applications, but two of the major challenges of the state-of-the-art contributions are image deformations and time efficiency. In the present work, we propose a new method for palm vein recognition by combining DAISY descriptor and the Coarse-to-fine PatchMatch (CPM) algorithm in a parallel matching process. Our proposal aims at providing an effective and efficient technique to obtain similarity of palm vein images considering their displacements as discriminatory information. Extensive evaluation on three publicly available databases demonstrates that the discriminability of the proposed approach reaches the state-of-the-art results while it is considerably superior in time efficiency.

PRNU based source camera attribution for image sets anonymized with patch-match algorithm

Patch-Match is an efficient algorithm used for structural image editing and available as a tool on popular commercial photo-editing software. The tool allows users to insert or remove objects from photos using information from similar scene content. Recently, a modified version of this algorithm was proposed as a counter-measure against Photo-Response Non-Uniformity (PRNU) based Source Camera Identification (SCI). The algorithm can provide anonymity at a great rate (97%) and impede PRNU based SCI without the need of any other information, hence leaving no-known recourse for the PRNU-based SCI. In this paper, we propose a method to identify sources of the Patch-Match-applied images by using randomized subsets of images and the traditional PRNU based SCI methods. We evaluate the proposed method on two forensics scenarios in which an adversary makes use of the Patch-Match algorithm and distorts the PRNU noise pattern in the incriminating images she took with his camera. Our results show that it is possible to link sets of Patch-Match-applied images back to their source camera even in the presence of images that come from unknown cameras. To our best knowledge, the proposed method represents the very first counter-measure against the usage of Patch-Match in the digital forensics literature.