Neighboring Patches Research Articles

Neighbor patches merging reduces spatial redundancy to accelerate vision transformer

Vision Transformers (ViTs) deliver outstanding performance but often require substantial computational resources. Various token pruning methods have been developed to enhance throughput by removing redundant tokens; however, these methods do not address the peak memory consumption, which remains equivalent to that of the unpruned networks. In this study, we introduce Neighbor Patches Merging (NEPAM), a method that significantly reduces the maximum memory footprint of ViTs while pruning tokens. NEPAM targets spatial redundancy within images and prunes redundant patches at the onset of the model, thereby achieving the optimal throughput-accuracy trade-off without fine-tuning. Experimental results demonstrate that NEPAM can accelerate the inference speed of the Vit-Base-Patch16-384 model by 25% with a negligible accuracy loss of 0.07% and a notable 18% reduction in memory usage. When applied to VideoMAE, NEPAM doubles the throughput with a 0.29% accuracy loss and a 48% reduction in memory usage. These findings underscore the efficacy of NEPAM in mitigating computational requirements while maintaining model performance.

Reinforced deep learning approach for analyzing spaceborne-derived crop phenology

Vegetation Index (VI) curves capture the crop-specific phenological events from multi-temporal Earth observation images. Although advanced machine learning classifiers are the existing state-of-the-art, most of them assume the samples to be independent and identically distributed, which is not true for VI curve classification. This research proposed the consideration of spatial non-stationarity for VI curve-based crop classification. The proposed approach does not assume the samples to be independent and identically distributed. Different state-of-the-art approaches attempted to solve this problem by introducing architectural variations and considering the neighborhood patches. However, the inherent nature of the problem demanded an online approach. In the current research, this problem is formulated in a reinforcement learning framework where the generalizability of the predictor is improved using a feedback system. Different approaches for feedback were investigated in this study. The predictor was composed of a graph convolutional framework where the VI curves were transformed into a graph representation, incorporating the spatial, spectral, and temporal context. Graph convolutional operations were then used to learn the embedded representations and assign labels to the unlabeled points based on the labeled ones in the proximity. Further, a semi-supervised strategy was proposed where some initial feedback was employed to improve the generalizability of the model. Additionally, neighborhood-based feedback was also considered. The experiments using the VI curves collected over three farms in Israel covering wheat, potato, and barley crops illustrated the need for accounting the spatial non-stationarity in VI curve classification. The proposed dynamic feedback system and the graph-based strategy to consider spatial heterogeneity significantly improved the accuracy of the proposed framework. Additionally, the proposed reinforcement learning strategy ensures the dynamic refinement of model weights to facilitate the real-time applications even when the target data varies significantly. Although the proposed approach is generic, the end-to-end framework, as adopted in this research, is suitable for aerial or agricultural datasets. Hence, further fine tuning and remodeling may be required for non-agriculture or non-spatial datasets. However, the dynamic weight update may be adopted for any applications irrespective of the datasets.

Photovoltaic power prediction based on sky images and tokens-to-token vision transformer

Photovoltaic (PV) power generation has high uncertainties due to the randomness and imbalance nature of solar energy and meteorological parameters. Hence, accurate PV power forecasts are essential in the operation of PV power plants (PVPP) for short-term dispatches and power generation schedules. In this paper, a new deep neural network structure based on vision transformer is proposed to combine sky images and Tokens-To-Token（T2T） for photovoltaic power prediction. The method uses an incremental tokenization module to aggregate neighboring image patches into tokens, which capture the local structural information of the clouds. Then, an efficient T2T-ViT backbone network is used to extract the global attentional relationships of the tokens for power prediction. In order to evaluate the performance of the proposed model, the method was compared with several deep learning architectures such as ResNet and GoogleNet on a dataset collected by the National Renewable Energy Laboratory in Colorado, USA. The results of power prediction were analysed using training loss, prediction error, and linear regression, and they show that the proposed method achieves higher prediction accuracy and lower error compared to the existing methods, especially in short- and ultra-short-term prediction. The paper demonstrates the potential of applying Transformer models to computer vision tasks for renewable energy forecasting. The results show that the proposed method achieves higher prediction accuracy and lower error than several deep learning architectures, such as ResNet and GoogleNet, especially in short- and ultra-short-term prediction.

GeoTransformer: Fast and Robust Point Cloud Registration With Geometric Transformer.

We study the problem of extracting accurate correspondences for point cloud registration. Recent keypoint-free methods have shown great potential through bypassing the detection of repeatable keypoints which is difficult to do especially in low-overlap scenarios. They seek correspondences over downsampled superpoints, which are then propagated to dense points. Superpoints are matched based on whether their neighboring patches overlap. Such sparse and loose matching requires contextual features capturing the geometric structure of the point clouds. We propose Geometric Transformer, or GeoTransformer for short, to learn geometric feature for robust superpoint matching. It encodes pair-wise distances and triplet-wise angles, making it invariant to rigid transformation and robust in low-overlap cases. The simplistic design attains surprisingly high matching accuracy such that no RANSAC is required in the estimation of alignment transformation, leading to 100 times acceleration. Extensive experiments on rich benchmarks encompassing indoor, outdoor, synthetic, multiway and non-rigid demonstrate the efficacy of GeoTransformer. Notably, our method improves the inlier ratio by 18∼31 percentage points and the registration recall by over 7 points on the challenging 3DLoMatch benchmark.

Front propagation in a spatial system of weakly interacting networks.

We consider the spread of epidemic in a spatial metapopulation system consisting of weakly interacting patches. Each local patch is represented by a network with a certain node degree distribution and individuals can migrate between neighboring patches. Stochastic particle simulations of the SIR model show that after a short transient, the spatial spread of epidemic has a form of a propagating front. A theoretical analysis shows that the speed of front propagation depends on the effective diffusion coefficient and on the local proliferation rate similarly to fronts described by the Fisher-Kolmogorov equation. To determine the speed of front propagation, first, the early-time dynamics in a local patch is computed analytically by employing degree based approximation for the case of a constant disease duration. The resulting delay differential equationis solved for early times to obtain the local growth exponent. Next, the reaction diffusion equationis derived from the effective master equationand the effective diffusion coefficient and the overall proliferation rate are determined. Finally, the fourth order derivative in the reaction diffusion equationis taken into account to obtain the discrete correction to the front propagation speed. The analytical results are in a good agreement with the results of stochastic particle simulations.

Contrasting Response of Mountain Plant-Pollinator Network to Fragmented Semi-Natural Grasslands

The majority of the world’s plants rely on animal pollinators for reproduction, making pollination a key ecosystem service for the maintenance of natural and cultivated plant communities. Mutual interactions between plants and pollinators, also called “plant-pollinator networks”, are becoming increasingly vulnerable due to the intensification of anthropogenic land use and climate change. Thus, due to the rapid decline of semi-natural grasslands in the Northern Apennines (Italy), we aimed at understanding how the fragmentation of these habitats, the spatial distribution, and the amount of semi- and natural areas surrounding them, could affect species diversity and plant-pollinator networks. Specifically, in the Northern Apennines, we monitored semi-natural grasslands belonging to the EU habitat type 6510 to evaluate the effect of fragmentation on plant and pollinator richness and on the plant-pollinator network. We carried out generalized linear models considering three taxonomical and six network descriptors as response variables and the combinations of grasslands size and isolation, as well as 10 other factors describing landscape composition as explanatory variables. We found a well-structured plant-pollinator network, characterized by a high diversity of both plants and pollinators, with mutual relations marginally specialized, highly affected by habitat fragmentation and the land use of surrounding grasslands. Moreover, large and neighboring patches increased pollinator richness and improved the overall network structure while the occurrence of meadows and shrubs around fragmented patches was important to ensure the continuity of floristic resources. Finally, extensive croplands and agricultural settlements significantly reduced plant and pollinator diversity, favoring generalist (probably invasive) species, which however increased the strength and stability of the network.

The contribution of dynamics to macaque body and face patch responses

Previous functional imaging studies demonstrated body-selective patches in the primate visual temporal cortex, comparing activations to static bodies and static images of other categories. However, the use of static instead of dynamic displays of moving bodies may have underestimated the extent of the body patch network. Indeed, body dynamics provide information about action and emotion and may be processed in patches not activated by static images. Thus, to map with fMRI the full extent of the macaque body patch system in the visual temporal cortex, we employed dynamic displays of natural-acting monkey bodies, dynamic monkey faces, objects, and scrambled versions of these videos, all presented during fixation. We found nine body patches in the visual temporal cortex, starting posteriorly in the superior temporal sulcus (STS) and ending anteriorly in the temporal pole. Unlike for static images, body patches were present consistently in both the lower and upper banks of the STS. Overall, body patches showed a higher activation by dynamic displays than by matched static images, which, for identical stimulus displays, was less the case for the neighboring face patches. These data provide the groundwork for future single-unit recording studies to reveal the spatiotemporal features the neurons of these body patches encode. These fMRI findings suggest that dynamics have a stronger contribution to population responses in body than face patches.

Spatially Explicit Habitat Selection: Testing Contagion and the Ideal Free Distribution with Culex Mosquitoes.

AbstractSince its inception, attempts have been made to improve ideal free distribution (IFD) theory to make it better fit real-world data. Spatial contagion is a newer ecological concept that suggests that the perceived quality of a patch can be affected by the quality of its neighbor patches. Here, we present a series of experiments testing for potential contagion effects, examining how contagion can interact with the IFD and determining whether spatial context affects assessment of habitat quality. First, we tested whether the presence of conspecific competitors negatively impacts oviposition habitat selection by female mosquitoes (Culex restuans). We then used a more complex spatial landscape to determine whether competition can create a spatial contagion effect. Finally, we examined whether the density of conspecifics can adjust the contagion effect of nutrient availability. While females avoided patches containing conspecifics, there was no effect of competition/density on neighboring patches. Additionally, we found that resource availability was a significant predictor of where egg rafts were laid, but resource availability did not have a contagion effect. These results provide further support for the utility of the IFD, as individuals were able to accurately assess patch-level habitat quality.

Change detection based on unsupervised sparse representation for fundus image pair

Detecting changes is an important issue for ophthalmology to compare longitudinal fundus images at different stages and obtain change regions. Illumination variations bring distractions on the change regions by the pixel-by-pixel comparison. In this paper, a new unsupervised change detection method based on sparse representation classification (SRC) is proposed for the fundus image pair. First, the local neighborhood patches are extracted from the reference image to build a dictionary of the local background. Then the current image patch is represented sparsely and its background is reconstructed by the obtained dictionary. Finally, change regions are given through background subtracting. The SRC method can correct automatically illumination variations through the representation coefficients and filter local contrast and global intensity effectively. In experiments of this paper, the AUC and mAP values of SRC method are 0.9858 and 0.8647 respectively for the image pair with small lesions; the AUC and mAP values of the fusion method of IRHSF and SRC are 0.9892 and 0.9692 separately for the image pair with the big change region. Experiments show that the proposed method in this paper is more robust than RPCA for the illumination variations and can detect change regions more effectively than pixel-wised image differencing.

Structure-Based Ligand Design Targeting Pseudomonas aeruginosa LpxA in Lipid A Biosynthesis.

Enzymes involved in lipid A biosynthesis are promising antibacterial drug targets in Gram-negative bacteria. In this study, we use a structure-based design approach to develop a series of novel tetrazole ligands with low μM affinity for LpxA, the first enzyme in the lipid A pathway. Aided by previous structural data, X-ray crystallography, and surface plasmon resonance bioanalysis, we identify 17 hit compounds. Two of these hits were subsequently modified to optimize interactions with three regions of the LpxA active site. This strategy ultimately led to the discovery of ligand L13, which had a KD of 3.0 μM. The results reveal new chemical scaffolds as potential LpxA inhibitors, important binding features for ligand optimization, and protein conformational changes in response to ligand binding. Specifically, they show that a tetrazole ring is well-accommodated in a small cleft formed between Met169, the "hydrophobic-ruler" and His156, both of which demonstrate significant conformational flexibility. Furthermore, we find that the acyl-chain binding pocket is the most tractable region of the active site for realizing affinity gains and, along with a neighboring patch of hydrophobic residues, preferentially binds aliphatic and aromatic groups. The results presented herein provide valuable chemical and structural information for future inhibitor discovery against this important antibacterial drug target.

Single Image Super-Resolution Algorithm based on Fixed-Point Multi-Model Gaussian Process Regression

To solve the problem of large amount of computation in matrix inversion of Gaussian process regression model, a super-resolution algorithm based on local Gaussian process regression of fixed point multi-model is proposed. Firstly, the training samples are classified by Gaussian mixture model, and image patches are randomly selected as fixed points in each type of training samples, and its K nearest neighbor patch are searched. Secondly, local Gaussian process regression model by using its low-mid-frequency components and the corresponding high frequency components. Again, low resolution test image patch is classified, only the K nearest neighbor test image patch is searched during reconstruction. And then, find the nearest fixed point in each kind of image patch, and use the local Gaussian process regression model based on the fixed point to predict its corresponding high-frequency component. Finally, corresponding high frequency information is predicted by using this trained model, which improves the reconstruction efficiency. Experimental results have demonstrated that the proposed algorithm is superior in both quantitative and qualitative aspects against other algorithms.

Similarities in the rupture process and cascading asperities between neighboring fault patches and seismic implications: The 2002–2009 Sumbawa (Indonesia) earthquakes with moment magnitudes of 6.2–6.6

The Flores Thrust is a southward-dipping, low-to-moderate angle submarine active fault in the eastern Sunda-Banda back-arc (Indonesia). Significant shallow-depth destructive earthquakes have been reported along this fault zone. From 2002 to 2009, one of its fault segments, called the Sumbawa segment, experienced five earthquakes with moment magnitude (MW) values of 6.2–6.6. In this study, we performed finite-fault rupture inversions for these earthquakes, constrained with teleseismic body and surface waveforms, to investigate the characteristics of earthquake ruptures along this fault zone. We obtained the source-time-functions and finite-fault rupture models for these five earthquakes. Results indicated that ruptures often propagated along-strike or down-dip. The ruptures were initiated from the middle crust (depth of approximately 12–17 km) and exhibited a comparable initiation behavior to their entire rupture. The rupture speeds and stress drops were approximately 2.0–2.5 km/s and 1.0–2.0 MPa, respectively. Five cascading asperities ruptured neighboring fault patches and did not overlap each other. The characteristics of earthquake source parameters and rupture processes obtained in this study are robust and helpful for future regional seismic hazard assessment and earthquake early warning studies. These cascading asperities might be related to the fault immaturity of the western Flores Thrust. Alternatively, these earthquakes may act as asperities located at the down-dip patches of the Sumbawa segment, and its shallower section still has a potential of ruptures with MW > 7.0.

Multi-attention multiple instance learning

A new multi-attention based method for solving the MIL problem (MAMIL), which takes into account the neighboring patches or instances of each analyzed patch in a bag, is proposed. In the method, one of the attention modules takes into account adjacent patches or instances, several attention modules are used to get a diverse feature representation of patches, and one attention module is used to unite different feature representations to provide an accurate classification of each patch (instance) and the whole bag. Due to MAMIL, a combined representation of patches and their neighbors in the form of embeddings of a small dimensionality for simple classification is realized. Moreover, different types of patches are efficiently processed, and a diverse feature representation of patches in a bag by using several attention modules is implemented. A simple approach for explaining the classification predictions of patches is proposed. Numerical experiments with various datasets illustrate the proposed method.

Ship Target Segmentation for SAR Images Based on Clustering Center Shift

Ship target segmentation plays an important role in synthetic aperture radar (SAR) image interpretation. However, existing segmentation methods for marine SAR images have the problem of inaccurate edge segmentation, a concern for real-world applications. In this letter, we propose a clustering center shifted adaptive target segmentation (CCSATS) method. Firstly, the proposed clustering center shift method is used to update the clustering centers of each iteration, which can quickly and accurately capture ship pixels. Then, based on regional homogeneity coefficients, we define a new similarity measurement criterion with two adaptive weight factors to ensure the homogeneity of segmentation results. Finally, neighborhood patches are used to represent pixel information, which can reduce the influence of speckle noise and enhance the target edge fitting ability. Our segmentation results of measured SAR images show that the proposed method effectively ensures segmentation accuracy. Compared with other existing methods, the proposed target segmentation method achieves better edge capture performance.

Dual Learning-Based Graph Neural Network for Remote Sensing Image Super-Resolution

High-resolution (HR) remote sensing imagery plays a critical role in remote sensing image interpretation, and single image super-resolution (SISR) reconstruction technology is becoming increasingly valuable and significant. The state-of-the-art deep-learning-based SISR methods have demonstrated remarkable advantages, while reconstructing complex texture details still remains a big challenge. Besides, as a typical ill-posed inverse problem, how to determine the optimal solution is another important topic. To address these problems, in this work, a dual learning-based graph neural network (DLGNN) is proposed, in which the GNN is utilized to consider the self-similarity patches in remote sensing imagery by aggregating cross-scale neighboring feature patches, and dual learning strategy is adopted to refine the reconstruction results by constraining the mapping process in terms of the loss function, transferring the typical ill-posed problem to a well-posed one. Abundant experiments on 3K VEHICLE_SR datasets and Massachusetts Roads demonstrate the validity and outstanding performance for remote sensing image super-resolution tasks compared with other state-of-the-art super-resolution construction methods. Code is available at https://github.com/CUG-RS/DLGNN.

Hybrid Spatiotemporal Graph Convolutional Network for Detecting Landscape Pattern Evolution From Long-Term Remote Sensing Images

The remote sensing time-series change detection algorithm based on the pixel or single landscape patch ignores the change analysis of spatial structure information. Inspired by graph convolutional network (GCN) modeling, a set of landscapes (nodes) and their relationships (edges) are proposed. In this study, a parallel strategy in spatial GCN and progressive strategy in temporal GCN, called the hybrid GCN model network as a holistic framework, was proposed to accurately capture both spatial and temporal variations in landscape patterns based on yearly Landsat time series. A super-patch (i.e., fixed patch class surrounded by a one-hop neighbor patch) was selected as the input of the GCN network. First, a spatial GCN model with three parallel graph convolutional layers was adopted to classify landscape pattern types. Three dominant categories of landscape patterns over the past three decades have been identified. Second, four landscape metrics in super-patches were proposed for the quantitative characterization of changes in landscape patterns. Finally, a temporal GCN model with two progressive graph convolutional layers was used to detect six types of patch changes, which were applied to continuously detect the landscape pattern evolution processes. Regardless of the spatial GCN and temporal GCN, they provided satisfactory performance using the training and validation sets with overall accuracy > 92% and Kappa coefficient > 0.90, and loss values converging to 0.049 and 0.128, respectively, based on NLLLoss function until 500 epochs. It is believed that the hybrid GCN model has great potential for mining possible implicit spatiotemporal relationships and future evolution of landscape patterns.

Video-Based Point Cloud Compression Artifact Removal

Photo-realistic point cloud capture and transmission are the fundamental enablers for immersive visual communication. The coding process of dynamic point clouds, especially video-based point cloud compression (V-PCC) developed by the MPEG standardization group, is now delivering state-of-the-art performance in compression efficiency. V-PCC is based on the projection of the point cloud patches to 2D planes and encoding the sequence as 2D texture and geometry patch sequences. However, the resulting quantization errors from coding can introduce compression artifacts, which can be very unpleasant for the quality of experience (QoE). In this work, we developed a novel out-of-the-loop point cloud geometry artifact removal solution that can significantly improve reconstruction quality without additional bandwidth cost. Our novel framework consists of a point cloud sampling scheme, an artifact removal network, and an aggregation scheme. The point cloud sampling scheme employs a cube-based neighborhood patch extraction to divide the point cloud into patches. The geometry artifact removal network then processes these patches to obtain artifact-removed patches. The artifact-removed patches are then merged together using an aggregation scheme to obtain the final artifact-removed point cloud. We employ 3D deep convolutional feature learning for geometry artifact removal that jointly recovers both the quantization direction and the quantization noise level by exploiting projection and quantization prior. The simulation results demonstrate that the proposed method is highly effective and can considerably improve the quality of the reconstructed point cloud.

Use of a DVH overlay technique for quality assurance of deformable image registration‐based dose accumulation

We present a DVH overlay technique as a quality assurance (QA) metric for deformable image registration-based dose accumulation (DIR-DA). We use the technique to estimate the uncertainty in a DIR-DA for a revised treatment plan, and to compare two different DIR algorithms. The required inputs to the DVH overlay workflow are deformably registered primary and secondary images, primary regions-of-interest (ROIs), and secondary dose distribution. The primary ROIs were forward warped to the secondary image, the secondary dose was inversely warped to the primary image, and the DVHs for each image were compiled. Congruent DVHs imply minimal inverse consistency error (ICE) within an ROI. For a pancreas case re-planned after 21 fractions of a 29-fraction course, the workflow was used to quantify dose accumulation error attributable to ICE, based on a hybrid contour-and-intensity-based DIR. The usefulness of the workflow was further demonstrated by assessing the performance of two DIR algorithms (one free-form intensity-based, FFIB, the other using normalized correlation coefficients, NCC, over small neighborhood patches) as applied toward kilovoltage computed tomography (kVCT)-to-megavoltage computed tomography (MVCT) registration and five-fraction dose accumulation of ten male pelvis cases. For the re-planned pancreas case, when applying the DVH-overlay-based uncertainties the resulting accumulated dose remained compliant with all but two of the original plan objectives. Among the male pelvis cases, FFIB and NCC DIR showed good invertibility within the planning target volume (PTV), according to the DVH overlay QA results. NCC DIR exhibited better invertibility for the bladder and rectum compared with FFIB. However, compared with FFIB, NCC DIR exhibited less regional deformation for the bladder and a tendency for increased local contraction of the rectum ROI. For the five-fraction summations, ICE for the PTV V100%Rx is comparable for both algorithms (FFIB 0.8±0.7%, NCC 0.7±0.3%). For the bladder and rectum V70%Rx , ICE is greater for FFIB (1.8±0.7% for bladder, 1.7±0.6% for rectum) than for NCC (1.0±0.3% for bladder, 1.0±0.4% for rectum). The DVH overlay technique identified instances in which a DIR exhibits favorable invertibility, implying low ICE in a DIR-based dose accumulation. Differences in the overlaid DVHs can also estimate dose accumulation errors attributable to ICE for given ROIs.

Improvement of the Zienkiewicz–Zhu Error Recovery Technique Using a Patch Configuration

The Zienkiewicz–Zhu (ZZ) super-convergent patch recovery technique based on a node neighborhood patch configuration is used most widely for recovery of the stress field of a finite element analysis. In this study, an improved ZZ recovery technique using element neighborhood patch configuration is proposed. The improved recovery procedure is based on recovery of the stress field in the least-squares sense over an element patch that consists of the union of the elements surrounding the element under consideration. The proposed patch configuration provides more sampling points and improves the performance of the standard ZZ recovery technique. The effectiveness and reliability of the improved ZZ recovery approach is demonstrated through plane elastic and plastic plate problems. The problem domain is discretized with triangular and quadrilateral elements of different sizes. A comparison of the quality of error estimation using the ZZ recovery of derivative field and recovery of the displacement field using similar element neighborhood patch configurations is also presented. The numerical results show that the ZZ recovery technique and the displacement recovery technique, using a modified patch configuration, yield better results, convergence rate, and effectivity as compared with the standard ZZ super-convergent patch recovery technique. It is concluded that the improved ZZ recovery technique-based adaptive finite element analysis is very effective for converging a predefined accuracy with a significantly smaller number of degrees of freedom, especially in an elastic problem. It is also concluded that the improved ZZ recovery technique captures the plastic deformation problem solution errors more reliably than the standard ZZ recovery technique.

Remote Sensing Image Classification with a Graph-Based Pre-Trained Neighborhood Spatial Relationship.

Previous knowledge of the possible spatial relationships between land cover types is one factor that makes remote sensing image classification “smarter”. In recent years, knowledge graphs, which are based on a graph data structure, have been studied in the community of remote sensing for their ability to build extensible relationships between geographic entities. This paper implements a classification scheme considering the neighborhood relationship of land cover by extracting information from a graph. First, a graph representing the spatial relationships of land cover types was built based on an existing land cover map. Empirical probability distributions of the spatial relationships were then extracted using this graph. Second, an image was classified based on an object-based fuzzy classifier. Finally, the membership of objects and the attributes of their neighborhood objects were joined to decide the final classes. Two experiments were implemented. Overall accuracy of the two experiments increased by 5.2% and 0.6%, showing that this method has the ability to correct misclassified patches using the spatial relationship between geo-entities. However, two issues must be considered when applying spatial relationships to image classification. The first is the “siphonic effect” produced by neighborhood patches. Second, the use of global spatial relationships derived from a pre-trained graph loses local spatial relationship in-formation to some degree.