Non-local Information Research Articles

Multigraph approximate-representation learning for hyperspectral band selection

ABSTRACT Unsupervised hyperspectral image (HSI) band selection methods have been attracting ever-increasing attention. However, the local structural features captured by most of the existing methods suffer from a certain degree of risk of feature ambiguity. Moreover, these methods typically have difficulty preserving the latent manifold structure of the HSI in selected subsets of the bands. To address these problems, firstly, the graph approximate representation learning (GARL) model is proposed in this article by incorporating graph-regularized sparse coding into self-representation learning to preserve the local manifold structure information of HSI in the selected subset of bands. Meanwhile, to alleviate the ambiguity of the local graph structures, we design a local-global feature relationship expression model to construct the latent long-distance contextual connectivity (LDCC) graphs between the obtained local graph structures. Then, to preserve HSI’s local and non-local manifold structure information as much as possible in the selected subset of bands, a multigraph approximate representation learning (MGARL) model is proposed in this article by incorporating the obtained LDCC graph into the proposed GARL model. Next, we design a solution method to solve the proposed formulation. Finally, extensive experimental results on four datasets reveal the promising performance of MGARL over some state-of-the-art methods.

Cosmological constant as quantum error correction from generalized gauge invariance in double field theory

The holographic principle and its realization as the anti-de Sitter/conformal field theory (AdS/CFT) correspondence leads to the existence of the so-called precursor operators. These are boundary operators that carry nonlocal information regarding events occurring deep inside the bulk and which cannot be causally connected to the boundary. Such nonlocal operators can distinguish nonvacuum-like excitations within the bulk that cannot be observed by any local gauge invariant operators in the boundary. The boundary precursors are expected to become increasingly nonlocal the further the bulk process is from the boundary. Such phenomena are expected to be related to the extended nature of the strings. Standard gauge invariance in the boundary theory equates to quantum error correction which furthermore establishes localization of bulk information. I show that when double field theory quantum error correction prescriptions are considered in the bulk, gauge invariance in the boundary manifests residual effects associated to stringy winding modes. Also, an effect of double field theory quantum error correction is the appearance of positive cosmological constant. The emergence of space–time from the entanglement structure of a dual quantum field theory appears in this context to generalize for de Sitter space–times as well.

Transversely heterogeneous nonlocal Timoshenko beam theory: A reduced-order modeling via distributed-order fractional operators

Nonlocal effects are widely identified in micro/nano systems and have been studied extensively in recent decades. Nevertheless, existing studies mostly focus on the modeling of homogeneous nonlocal systems and as a result, cannot model micro/nanostructures with heterogeneously distributed nonlocal effects. This study introduces a reduced-order nonlocal Timoshenko beam model that facilitates precise modeling of micro-elastic beams with layer-wise heterogeneous nonlocal effects. First, a fully-resolved two-dimensional (2D) plane-strain fractional-order nonlocal elastic framework is developed to provide reference solutions for 2D nonlocal beam problems. Building upon this general nonlocal elasticity theory, an equivalent one-dimensional (1D) nonlocal Timoshenko model is then developed. The layer-wise heterogeneous nonlocal information is captured by using distributed-order (DO) operators, and the impacts arising from the nonlocal heterogeneity are further characterized by introducing two auxiliary parameters. Both 1D and 2D approaches are applied to simulate the mechanical responses of nonlocal beams. Direct comparisons of numerical simulations produced by either the 1D or the fully-resolved 2D model confirm that the DO Timoshenko beam formulation (together with the two auxiliary parameters) can capture not only the overall beam deflection but also the additional shear effect induced by the heterogeneous nonlocality. Computational cost assessment also indicates the proposed approach’s superior performance. The proposed DO Timoshenko model enables model-order reduction without compromising the heterogeneous nonlocal description of the material hence leading to an efficient and accurate reduced-order nonlocal modeling approach that addresses the limitations of prior nonlocal microstructural theories and extends applicability to a broader range of heterogeneous nano/microstructures.

Adaptively Enhancing Facial Expression Crucial Regions via a Local Non-local Joint Network

AbstractFacial expression recognition (FER) is still challenging due to the small interclass discrepancy in facial expression data. In view of the significance of facial crucial regions for FER, many existing studies utilize the prior information from some annotated crucial points to improve the performance of FER. However, it is complicated and time-consuming to manually annotate facial crucial points, especially for vast wild expression images. Based on this, a local non-local joint network is proposed to adaptively enhance the facial crucial regions in feature learning of FER in this paper. In the proposed method, two parts are constructed based on facial local and non-local information, where an ensemble of multiple local networks is proposed to extract local features corresponding to multiple facial local regions and a non-local attention network is addressed to explore the significance of each local region. In particular, the attention weights obtained by the non-local network are fed into the local part to achieve interactive feedback between the facial global and local information. Interestingly, the non-local weights corresponding to local regions are gradually updated and higher weights are given to more crucial regions. Moreover, U-Net is employed to extract the integrated features of deep semantic information and low hierarchical detail information of expression images. Finally, experimental results illustrate that the proposed method achieves more competitive performance than several state-of-the-art methods on five benchmark datasets.

Deep convolutional dictionary learning network for sparse view CT reconstruction with a group sparse prior

Purpose Numerous techniques based on deep learning have been utilized in sparse view computed tomography (CT) imaging. Nevertheless, the majority of techniques are instinctively constructed utilizing state-of-the-art opaque convolutional neural networks (CNNs) and lack interpretability. Moreover, CNNs tend to focus on local receptive fields and neglect nonlocal self-similarity prior information. Obtaining diagnostically valuable images from sparsely sampled projections is a challenging and ill-posed task.Method To address this issue, we propose a unique and understandable model named DCDL-GS for sparse view CT imaging. This model relies on a network comprised of convolutional dictionary learning and a nonlocal group sparse prior. To enhance the quality of image reconstruction, we utilize a neural network in conjunction with a statistical iterative reconstruction framework and perform a set number of iterations. Inspired by group sparsity priors, we adopt a novel group thresholding operation to improve the feature representation and constraint ability and obtain a theoretical interpretation. Furthermore, our DCDL-GS model incorporates filtered backprojection (FBP) reconstruction, fast sliding window nonlocal self-similarity operations, and a lightweight and interpretable convolutional dictionary learning network to enhance the applicability of the model.Results The efficiency of our proposed DCDL-GS model in preserving edges and recovering features is demonstrated by the visual results obtained on the LDCT-P and UIH datasets. Compared to the results of the most advanced techniques, the quantitative results are enhanced, with increases of 0.6-0.8 dB for the peak signal-to-noise ratio (PSNR), 0.005-0.01 for the structural similarity index measure (SSIM), and 1-1.3 for the regulated Fréchet inception distance (rFID) on the test dataset. The quantitative results also show the effectiveness of our proposed deep convolution iterative reconstruction module and nonlocal group sparse prior.Conclusion In this paper, we create a consolidated and enhanced mathematical model by integrating projection data and prior knowledge of images into a deep iterative model. The model is more practical and interpretable than existing approaches. The results from the experiment show that the proposed model performs well in comparison to the others.

A non-local total generalized variation regularization reconstruction method for sparse-view x-ray CT

Sparse-view x-ray computed tomography (CT) reconstruction, employing total generalized variation (TGV), effectively mitigates the stepwise artifacts associated with total variation (TV) regularization while preserving structural features within transitional regions of the reconstructed image. Despite TGV surpassing TV in reconstruction quality, it neglects the non-local self-similarity prior, recognized for its efficacy in restoring details during CT reconstruction. This study introduces a non-local TGV (NLTGV) to address the limitation of TGV regularization method. Specifically, we propose an NLTGV-regularized method for sparse-view CT reconstruction, utilizing non-local high-order derivative information to maintain image features and non-local self-similarity for detail recovery. Owing to the non-differentiability of the NLTGV regularized, we employ an alternating direction method of multipliers optimization method, facilitating an efficient solution by decomposing the reconstruction model into sub-problems. The proposed method undergoes evaluation using both simulated and real-world projection data. Simulation and experimental results demonstrate the efficacy of the proposed approach in enhancing the quality of reconstructed images compared to other competitive variational reconstruction methods. In conclusion, the simultaneous incorporation of sparsity priors of high-order TV and non-local similarity proves advantageous for structural detail recovery in sparse-view CT reconstruction.

Comparing reinforcement learning algorithms for a trip building task: A multi-objective approach using non-local information

Using reinforcement learning (RL) to support agents in making decisions that consider more than one objective poses challenges. We formulate the problem of multiple agents learning how to travel from A to B as a reinforcement learning task modeled as a stochastic game, in which we take into account: (i) more than one objective, (ii) non-stationarity, (iii) communication of local and non-local information among the various actors. We use and compare RL algorithms, both for the single objective (Q-learning), as well as for multiple objectives (Pareto Qlearning), with and without non-local communication. We evaluate these methods in a scenario in which hundreds of agents have to learn how to travel from their origins to their destinations, aiming at minimizing their travel times, as well as the carbon monoxide vehicles emit. Results show that the use of non-local communication reduces both travel time and emissions.

Cross-Component Prediction Boosted With Local and Non-Local Information in Video Coding.

In the versatile video coding (VVC) standard, cross-component prediction (CCP) is introduced to utilize the correlation between luma and chroma components. To further exploit the cross-component redundancy, a bundle of novel CCP methods with more sophisticated models are studied in enhanced compression model (ECM), which is a platform targeting at the next generation video coding standard, developed by JVET. This paper presents two methods, known as local-boosting CCP (LB-CCP) and non-local CCP (NL-CCP) to improve CCP with local and non-local information. With LB-CCP, prediction samples of CCP can be filtered with neighbouring samples. Besides, neighbouring template costs are calculated to determine the range of training samples, as well as the cross-component prediction method used in the chroma fusion mode. With NL-CCP, a CCP model can be derived with samples non-adjacent to the current block. As an equivalent but simpler implementation, the CCP model can be inherited from a CCP-coded neighbouring block as a spatial CCP candidate. Moreover, the CCP model of a previous CCP-coded block can be stored in a history-based table, which can be fetched and used by the current block as a history-based CCP candidate. A NL-CCP candidate list is built with the two kinds of candidates. The encoder can select the optimal candidate and send an index to the decoder. Experimental results show that LB-CCP together with NL-CCP can provide an average Bjontegaard delta rate (BD-rate) reduction of 0.27%, 2.31%, 2.44% on Y, Cb, Cr components, respectively, with a negligible change in the running time, compared with ECM-8.0 in all intra configurations. Both LB-CCP and NL-CCP have been adopted into ECM.

Deblurring Videos Using Spatial-Temporal Contextual Transformer With Feature Propagation.

We present a simple and effective approach to explore both local spatial-temporal contexts and non-local temporal information for video deblurring. First, we develop an effective spatial-temporal contextual transformer to explore local spatial-temporal contexts from videos. As the features extracted by the spatial-temporal contextual transformer does not model the non-local temporal information of video well, we then develop a feature propagation method to aggregate useful features from the long-range frames so that both local spatial-temporal contexts and non-local temporal information can be better utilized for video deblurring. Finally, we formulate the spatial-temporal contextual transformer with the feature propagation into a unified deep convolutional neural network (CNN) and train it in an end-to-end manner. We show that using the spatial-temporal contextual transformer with the feature propagation is able to generate useful features and makes the deep CNN model more compact and effective for video deblurring. Extensive experimental results show that the proposed method performs favorably against state-of-the-art ones on the benchmark datasets in terms of accuracy and model parameters.

A Nonlocal Self-Similarity-Based Weighted Tensor Low-Rank Decomposition for Multichannel Image Completion With Mixture Noise.

Multichannel image completion with mixture noise is a challenging problem in the fields of machine learning, computer vision, image processing, and data mining. Traditional image completion models are not appropriate to deal with this problem directly since their reconstruction priors may mismatch corruption priors. To address this issue, we propose a novel nonlocal self-similarity-based weighted tensor low-rank decomposition (NSWTLD) model that can achieve global optimization and local enhancement. In the proposed model, based on the corruption priors and the reconstruction priors, a pixel weighting strategy is given to characterize the joint effects of missing data, the Gaussian noise, and the impulse noise. To discover and utilize the accurate nonlocal self-similarity information to enhance the restoration quality of the details, the traditional nonlocal learning framework is optimized by employing improved index determination of patch group and handling strip noise caused by patch overlapping. In addition, an efficient and convergent algorithm is presented to solve the NSWTLD model. Comprehensive experiments are conducted on four types of multichannel images under various corruption scenarios. The results demonstrate the efficiency and effectiveness of the proposed model.

SAB Net: A Semantic Attention Boosting Framework for Semantic Segmentation.

Semantic segmentation has achieved great progress by effectively fusing features of contextual information. In this article, we propose an end-to-end semantic attention boosting (SAB) framework to adaptively fuse the contextual information iteratively across layers with semantic regularization. Specifically, we first propose a pixelwise semantic attention (SAP) block, with a semantic metric representing the pixelwise category relationship, to aggregate the nonlocal contextual information. In addition, we improve the computation complexity of SAP block from O(n²) to O(n) for images with size n. Second, we present a categorywise semantic attention (SAC) block to adaptively balance the nonlocal contextual dependencies and the local consistency with a categorywise weight, overcoming the contextual information confusion caused by the feature imbalance within intra-category. Furthermore, we propose the SAB module to refine the segmentation with SAC and SAP blocks. By applying the SAB module iteratively across layers, our model shrinks the semantic gap and enhances the structure reasoning by fully utilizing the coarse segmentation information. Extensive quantitative evaluations demonstrate that our method significantly improves the segmentation results and achieves superior performance on the PASCAL VOC 2012, Cityscapes, PASCAL Context, and ADE20K datasets.

Lightweight Neural Network for Enhancing Imaging Performance of Under-Display Camera

Under-Display Camera (UDC) is an emerging feature of cellphone. This technology makes full-screen cellphones possible by hiding the front-facing camera below the display panel, which is in contrast to the conventional designs that place the camera in a bezel or punch-hole on the screen border. However, this novel imaging paradigm also causes degradation. The display panel attenuates and diffracts incoming light, so the images captured by UDC contain multiple artifacts, such as blurring, color shift, and low intensities. This paper proposes a lightweight deep learning approach to restore UDC images in a blind setting. The restoration network uses cross-scale modulation to exploit complementary information from multi-scale representations and capture the self-similarity across scales, aiming to find the cues for recovering distortion-free images. To facilitate the deployment of this scheme across mobile devices, especially on those with limited memory space and computing power, we compress the restoration network by reducing architectural redundancy. An adaptive distillation algorithm is designed to exploit knowledge from a pre-trained full-size model. The proposed work also interprets the behavior of the neural network in utilizing local and non-local information to restore UDC images. The proposed algorithm is evaluated over three datasets of the images captured by the cameras below different types of display panels. The results of comparative experiments demonstrate that our algorithm shows comparable or superior performance to the competing ones that are much heavier in parameter amount and computational complexities.

MS-RAFT+: High Resolution Multi-Scale RAFT

AbstractHierarchical concepts have proven useful in many classical and learning-based optical flow methods regarding both accuracy and robustness. In this paper we show that such concepts are still useful in the context of recent neural networks that follow RAFT’s paradigm refraining from hierarchical strategies by relying on recurrent updates based on a single-scale all-pairs transform. To this end, we introduce MS-RAFT+: a novel recurrent multi-scale architecture based on RAFT that unifies several successful hierarchical concepts. It employs a coarse-to-fine estimation to enable the use of finer resolutions by useful initializations from coarser scales. Moreover, it relies on RAFT’s correlation pyramid that allows to consider non-local cost information during the matching process. Furthermore, it makes use of advanced multi-scale features that incorporate high-level information from coarser scales. And finally, our method is trained subject to a sample-wise robust multi-scale multi-iteration loss that closely supervises each iteration on each scale, while allowing to discard particularly difficult samples. In combination with an appropriate mixed-dataset training strategy, our method performs favorably. It not only yields highly accurate results on the four major benchmarks (KITTI 2015, MPI Sintel, Middlebury and VIPER), it also allows to achieve these results with a single model and a single parameter setting. Our trained model and code are available at https://github.com/cv-stuttgart/MS_RAFT_plus.

High-Fidelity compressive spectral image reconstruction through a novel Non-Convex Non-Local Low-Rank tensor approximation model

Low-quality reconstruction is one main factor that greatly limits the development of snapshot compressive imaging (SCI) systems. The common algorithms usually unfold a 3D hyperspectral image (HSI) into a vector or matrix which inevitably corrupts the intrinsic structure of the HSI and insufficiently discovers its prior information. To fully exploit the underlying structures and internal correlations of the HSIs, this paper proposes a novel non-convex model based on Non-Local low-rank Tensor Approximation with Hyper-Laplacian priori (HL), named HL-NLTA, which effectively combines local and non-local similarity information from fused spectral and spatial perspectives. Specifically, a non-local low-rank tensor model based on the minimax concave plus (MCP) penalty and log sum (LS) penalty separately is developed which can simultaneously exploit two intrinsic properties of HSI, i.e., global correlation along the spectral domain (GCAS) and non-local self-similarity along the spatial domain (NSAS). Furthermore, the Hyper-Laplacian priori regularized through a non-convex ℓp (0 < p < 1) norm can preserve well the spectral and spatial structure. An optimization algorithm based on the alternating direction multiplier method (ADMM) framework is designed and accelerated by the Generalized shrinkage/thresholding (GST) algorithm and the fast Fourier transform (FFT) to solve the new nonconvex model. Extensive simulation results on public datasets and experimental results on our real CASSI system indicate that the proposed method enables high-fidelity HSI reconstructions in terms of simultaneous spatial structure detail recovery and spectral preservation. These results verify the effectiveness of the proposed model for snapshot compressive imaging (e.g., Coded aperture snapshot spectral imaging CASSI).

Fisher and Shannon Functionals for Hyperbolic Diffusion.

The complexity measure for the distribution in space-time of a finite-velocity diffusion process is calculated. Numerical results are presented for the calculation of Fisher's information, Shannon's entropy, and the Cramér-Rao inequality, all of which are associated with a positively normalized solution to the telegrapher's equation. In the framework of hyperbolic diffusion, the non-local Fisher's information with the x-parameter is related to the local Fisher's information with the t-parameter. A perturbation theory is presented to calculate Shannon's entropy of the telegrapher's equation at long times, as well as a toy model to describe the system as an attenuated wave in the ballistic regime (short times).

Transformer-based network with difference information aware attention module

Existing great progress has been made through remote sensing image change detection (CD) methods, which use convolutions and transformations appropriately. However, due to the complexity of the objects photographed on satellites, the out-of-phase of the same object may exhibit a fine feature diversity and a wide range of variations. Meanwhile, in the previous work, pure convolutional networks and Transformer networks that focus on local information perform better on non-local information. Therefore, current methods still have the problem of insufficient discrimination between irregular changes and diverse boundaries. Here, to solve this biphasic dilemma, we put forward a network that uses the difference information between bitemporal images as a guide to refine features. We use one feature map to subtract another and use multi-scale convolution to refine the difference map. We then designed an attention module similar to self-attention using differential information. In addition, considering the need for both local and global information in change detection, we use the full Swin Transformer framework with shift window self-attention to pass information between local areas. In addition, in order to enhance the network’s ability to recognize, members of our team reformulate supervised, traditional saliency map prediction through saliency and edge prediction. Notably, our proposed model significantly outperforms existing single-branch methods and state-of-the-art performance was achieved on four famous public datasets in terms of the F1-score index, interest in joint indices, and population indices.

Shatavadhan Could Imply A New State of Consciousness

Recent advances in neurosciences and an attempt to explain properties of consciousness has opened new frontiers to understand the properties of consciousness from various perspectives like quantum biology, neurophysiology, neuropsychology, quantum physics, computer science and several other disciplines. Here we describe a state of consciousness known as Shatvadhan. Shatvadhan in Jainism means one hundred states of extraordinary attention of the brain. In this state a person will listen or observe one hundred questions of different types from one hundred persons or more for over a period of say 2-3 hours without break and without using any paper or pen or electronic device. This is a demonstration of extraordinary memory to retain and recall questions and their answers in same sequence, reverse sequence or random sequences involving complex questions. This appears to represent a new state of consciousness which involves demonstration of extra ordinary memory, highly sequential ordering of memory, recalling, remembering sequences also in which memories of events are stored, simultaneous storing information from different sense organs and others. This also involves mathematical computations which are done in parts in different activities and simultaneous observations of pictures and listening of audio sounds. It appears that this process represents a new state of consciousness mostly found in saints who are highly advanced in the process of spiritual development. An analysis of this may provide new avenues of thought regarding storage and recalling mechanism of memory in the human brain. It may also involve non-local information transfer in the brain through quantum holography and similar mechanism.

Combining external-latent attention for medical image segmentation

The attention mechanism comes as a new entry point for improving the performance of medical image segmentation. How to reasonably assign weights is a key element of the attention mechanism, and the current popular schemes include the global squeezing and the non-local information interactions using self-attention (SA) operation. However, these approaches over-focus on external features and lack the exploitation of latent features. The global squeezing approach crudely represents the richness of contextual information by the global mean or maximum value, while non-local information interactions focus on the similarity of external features between different regions. Both ignore the fact that the contextual information is presented more in terms of the latent features like the frequency change within the data. To tackle above problems and make proper use of attention mechanisms in medical image segmentation, we propose an external-latent attention collaborative guided image segmentation network, named TransGuider. This network consists of three key components: 1) a latent attention module that uses an improved entropy quantification method to accurately explore and locate the distribution of latent contextual information. 2) an external self-attention module using sparse representation, which can preserve external global contextual information while reducing computational overhead by selecting representative feature description map for SA operation. 3) a multi-attention collaborative module to guide the network to continuously focus on the region of interest, refining the segmentation mask. Our experimental results on several benchmark medical image segmentation datasets show that TransGuider outperforms the state-of-the-art methods, and extensive ablation experiments demonstrate the effectiveness of the proposed components. Our code will be available at https://github.com/chasingone/TransGuider.

Densely Connected U-Net With Criss-Cross Attention for Automatic Liver Tumor Segmentation in CT Images.

Automatic liver tumor segmentation plays a key role in radiation therapy of hepatocellular carcinoma. In this paper, we propose a novel densely connected U-Net model with criss-cross attention (CC-DenseUNet) to segment liver tumors in computed tomography (CT) images. The dense interconnections in CC-DenseUNet ensure the maximum information flow between encoder layers when extracting intra-slice features of liver tumors. Moreover, the criss-cross attention is used in CC-DenseUNet to efficiently capture only the necessary and meaningful non-local contextual information of CT images containing liver tumors. We evaluated the proposed CC-DenseUNet on the LiTS dataset and the 3DIRCADb dataset. Experimental results show that the proposed method reaches the state-of-the-art performance for liver tumor segmentation. We further experimentally demonstrate the robustness of the proposed method on a clinical dataset comprising 20 CT volumes.

Frequency Learning via Multi-scale Fourier Transformer for MRI Reconstruction.

Since Magnetic Resonance Imaging(MRI) requires a long acquisition time, various methods were proposed to reduce the time, but they ignored the frequency information and non-local similarity, so that they failed to reconstruct images with a clear structure. In this paper, we propose Frequency Learning via Multi-scale Fourier Transformer for MRI Reconstruction(FMTNet), which focuses on repairing the low-frequency and high-frequency information. Specifically, FMTNet is composed of a high-frequency learning branch(HFLB) and a low-frequency learning branch(LFLB). Meanwhile, we propose a Multi-scale Fourier Transformer(MFT) as the basic module to learn the non-local information. Unlike normal Transformers, MFT adopts Fourier convolution to replace self-attention to efficiently learn global information. Moreover, we further introduce a multi-scale learning and cross-scale linear fusion strategy in MFT to interact information between features of different scales and strengthen the representation of features. Compared with normal Transformers, the proposed MFT occupies fewer computing resources. Based on MFT, we design a Residual Multi-scale Fourier Transformer module as the main component of HFLB and LFLB. We conduct several experiments under different acceleration rates and different sampling patterns on different datasets, and the experiment results show that our method is superior to the previous state-of-the-art method. The code and dataset will be available at: https://github.com/Joyies/FMTNet.