Nonlocal Mechanism Research Articles

A Novel Hybridoma Cell Segmentation Method Based on Multi-Scale Feature Fusion and Dual Attention Network

The hybridoma cell screening method is usually done manually by human eyes during the production process for monoclonal antibody drugs. This traditional screening method has certain limitations, such as low efficiency and subjectivity bias. Furthermore, most of the existing deep learning-based image segmentation methods have certain drawbacks, due to different shapes of hybridoma cells and uneven location distribution. In this paper, we propose a deep hybridoma cell image segmentation method based on residual and attention U-Net (RA-UNet). Firstly, the feature maps of the five modules in the network encoder are used for multi-scale feature fusion in a feature pyramid form and then spliced into the network decoder to enrich the semantic level of the feature maps in the decoder. Secondly, a dual attention mechanism module based on global and channel attention mechanisms is presented. The global attention mechanism (non-local neural network) is connected to the network decoder to expand the receptive field of the feature map and bring more rich information to the network. Then, the channel attention mechanism SENet (the squeeze-and-excitation network) is connected to the non-local attention mechanism. Consequently, the important features are enhanced by the learning of the feature channel weights, and the secondary features are suppressed, hence improving the cell segmentation performance and accuracy. Finally, the focal loss function is used to guide the network to learn the hard-to-classify cell categories. Furthermore, we evaluate the performance of the proposed RA-UNet method on a newly established hybridoma cell image dataset. Experimental results show that the proposed method has good reliability and improves the efficiency of hybridoma cell segmentation compared with state-of-the-art networks such as FCN, UNet, and UNet++. The results show that the proposed RA-UNet model has improvements of 0.8937%, 0.9926%, 0.9512%, and 0.9007% in terms of the dice coefficients, PA, MPA, and MIoU, respectively.

Learning Enriched Features for Fast Image Restoration and Enhancement.

Given a degraded input image, image restoration aims to recover the missing high-quality image content. Numerous applications demand effective image restoration, e.g., computational photography, surveillance, autonomous vehicles, and remote sensing. Significant advances in image restoration have been made in recent years, dominated by convolutional neural networks (CNNs). The widely-used CNN-based methods typically operate either on full-resolution or on progressively low-resolution representations. In the former case, spatial details are preserved but the contextual information cannot be precisely encoded. In the latter case, generated outputs are semantically reliable but spatially less accurate. This paper presents a new architecture with a holistic goal of maintaining spatially-precise high-resolution representations through the entire network, and receiving complementary contextual information from the low-resolution representations. The core of our approach is a multi-scale residual block containing the following key elements: (a) parallel multi-resolution convolution streams for extracting multi-scale features, (b) information exchange across the multi-resolution streams, (c) non-local attention mechanism for capturing contextual information, and (d) attention based multi-scale feature aggregation. Our approach learns an enriched set of features that combines contextual information from multiple scales, while simultaneously preserving the high-resolution spatial details. Extensive experiments on six real image benchmark datasets demonstrate that our method, named as MIRNet-v2, achieves state-of-the-art results for a variety of image processing tasks, including defocus deblurring, image denoising, super-resolution, and image enhancement. The source code and pre-trained models are available at https://github.com/swz30/MIRNetv2.

Reduced volume of diabetic pancreatic islets in rodents detected by synchrotron X-ray phase-contrast microtomography and deep learning network

The pancreatic islet is a highly structured micro-organ that produces insulin in response to rising blood glucose. Here we develop a label-free and automatic imaging approach to visualize the islets in situ in diabetic rodents by the synchrotron radiation X-ray phase-contrast microtomography (SRμCT) at the ID17 station of the European Synchrotron Radiation Facility. The large-size images (3.2mm×15.97mm) were acquired in the pancreas in STZ-treated mice and diabetic GK rats. Each pancreas was dissected by 3000 reconstructed images. The image datasets were further analysed by a self-developed deep learning method, AA-Net. All islets in the pancreas were segmented and visualized by the three-dimension (3D) reconstruction. After quantifying the volumes of the islets, we found that the number of larger islets (=>1500 μm3) was reduced by 2-fold (wt 1004±94 vs GK 419±122, P<0.001) in chronically developed diabetic GK rat, while in STZ-treated diabetic mouse the large islets were decreased by half (189±33 vs 90±29, P<0.001) compared to the untreated mice. Our study provides a label-free tool for detecting and quantifying pancreatic islets in situ. It implies the possibility of monitoring the state of pancreatic islets in vivo diabetes without labelling.

(Retracted) Design of street art image retrieval system based on virtual simulation technology for the public health environment

Creating a good public health environment can improve the public’s environmental health literacy level. Therefore, we propose a street art image retrieval system to address the problems of artists’ single creation method and the exhaustion of creative inspiration in the process of creating street art images in a public health environment. The system can retrieve the relevant image categories and cultural backgrounds, according to the street art images to be drawn, which reduces the artists’ creation burden and helps the public to better appreciate and understand art. In the image classification module, a ResNet34 street art image classification network with a nonlocal attention mechanism is proposed by combining the characteristics of street art images. The experimental results show that the method can achieve accurate classification of art images and can accurately retrieve relevant images based on the input art images, helping artists to better create and improve the public health environment.

Fast instruments and tissues segmentation of micro-neurosurgical scene using high correlative non-local network

Surgical scene segmentation provides critical information for guidance in micro-neurosurgery. Segmentation of instruments and critical tissues contributes further to robot assisted surgery and surgical evaluation. However, due to the lack of relevant scene segmentation dataset, scale variation and local similarity, micro-neurosurgical segmentation faces many challenges. To address these issues, a high correlative non-local network (HCNNet), is proposed to aggregate multi-scale feature by optimized non-local mechanism. HCNNet adopts two-branch design to generate features of different scale efficiently, while the two branches share common weights in shallow layers. Several short-term dense concatenate (STDC) modules are combined as the backbone to capture both semantic and spatial information. Besides, a high correlative non-local module (HCNM) is designed to guide the upsampling process of the high-level feature by modeling global context generated from the low-level feature. It filters out confused pixels of different classes in the non-local correlation map. Meanwhile, a large segmentation dataset named NeuroSeg is constructed, which contains 15 types of instruments and 3 types of tissues that appear in meningioma resection surgery. The proposed HCNNet achieves the state-of-the-art performance on NeuroSeg, it reaches an inference speed of 54.85 FPS with the highest accuracy of 59.62% mIoU, 74.7% Dice, 70.55% mAcc and 87.12% aAcc.

Nonlocal Impacts of Soil Moisture Variability in South America: Linking Two Land–Atmosphere Coupling Hot Spots

Abstract The land–atmosphere interactions play an important role in modulating climate variability at different spatial and temporal scales. In South America, two recognized hot spots of soil moisture–atmosphere coupling are located in southeastern South America (SESA) and eastern Brazil. Soil moisture variability may not only alter the climate locally but may also have nonlocal impacts through changes in the regional circulation. Here we explore how these two local coupling hot spots interact with each other, how soil moisture variability modulates the regional circulation, and what is the consequent nonlocal impact on precipitation. To this end, we analyze numerical experiments, performed with a regional climate model for the period October–March of 1983–2012, that allow us to isolate the influence of the soil moisture interannual variability on the regional climate. When the soil moisture–atmosphere interaction is enabled, we find a nonlocal coupling mechanism that links both hot spots at different temporal scales, favoring precipitation in eastern Brazil to the detriment of the precipitation in SESA through shifts in the regional circulation, when compared with a simulation with constrained soil moisture–atmosphere interaction. In northeastern Argentina, a subregion of SESA located at the exit of the South American low-level jet, it was found that the amount of nighttime precipitation is modulated by the proposed nonlocal coupling mechanism. A better understanding of the variability of precipitation due to the influence of land–atmosphere interaction processes may contribute to improving the predictability of precipitation and the interpretation of climate projections.

Action at a distance: organic cation induced long range organization of interfacial water enhances hydrogen evolution and oxidation kinetics.

Engineering efficient electrode-electrolyte interfaces for the hydrogen evolution and oxidation reactions (HOR/HER) is central to the growing hydrogen economy. Existing descriptors for HOR/HER catalysts focused on species that could directly impact the immediate micro-environment of surface-mediated reactions, such as the binding energies of adsorbates. In this work, we demonstrate that bulky organic cations, such as tetrapropyl ammonium, are able to induce a long-range structure of interfacial water molecules and enhance the HOR/HER kinetics even though they are located outside the outer Helmholtz plane. Through a combination of electrokinetic analysis, molecular dynamics and in situ spectroscopic investigations, we propose that the structure-making ability of bulky hydrophobic cations promotes the formation of hydrogen-bonded water chains connecting the electrode surface to the bulk electrolyte. In alkaline electrolytes, the HOR/HER involve the activation of interfacial water by donating or abstracting protons. The structural diffusion mechanism of protons in aqueous electrolytes enables water molecules and cations located at a distance from the electrode to influence surface-mediated reactions. The findings reported in this work highlight the prospect of leveraging the nonlocal mechanism to enhance electrocatalytic performance.

Yaglom limit for critical nonlocal branching Markov processes

We consider the classical Yaglom limit theorem for a branching Markov process X=(Xt,t≥0), with nonlocal branching mechanism in the setting that the mean semigroup is critical, that is, its leading eigenvalue is zero. In particular, we show that there exists a constant c(f) such that Law(⟨f,Xt⟩ t|⟨1,Xt⟩>0)→ec(f),t→∞, where ec(f) is an exponential random variable with rate c(f) and the convergence is in distribution. As part of the proof, we also show that the probability of survival decays inversely proportionally to time. Although Yaglom limit theorems have recently been handled in the setting of branching Brownian motion in a bounded domain and superprocesses, (Probab. Theory Related Fields 173 (2019) 999–1062; Electron. Commun. Probab. 23 (2018) 42), these results do not allow for nonlocal branching which complicates the analysis. Our approach and the main novelty of this work is based around a precise result for the scaled asymptotics for the kth martingale moments of X (rather than the Yaglom limit itself). We then illustrate our results in the setting of neutron transport for which the nonlocality is essential, complementing recent developments in this domain (Ann. Appl. Probab. 30 (2020) 2573–2612; Ann. Appl. Probab. 30 (2020) 2815–2845; SIAM J. Appl. Math. 81 (2021) 982–1001; Cox et al. (2021); J. Stat. Phys. 176 (2019) 425–455).

Channel exchange and adversarial learning guided cross-modal person re-identification

The extensive progress of Re-ID has been obtained in the visible modality. Because the security monitoring system automatically switches from visible modality to infrared modality in the dark situation, the research of the infrared-visible modality-based cross-modal person Re-ID (IV-Re-ID) task increases much attention. However, the existing heterogeneous physical properties result in a substantial semantic gap between the visible and infrared modality data, further increasing the challenge of IV-Re-ID. This paper proposes a Cross-modal Channel Exchange Network (CmCEN) for the task of IV-Re-ID. First, a non-local attention mechanism and a semi-weight share mechanism are utilized in the backbone network to enhance the discriminative capability of both local and global data representations. Then, a channel exchange module is designed to measure each channel’s significance effectively, and the useless channels could be replaced by the critical channels from the other modality data. Finally, a discriminator with adversarial loss is designed in the generative adversarial module. It can generate a similar distribution of two different modality images with the same person identity and a different distribution of two different modality images with different identities under the supervision of the adversarial loss to further improve the robustness of the learned latent feature space. The evaluation results on two cross-modal datasets demonstrated that the CmCEN achieves competitively and even higher performance comparing the SOTA models in terms of accuracy and mAP of IV-Re-ID.

MANet: A Network Architecture for Remote Sensing Spatiotemporal Fusion Based on Multiscale and Attention Mechanisms

Obtaining high-spatial–high-temporal (HTHS) resolution remote sensing images from a single sensor remains a great challenge due to the cost and technical limitations. Spatiotemporal fusion (STF) technology breaks through the technical limitations of existing sensors and provides a convenient and economical solution for obtaining HTHS resolution images. At present, most STF methods use stacked convolutional layers to extract image features and then obtain fusion images by using a summation strategy. However, these convolution operations may lead to the loss of feature information, and the summation strategy results in poorly fused images due to a lack of consideration of global spatial feature information. To address these issues, this article proposes a STF network architecture based on multiscale and attention mechanisms (MANet). The multiscale mechanism module composed of dilated convolutions is used to extract the detailed features of low-spatial resolution remote sensing images at multiple scales. The channel attention mechanism adaptively adjusts the weights of the feature map channels to retain more temporal and spatial information in the upsampling process, while the non-local attention mechanism adjusts the initial fusion images to obtain more accurate predicted images by calculating the correlation between pixels. We use two datasets with different characteristics to conduct the experiments, and the results prove that the proposed MANet method with fewer parameters obtains better fusion results than the existing machine learning-based and deep learning-based fusion methods.

Refine-FPN: Instance Segmentation Based on a Non-local Multi-feature Aggregation Mechanism

Refine-FPN: Instance Segmentation Based on a Non-local Multi-feature Aggregation Mechanism

A new end-to-end framework based on non-local network structure and spatial attention mechanism for image rain removal

Image rain removal is an important topic in the field of computer vision. In the rainy environment, the rain will seriously affect the quality of imaging, resulting in image deformation, blur, poor visibility, and other problems. So the outdoor vision system cannot accurately detect object, monitor, and other works. Therefore, how to effectively eliminate the rain-weather interference to imaging has a very important practical value. In the absence of the time series information between frame and frame, the bottleneck problem of image rain removal technology is how to effectively remove multi-density rain-fringes while preserving the detailed structure information of the image background. To solve the above problems, a new image rain removal algorithm based on non-local network structure and spatial attention mechanism is proposed in this paper. Firstly, the position relation information between different pixels is obtained by the non-local operation to obtain the global image representation. Secondly, the spatial attention mechanism is used to recalibrate the global information in the spatial dimension. In other words, the nonlinear modeling is carried out on the channel dimension to gather similar features and useful information. Finally, deconvolution and long-distance residual connection are used to restore the size of the rain-removing image layer by layer. The analysis and experimental results show that the proposed algorithm in this paper is effective in removing rain marks, effectively solves the practical difficulties of removing rain stripes with different rain densities, and preserves the details and edge information of the image well.

An Unsupervised Monocular Visual Odometry Based on Multi-Scale Modeling.

Unsupervised deep learning methods have shown great success in jointly estimating camera pose and depth from monocular videos. However, previous methods mostly ignore the importance of multi-scale information, which is crucial for pose estimation and depth estimation, especially when the motion pattern is changed. This article proposes an unsupervised framework for monocular visual odometry (VO) that can model multi-scale information. The proposed method utilizes densely linked atrous convolutions to increase the receptive field size without losing image information, and adopts a non-local self-attention mechanism to effectively model the long-range dependency. Both of them can model objects of different scales in the image, thereby improving the accuracy of VO, especially in rotating scenes. Extensive experiments on the KITTI dataset have shown that our approach is competitive with other state-of-the-art unsupervised learning-based monocular methods and is comparable to supervised or model-based methods. In particular, we have achieved state-of-the-art results on rotation estimation.

Two stream Non-Local CNN-LSTM network for the auxiliary assessment of mental retardation

At present, the assessment of mental retardation is mainly based on clinical interview, which requires the participation of experienced psychiatrist and is laborious. Studies have shown that there are correlations between mental retardation and abnormal behaviors (such as, hyperkinetic, tics, stereotypes, etc.). On the basis of this fact, a two stream Non-Local CNN-LSTM network has been proposed to learn the features of upper body behavior and facial expression of patients, thus, to achieve the preliminary screening of mental retardation. Specifically, RGB and optical flow are extracted separately from interview videos, and a two stream network based on contribution mechanism is designed to effectively fuse the information of two kinds of images, which may update the network in a new approach of alternating iteration training to find the optimal model. Besides, by introducing non-local mechanism and adopting it to the network, the global feature sensing can be established more effectively to reduce the background interference for video clip in a short time zone. Experiments on clinical video dataset show that the performance of proposed model is better than other prevalent deep learning methods of behavioral feature learning, the accuracy reaches 89.15% in basic experiment, and is further improved to 89.52% in the supplementary experiment. Furthermore, the experimental results show that this method still has a lot of room for improvement. In general, our work indicates that the proposed model has potential value for the clinical diagnosis and screening of mental retardation.

Automated polyp segmentation in colonoscopy images via deep network with lesion-aware feature selection and refinement

Clinically, proper polyp localization in colonoscopy images is crucial for early diagnosis and follow-up treatment of colorectal cancer. However, visual inspection is subjective, error-prone, and burdensome. In this paper, we propose an automated polyp segmentation method (named LFSRNet) to assist physicians to accurately segment polyps in colonoscopy images. The proposed LFSRNet follows an encoder–decoder architecture and benefits from two pivotal modules, i.e., a lesion-aware feature selection module (LFSM) and a lesion-aware feature refinement module (LFRM). Specifically, the LFSM selects lesion-aware features from the top-three highest layers of the encoder via a non-local attention mechanism and fuses them to generate the initial segmentation map for the decoder. The LFRM embedded in the decoder incorporates the guided context information and the output of LFRM from the adjacent higher layer to refine the lesion-aware features. Through top-down deep supervision, our LFSRNet can adaptively select and refine lesion-aware features and precisely localize the polyp regions. Experimental results on the Kvasir-SEG dataset (with the 80%–20% train-test split) show that LFSRNet is superior to six state-of-the-art competing methods and achieves a dice score of 0.9127, an intersection-over-union score of 0.8615, a sensitivity score of 0.9174, an accuracy score of 0.9728, an F2 score of 0.9123, and an MAE score of 0.0291, respectively. More extensive results show that LFSRNet also holds better generalization than competing methods when trained on the Kvasir-SEG dataset and tested on both the CVC-ClinicDB dataset and EndoScene dataset.

Nonlinear aeroelastic analysis of a damped elastica-aerofoil system

This work formulates a comprehensive model of a nonlinear aeroelastic system developed for the analysis of complex aeroelastic phenomena related to structural and aerodynamic nonlinearities. The system is formulated as a two-dimensional cantilevered elastica with a rigid airfoil section firmly attached at its tip undergoing large displacements in the crosswind conditions. The system can demonstrate a wide range of domain specific as well as coupled nonlinear phenomena. The structural model is developed by means of the Rayleigh–Ritz approach, with shape functions discretizing both vertical and horizontal displacements and Lagrangian multipliers enforcing inextensibility. Damping is modeled based on a non-local strain-based mechanism in the Kelvin–Voigt arrangement. The resulting structural model is examined through studying the behavior under a follower load and with a tip-attached tendon under tension to study the shape convergence properties and the alignment of the results with known characteristics in the literature. The ONERA dynamic stall model is used to model the aerodynamics of the problem to accurately capture post-stall behavior at large deformations. The LCO responses of the aeroelastic problem are evaluated through time-marched simulations, and the combined airspeed–damping interactions are studied in this manner.

Concerted Double Hydrogen-Bond Breaking by Intermolecular Coulombic Decay in the Formic Acid Dimer.

Hydrogen bonds are ubiquitous in nature and of fundamental importance to the chemical and physical properties of molecular systems in the condensed phase. Nevertheless, our understanding of the structural and dynamical properties of hydrogen-bonded complexes in particular in electronic excited states remains very incomplete. Here, by using formic acid (FA) dimer as a prototype of DNA base pair, we investigate the ultrafast decay process initiated by removal of an electron from the inner-valence shell of the molecule upon electron-beam irradiation. Through fragment-ion and electron coincident momentum measurements and ab initio calculations, we find that de-excitation of an outer-valence electron at the same site can initiate ultrafast energy transfer to the neighboring molecule, which is in turn ionized through the emission of low-energy electrons. Our study reveals a concerted breaking of double hydrogen-bond in the dimer initiated by the ultrafast molecular rotations of two FA+ cations following this nonlocal decay mechanism.

NHNet: A non‐local hierarchical network for image denoising

With the fast development of deep learning models, hierarchical convolutional neural networks have achieved great success in image denoising tasks. To further boost the performance of image denoising, a novel non-local hierarchical network (NHNet) is proposed. Unlike existing U-Net-based hierarchical methods, which mainly focus on downsampling operations, NHNet adopts an initial resolution path and a high resolution path. Specifically, the high-resolution features are obtained through upsampling, where the non-local mechanism is adopted to capture the self-similarity properties, which contribute to a better denoising performance. Cross connections and channel attention layers are added between the two paths to integrate features in different resolutions. Compared with other U-Net-based hierarchical networks, NHNet requires fewer parameters. Experiments show that NHNet achieves state-of-the-art performance in Gaussian denoising tasks and gets competitive results when dealing with real image denoising.

Arbitrary large-gradient wavefront shaping: from local phase modulation to nonlocal diffraction engineering

The powerful wavefront manipulation capability of metasurfaces originates from their subwavelength or deep subwavelength elements with designable optical responses, especially phase responses. However, they usually suffer from performance degradation as the spatial phase gradient is large. To solve this issue, we propose an accurate and efficient nonlocal diffraction engineering mechanism to tailor an arbitrary large-gradient wavefront utilizing superwavelength-scale elements. The fast-varying phase profile is cut into segments according to 2 π zones rather than subwavelength discretization. Each phase segment is accurately implemented by precisely tailoring the diffraction pattern of the element, where diffraction angles, efficiencies, and phases are controlled simultaneously. As proof of the concept, high numerical aperture cylindrical metalenses are designed using this method and experimentally validated at the terahertz band. The cylindrical metalens is further extended to a full-space metalens, which enables high-quality subwavelength imaging with resolved details of 0.65 λ . The proposed mechanism offers an efficient way to capture the fast-varying wavefront using relatively coarse geometries with new physical insights.

The effect of CT high-resolution imaging diagnosis based on deep residual network on the pathology of bladder cancer classification and staging

Background and objectiveTo study the high-resolution CT image based on deep residual network to efficiently and accurately predict the staging diagnosis of bladder tumors. MethodsThe image was processed with super-resolution to restore the missing details of the image. The CT data of 75 bladder patients who were treated in our hospital from June to December 2013 were collected. And obtain the patient's classification and staging information through pathology, which is used to establish a model of ResNet structure combined with non-Local attention mechanism. The clinical data of 76 patients with bladder disease admitted to our hospital from May 2018 to August 2021 were randomly selected, and the imaging and accuracy of CT diagnosis were retrospectively analyzed. Results52 cases were diagnosed <T1 stage, 16 cases belonged to T2 stage, 2 cases T3 stage, and 2 cases T4 stage. The sensitivity rate of experimental diagnosis was 94.74%, which was not significantly different from the sensitivity rate of preoperative pathological diagnosis. ConclusionCT based on deep residual network has high application value in the diagnosis and staging of bladder cancer, can effectively improve the diagnostic accuracy, and is worthy of clinical application.