Local Convolution Research Articles

An Evidential Mask Transformer for Left Atrium Segmentation

The segmentation of the left atrium (LA) is required to calculate the clinical parameters of the LA, to identify diseases related to its remodeling. Generally, convolutional networks have been used for this task. However, their performance may be limited as a result of the use of local convolution operations for feature extraction. Also, such models usually need extra steps to provide uncertainty maps such as multiple forward passes for Monte Carlo dropouts or training multiple models for ensemble learning. To address these issues, we adapt mask transformers for LA segmentation which effectively use both local and global information, and train them with evidential learning to generate uncertainty maps from the learned Dirichlet distribution, with a single forward pass. We validated our approach on the STACOM 2013 dataset and found that our method can produce better segmentation performance than baseline models, and can identify locations our model’s responses are not trustable.

Weakly supervised semantic segmentation by knowledge graph inference

The weakly supervised semantic segmentation (WSSS) training based on image-level labels in convolutional neural network (CNN) is usually divided into two stages: multi-label classification and semantic segmentation. However, most of the existing work focuses on the improvement of the multi-label classification network stage, and little effort has been done to improve the performance of the downstream segmentation networks. In addition, CNN-based local convolution lacks in modeling extensive dependencies among categories. Therefore, in this paper, we propose a graph reasoning method to improve both the upstream and the downstream stages of the multi-label classification network and the semantic segmentation networks. In the multi-label classification network, we utilize external knowledge combined with a graph convolutional network (GCN) to perform global reasoning on the dependencies of each category. In the segmentation network, the Graph Reasoning Mapping Module (GRM) is proposed to explore the knowledge acquired from text corpora and facilitate contextual reasoning in various categories of image regions. The proposed GRM module is able to enhance the feature representation of local convolutions on the high-level semantics of the segmentation network, and adaptively learn the semantic consistency of each sample. We achieve state-of-the-art WSSS performance on PASCAL VOC 2012 and MS-COCO 2014 datasets with only image-level supervision. Extensive experiments on multi-label classification networks and semantic segmentation networks demonstrate the effectiveness of our proposed graph reasoning method on WSSS. Our code is available at:https://github.com/JIA-ZHANG666/GRM_layer.

CloudSwinNet: A hybrid CNN-transformer framework for ground-based cloud images fine-grained segmentation

Solar irradiance is the main factor affecting the output of a photovoltaic (PV) power station. The dominant role of ground-based clouds on the variation of direct solar radiation in the whole sky. Ground-based cloud segmentation plays a crucial role in photovoltaic power generation prediction. Despite advancements, current cloud segmentation methods fail to meet the requirements of this application. While convolutional neural networks demonstrate impressive segmentation capabilities in ground-based cloud segmentation tasks, their inherent limitations hinder further performance enhancement. Hence, this paper introduces CloudSwinNet, a hybrid CNN-Transformer framework for ground-based cloud images fine-grained segmentation. CloudSwinNet leverages concepts from convolutional neural networks, including stepwise downsampling, local convolution, and skip connections. To further explore the fine-grained features in the ground-based cloud images, we incorporates the Fine-grained Feature Fusion Module (Fg-FFM) in encoder structure, while the jump connection structure integrates the GloRe spatial graph inference module. These additions enable comprehensive learning of multi-scale and long-range dependencies. We conducted extensive experiments on the fine-grained ground-based cloud dataset, demonstrating that CloudSwinNet outperforms six semantic segmentation networks in both qualitative and quantitative evaluations. The results of ablation experiments prove the effectiveness of the module introduced in this paper.

Semantic dependency and local convolution for enhancing naturalness and tone in text-to-speech synthesis

Self-attention-based networks have become increasingly popular due to their exceptional performance in parallel training and global context modeling. However, it may fall short of capturing local dependencies, particularly in datasets with strong local correlations. To address this challenge, we propose a novel method that utilizes semantic dependency to extract linguistic information from the original text. The semantic relationship between nodes serves as prior knowledge to refine the self-attention distribution. Additionally, to better fuse local contextual information, we introduce a one-dimensional convolution neural network to generate the query and value matrices in the self-attention mechanism, taking advantage of the strong correlation between input characters. We apply this variant of the self-attention network to text-to-speech tasks and propose a non-autoregressive neural text-to-speech model. To enhance pronunciation accuracy, we separate tones from phonemes as independent features in model training. Experimental results show that our model yields good performance in speech synthesis. Specifically, the proposed method significantly improves the processing of pause, stress, and intonation in speech.

Lightweight Corn Leaf Detection and Counting Using Improved YOLOv8.

The number of maize leaves is an important indicator for assessing plant growth and regulating population structure. However, the traditional leaf counting method mainly relies on manual work, which is both time-consuming and straining, while the existing image processing methods have low accuracy and poor adaptability, making it difficult to meet the standards for practical application. To accurately detect the growth status of maize, an improved lightweight YOLOv8 maize leaf detection and counting method was proposed in this study. Firstly, the backbone of the YOLOv8 network is replaced using the StarNet network and the convolution and attention fusion module (CAFM) is introduced, which combines the local convolution and global attention mechanisms to enhance the ability of feature representation and fusion of information from different channels. Secondly, in the neck network part, the StarBlock module is used to improve the C2f module to capture more complex features while preserving the original feature information through jump connections to improve training stability and performance. Finally, a lightweight shared convolutional detection head (LSCD) is used to reduce repetitive computations and improve computational efficiency. The experimental results show that the precision, recall, and mAP50 of the improved model are 97.9%, 95.5%, and 97.5%, and the numbers of model parameters and model size are 1.8 M and 3.8 MB, which are reduced by 40.86% and 39.68% compared to YOLOv8. This study shows that the model improves the accuracy of maize leaf detection, assists breeders in making scientific decisions, provides a reference for the deployment and application of maize leaf number mobile end detection devices, and provides technical support for the high-quality assessment of maize growth.

Neural waves and computation in a neural net model II: Data-like structures and the dynamics of episodic memory.

The computational resources of a neuromorphic network model introduced earlier were investigated in the first paper of this series. It was argued that a form of ubiquitous spontaneous local convolution enabled logical gate-like neural motifs to form into hierarchical feed-forward structures of the Hubel-Wiesel type. Here we investigate concomitant data-like structures and their dynamic rôle in memory formation, retrieval, and replay. The mechanisms give rise to the need for general inhibitory sculpting, and the simulation of the replay of episodic memories, well known in humans and recently observed in rats. Other consequences include explanations of such findings as the directional flows of neural waves in memory formation and retrieval, visual anomalies and memory deficits in schizophrenia, and the operation of GABA agonist drugs in suppressing episodic memories. We put forward the hypothesis that all neural logical operations and feature extractions are of the convolutional hierarchical type described here and in the earlier paper, and exemplified by the Hubel-Wiesel model of the visual cortex, but that in more general cases the precise geometric layering might be obscured and so far undetected.

Lightweight Food Recognition via Aggregation Block and Feature Encoding

Food image recognition has recently been given considerable attention in the multimedia field in light of its possible implications on health. The characteristics of the dispersed distribution of ingredients in food images put forward higher requirements on the long-range information extraction ability of neural networks, leading to more complex and deeper models. Nevertheless, the lightweight version of food image recognition is essential for improved implementation on end devices and sustained server-side expansion. To address this issue, we present Aggregation Feature Net(AFNet), a lightweight network that is capable of effectively capturing both global and local features from food images. In AFNet, we develop a novel convolution based on a residual model by encoding global features through row-wise and column-wise information integration. Merging aggregation block with classic local convolution yields a framework that works as the backbone of the network. Based on the efficient use of parameters by the aggregation block, we constructed a lightweight food image recognition network with fewer layers and a smaller scale, assisted by a new type of activation function. Experimental results on four popular food recognition datasets demonstrate that our approach achieves state-of-the-art performance with higher accuracy and fewer FLOPs and parameters. For example, in comparison to the current state-of-the-art model of MobileViTv2, AFNet achieved 88.4% accuracy of the top-1 level on the ETHZ Food-101 dataset, with similar parameters and FLOPs but 1.4% more accuracy. The source code will be provided in supplementary materials.

Collaborative spatial-temporal video salient object detection with cross attention transformer

Noticing the effectiveness of explicit motion encoding in optical flow, a variety of recent works employ flow-guided two-branch structures to handle the video salient object detection (VSOD) task. However, most of them ignore the semantic gap between the moving objects and the salient objects. Besides, the long-range dependency is not sufficiently investigated due to the local convolution operation. To tackle these problems, in this paper, we propose a novel collaborative spatio-temporal feature fusion method based on the cross-attention transformer for VSOD. A Siamese feature extractor is introduced to jointly optimize the motion and static feature extraction with reduced network parameters. The deep level set method is employed to segment the moving objects from optical flow, and thus the dependence on saliency groundtruth is greatly reduced. Moreover, a cross-attention transformer is proposed to jointly optimize and fuse the static and motion features, as well as investigate long-range dependency. Experimental results on six commonly used video salient object datasets demonstrate that our method achieves state-of-the-art performance among all VSOD algorithms.

Mining local and global spatiotemporal features for tactile object recognition.

The tactile object recognition (TOR) is highly important for environmental perception of robots. The previous works usually utilize single scale convolution which cannot simultaneously extract local and global spatiotemporal features of tactile data, which leads to low accuracy in TOR task. To address above problem, this article proposes a local and global residual (LGR-18) network which is mainly consisted of multiple local and global convolution (LGC) blocks. An LGC block contains two pairs of local convolution (LC) and global convolution (GC) modules. The LC module mainly utilizes a temporal shift operation and a 2D convolution layer to extract local spatiotemporal features. The GC module extracts global spatiotemporal features by fusing multiple 1D and 2D convolutions which can expand the receptive field in temporal and spatial dimensions. Consequently, our LGR-18 network can extract local-global spatiotemporal features without using 3D convolutions which usually require a large number of parameters. The effectiveness of LC module, GC module and LGC block is verified by ablation studies. Quantitative comparisons with state-of-the-art methods reveal the excellent capability of our method.

Learnable weight initialization for volumetric medical image segmentation

Hybrid volumetric medical image segmentation models, combining the advantages of local convolution and global attention, have recently received considerable attention. While mainly focusing on architectural modifications, most existing hybrid approaches still use conventional data-independent weight initialization schemes which restrict their performance due to ignoring the inherent volumetric nature of the medical data. To address this issue, we propose a learnable weight initialization approach that utilizes the available medical training data to effectively learn the contextual and structural cues via the proposed self-supervised objectives. Our approach is easy to integrate into any hybrid model and requires no external training data. Experiments on multi-organ and lung cancer segmentation tasks demonstrate the effectiveness of our approach, leading to state-of-the-art segmentation performance. Our proposed data-dependent initialization approach performs favorably as compared to the Swin-UNETR model pretrained using large-scale datasets on multi-organ segmentation task. Our source code and models are available at: https://github.com/ShahinaKK/LWI-VMS.

Earthfarsser: Versatile Spatio-Temporal Dynamical Systems Modeling in One Model

Efficiently modeling spatio-temporal (ST) physical processes and observations presents a challenging problem for the deep learning community. Many recent studies have concentrated on meticulously reconciling various advantages, leading to designed models that are neither simple nor practical. To address this issue, this paper presents a systematic study on existing shortcomings faced by off-the-shelf models, including lack of local fidelity, poor prediction performance over long time-steps, low scalability, and inefficiency. To systematically address the aforementioned problems, we propose an EarthFarseer, a concise framework that combines parallel local convolutions and global Fourier-based transformer architectures, enabling dynamically capture the local-global spatial interactions and dependencies. EarthFarseer also incorporates a multi-scale fully convolutional and Fourier architectures to efficiently and effectively capture the temporal evolution. Our proposal demonstrates strong adaptability across various tasks and datasets, with fast convergence and better local fidelity in long time-steps predictions. Extensive experiments and visualizations over eight human society physical and natural physical datasets demonstrates the state-of-the-art performance of EarthFarseer. We release our code at https://github.com/easylearningscores/EarthFarseer.

SFFNet: Staged Feature Fusion Network of Connecting Convolutional Neural Networks and Graph Convolutional Neural Networks for Hyperspectral Image Classification

The immense representation power of deep learning frameworks has kept them in the spotlight in hyperspectral image (HSI) classification. Graph Convolutional Neural Networks (GCNs) can be used to compensate for the lack of spatial information in Convolutional Neural Networks (CNNs). However, most GCNs construct graph data structures based on pixel points, which requires the construction of neighborhood matrices on all data. Meanwhile, the setting of GCNs to construct similarity relations based on spatial structure is not fully applicable to HSIs. To make the network more compatible with HSIs, we propose a staged feature fusion model called SFFNet, a neural network framework connecting CNN and GCN models. The CNN performs the first stage of feature extraction, assisted by adding neighboring features and overcoming the defects of local convolution; then, the GCN performs the second stage for classification, and the graph data structure is constructed based on spectral similarity, optimizing the original connectivity relationships. In addition, the framework enables the batch training of the GCN by using the extracted spectral features as nodes, which greatly reduces the hardware requirements. The experimental results on three publicly available benchmark hyperspectral datasets show that our proposed framework outperforms other relevant deep learning models, with an overall classification accuracy of over 97%.

Global domain adaptation attention with data-dependent regulator for scene segmentation.

Most semantic segmentation works have obtained accurate segmentation results through exploring the contextual dependencies. However, there are several major limitations that need further investigation. For example, most approaches rarely distinguish different types of contextual dependencies, which may pollute the scene understanding. Moreover, local convolutions are commonly used in deep learning models to learn attention and capture local patterns in the data. These convolutions operate on a small neighborhood of the input, focusing on nearby information and disregarding global structural patterns. To address these concerns, we propose a Global Domain Adaptation Attention with Data-Dependent Regulator (GDAAR) method to explore the contextual dependencies. Specifically, to effectively capture both the global distribution information and local appearance details, we suggest using a stacked relation approach. This involves incorporating the feature node itself and its pairwise affinities with all other feature nodes within the network, arranged in raster scan order. By doing so, we can learn a global domain adaptation attention mechanism. Meanwhile, to improve the features similarity belonging to the same segment region while keeping the discriminative power of features belonging to different segments, we design a data-dependent regulator to adjust the global domain adaptation attention on the feature map during inference. Extensive ablation studies demonstrate that our GDAAR better captures the global distribution information for the contextual dependencies and achieves the state-of-the-art performance on several popular benchmarks.

Neural waves and computation in a neural net model I: Convolutional hierarchies.

The computational resources of a neuromorphic network model introduced earlier are investigated in the context of such hierarchical systems as the mammalian visual cortex. It is argued that a form of ubiquitous spontaneous local convolution, driven by spontaneously arising wave-like activity-which itself promotes local Hebbian modulation-enables logical gate-like neural motifs to form into hierarchical feed-forward structures of the Hubel-Wiesel type. Extra-synaptic effects are shown to play a significant rôle in these processes. The type of logic that emerges is not Boolean, confirming and extending earlier findings on the logic of schizophrenia.

An Efficient Latent Style Guided Transformer-CNN Framework for Face Super-Resolution

In the Face Super-Resolution (FSR) task, it is important to precisely recover facial textures while maintaining facial contours for realistic high resolution faces. Although several CNN-based FSR methods have achieved great performance, they fail in restoring the facial contours due to the limitation of local convolutions. In contrast, Transformer-based methods which use self-attention as the basic component, are expert in modeling long-range dependencies between image patches. However, learning long-range dependencies often deteriorates facial textures due to the lack of locality. Therefore, a question is naturally raised: <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">how to effectively combine the superiority of CNN and Transformer for better reconstructing faces?</i> To address this issue, we propose an Efficient Latent Style guided Transformer-CNN framework for FSR called <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ELSFace</b> , which can sufficiently integrate the advantages of CNN and Transformer. The framework consists of a Feature Preparation Stage and a Feature Carving Stage. Basic facial contours and textures are generated in the Feature Preparation Stage, and separately guided by latent styles, so that facial details are better represented in reconstruction. CNN and Transformer streams in the Feature Carving Stage are used to individually restore facial textures and facial contours, respectively in a parallel recursive way. Considering the negligence of high-frequency features when learning the long-range dependencies, we design the High-Frequency Enhancement Block (HFEB) in the Transformer stream. The Sharp Loss is also proposed for better perceptual quality in optimization. Extensive experimental results demonstrate that our ELSFace can achieve the best results among all metrics compared to the state-of-the-art CNN and Transformer-based methods on commonly used datasets and real-world tasks. Meanwhile, our ELSFace method has the least model parameters and running time. The codes are released at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/FVL2020/ELSFace</uri> .

Deep learning-based projection synthesis for low-dose cone-beam computed tomography imaging in image-guided radiotherapy.

The imaging dose of cone-beam computed tomography (CBCT) in image-guided radiotherapy (IGRT) poses adverse effects on patient health. To improve the quality of sparse-view low-dose CBCT images, a projection synthesis convolutional neural network (SynCNN) model is proposed. Included in this retrospective, single-center study were 223 patients diagnosed with brain tumours from Beijing Cancer Hospital. The proposed SynCNN model estimated two pairs of orthogonally direction-separable spatial kernels to synthesize the missing projection in between the input neighboring sparse-view projections via local convolution operations. The SynCNN model was trained on 150 real patients to learn patterns for inter-view projection synthesis. CBCT data from 30 real patients were used to validate the SynCNN, while data from a phantom and 43 real patients were used to test the SynCNN externally. Sparse-view projection datasets with 1/2, 1/4, and 1/8 of the original sampling rate were simulated, and the corresponding full-view projection datasets were restored using the SynCNN model. The tomographic images were then reconstructed with the Feldkamp-Davis-Kress algorithm. The root-mean-square error (RMSE), peak signal-to-noise ratio (PSNR), and structural similarity (SSIM) metrics were measured in both the projection and image domains. Five experts were invited to grade the image quality blindly for 40 randomly selected evaluation groups with a four-level rubric, where a score greater than or equal to 2 was considered acceptable image quality. The running time of the SynCNN model was recorded. The SynCNN model was directly compared with the three other methods on 1/4 sparse-view reconstructions. The phantom and patient studies showed that the missing projections were accurately synthesized. In the image domain, for the phantom study, compared with images reconstructed from sparse-view projections, images with SynCNN synthesis exhibited significantly improved qualities with decreased values in RMSE and increased values in PSNR and SSIM. For the patient study, between the results with and without the SynCNN synthesis, the averaged RMSE decreased by 3.4×10-4, 10.3×10-4, and 21.7×10-4, the averaged PSNR increased by 3.4, 6.6, and 9.4 dB, and the averaged SSIM increased by 5.2×10-2, 18.9×10-2 and 33.9×10-2, for the 1/2, 1/4, and 1/8 sparse-view reconstructions, respectively. In expert subjective evaluation, both the median scores and acceptance rates of the images with SynCNN synthesis were higher than those reconstructed from sparse-view projections. It took the model less than 0.01 s to synthesize an inter-view projection. Compared with the three other methods, the SynCNN model obtained the best scores in terms of the three metrics in both domains. The proposed SynCNN model effectively improves the quality of sparse-view CBCT images at a low time cost. With the SynCNN model, the CBCT imaging dose in IGRT could be reduced potentially.

Convolutional Transformer-based Cross Subject Model for SSVEP-based BCI Classification.

Steady-state visual evoked potential (SSVEP) is a commonly used brain-computer interface (BCI) paradigm. The performance of cross-subject SSVEP classification has a strong impact on SSVEP-BCI. This study designed a cross subject generalization SSVEP classification model based on an improved transformer structure that uses domain generalization (DG). The global receptive field of multi-head self-attention is used to learn the global generalized SSVEP temporal information across subjects. This is combined with a parallel local convolution module, designed to avoid oversmoothing the oscillation characteristics of temporal SSVEP data and better fit the feature. Moreover, to improve the cross-subject calibration-free SSVEP classification performance, an DG method named StableNet is combined with the proposed convolutional transformer structure to form the DG-Conformer method, which can eliminate spurious correlations between SSVEP discriminative information and background noise to improve cross-subject generalization. Experiments on two public datasets, Benchmark and BETA, demonstrated the outstanding performance of the proposed DG-Conformer compared with other calibration-free methods, FBCCA, tt-CCA, Compact-CNN, FB-tCNN, and SSVEPNet. Additionally, DG-Conformer outperforms the classic calibration-required algorithms eCCA, eTRCA and eSSCOR when calibration is used. An incomplete partial stimulus calibration scheme was also explored on the Benchmark dataset, and it was demonstrated to be a potential solution for further high-performance personalized SSVEP-BCI with quick calibration.

SMILE: Siamese Multi-scale Interactive-representation LEarning for Hierarchical Diffeomorphic Deformable image registration

Deformable medical image registration plays an important role in many clinical applications. It aims to find a dense deformation field to establish point-wise correspondences between a pair of fixed and moving images. Recently, unsupervised deep learning-based registration methods have drawn more and more attention because of fast inference at testing stage. Despite remarkable progress, existing deep learning-based methods suffer from several limitations including: (a) they often overlook the explicit modeling of feature correspondences due to limited receptive fields; (b) the performance on image pairs with large spatial displacements is still limited since the dense deformation field is regressed from features learned by local convolutions; and (c) desirable properties, including topology-preservation and the invertibility of transformation, are often ignored. To address above limitations, we propose a novel Convolutional Neural Network (CNN) consisting of a Siamese Multi-scale Interactive-representation LEarning (SMILE) encoder and a Hierarchical Diffeomorphic Deformation (HDD) decoder. Specifically, the SMILE encoder aims for effective feature representation learning and spatial correspondence establishing while the HDD decoder seeks to regress the dense deformation field in a coarse-to-fine manner. We additionally propose a novel Local Invertible Loss (LIL) to encourage topology-preservation and local invertibility of the regressed transformation while keeping high registration accuracy. Extensive experiments conducted on two publicly available brain image datasets demonstrate the superiority of our method over the state-of-the-art (SOTA) approaches. Specifically, on the Neurite-OASIS dataset, our method achieved an average DSC of 0.815 and an average ASSD of 0.633 mm.

A combination network of CNN and transformer for interference identification.

Communication interference identification is critical in electronic countermeasures. However, existed methods based on deep learning, such as convolutional neural networks (CNNs) and transformer, seldom take both local characteristics and global feature information of the signal into account. Motivated by the local convolution property of CNNs and the attention mechanism of transformer, we designed a novel network that combines both architectures, which make better use of both local and global characteristics of the signals. Additionally, recognizing the challenge of distinguishing contextual semantics within the one-dimensional signal data used in this study, we advocate the use of CNNs in place of word embedding, aligning more closely with the intrinsic features of the signal data. Furthermore, to capture the time-frequency characteristics of the signals, we integrate the proposed network with a cross-attention mechanism, facilitating the fusion of temporal and spectral domain feature information through multiple cross-attention computational layers. This innovation obviates the need for specialized time-frequency analysis. Experimental results demonstrate that our approach significantly improves recognition accuracy compared to existing methods, highlighting its efficacy in addressing the challenge of communication interference identification in electronic warfare.

Patch-based stochastic attention for image editing

Attention mechanisms have become of crucial importance in deep learning in recent years. These non-local operations, which are similar to traditional patch-based methods in image processing, complement local convolutions. However, computing the full attention matrix is an expensive step with heavy memory and computational loads. These limitations curb network architectures and performances, in particular for the case of high resolution images. We propose an efficient attention layer based on the stochastic algorithm PatchMatch, which is used for determining approximate nearest neighbors. We refer to our proposed layer as a “Patch-based Stochastic Attention Layer” (PSAL). Furthermore, we propose different approaches, based on patch aggregation, to ensure the differentiability of PSAL, thus allowing end-to-end training of any network containing our layer. PSAL has a small memory footprint and can therefore scale to high resolution images. It maintains this footprint without sacrificing spatial precision and globality of the nearest neighbors, which means that it can be easily inserted in any level of a deep architecture, even in shallower levels. We demonstrate the usefulness of PSAL on several image editing tasks, such as image inpainting, guided image colorization, and single-image super-resolution. Our code is available at: https://github.com/ncherel/psal.